Re: Strange wait time in my application - Tomcat 7.0.67
On 23.10.2016 15:52, Tullio Bettinazzi wrote:
I'm sure is noty a Db issue, the delay is not only concentrated in db functions.
The same DB operations seems to take from 4millsec to 4 sec with same data and
parameters.
Ok, so let's summarise what you have told us so far :
- this is under Tomcat 7.0.67 (missing : Java version and platform)
- the application works fine, without slowdowns (or "hiccups") when a single
user is using it
- when several users are using it simultaneously, unexplained "hiccups" happen for the
users : some (not always the same) operation seems to block for up to 4 seconds, instead
of the few milliseconds taken normally by that same operation.
- the block does not always happen at the same point in the application. It also does not
happen systematically in DB operations (but it happens sometimes in a DB operation also)
- when the block happens for one user, it usually also happens afterward for the other
users, in sequence
- the memory usage and Garbage Collection does not seem to be the cause : GC appears
"normal" and there is plenty of memory available on the system
It looks like there is some contention for a resource, between the user
sessions.
If it is not the memory or GC, nor the database, then you have to look for
something else.
I am going to assume - for now - that it is something happening in the application (as
opposed to Tomcat), which causes these hiccups.
Does the application write a logfile ? could it be that several user sessions are
competing for access to the logfile ?
(may be easy to check : turn off the logging, and see if it still happens)
Is the database on the same server ? if not, how is the connection with the database made
? (and even if it is on the same server). I am thinking of DNS as one (far-fetched)
possible issue.
Does one of the system logs indicate any problem ?
(e.g. "dmesg" under Linux; or Windows Event logs)
Can you write some little script (e.g. in perl) which just loops, and writes a log line
(with a timestamp) say every second ? that would be to check if when your java application
has a "hiccup", this script also has one at the same time, or not.
Example :
#!/usr/bin/perl
while (1) {
my ($sec,$min,$hour,$mday,$mon,$year) = localtime(time);
my $stamp = sprintf("%04d-%02d-%02d -
%02d:%02d:%02d",$year+1900,$mon+1,$mday,$hour,$min,$sec);
print STDERR $stamp," one second later\n";
sleep 1;
}
Note : this script can easily be modified, e.g. to /also/ print the log message to a disk
file. This way it would be affected also if the problem is with the disk.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange wait time in my application - Tomcat 7.0.67
I'm sure is noty a Db issue, the delay is not only concentrated in db functions. The same DB operations seems to take from 4millsec to 4 sec with same data and parameters. Tks Da: Christopher Schultz <ch...@christopherschultz.net> Inviato: venerdì 21 ottobre 2016 23.07 A: Tomcat Users List Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Tullio, On 10/17/16 12:24 PM, Tullio Bettinazzi wrote: > I monitored it using Yourkit profiler and I didn't see any extreme > GC. > > I noticed anotehr thing : looking for 4 users which had the > problem they seems to have the problem "in sequence". > > The first user "stops" for 4 seconds then another one "stops" and > so on : they don't seem to stop at the sam time (as it would be > expected in case of GC) but in sequence. My first reaction is that this has to be a database issue. Are you using a traditional RDBMS under the hood? Can you look for long-running queries on the db? Maybe you have a table that is being locked during a transaction which is holding-up users. Maybe you have a database connection pool without enough connections in it. During times of heavy traffic, when you have users "stalling", what does a thread dump show you? - -chris > Da: André Warnier (tomcat) > <a...@ice-sa.com> Inviato: lunedì 17 ottobre 2016 18.01 A: > users@tomcat.apache.org Oggetto: Re: Strange wait time in my > application - Tomcat 7.0.67 > > On 17.10.2016 17:52, Tullio Bettinazzi wrote: >> I didn't find any solution to my problem. >> >> Could someone provide suggestions or a strategy to find the >> solution ? >> > > "I don't see relevant garbage collection : heap size and permgen > have correct dimentions." > > Define "correct". Are you really *logging* the JVM Garbage > Collection, and do you *know* that this is not the issue ? > > (Note that 4 seconds seems an awfully long time for a GC; but one > would want to eliminate this with certainty, before looking any > further). > > >> >> >> Da: Martijn Bos >> <mart...@maboc.nl> Inviato: lunedì 3 ottobre 2016 21.05 A: Tomcat >> Users List Oggetto: Re: Strange wait time in my application - >> Tomcat 7.0.67 >> >> On 2016-10-03 07:56:34, Tullio Bettinazzi wrote: >>> I've an application under tomcat. When only a one or two users >>> works on it everithing is ok. When the number of users grows >>> the application slows down. Is not a memory nor a cpu problem : >>> using top I see the system resources quite free. I don't see >>> relevant garbage collection : heap size and permgen have >>> correct dimentions. No other applications are running on the >>> system. I log more or less every relevant operation in my >>> system (db query and so on) and I see that every slowdown is >>> concentered in a single operation. I mean all operations take >>> "normal" time but one or two of them take 4 seconds more. The >>> "slowing" operations are not the same in different executions, >>> and theydo not have a specific type (not only DB query, not >>> only DB stored procedures, not only.). It seems like if the >>> thread is frozen for a fixed amount fo time (4 seconds more or >>> less) and then it restarts. I don't think it's a "queue" >>> problem because otherwise the wait time would be unperdictable >>> and not a "fixed" 4 seconds time. I don't know any parameter >>> impacting on that behaviour. I use Tomcat 7.0.32 with JVM >>> 1.7.0.67 on a Linux server. Could someone suggest a solution >>> for my problem or, at least, an investigation strategy. Tks >>> >>> Tullio >>> >> >> >> The few examples that you mention are all database related >> (query/stored procedure). Can it be that your connection pool (if >> used) combined with not closing connections is part of the >> problem. >> >> I can imagine : 1) Maybe you run out of conenctions, because >> connections are not properly closed. And also the connection pool >> teminates connections when they are not used for 4 seconds. After >> 4 seconds the pool can recreate connections again. >> >> or 2) Maybe your connection pool has very limited connections. >> With one or two users this limited number of connections in the >> pool will suffice. If there are more users, the max. number of >> connections isn't enough. The pool then has to wait for >> connections to
Re: Strange wait time in my application - Tomcat 7.0.67
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Tullio, On 10/17/16 12:24 PM, Tullio Bettinazzi wrote: > I monitored it using Yourkit profiler and I didn't see any extreme > GC. > > I noticed anotehr thing : looking for 4 users which had the > problem they seems to have the problem "in sequence". > > The first user "stops" for 4 seconds then another one "stops" and > so on : they don't seem to stop at the sam time (as it would be > expected in case of GC) but in sequence. My first reaction is that this has to be a database issue. Are you using a traditional RDBMS under the hood? Can you look for long-running queries on the db? Maybe you have a table that is being locked during a transaction which is holding-up users. Maybe you have a database connection pool without enough connections in it. During times of heavy traffic, when you have users "stalling", what does a thread dump show you? - -chris > Da: André Warnier (tomcat) > <a...@ice-sa.com> Inviato: lunedì 17 ottobre 2016 18.01 A: > users@tomcat.apache.org Oggetto: Re: Strange wait time in my > application - Tomcat 7.0.67 > > On 17.10.2016 17:52, Tullio Bettinazzi wrote: >> I didn't find any solution to my problem. >> >> Could someone provide suggestions or a strategy to find the >> solution ? >> > > "I don't see relevant garbage collection : heap size and permgen > have correct dimentions." > > Define "correct". Are you really *logging* the JVM Garbage > Collection, and do you *know* that this is not the issue ? > > (Note that 4 seconds seems an awfully long time for a GC; but one > would want to eliminate this with certainty, before looking any > further). > > >> >> >> Da: Martijn Bos >> <mart...@maboc.nl> Inviato: lunedì 3 ottobre 2016 21.05 A: Tomcat >> Users List Oggetto: Re: Strange wait time in my application - >> Tomcat 7.0.67 >> >> On 2016-10-03 07:56:34, Tullio Bettinazzi wrote: >>> I've an application under tomcat. When only a one or two users >>> works on it everithing is ok. When the number of users grows >>> the application slows down. Is not a memory nor a cpu problem : >>> using top I see the system resources quite free. I don't see >>> relevant garbage collection : heap size and permgen have >>> correct dimentions. No other applications are running on the >>> system. I log more or less every relevant operation in my >>> system (db query and so on) and I see that every slowdown is >>> concentered in a single operation. I mean all operations take >>> "normal" time but one or two of them take 4 seconds more. The >>> "slowing" operations are not the same in different executions, >>> and theydo not have a specific type (not only DB query, not >>> only DB stored procedures, not only.). It seems like if the >>> thread is frozen for a fixed amount fo time (4 seconds more or >>> less) and then it restarts. I don't think it's a "queue" >>> problem because otherwise the wait time would be unperdictable >>> and not a "fixed" 4 seconds time. I don't know any parameter >>> impacting on that behaviour. I use Tomcat 7.0.32 with JVM >>> 1.7.0.67 on a Linux server. Could someone suggest a solution >>> for my problem or, at least, an investigation strategy. Tks >>> >>> Tullio >>> >> >> >> The few examples that you mention are all database related >> (query/stored procedure). Can it be that your connection pool (if >> used) combined with not closing connections is part of the >> problem. >> >> I can imagine : 1) Maybe you run out of conenctions, because >> connections are not properly closed. And also the connection pool >> teminates connections when they are not used for 4 seconds. After >> 4 seconds the pool can recreate connections again. >> >> or 2) Maybe your connection pool has very limited connections. >> With one or two users this limited number of connections in the >> pool will suffice. If there are more users, the max. number of >> connections isn't enough. The pool then has to wait for >> connections to become fee again. >> >> >> (uhhI'm not an expert at all, but the above came immediately >> to my mind) >> >> -- Met vriendelijke groet, >> >> Martijn Bos +31 6 39477001 >> >> (Public pgp-key : http://maboc.nl/pubkey.maboc.asc) > maboc.nl<http://maboc.nl/pubkey.maboc.asc> mabo
Re: Strange wait time in my application - Tomcat 7.0.67
No further help ? Tks Da: André Warnier (tomcat) <a...@ice-sa.com> Inviato: lunedì 17 ottobre 2016 18.01 A: users@tomcat.apache.org Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 On 17.10.2016 17:52, Tullio Bettinazzi wrote: > I didn't find any solution to my problem. > > Could someone provide suggestions or a strategy to find the solution ? > "I don't see relevant garbage collection : heap size and permgen have correct dimentions." Define "correct". Are you really *logging* the JVM Garbage Collection, and do you *know* that this is not the issue ? (Note that 4 seconds seems an awfully long time for a GC; but one would want to eliminate this with certainty, before looking any further). > > > > Da: Martijn Bos <mart...@maboc.nl> > Inviato: lunedì 3 ottobre 2016 21.05 > A: Tomcat Users List > Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 > > On 2016-10-03 07:56:34, Tullio Bettinazzi wrote: >> I've an application under tomcat. >> When only a one or two users works on it everithing is ok. >> When the number of users grows the application slows down. >> Is not a memory nor a cpu problem : using top I see the system resources >> quite free. >> I don't see relevant garbage collection : heap size and permgen have correct >> dimentions. >> No other applications are running on the system. >> I log more or less every relevant operation in my system (db query and so >> on) and I see that every slowdown is concentered in a single operation. >> I mean all operations take "normal" time but one or two of them take 4 >> seconds more. >> The "slowing" operations are not the same in different executions, and >> theydo not have a specific type (not only DB query, not only DB stored >> procedures, not only.). >> It seems like if the thread is frozen for a fixed amount fo time (4 seconds >> more or less) and then it restarts. >> I don't think it's a "queue" problem because otherwise the wait time would >> be unperdictable and not a "fixed" 4 seconds time. >> I don't know any parameter impacting on that behaviour. >> I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. >> Could someone suggest a solution for my problem or, at least, an >> investigation strategy. >> Tks >> >> Tullio >> > > > The few examples that you mention are all database related (query/stored > procedure). > Can it be that your connection pool (if used) combined with not closing > connections is part of the problem. > > I can imagine : > 1) > Maybe you run out of conenctions, because connections are not properly closed. > And also the connection pool teminates connections when they are not used for > 4 seconds. > After 4 seconds the pool can recreate connections again. > > or 2) > Maybe your connection pool has very limited connections. > With one or two users this limited number of connections in the pool will > suffice. > If there are more users, the max. number of connections isn't enough. > The pool then has to wait for connections to become fee again. > > > (uhhI'm not an expert at all, but the above came immediately to my mind) > > -- > Met vriendelijke groet, > > Martijn Bos > +31 6 39477001 > > (Public pgp-key : http://maboc.nl/pubkey.maboc.asc) maboc.nl<http://maboc.nl/pubkey.maboc.asc> maboc.nl -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1 mQENBFJ6LpgBCADL9w2eicatZKiw4xijCVC8WZpcPOr2So6jFfQ6nWk3bTXoVsHk sYgdLIFeCn9Wn+EeC+CSoosyMcUeijKH5yVqc/mcg0 ... > maboc.nl<http://maboc.nl/pubkey.maboc.asc> > maboc.nl > -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1 > mQENBFJ6LpgBCADL9w2eicatZKiw4xijCVC8WZpcPOr2So6jFfQ6nWk3bTXoVsHk > sYgdLIFeCn9Wn+EeC+CSoosyMcUeijKH5yVqc/mcg0 ... > > > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange wait time in my application - Tomcat 7.0.67
I monitored it using Yourkit profiler and I didn't see any extreme GC. I noticed anotehr thing : looking for 4 users which had the problem they seems to have the problem "in sequence". The first user "stops" for 4 seconds then another one "stops" and so on : they don't seem to stop at the sam time (as it would be expected in case of GC) but in sequence. Tks Da: André Warnier (tomcat) <a...@ice-sa.com> Inviato: lunedì 17 ottobre 2016 18.01 A: users@tomcat.apache.org Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 On 17.10.2016 17:52, Tullio Bettinazzi wrote: > I didn't find any solution to my problem. > > Could someone provide suggestions or a strategy to find the solution ? > "I don't see relevant garbage collection : heap size and permgen have correct dimentions." Define "correct". Are you really *logging* the JVM Garbage Collection, and do you *know* that this is not the issue ? (Note that 4 seconds seems an awfully long time for a GC; but one would want to eliminate this with certainty, before looking any further). > > > > Da: Martijn Bos <mart...@maboc.nl> > Inviato: lunedì 3 ottobre 2016 21.05 > A: Tomcat Users List > Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 > > On 2016-10-03 07:56:34, Tullio Bettinazzi wrote: >> I've an application under tomcat. >> When only a one or two users works on it everithing is ok. >> When the number of users grows the application slows down. >> Is not a memory nor a cpu problem : using top I see the system resources >> quite free. >> I don't see relevant garbage collection : heap size and permgen have correct >> dimentions. >> No other applications are running on the system. >> I log more or less every relevant operation in my system (db query and so >> on) and I see that every slowdown is concentered in a single operation. >> I mean all operations take "normal" time but one or two of them take 4 >> seconds more. >> The "slowing" operations are not the same in different executions, and >> theydo not have a specific type (not only DB query, not only DB stored >> procedures, not only.). >> It seems like if the thread is frozen for a fixed amount fo time (4 seconds >> more or less) and then it restarts. >> I don't think it's a "queue" problem because otherwise the wait time would >> be unperdictable and not a "fixed" 4 seconds time. >> I don't know any parameter impacting on that behaviour. >> I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. >> Could someone suggest a solution for my problem or, at least, an >> investigation strategy. >> Tks >> >> Tullio >> > > > The few examples that you mention are all database related (query/stored > procedure). > Can it be that your connection pool (if used) combined with not closing > connections is part of the problem. > > I can imagine : > 1) > Maybe you run out of conenctions, because connections are not properly closed. > And also the connection pool teminates connections when they are not used for > 4 seconds. > After 4 seconds the pool can recreate connections again. > > or 2) > Maybe your connection pool has very limited connections. > With one or two users this limited number of connections in the pool will > suffice. > If there are more users, the max. number of connections isn't enough. > The pool then has to wait for connections to become fee again. > > > (uhhI'm not an expert at all, but the above came immediately to my mind) > > -- > Met vriendelijke groet, > > Martijn Bos > +31 6 39477001 > > (Public pgp-key : http://maboc.nl/pubkey.maboc.asc) maboc.nl<http://maboc.nl/pubkey.maboc.asc> maboc.nl -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1 mQENBFJ6LpgBCADL9w2eicatZKiw4xijCVC8WZpcPOr2So6jFfQ6nWk3bTXoVsHk sYgdLIFeCn9Wn+EeC+CSoosyMcUeijKH5yVqc/mcg0 ... > maboc.nl<http://maboc.nl/pubkey.maboc.asc> maboc.nl<http://maboc.nl/pubkey.maboc.asc> maboc.nl -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1 mQENBFJ6LpgBCADL9w2eicatZKiw4xijCVC8WZpcPOr2So6jFfQ6nWk3bTXoVsHk sYgdLIFeCn9Wn+EeC+CSoosyMcUeijKH5yVqc/mcg0 ... > maboc.nl > -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1 > mQENBFJ6LpgBCADL9w2eicatZKiw4xijCVC8WZpcPOr2So6jFfQ6nWk3bTXoVsHk > sYgdLIFeCn9Wn+EeC+CSoosyMcUeijKH5yVqc/mcg0 ... > > > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange wait time in my application - Tomcat 7.0.67
On 17.10.2016 17:52, Tullio Bettinazzi wrote: I didn't find any solution to my problem. Could someone provide suggestions or a strategy to find the solution ? "I don't see relevant garbage collection : heap size and permgen have correct dimentions." Define "correct". Are you really *logging* the JVM Garbage Collection, and do you *know* that this is not the issue ? (Note that 4 seconds seems an awfully long time for a GC; but one would want to eliminate this with certainty, before looking any further). Da: Martijn Bos <mart...@maboc.nl> Inviato: lunedì 3 ottobre 2016 21.05 A: Tomcat Users List Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 On 2016-10-03 07:56:34, Tullio Bettinazzi wrote: I've an application under tomcat. When only a one or two users works on it everithing is ok. When the number of users grows the application slows down. Is not a memory nor a cpu problem : using top I see the system resources quite free. I don't see relevant garbage collection : heap size and permgen have correct dimentions. No other applications are running on the system. I log more or less every relevant operation in my system (db query and so on) and I see that every slowdown is concentered in a single operation. I mean all operations take "normal" time but one or two of them take 4 seconds more. The "slowing" operations are not the same in different executions, and theydo not have a specific type (not only DB query, not only DB stored procedures, not only.). It seems like if the thread is frozen for a fixed amount fo time (4 seconds more or less) and then it restarts. I don't think it's a "queue" problem because otherwise the wait time would be unperdictable and not a "fixed" 4 seconds time. I don't know any parameter impacting on that behaviour. I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. Could someone suggest a solution for my problem or, at least, an investigation strategy. Tks Tullio The few examples that you mention are all database related (query/stored procedure). Can it be that your connection pool (if used) combined with not closing connections is part of the problem. I can imagine : 1) Maybe you run out of conenctions, because connections are not properly closed. And also the connection pool teminates connections when they are not used for 4 seconds. After 4 seconds the pool can recreate connections again. or 2) Maybe your connection pool has very limited connections. With one or two users this limited number of connections in the pool will suffice. If there are more users, the max. number of connections isn't enough. The pool then has to wait for connections to become fee again. (uhhI'm not an expert at all, but the above came immediately to my mind) -- Met vriendelijke groet, Martijn Bos +31 6 39477001 (Public pgp-key : http://maboc.nl/pubkey.maboc.asc) maboc.nl<http://maboc.nl/pubkey.maboc.asc> maboc.nl -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1 mQENBFJ6LpgBCADL9w2eicatZKiw4xijCVC8WZpcPOr2So6jFfQ6nWk3bTXoVsHk sYgdLIFeCn9Wn+EeC+CSoosyMcUeijKH5yVqc/mcg0 ... - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange wait time in my application - Tomcat 7.0.67
I didn't find any solution to my problem. Could someone provide suggestions or a strategy to find the solution ? Tks Da: Martijn Bos <mart...@maboc.nl> Inviato: lunedì 3 ottobre 2016 21.05 A: Tomcat Users List Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 On 2016-10-03 07:56:34, Tullio Bettinazzi wrote: > I've an application under tomcat. > When only a one or two users works on it everithing is ok. > When the number of users grows the application slows down. > Is not a memory nor a cpu problem : using top I see the system resources > quite free. > I don't see relevant garbage collection : heap size and permgen have correct > dimentions. > No other applications are running on the system. > I log more or less every relevant operation in my system (db query and so on) > and I see that every slowdown is concentered in a single operation. > I mean all operations take "normal" time but one or two of them take 4 > seconds more. > The "slowing" operations are not the same in different executions, and theydo > not have a specific type (not only DB query, not only DB stored procedures, > not only.). > It seems like if the thread is frozen for a fixed amount fo time (4 seconds > more or less) and then it restarts. > I don't think it's a "queue" problem because otherwise the wait time would be > unperdictable and not a "fixed" 4 seconds time. > I don't know any parameter impacting on that behaviour. > I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. > Could someone suggest a solution for my problem or, at least, an > investigation strategy. > Tks > > Tullio > The few examples that you mention are all database related (query/stored procedure). Can it be that your connection pool (if used) combined with not closing connections is part of the problem. I can imagine : 1) Maybe you run out of conenctions, because connections are not properly closed. And also the connection pool teminates connections when they are not used for 4 seconds. After 4 seconds the pool can recreate connections again. or 2) Maybe your connection pool has very limited connections. With one or two users this limited number of connections in the pool will suffice. If there are more users, the max. number of connections isn't enough. The pool then has to wait for connections to become fee again. (uhhI'm not an expert at all, but the above came immediately to my mind) -- Met vriendelijke groet, Martijn Bos +31 6 39477001 (Public pgp-key : http://maboc.nl/pubkey.maboc.asc) maboc.nl<http://maboc.nl/pubkey.maboc.asc> maboc.nl -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1 mQENBFJ6LpgBCADL9w2eicatZKiw4xijCVC8WZpcPOr2So6jFfQ6nWk3bTXoVsHk sYgdLIFeCn9Wn+EeC+CSoosyMcUeijKH5yVqc/mcg0 ...
Re: Strange wait time in my application - Tomcat 7.0.67
Tks for Your help but : . problems comes out mainly in Db activities because are the longest one but I saw the same problem also in different operations like xsl parsing . I don't use a connection pool but only private connections one for each user Any further suggestion ? Tkls Da: Martijn Bos <mart...@maboc.nl> Inviato: lunedì 3 ottobre 2016 21.05 A: Tomcat Users List Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 On 2016-10-03 07:56:34, Tullio Bettinazzi wrote: > I've an application under tomcat. > When only a one or two users works on it everithing is ok. > When the number of users grows the application slows down. > Is not a memory nor a cpu problem : using top I see the system resources > quite free. > I don't see relevant garbage collection : heap size and permgen have correct > dimentions. > No other applications are running on the system. > I log more or less every relevant operation in my system (db query and so on) > and I see that every slowdown is concentered in a single operation. > I mean all operations take "normal" time but one or two of them take 4 > seconds more. > The "slowing" operations are not the same in different executions, and theydo > not have a specific type (not only DB query, not only DB stored procedures, > not only.). > It seems like if the thread is frozen for a fixed amount fo time (4 seconds > more or less) and then it restarts. > I don't think it's a "queue" problem because otherwise the wait time would be > unperdictable and not a "fixed" 4 seconds time. > I don't know any parameter impacting on that behaviour. > I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. > Could someone suggest a solution for my problem or, at least, an > investigation strategy. > Tks > > Tullio > The few examples that you mention are all database related (query/stored procedure). Can it be that your connection pool (if used) combined with not closing connections is part of the problem. I can imagine : 1) Maybe you run out of conenctions, because connections are not properly closed. And also the connection pool teminates connections when they are not used for 4 seconds. After 4 seconds the pool can recreate connections again. or 2) Maybe your connection pool has very limited connections. With one or two users this limited number of connections in the pool will suffice. If there are more users, the max. number of connections isn't enough. The pool then has to wait for connections to become fee again. (uhhI'm not an expert at all, but the above came immediately to my mind) -- Met vriendelijke groet, Martijn Bos +31 6 39477001 (Public pgp-key : http://maboc.nl/pubkey.maboc.asc) maboc.nl<http://maboc.nl/pubkey.maboc.asc> maboc.nl -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1 mQENBFJ6LpgBCADL9w2eicatZKiw4xijCVC8WZpcPOr2So6jFfQ6nWk3bTXoVsHk sYgdLIFeCn9Wn+EeC+CSoosyMcUeijKH5yVqc/mcg0 ...
Re: Strange wait time in my application - Tomcat 7.0.67
On 2016-10-03 07:56:34, Tullio Bettinazzi wrote: > I've an application under tomcat. > When only a one or two users works on it everithing is ok. > When the number of users grows the application slows down. > Is not a memory nor a cpu problem : using top I see the system resources > quite free. > I don't see relevant garbage collection : heap size and permgen have correct > dimentions. > No other applications are running on the system. > I log more or less every relevant operation in my system (db query and so on) > and I see that every slowdown is concentered in a single operation. > I mean all operations take "normal" time but one or two of them take 4 > seconds more. > The "slowing" operations are not the same in different executions, and theydo > not have a specific type (not only DB query, not only DB stored procedures, > not only.). > It seems like if the thread is frozen for a fixed amount fo time (4 seconds > more or less) and then it restarts. > I don't think it's a "queue" problem because otherwise the wait time would be > unperdictable and not a "fixed" 4 seconds time. > I don't know any parameter impacting on that behaviour. > I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. > Could someone suggest a solution for my problem or, at least, an > investigation strategy. > Tks > > Tullio > The few examples that you mention are all database related (query/stored procedure). Can it be that your connection pool (if used) combined with not closing connections is part of the problem. I can imagine : 1) Maybe you run out of conenctions, because connections are not properly closed. And also the connection pool teminates connections when they are not used for 4 seconds. After 4 seconds the pool can recreate connections again. or 2) Maybe your connection pool has very limited connections. With one or two users this limited number of connections in the pool will suffice. If there are more users, the max. number of connections isn't enough. The pool then has to wait for connections to become fee again. (uhhI'm not an expert at all, but the above came immediately to my mind) -- Met vriendelijke groet, Martijn Bos +31 6 39477001 (Public pgp-key : http://maboc.nl/pubkey.maboc.asc) signature.asc Description: Digital signature
Re: Strange wait time in my application - Tomcat 7.0.67
Hi Tullio, well I am biased of course, but there are multiple. First, profiles are never used in production, MoSKito is aimed for production use. Not test lab, not dev machine, production, hardcore on all servers you've got. Than MoSKito also offers analysis tools, thresholds, alerts, charts etc. Finally MoSKito allows you to integrate business data into monitoring, things like registration count, checkout, gender of user, and so on... 'standard profiler' - is something designed to be used in the development environment. moskito is an apm tool. hope that helps regards Leon On Mon, Oct 3, 2016 at 4:21 PM, Tullio Bettinazzi <tullio0...@live.it> wrote: > Please help me to understand diffrences beween Moskito and standard > profilers. > > Tks > > Tullio > > > > Da: Leon Rosenberg <rosenberg.l...@gmail.com> > Inviato: lunedì 3 ottobre 2016 14.29 > A: Tomcat Users List > Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 > > Hi Tullio, > > you could download and integrate MoSKito -> http://www.moskito.org. After > Health and Performance Monitoring for Java Applications | MoSKito< > http://www.moskito.org/> > www.moskito.org > MoSKito: Health and Performance Monitoring for Java applications. Complete > ecosystem for DevOps. Free and open source > > > > integration as described here > blog.anotheria.net/msk/the-complete-moskito-integration-guide-step-1/ or > here: http://www.moskito.org/integration.html you can start your > Integration for MoSKito, the Open Source Health and Performance Monitoring > System for Java Applications<http://www.moskito.org/integration.html> > www.moskito.org > MoSKito Integration > > > > application and setup a tracer. > A tracer, explained here: > http://blog.anotheria.net/msk/newest-hottest-tracers/, can be used to > automatically trace all requests passing a specific point and compare the > execution times. It will keep the longest requests along with the execution > time in each monitored component. > So if there is some part of code you have, that lasts longer from time to > time, the tracer will find it. > > regards > Leon > > > On Mon, Oct 3, 2016 at 2:12 PM, Mark Thomas <ma...@apache.org> wrote: > > > On 03/10/2016 11:19, Tullio Bettinazzi wrote: > > > I already use Yourkit but it doesn't seems a load problem. > > > > > > The delay is not spread over all operaions but concentred in only one > or > > two and allways takes 4 secs more than the normal operation time. > > > > > > Could You suggest how to use Yourkit in this schenario ? > > > > I'd look at GC activity and detailed CPU profiling. > > > > Mark > > > > > > > > Tks > > > > > > Tullio > > > > > > > > > > > > Da: Mark Thomas <ma...@apache.org> > > > Inviato: lunedì 3 ottobre 2016 10.39 > > > A: Tomcat Users List > > > Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 > > > > > > On 03/10/2016 08:56, Tullio Bettinazzi wrote: > > >> I've an application under tomcat. > > >> When only a one or two users works on it everithing is ok. > > >> When the number of users grows the application slows down. > > >> Is not a memory nor a cpu problem : using top I see the system > > resources quite free. > > >> I don't see relevant garbage collection : heap size and permgen have > > correct dimentions. > > >> No other applications are running on the system. > > >> I log more or less every relevant operation in my system (db query and > > so on) and I see that every slowdown is concentered in a single > operation. > > >> I mean all operations take "normal" time but one or two of them take 4 > > seconds more. > > >> The "slowing" operations are not the same in different executions, and > > theydo not have a specific type (not only DB query, not only DB stored > > procedures, not only.). > > >> It seems like if the thread is frozen for a fixed amount fo time (4 > > seconds more or less) and then it restarts. > > >> I don't think it's a "queue" problem because otherwise the wait time > > would be unperdictable and not a "fixed" 4 seconds time. > > >> I don't know any parameter impacting on that behaviour. > > >> I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. > > >> Could someone suggest a solution for my problem or, at least
Re: Strange wait time in my application - Tomcat 7.0.67
Please help me to understand diffrences beween Moskito and standard profilers. Tks Tullio Da: Leon Rosenberg <rosenberg.l...@gmail.com> Inviato: lunedì 3 ottobre 2016 14.29 A: Tomcat Users List Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 Hi Tullio, you could download and integrate MoSKito -> http://www.moskito.org. After Health and Performance Monitoring for Java Applications | MoSKito<http://www.moskito.org/> www.moskito.org MoSKito: Health and Performance Monitoring for Java applications. Complete ecosystem for DevOps. Free and open source integration as described here blog.anotheria.net/msk/the-complete-moskito-integration-guide-step-1/ or here: http://www.moskito.org/integration.html you can start your Integration for MoSKito, the Open Source Health and Performance Monitoring System for Java Applications<http://www.moskito.org/integration.html> www.moskito.org MoSKito Integration application and setup a tracer. A tracer, explained here: http://blog.anotheria.net/msk/newest-hottest-tracers/, can be used to automatically trace all requests passing a specific point and compare the execution times. It will keep the longest requests along with the execution time in each monitored component. So if there is some part of code you have, that lasts longer from time to time, the tracer will find it. regards Leon On Mon, Oct 3, 2016 at 2:12 PM, Mark Thomas <ma...@apache.org> wrote: > On 03/10/2016 11:19, Tullio Bettinazzi wrote: > > I already use Yourkit but it doesn't seems a load problem. > > > > The delay is not spread over all operaions but concentred in only one or > two and allways takes 4 secs more than the normal operation time. > > > > Could You suggest how to use Yourkit in this schenario ? > > I'd look at GC activity and detailed CPU profiling. > > Mark > > > > > Tks > > > > Tullio > > > > > > > > Da: Mark Thomas <ma...@apache.org> > > Inviato: lunedì 3 ottobre 2016 10.39 > > A: Tomcat Users List > > Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 > > > > On 03/10/2016 08:56, Tullio Bettinazzi wrote: > >> I've an application under tomcat. > >> When only a one or two users works on it everithing is ok. > >> When the number of users grows the application slows down. > >> Is not a memory nor a cpu problem : using top I see the system > resources quite free. > >> I don't see relevant garbage collection : heap size and permgen have > correct dimentions. > >> No other applications are running on the system. > >> I log more or less every relevant operation in my system (db query and > so on) and I see that every slowdown is concentered in a single operation. > >> I mean all operations take "normal" time but one or two of them take 4 > seconds more. > >> The "slowing" operations are not the same in different executions, and > theydo not have a specific type (not only DB query, not only DB stored > procedures, not only.). > >> It seems like if the thread is frozen for a fixed amount fo time (4 > seconds more or less) and then it restarts. > >> I don't think it's a "queue" problem because otherwise the wait time > would be unperdictable and not a "fixed" 4 seconds time. > >> I don't know any parameter impacting on that behaviour. > >> I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. > >> Could someone suggest a solution for my problem or, at least, an > investigation strategy. > > > > https://www.yourkit.com/java/profiler/features/ > > Performance and Memory Java Profiler - YourKit Java Profiler< > https://www.yourkit.com/java/profiler/features/> > > www.yourkit.com<http://www.yourkit.com> > > Easy to use performance and memory profiler for .NET framework. Supports > ASP.NET, Silverlight, .NET Windows services and more. > > > > > > > > > > Mark > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Strange wait time in my application - Tomcat 7.0.67
Hi Tullio, you could download and integrate MoSKito -> http://www.moskito.org. After integration as described here blog.anotheria.net/msk/the-complete-moskito-integration-guide-step-1/ or here: http://www.moskito.org/integration.html you can start your application and setup a tracer. A tracer, explained here: http://blog.anotheria.net/msk/newest-hottest-tracers/, can be used to automatically trace all requests passing a specific point and compare the execution times. It will keep the longest requests along with the execution time in each monitored component. So if there is some part of code you have, that lasts longer from time to time, the tracer will find it. regards Leon On Mon, Oct 3, 2016 at 2:12 PM, Mark Thomas <ma...@apache.org> wrote: > On 03/10/2016 11:19, Tullio Bettinazzi wrote: > > I already use Yourkit but it doesn't seems a load problem. > > > > The delay is not spread over all operaions but concentred in only one or > two and allways takes 4 secs more than the normal operation time. > > > > Could You suggest how to use Yourkit in this schenario ? > > I'd look at GC activity and detailed CPU profiling. > > Mark > > > > > Tks > > > > Tullio > > > > > > > > Da: Mark Thomas <ma...@apache.org> > > Inviato: lunedì 3 ottobre 2016 10.39 > > A: Tomcat Users List > > Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 > > > > On 03/10/2016 08:56, Tullio Bettinazzi wrote: > >> I've an application under tomcat. > >> When only a one or two users works on it everithing is ok. > >> When the number of users grows the application slows down. > >> Is not a memory nor a cpu problem : using top I see the system > resources quite free. > >> I don't see relevant garbage collection : heap size and permgen have > correct dimentions. > >> No other applications are running on the system. > >> I log more or less every relevant operation in my system (db query and > so on) and I see that every slowdown is concentered in a single operation. > >> I mean all operations take "normal" time but one or two of them take 4 > seconds more. > >> The "slowing" operations are not the same in different executions, and > theydo not have a specific type (not only DB query, not only DB stored > procedures, not only.). > >> It seems like if the thread is frozen for a fixed amount fo time (4 > seconds more or less) and then it restarts. > >> I don't think it's a "queue" problem because otherwise the wait time > would be unperdictable and not a "fixed" 4 seconds time. > >> I don't know any parameter impacting on that behaviour. > >> I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. > >> Could someone suggest a solution for my problem or, at least, an > investigation strategy. > > > > https://www.yourkit.com/java/profiler/features/ > > Performance and Memory Java Profiler - YourKit Java Profiler< > https://www.yourkit.com/java/profiler/features/> > > www.yourkit.com > > Easy to use performance and memory profiler for .NET framework. Supports > ASP.NET, Silverlight, .NET Windows services and more. > > > > > > > > > > Mark > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Strange wait time in my application - Tomcat 7.0.67
On 03/10/2016 11:19, Tullio Bettinazzi wrote: > I already use Yourkit but it doesn't seems a load problem. > > The delay is not spread over all operaions but concentred in only one or two > and allways takes 4 secs more than the normal operation time. > > Could You suggest how to use Yourkit in this schenario ? I'd look at GC activity and detailed CPU profiling. Mark > > Tks > > Tullio > > > > Da: Mark Thomas <ma...@apache.org> > Inviato: lunedì 3 ottobre 2016 10.39 > A: Tomcat Users List > Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 > > On 03/10/2016 08:56, Tullio Bettinazzi wrote: >> I've an application under tomcat. >> When only a one or two users works on it everithing is ok. >> When the number of users grows the application slows down. >> Is not a memory nor a cpu problem : using top I see the system resources >> quite free. >> I don't see relevant garbage collection : heap size and permgen have correct >> dimentions. >> No other applications are running on the system. >> I log more or less every relevant operation in my system (db query and so >> on) and I see that every slowdown is concentered in a single operation. >> I mean all operations take "normal" time but one or two of them take 4 >> seconds more. >> The "slowing" operations are not the same in different executions, and >> theydo not have a specific type (not only DB query, not only DB stored >> procedures, not only.). >> It seems like if the thread is frozen for a fixed amount fo time (4 seconds >> more or less) and then it restarts. >> I don't think it's a "queue" problem because otherwise the wait time would >> be unperdictable and not a "fixed" 4 seconds time. >> I don't know any parameter impacting on that behaviour. >> I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. >> Could someone suggest a solution for my problem or, at least, an >> investigation strategy. > > https://www.yourkit.com/java/profiler/features/ > Performance and Memory Java Profiler - YourKit Java > Profiler<https://www.yourkit.com/java/profiler/features/> > www.yourkit.com > Easy to use performance and memory profiler for .NET framework. Supports > ASP.NET, Silverlight, .NET Windows services and more. > > > > > Mark > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange wait time in my application - Tomcat 7.0.67
I already use Yourkit but it doesn't seems a load problem. The delay is not spread over all operaions but concentred in only one or two and allways takes 4 secs more than the normal operation time. Could You suggest how to use Yourkit in this schenario ? Tks Tullio Da: Mark Thomas <ma...@apache.org> Inviato: lunedì 3 ottobre 2016 10.39 A: Tomcat Users List Oggetto: Re: Strange wait time in my application - Tomcat 7.0.67 On 03/10/2016 08:56, Tullio Bettinazzi wrote: > I've an application under tomcat. > When only a one or two users works on it everithing is ok. > When the number of users grows the application slows down. > Is not a memory nor a cpu problem : using top I see the system resources > quite free. > I don't see relevant garbage collection : heap size and permgen have correct > dimentions. > No other applications are running on the system. > I log more or less every relevant operation in my system (db query and so on) > and I see that every slowdown is concentered in a single operation. > I mean all operations take "normal" time but one or two of them take 4 > seconds more. > The "slowing" operations are not the same in different executions, and theydo > not have a specific type (not only DB query, not only DB stored procedures, > not only.). > It seems like if the thread is frozen for a fixed amount fo time (4 seconds > more or less) and then it restarts. > I don't think it's a "queue" problem because otherwise the wait time would be > unperdictable and not a "fixed" 4 seconds time. > I don't know any parameter impacting on that behaviour. > I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. > Could someone suggest a solution for my problem or, at least, an > investigation strategy. https://www.yourkit.com/java/profiler/features/ Performance and Memory Java Profiler - YourKit Java Profiler<https://www.yourkit.com/java/profiler/features/> www.yourkit.com Easy to use performance and memory profiler for .NET framework. Supports ASP.NET, Silverlight, .NET Windows services and more. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange wait time in my application - Tomcat 7.0.67
On 03/10/2016 08:56, Tullio Bettinazzi wrote: > I've an application under tomcat. > When only a one or two users works on it everithing is ok. > When the number of users grows the application slows down. > Is not a memory nor a cpu problem : using top I see the system resources > quite free. > I don't see relevant garbage collection : heap size and permgen have correct > dimentions. > No other applications are running on the system. > I log more or less every relevant operation in my system (db query and so on) > and I see that every slowdown is concentered in a single operation. > I mean all operations take "normal" time but one or two of them take 4 > seconds more. > The "slowing" operations are not the same in different executions, and theydo > not have a specific type (not only DB query, not only DB stored procedures, > not only.). > It seems like if the thread is frozen for a fixed amount fo time (4 seconds > more or less) and then it restarts. > I don't think it's a "queue" problem because otherwise the wait time would be > unperdictable and not a "fixed" 4 seconds time. > I don't know any parameter impacting on that behaviour. > I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. > Could someone suggest a solution for my problem or, at least, an > investigation strategy. https://www.yourkit.com/java/profiler/features/ Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Strange wait time in my application - Tomcat 7.0.67
I've an application under tomcat. When only a one or two users works on it everithing is ok. When the number of users grows the application slows down. Is not a memory nor a cpu problem : using top I see the system resources quite free. I don't see relevant garbage collection : heap size and permgen have correct dimentions. No other applications are running on the system. I log more or less every relevant operation in my system (db query and so on) and I see that every slowdown is concentered in a single operation. I mean all operations take "normal" time but one or two of them take 4 seconds more. The "slowing" operations are not the same in different executions, and theydo not have a specific type (not only DB query, not only DB stored procedures, not only.). It seems like if the thread is frozen for a fixed amount fo time (4 seconds more or less) and then it restarts. I don't think it's a "queue" problem because otherwise the wait time would be unperdictable and not a "fixed" 4 seconds time. I don't know any parameter impacting on that behaviour. I use Tomcat 7.0.32 with JVM 1.7.0.67 on a Linux server. Could someone suggest a solution for my problem or, at least, an investigation strategy. Tks Tullio
