Re: [Seriously OT] Help in diagnosing server unresponsiveness
On 2/12/2013 9:36 AM, Christopher Schultz wrote: On 2/11/13 4:30 PM, Terence M. Bandoian wrote: I understand the considerations above and they are a part of the prevailing thinking. However, one underlying assumption of the supporting argument appears to be that today's programmers are not capable of developing maintainable code which I don't believe is true. As I understand it, programmer productivity is one of the most significant factors in the decision making process and it is a valid concern. IF (that's a big if) an application can be developed in half the time using a generalized solution, then that approach has to be considered along with a host of other concerns including the end product and the effect on the organization. I say reliance on generalized solutions is short-sighted because knowledge of the underlying technologies is lost, or never gained, along with the skills to work in those spheres. Are you suggesting that people who program using Java are oblivious to the innards of hardware architecture and are remain ignorant of these important details? That's the logical conclusion to your argument. I'm not saying you're wrong, but you have to admit that a Java programmer (of which I'm one) saying that using a generalized solution makes you ignorant is a bit like the pot calling the kettle black. Not at all. I probably should have said there is a potential for lost knowledge. Here are a couple of anecdotal examples that I hope will help illustrate what I mean. - I was told recently by a person in a software architect position that they use Hibernate because it prevents SQL injection. I'll give them the benefit of the doubt and assume they have other motivations but still, is escaping the input strings to a query really advanced knowledge? Think about that from the perspective of a junior or mid-level programmer who has only ever used an ORM. What happens when something goes wrong or performance has to be optimized. For that matter, what does happen? Now you have to know SQL, the DBMS, and HQL and understand Hibernate behavior. Double the complexity? Disclaimer: I don't have anything against Hibernate or JPA. I worked with an early implementation of JDO for a short time and am beginning some work with Hibernate. - A JavaScript programmer told me not too long ago that you really have to use a JavaScript library (e.g. jQuery) if you're going to use AJAX in an interface because it's just too complicated. Is instantiating and using an XMLHttpRequest object really that difficult? What about those programmers who have only ever used a JavaScript library? Another developer says he tells clients that they shouldn't consider a feature if it isn't supported by jQuery. Still another says that one of the reasons their organization uses a full-blown framework is that their programmers can't develop cross-browser compatible JavaScript. Disclaimer: I think jQuery is a wonderful library and, if you plan to make good use of the features available, it should definitely be considered. What I'm saying is there should be a good reason (really good) to add significant complexity, performance overhead, memory requirements and megabytes of code to an application. Efficiency, flexibility, repairability, extensibility and reliability are all components of software quality and all are affected by complexity. Less complex systems are easier to maintain. To continue the aside, wasted energy is wasted energy and it may become a factor in software development at some point. I think decision makers should be taught that there is more to the bottom line than dollars and cents. In my experience, by far the biggest time waster is trying to deal with code that is (or has become) unmaintainable. Re-writing just because a piece of code has become out-of-touch with current standards or because nobody understands how it works is entirely wasted effort. We have lots of places in our code where we have been spending - literally - years recording from bad decisions in the past. Granted. Reading other people's code is a learned skill and can be problematic. Isn't that where design and code reviews and coding standards come into play? Also, apples and oranges. Energy is precious resource that deserves special consideration. I'm just blue-skying here and don't have the answers but how much electricity is wasted by inefficient programming? We may have to factor that in some day. -Terence From Christopher Shultz 2/14/2013 (comments inline): Terrence, On 2/14/13 1:23 AM, Terence M. Bandoian wrote: I probably should have said there is a potential for lost knowledge. Here are a couple of anecdotal examples that I hope will help illustrate what I mean. - I was told recently by a person in a software architect position that they use Hibernate because it prevents SQL injection. :) So, it doesn't support custom WHERE clauses? Sounds like a deal-breaker for
Re: Help in diagnosing server unresponsiveness
Hi Guys, It's been a while but the nature of this problem means it may be a while between crashes. But we just had a big one which hung the system and required a reboot. I have changed the tomcat options as follows inline with all the advice and material I read to be as follows: -server -Xms1460m -Xmx11460m -Djava.awt.headless=true -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:MaxPermSize=512M -XX:NewSize=4500m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=80 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/usr/local/tomcat/logs/gc.log The garbage collection log had the following details just prior to the crash: 4163.757: [GC [1 CMS-initial-mark: 0K(5376K)] 1834200K(4152576K), 1.9237250 secs] [Times: user=1.92 sys=0.00, real=1.92 secs] 4165.682: [CMS-concurrent-mark-start] 4165.834: [CMS-concurrent-mark: 0.152/0.152 secs] [Times: user=0.15 sys=0.00, real=0.16 secs] 4165.834: [CMS-concurrent-preclean-start] 4165.849: [CMS-concurrent-preclean: 0.015/0.015 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 4165.849: [CMS-concurrent-abortable-preclean-start] CMS: abort preclean due to time 4171.285: [CMS-concurrent-abortable-preclean: 5.035/5.436 secs] [Times: user=5.05 sys=0.00, real=5.44 secs] 4171.285: [GC[YG occupancy: 1834200 K (4147200 K)]4171.286: [Rescan (parallel) , 1.5184720 secs]4172.804: [weak refs processing, 0.0001420 secs]4172.804: [class unloading, 0.0118860 secs]4172.816: [scrub symbol string tables, 0.0141570 secs] [1 CMS-remark: 0K(5376K)] 1834200K(4152576K), 1.5484470 secs] And the JavaMelody monitoring indicated the crash occurred at the same time as garbage collection took place. Basically the Garbage collector time chart peaked at 20 and ran for about 15minutes. I has a look at the garbage collector chart over a longer period and when the collector runs more frequently it appears to be more stable. Any advice on where to go next? Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Also, I forgot to add the details of the memory histogram: Heap Classes: 3,999, Instances: 6,333,516, Kilo-Bytes: 592,665 Class Size (Kb) % size Instances % instances int[]243,29641151,0842 char[]153,148251,699,59426 java.lang.String36,70861,174,68318 byte[]29,6505120,3661 java.lang.Object[]15,6992263,6124 java.util.HashMap$Entry[]9,6721117,7871 java.util.HashMap$Entry5,9701191,0613 PermGen Classes: 11, Instances: 343,293, Kilo-Bytes: 56,733 Class Size (Kb) % size Instances % instances constMethodKlass13,2402389,72226 methodKlass11,9282189,72226 constantPoolKlass9,403168,2522 symbolKlass7,27212134,88039 instanceKlassKlass6,202108,2522 constantPoolCacheKlass5,826107,3242 methodDataKlass2,52444,3441 Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
AFAIK, as best practice is recommended that if you have dedicated server, let -Xms as close as possible to -Xmx to avoid extra effort in releasing memory. I remember to read this information as recommended by Oracle (JRockit) and IBM (WebSphere) documentation (unfortunately, I don't have the papers link right here). Regards, Edson Em 20/02/2013 05:52, Zoran Avtarovski escreveu: Hi Guys, It's been a while but the nature of this problem means it may be a while between crashes. But we just had a big one which hung the system and required a reboot. I have changed the tomcat options as follows inline with all the advice and material I read to be as follows: -server -Xms1460m -Xmx11460m -Djava.awt.headless=true -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:MaxPermSize=512M -XX:NewSize=4500m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=80 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/usr/local/tomcat/logs/gc.log The garbage collection log had the following details just prior to the crash: 4163.757: [GC [1 CMS-initial-mark: 0K(5376K)] 1834200K(4152576K), 1.9237250 secs] [Times: user=1.92 sys=0.00, real=1.92 secs] 4165.682: [CMS-concurrent-mark-start] 4165.834: [CMS-concurrent-mark: 0.152/0.152 secs] [Times: user=0.15 sys=0.00, real=0.16 secs] 4165.834: [CMS-concurrent-preclean-start] 4165.849: [CMS-concurrent-preclean: 0.015/0.015 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 4165.849: [CMS-concurrent-abortable-preclean-start] CMS: abort preclean due to time 4171.285: [CMS-concurrent-abortable-preclean: 5.035/5.436 secs] [Times: user=5.05 sys=0.00, real=5.44 secs] 4171.285: [GC[YG occupancy: 1834200 K (4147200 K)]4171.286: [Rescan (parallel) , 1.5184720 secs]4172.804: [weak refs processing, 0.0001420 secs]4172.804: [class unloading, 0.0118860 secs]4172.816: [scrub symbol string tables, 0.0141570 secs] [1 CMS-remark: 0K(5376K)] 1834200K(4152576K), 1.5484470 secs] And the JavaMelody monitoring indicated the crash occurred at the same time as garbage collection took place. Basically the Garbage collector time chart peaked at 20 and ran for about 15minutes. I has a look at the garbage collector chart over a longer period and when the collector runs more frequently it appears to be more stable. Any advice on where to go next? Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On Feb 20, 2013, at 3:52 AM, Zoran Avtarovski wrote: Hi Guys, It's been a while but the nature of this problem means it may be a while between crashes. But we just had a big one which hung the system and required a reboot. Can you elaborate more on this? What OS are you running? What do you mean by hung the system? Did you get a kernel panic / Bsod? I have changed the tomcat options as follows inline with all the advice and material I read to be as follows: This can be dangerous. Especially, if you haven't tested the settings and verified that they help to increase performance and lower GC overhead for your system and applications. Applications are unique and what works to tune one of them may not work well for others. -server -Xms1460m -Xmx11460m -Djava.awt.headless=true -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:MaxPermSize=512M -XX:NewSize=4500m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=80 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/usr/local/tomcat/logs/gc.log First, what JVM are you running? vendor and version. If you are running anything but the latest version of that JVM, upgrade to the latest version. See if the problem is still present. Some comments on your JVM options... 1.) You have -XX:+UseConcMarkSweepGC listed twice 2.) You have -XX:+CMSIncrementalMode, does the following describe your system? If not, remove this setting. This feature is useful when applications that need the low pause times provided by the concurrent collector are run on machines with small numbers of processors (e.g., 1 or 2). http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#icms 3.) I'm not a fan of specifying -XX:NewSize=4500m. I think the JVM's default usually works fine, plus it's difficult to manually specify this value and get it correct. My suggestion would be to remove this option, unless you have load tested your application with and without the setting and you can 100% guarantee that it is helping performance. 4.) You have set -XX:-UseGCOverheadLimit, which could be dangerous. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms.oom Disabling this would seem unnecessary if your JVM options are tuned correctly. 5.) This option, -XX:CMSInitiatingOccupancyFraction, is another one where I would suggest using the JVM default. Unless you have load tested with and without the setting and can guarantee that setting this value improves performance. The garbage collection log had the following details just prior to the crash: 4163.757: [GC [1 CMS-initial-mark: 0K(5376K)] 1834200K(4152576K), 1.9237250 secs] [Times: user=1.92 sys=0.00, real=1.92 secs] 4165.682: [CMS-concurrent-mark-start] 4165.834: [CMS-concurrent-mark: 0.152/0.152 secs] [Times: user=0.15 sys=0.00, real=0.16 secs] 4165.834: [CMS-concurrent-preclean-start] 4165.849: [CMS-concurrent-preclean: 0.015/0.015 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 4165.849: [CMS-concurrent-abortable-preclean-start] CMS: abort preclean due to time 4171.285: [CMS-concurrent-abortable-preclean: 5.035/5.436 secs] [Times: user=5.05 sys=0.00, real=5.44 secs] 4171.285: [GC[YG occupancy: 1834200 K (4147200 K)]4171.286: [Rescan (parallel) , 1.5184720 secs]4172.804: [weak refs processing, 0.0001420 secs]4172.804: [class unloading, 0.0118860 secs]4172.816: [scrub symbol string tables, 0.0141570 secs] [1 CMS-remark: 0K(5376K)] 1834200K(4152576K), 1.5484470 secs] And the JavaMelody monitoring indicated the crash occurred at the same time as garbage collection took place. Basically the Garbage collector time chart peaked at 20 and ran for about 15minutes. I has a look at the garbage collector chart over a longer period and when the collector runs more frequently it appears to be more stable. Any advice on where to go next? 1.) Look at the Basic Troubleshooting section here. http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#icms.troubleshooting 2.) If possible, take some heap dumps when you start to notice a problem. Then you can analyze them with a profiler and see what is happening in the heap. 3.) Load test with a profiler hooked directly up to your application. Try to recreate the problem. Hope that helps. Dan Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Seriously OT] Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Terrence, On 2/14/13 1:23 AM, Terence M. Bandoian wrote: I probably should have said there is a potential for lost knowledge. Here are a couple of anecdotal examples that I hope will help illustrate what I mean. - I was told recently by a person in a software architect position that they use Hibernate because it prevents SQL injection. :) So, it doesn't support custom WHERE clauses? Sounds like a deal-breaker for me. If it does support custom WHERE clauses, then it's open to SQL Injection because programmers can still suck. - A JavaScript programmer told me not too long ago that you really have to use a JavaScript library (e.g. jQuery) if you're going to use AJAX in an interface because it's just too complicated. Is instantiating and using an XMLHttpRequest object really that difficult? No, but if you've ever done your own AJAX work (and not just something silly like sending a ping to the server or trading a 2-line XML document), you'll know that using a support library makes your life s much easier, especially when you have to support multiple browsers/versions/etc. Good news is that you can mix-and-match: if jQuery (et al) doesn't support some feature, you can write it yourself and (probably) not interfere. Still another says that one of the reasons their organization uses a full-blown framework is that their programmers can't develop cross-browser compatible JavaScript. It *is* very difficult: when you can leverage the work others are doing, I think everybody wins. Let the jQuery folks figure out how to make XML parsing and call-backs work on both MSIE 6 and NCSA Mosaic: NMFP. ;) What I'm saying is there should be a good reason (really good) to add significant complexity, performance overhead, memory requirements and megabytes of code to an application. Leverage is a powerful thing. In my experience, by far the biggest time waster is trying to deal with code that is (or has become) unmaintainable. Re-writing just because a piece of code has become out-of-touch with current standards or because nobody understands how it works is entirely wasted effort. We have lots of places in our code where we have been spending - literally - years recording from bad decisions in the past. Granted. Reading other people's code is a learned skill and can be problematic. Isn't that where design and code reviews and coding standards come into play? Also, apples and oranges. Energy is precious resource that deserves special consideration. I'm just blue-skying here and don't have the answers but how much electricity is wasted by inefficient programming? We may have to factor that in some day. Future-proofing is not possible, but it is a reasonable goal. All software rots, even if it's great software. Take Java code: anything written before generics were introduced and uses a lot of collections, etc. will now be a PITA to work with. I have code like this: we are updating it as we have to edit it for other reasons (e.g. we aren't going on a SD mission for non-generic-y code just to fix it), but it does represent some technical dept that does need to be paid at some point. Maybe the existing code is so good that it never needs to be edited, so it sits there and rots, and there are more things wrong with the code but it works perfectly well. It becomes something that irritates everyone but everyone tolerates because it's good code. If you can make some code someone else's problem, I think that's a win. It's very cynical, but having grown-up in the Java world before any good libraries were available (other than the standard Java API, which is great by any standard IMO), I've been party to a lot of time and energy spent just doing simple things like trashing some cowboy's home-grown logging API, O-R mapper, and (yes, true story) XML transformation engine. Not invented here is a pesky disease and it's really okay to leverage other people's code. There are always exceptions (I've written my own rules engine from scratch when DROOLS, JESS, etc. already existed) but you (not YOU) are a fool if you don't consider using something off-the-shelf -- especially if that shelf contains lots of great OSS and you can help support it. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEdF9UACgkQ9CaO5/Lv0PAQlQCgteFyWEhnbU+qNIXX8CGzmKWp nUYAn2TcezTK6fspfcMV9JiHEA7SRUL6 =MMge -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Seriously OT] Help in diagnosing server unresponsiveness
On 2/12/2013 9:36 AM, Christopher Schultz wrote: On 2/11/13 4:30 PM, Terence M. Bandoian wrote: I understand the considerations above and they are a part of the prevailing thinking. However, one underlying assumption of the supporting argument appears to be that today's programmers are not capable of developing maintainable code which I don't believe is true. As I understand it, programmer productivity is one of the most significant factors in the decision making process and it is a valid concern. IF (that's a big if) an application can be developed in half the time using a generalized solution, then that approach has to be considered along with a host of other concerns including the end product and the effect on the organization. I say reliance on generalized solutions is short-sighted because knowledge of the underlying technologies is lost, or never gained, along with the skills to work in those spheres. Are you suggesting that people who program using Java are oblivious to the innards of hardware architecture and are remain ignorant of these important details? That's the logical conclusion to your argument. I'm not saying you're wrong, but you have to admit that a Java programmer (of which I'm one) saying that using a generalized solution makes you ignorant is a bit like the pot calling the kettle black. Not at all. I probably should have said there is a potential for lost knowledge. Here are a couple of anecdotal examples that I hope will help illustrate what I mean. - I was told recently by a person in a software architect position that they use Hibernate because it prevents SQL injection. I'll give them the benefit of the doubt and assume they have other motivations but still, is escaping the input strings to a query really advanced knowledge? Think about that from the perspective of a junior or mid-level programmer who has only ever used an ORM. What happens when something goes wrong or performance has to be optimized. For that matter, what does happen? Now you have to know SQL, the DBMS, and HQL and understand Hibernate behavior. Double the complexity? Disclaimer: I don't have anything against Hibernate or JPA. I worked with an early implementation of JDO for a short time and am beginning some work with Hibernate. - A JavaScript programmer told me not too long ago that you really have to use a JavaScript library (e.g. jQuery) if you're going to use AJAX in an interface because it's just too complicated. Is instantiating and using an XMLHttpRequest object really that difficult? What about those programmers who have only ever used a JavaScript library? Another developer says he tells clients that they shouldn't consider a feature if it isn't supported by jQuery. Still another says that one of the reasons their organization uses a full-blown framework is that their programmers can't develop cross-browser compatible JavaScript. Disclaimer: I think jQuery is a wonderful library and, if you plan to make good use of the features available, it should definitely be considered. What I'm saying is there should be a good reason (really good) to add significant complexity, performance overhead, memory requirements and megabytes of code to an application. Efficiency, flexibility, repairability, extensibility and reliability are all components of software quality and all are affected by complexity. Less complex systems are easier to maintain. To continue the aside, wasted energy is wasted energy and it may become a factor in software development at some point. I think decision makers should be taught that there is more to the bottom line than dollars and cents. In my experience, by far the biggest time waster is trying to deal with code that is (or has become) unmaintainable. Re-writing just because a piece of code has become out-of-touch with current standards or because nobody understands how it works is entirely wasted effort. We have lots of places in our code where we have been spending - literally - years recording from bad decisions in the past. Granted. Reading other people's code is a learned skill and can be problematic. Isn't that where design and code reviews and coding standards come into play? Also, apples and oranges. Energy is precious resource that deserves special consideration. I'm just blue-skying here and don't have the answers but how much electricity is wasted by inefficient programming? We may have to factor that in some day. -Terence - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Build vs. buy (Was: [Seriously OT] Help in diagnosing server unresponsiveness)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 2/11/13 4:30 PM, Terence M. Bandoian wrote: I understand the considerations above and they are a part of the prevailing thinking. However, one underlying assumption of the supporting argument appears to be that today's programmers are not capable of developing maintainable code which I don't believe is true. As I understand it, programmer productivity is one of the most significant factors in the decision making process and it is a valid concern. IF (that's a big if) an application can be developed in half the time using a generalized solution, then that approach has to be considered along with a host of other concerns including the end product and the effect on the organization. I say reliance on generalized solutions is short-sighted because knowledge of the underlying technologies is lost, or never gained, along with the skills to work in those spheres. Are you suggesting that people who program using Java are oblivious to the innards of hardware architecture and are remain ignorant of these important details? That's the logical conclusion to your argument. I'm not saying you're wrong, but you have to admit that a Java programmer (of which I'm one) saying that using a generalized solution makes you ignorant is a bit like the pot calling the kettle black. Efficiency, flexibility, repairability, extensibility and reliability are all components of software quality and all are affected by complexity. Less complex systems are easier to maintain. To continue the aside, wasted energy is wasted energy and it may become a factor in software development at some point. I think decision makers should be taught that there is more to the bottom line than dollars and cents. In my experience, by far the biggest time waster is trying to deal with code that is (or has become) unmaintainable. Re-writing just because a piece of code has become out-of-touch with current standards or because nobody understands how it works is entirely wasted effort. We have lots of places in our code where we have been spending - literally - years recording from bad decisions in the past. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEaYWUACgkQ9CaO5/Lv0PC0CQCfX91lU8Tbik1CDe3g8ASV6pxQ rOkAn2PPdBNrP4rVPRJ6GWzXqFx/8HyQ =hcps -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Build vs. buy (Was: [Seriously OT] Help in diagnosing server unresponsiveness)
Em 12/02/2013 13:36, Christopher Schultz escreveu: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 2/11/13 4:30 PM, Terence M. Bandoian wrote: I understand the considerations above and they are a part of the prevailing thinking. However, one underlying assumption of the supporting argument appears to be that today's programmers are not capable of developing maintainable code which I don't believe is true. As I understand it, programmer productivity is one of the most significant factors in the decision making process and it is a valid concern. IF (that's a big if) an application can be developed in half the time using a generalized solution, then that approach has to be considered along with a host of other concerns including the end product and the effect on the organization. I say reliance on generalized solutions is short-sighted because knowledge of the underlying technologies is lost, or never gained, along with the skills to work in those spheres. Are you suggesting that people who program using Java are oblivious to the innards of hardware architecture and are remain ignorant of these important details? That's the logical conclusion to your argument. I'm not saying you're wrong, but you have to admit that a Java programmer (of which I'm one) saying that using a generalized solution makes you ignorant is a bit like the pot calling the kettle black. Efficiency, flexibility, repairability, extensibility and reliability are all components of software quality and all are affected by complexity. Less complex systems are easier to maintain. To continue the aside, wasted energy is wasted energy and it may become a factor in software development at some point. I think decision makers should be taught that there is more to the bottom line than dollars and cents. In my experience, by far the biggest time waster is trying to deal with code that is (or has become) unmaintainable. Re-writing just because a piece of code has become out-of-touch with current standards or because nobody understands how it works is entirely wasted effort. We have lots of places in our code where we have been spending - literally - years recording from bad decisions in the past. Most companies are based on believes of the past: development is costly and non profitable. While this is true for small companies (where each employee salary present a risk for the profitability), for medium to big companies this is not true anymore. The cost for constraining the company to software produced by big players (I wont cite names) is much bigger than having a (well organized) development team capable of integrating standards (like accounting and taxes) to the wild (sales, production, research). Using libraries like JPA cannot be considered a danger unless used without proper analysis. This is true for everything in life (even water consumed in excess cause damage to health). I do use JPA in the development of high performance applications, and I do sacrifice some nanoseconds in prol of well maintainable code - for the user, anything below 200ms will look instantaneous. This makes my company profitable where my customers failed when working in house. I hope they never learn how to do that, because this guarantees my money at the end of each month. Edson Richter - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEaYWUACgkQ9CaO5/Lv0PC0CQCfX91lU8Tbik1CDe3g8ASV6pxQ rOkAn2PPdBNrP4rVPRJ6GWzXqFx/8HyQ =hcps -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Build vs. buy (Was: [Seriously OT] Help in diagnosing server unresponsiveness)
-Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Build vs. buy (Was: [Seriously OT] Help in diagnosing server unresponsiveness) Re-writing just because a piece of code has become out-of-touch with current standards or because nobody understands how it works is entirely wasted effort. - -chris And, not to mention the technology an application uses eventually reaches EOL, then what? It's easier to keep it limping along until the point at which someone decides it's worth spending money to update it. It has been my observation that the trend where I work is buy and try to configure or enhance the product to make it do something it didn't do before, because I believe some people think building solutions are too complex or too costly. Buying and maintaining in my opinion is harder when the vendor product changes. You end up building additional complex functionality around a product that did not do 100% of what you wanted when you bought it, now the vendor changes something and you're faced with nearly redoing everything you did before to keep maintenance on the current vendor product version.
Re: [Seriously OT] Help in diagnosing server unresponsiveness
On 2/10/2013 7:13 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Terrence, On 2/9/13 8:01 PM, Terence M. Bandoian wrote: On 2/6/2013 9:26 AM, Jeffrey Janner wrote: IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris Chris, I'd like to differ with you on this last point. As someone who's been a developer, support person, and admin, I've got a pretty good perspective on this subject. While servers may be cheap, they will never be cheap enough to overcome poor programming practices. I've worked with systems so poorly designed that we couldn't purchase a system big enough to run the software adequately, once you got above a handful of users. Yes, it's gotten to the point where systems are much cheaper than they used to be, while developer salaries are only increasing (supposedly), so wasting time on some minor performance improvement may not be cost-effective. However, when you aggregate the time that hundreds of users spend waiting on a response from a poorly designed, unresponsive system, I think you'll find that it trumps the cost of having the developer spending a few extra minutes to get it right the first time. Generalized solutions, I think, include a substantial amount of code that isn't required for a given application. The additional code affects performance but with the speed, availability and low cost of hardware, people seem to be opting for packaged solutions that don't require their programmers to understand the details of the implementation. That seems short-sighted to me. As an aside, I wonder if, at some point, the energy costs of inefficient code will come into play. Don't wasted CPU cycles == wasted energy? There is always a balance between a number of criteria: 1. Cost to run (CPU time) 2. Cost to build (programmer time) 3. Performance 4. Maintainability IMO, issues #2-#4 trump #1 every time: maintainability is the most important aspect of software design. If using JPA means that you can maintain your software more effectively, then it is totally worth the extra cost of CPU time. Anyone who has ever worked at an outfit where there is some legacy code that you can't touch because nobody understands how it works knows that this is the truth: there is unmaintainable code that someone wrote in the past and then left/got fired/died/etc. and not the only options are a) leave the code alone forever in fear that it will break if you touch it or b) completely re-write the code. If you have some cowboy who writes thousands of lines of impenetrable JDBC code, you have a recipe for code rot. Use of Hibernate, JPA, etc. can significantly alleviate that because - at least for non-abandoned projects - there are many people always looking at the code, and they are experts in that code. The 3000¬ per annum is nothing compared to the cost of re-writing all that legacy code, even if you switch from (e.g.) straight JDBC to (e.g.) Hibernate or JPA. If you have to pay a consultant to do that, it's going to cost you an order or two of magnitude more than the cost of hardware: you can run more hardware for the lifetime of the software rather than make a big change like that across all your code. That said, when (or if) you run into performance problems, then you have to decide whether you need to ditch the framework (perhaps just for a portion of your code). We ditched O-R mappers for different core reasons, but maintainability was the primary one: they all required too many hacks at the time to make it worth it. - -chris Chris- I understand the considerations above and they are a part of the prevailing thinking. However, one underlying assumption of the supporting argument appears to be that today's programmers are not capable of developing maintainable code which I don't believe is true. As I understand it, programmer productivity is one of the most significant factors in the decision making process and it is a valid concern. IF (that's a big if) an application can be developed in half the time using a generalized solution, then that approach has to be considered along with a host of other concerns including the end product and the effect on the organization. I say reliance on generalized solutions is short-sighted because knowledge of the underlying technologies is lost, or never gained, along with the skills to work in those spheres. Efficiency, flexibility, repairability, extensibility and reliability are all components of software quality and all are affected by complexity. Less complex systems are easier to maintain. To continue the aside, wasted energy is wasted energy and it may become a factor in software development at some point. I think decision makers should be taught that there is more to the bottom line than dollars and
Re: [Seriously OT] Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Terrence, On 2/9/13 8:01 PM, Terence M. Bandoian wrote: On 2/6/2013 9:26 AM, Jeffrey Janner wrote: IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris Chris, I'd like to differ with you on this last point. As someone who's been a developer, support person, and admin, I've got a pretty good perspective on this subject. While servers may be cheap, they will never be cheap enough to overcome poor programming practices. I've worked with systems so poorly designed that we couldn't purchase a system big enough to run the software adequately, once you got above a handful of users. Yes, it's gotten to the point where systems are much cheaper than they used to be, while developer salaries are only increasing (supposedly), so wasting time on some minor performance improvement may not be cost-effective. However, when you aggregate the time that hundreds of users spend waiting on a response from a poorly designed, unresponsive system, I think you'll find that it trumps the cost of having the developer spending a few extra minutes to get it right the first time. Generalized solutions, I think, include a substantial amount of code that isn't required for a given application. The additional code affects performance but with the speed, availability and low cost of hardware, people seem to be opting for packaged solutions that don't require their programmers to understand the details of the implementation. That seems short-sighted to me. As an aside, I wonder if, at some point, the energy costs of inefficient code will come into play. Don't wasted CPU cycles == wasted energy? There is always a balance between a number of criteria: 1. Cost to run (CPU time) 2. Cost to build (programmer time) 3. Performance 4. Maintainability IMO, issues #2-#4 trump #1 every time: maintainability is the most important aspect of software design. If using JPA means that you can maintain your software more effectively, then it is totally worth the extra cost of CPU time. Anyone who has ever worked at an outfit where there is some legacy code that you can't touch because nobody understands how it works knows that this is the truth: there is unmaintainable code that someone wrote in the past and then left/got fired/died/etc. and not the only options are a) leave the code alone forever in fear that it will break if you touch it or b) completely re-write the code. If you have some cowboy who writes thousands of lines of impenetrable JDBC code, you have a recipe for code rot. Use of Hibernate, JPA, etc. can significantly alleviate that because - at least for non-abandoned projects - there are many people always looking at the code, and they are experts in that code. The 3000€ per annum is nothing compared to the cost of re-writing all that legacy code, even if you switch from (e.g.) straight JDBC to (e.g.) Hibernate or JPA. If you have to pay a consultant to do that, it's going to cost you an order or two of magnitude more than the cost of hardware: you can run more hardware for the lifetime of the software rather than make a big change like that across all your code. That said, when (or if) you run into performance problems, then you have to decide whether you need to ditch the framework (perhaps just for a portion of your code). We ditched O-R mappers for different core reasons, but maintainability was the primary one: they all required too many hacks at the time to make it worth it. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEYRcYACgkQ9CaO5/Lv0PBpVgCghvN48DZnEs8OUgl1aZJPWaJA Jq4AniQEI++Gez6es5U+gxCAry+LaflY =+UJg -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [Seriously OT] Help in diagnosing server unresponsiveness
On 2/6/2013 9:26 AM, Jeffrey Janner wrote: IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris Chris, I'd like to differ with you on this last point. As someone who's been a developer, support person, and admin, I've got a pretty good perspective on this subject. While servers may be cheap, they will never be cheap enough to overcome poor programming practices. I've worked with systems so poorly designed that we couldn't purchase a system big enough to run the software adequately, once you got above a handful of users. Yes, it's gotten to the point where systems are much cheaper than they used to be, while developer salaries are only increasing (supposedly), so wasting time on some minor performance improvement may not be cost-effective. However, when you aggregate the time that hundreds of users spend waiting on a response from a poorly designed, unresponsive system, I think you'll find that it trumps the cost of having the developer spending a few extra minutes to get it right the first time. Generalized solutions, I think, include a substantial amount of code that isn't required for a given application. The additional code affects performance but with the speed, availability and low cost of hardware, people seem to be opting for packaged solutions that don't require their programmers to understand the details of the implementation. That seems short-sighted to me. As an aside, I wonder if, at some point, the energy costs of inefficient code will come into play. Don't wasted CPU cycles == wasted energy? -Terence Bandoian - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Seriously OT] Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 2/6/13 3:05 PM, André Warnier wrote: As maybe one more rant to add to this series, and as a reply to the just throw more hardware at it, it's cheap line. In my defense, I was mostly talking about the trade-offs between something that is relatively easy to implement (e.g. Hibernate, JPA, etc.) taking more resources than something that was hand-coded, say with straight JDBC calls. Since CPU cycles are cheap relative to programmers, go with JPA or Hibernate unless there is a particular reason to do straight JDBC. In our case, the O-R mappers were dogs and we found ourselves hacking-around them all the time. I suspect things have gotten better, now, but we do have that pesky thing called time that is to valuable. So, it's cheaper to leave the (fast!) code along than it is to upgrade to using Hibernate or JPA. So the additional 1 GB of Tomcat heap (just as an example) ends up costing 850€, not 50€. Additionally, this bigger server could be 2U in height instead of 1U, and cost 75% more per month to host in a rack. And so it goes for other server resources (disk, network bandwidth, etc.). You must admit that your example is fairly contrived, but it is a good point. I am not even counting the time that will be needed to move the applications from server 1 to server 2. You aren't using some kind of cloning or scripted-setup? Yikes! (And the cloud turns out not to be much cheaper, once you seriously factor in the various cost elements). Speaking of the prior statement and the cloud: using AWS is really pretty cool: you set up a server the way you want it and take a snapshot. Then, cloning that server is as easy as saying gimmie another one of these and it just happens. Need to upgrade an entire fleet of app servers? No problem: upgrade one, snapshot, then launch 10 new ones just like it. Let the other ones shut down and disappear... That is also why I sometimes look in wonder as someone on this forum will tell someone else something like 4 GB of heap is really not very much to run your webapp; you should at least double that for any serious work. Current heap allocations on a real webapp: JVM 1: 384M JVM 2: 256M JVM 3: 256M JVM 4: 64M Lean and mean. And one of those (256M) does nothing but XSL transformations which are pretty heavy on RAM usage. Anyone who tells you anything about how much heap memory you need is probably selling RAM. What about instead Dude, 4 GB is four thousand million bytes (plus spares). Would you not have a quick look at your webapp, to see if it really needs that much workspace to send back a single html page ?. +1 - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEVYqYACgkQ9CaO5/Lv0PAYRgCfYM3yO0A7pc6redJFKlN9Rc+O N3sAnju6LYpMUTzvWHjd8PYSlYzv/CPX =eC5y -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Zoran Avtarovski wrote 5. Garbage collector time spikes to 24.0. I think with JavaMelody it means that GC took 24% of of the CPU?? Yes, 24% in % GC time in JavaMelody means that 24% of the CPU. And this is a lot, if longer than a few seconds. Zoran Avtarovski wrote So I think our issues are related to GC. Is there a way to trigger more frequent GC which will hopefully be less resource intensive? Triggering frequent GC is a bad idea in general. Zoran Avtarovski wrote And why have the memory usage levels not recovered? Take a heap dump as said by Chuck, then use Memory Analyzer Tool. Without a heap dump, you can also open the heap histogram in your JavaMelody reports to have a first look, Here is an example in the demo: http://demo.javamelody.cloudbees.net/monitoring?part=heaphisto For analysis of GC trends, you can enable the GC logs of the JVM and use HPJMeter tool. But at the moment, you already know that the memory was not good, so having the trends may be for a later time. Emeric -- View this message in context: http://tomcat.10.n6.nabble.com/Help-in-diagnosing-server-unresponsiveness-tp4993508p4993829.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 06/02/2013 01:18, Zoran Avtarovski escreveu: Thanks Igor, I just stumbled upon that same document. I think you may be on to something here. I have a feeling that the GC may not be configured well. Also read the following: http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/bestpractices.html And check this (if you are using RHEL 6 or CentOS 6, worth to take a look): https://rhn.redhat.com/errata/RHSA-2013-0223.html Edson Z. On 6/02/13 2:15 PM, Igor Cicimov icici...@gmail.com wrote: On Wed, Feb 6, 2013 at 1:15 PM, Zoran Avtarovski zo...@sparecreative.comwrote: Here's some updated observations after a not quite incident (CPU and memory spiked but the app is still running): 1. Yesterday we had a 90% CPU spike at a time where there was absolutely no server traffic. Verified through both the HTTP logs and the mod_jk logs. The CPU spiked and recovered back to average levels. 2. Used memory spiked at 10GB from a pre incident average of 500MB throughout 2 busy days without incident 3. Used memory has only gone back down to 4GB and is holding at this level 4. The Used physical memory went up from 2GB to 14GB and has stayed there 5. Garbage collector time spikes to 24.0. I think with JavaMelody it means that GC took 24% of of the CPU?? So I think our issues are related to GC. Is there a way to trigger more frequent GC which will hopefully be less resource intensive? And why have the memory usage levels not recovered? Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Zoran, First I would like to recommend the following document for reading: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms It explains the GC in JVM 1.6 including the Concurrent Collector settings which is the one you are using. The values for the GC in the log file are the time the particular collector spent for the operation so the 24.0 would probably mean 24 seconds, which might mean that for 24 seconds your applications might be unresponsive during the CMS GC. As explained in the document from the above link the GC has minor and major collecting phases. During the minor collection the objects are GC'ed from the so called young generation or promoted in the old (tenured) generation. More of this objects pile up in the old generation more frequent the major GC needs to run. The major ones take usually much longer (they have to clean much bigger space) time than the minor ones but the minor ones have to run more frequently. Now, what you need to find is what is causing your problem really? If your application creates lot of new objects then you'll have lots of minor GC running. More of this objects survive, they get moved to the old generation space and then you'll have lots of major GC running as well. The danger here is that you might end up with constantly running GC which will render your application unusable due to pauses. So basically badly written application can cause lots of problems, not closing connections and freeing objects etc etc, and in that case even the best GC tunning in the world will not help you, your application(s) will eventually get to halt. So read the document carefully and decide which user case is best for you. If you are creating lots of new objects then maybe increasing the minor space (default new/old ratio is 1:3) can help. Also paste here the results of the GC logs. The link I provided has some more useful settings and recommendations for the CMS collector. This collector stops the application threads twice during the operation so you need to check those times too. Cheers Pozdrav, Igor - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [Seriously OT] Help in diagnosing server unresponsiveness
-Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, February 05, 2013 4:59 PM To: Tomcat Users List Subject: Re: [Seriously OT] Help in diagnosing server unresponsiveness -BEGIN PGP SIGNED MESSAGE- IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris Chris, I'd like to differ with you on this last point. As someone who's been a developer, support person, and admin, I've got a pretty good perspective on this subject. While servers may be cheap, they will never be cheap enough to overcome poor programming practices. I've worked with systems so poorly designed that we couldn't purchase a system big enough to run the software adequately, once you got above a handful of users. Yes, it's gotten to the point where systems are much cheaper than they used to be, while developer salaries are only increasing (supposedly), so wasting time on some minor performance improvement may not be cost-effective. However, when you aggregate the time that hundreds of users spend waiting on a response from a poorly designed, unresponsive system, I think you'll find that it trumps the cost of having the developer spending a few extra minutes to get it right the first time.
Re: [Seriously OT] Help in diagnosing server unresponsiveness
Em 06/02/2013 13:26, Jeffrey Janner escreveu: -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, February 05, 2013 4:59 PM To: Tomcat Users List Subject: Re: [Seriously OT] Help in diagnosing server unresponsiveness -BEGIN PGP SIGNED MESSAGE- IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris Chris, I'd like to differ with you on this last point. As someone who's been a developer, support person, and admin, I've got a pretty good perspective on this subject. While servers may be cheap, they will never be cheap enough to overcome poor programming practices. I've worked with systems so poorly designed that we couldn't purchase a system big enough to run the software adequately, once you got above a handful of users. Yes, it's gotten to the point where systems are much cheaper than they used to be, while developer salaries are only increasing (supposedly), so wasting time on some minor performance improvement may not be cost-effective. However, when you aggregate the time that hundreds of users spend waiting on a response from a poorly designed, unresponsive system, I think you'll find that it trumps the cost of having the developer spending a few extra minutes to get it right the first time. I've partially agree. The systems architect should be able to create a balance between maintainable code and performance. We can't crucify JPA/EJB, indeed bad use of JPA/EJB (as any other development library) would induce to problems. Developers, in general, create tests for their own, to check if the logic if fulfilled. But then, it is mandatory to have another phase with load test - and then, in this situation, you will find the problems of memory, CPU, disk, network overload. In our case, any window that takes more than 0,250s (250ms) to popup is considered an issue, and must be reworked. That doesn't mean we need to write SQL on our own, means that the communication flow is wrong, or there is more information in the window than the user need at that point. No matter what some could argue, nobody reads more than 50 lines of anything (customers, sales, products) per page. So why execute a query that returns 35000 objects? So, the architect must have in my these constraints and requirements and then give the vision to the programmers, so they could write efficient code. As said, I partially agree. I do prefer to spend few bucks more than have an application that will take hundreds of thousands of (what-ever-your-money-is) to recover user confidence. Regards, Edson - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Seriously OT] Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jeffrey, On 2/6/13 10:26 AM, Jeffrey Janner wrote: -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, February 05, 2013 4:59 PM To: Tomcat Users List Subject: Re: [Seriously OT] Help in diagnosing server unresponsiveness -BEGIN PGP SIGNED MESSAGE- IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris Chris, I'd like to differ with you on this last point. As someone who's been a developer, support person, and admin, I've got a pretty good perspective on this subject. While servers may be cheap, they will never be cheap enough to overcome poor programming practices. I've worked with systems so poorly designed that we couldn't purchase a system big enough to run the software adequately, once you got above a handful of users. Yes, it's gotten to the point where systems are much cheaper than they used to be, while developer salaries are only increasing (supposedly), so wasting time on some minor performance improvement may not be cost-effective. However, when you aggregate the time that hundreds of users spend waiting on a response from a poorly designed, unresponsive system, I think you'll find that it trumps the cost of having the developer spending a few extra minutes to get it right the first time. How about servers are a lot cheaper than competent software developers? ;) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlESjGwACgkQ9CaO5/Lv0PDqUACgrpSMfS+87G/aofbuyZT/4C9n N9QAmwdTFk7t+JRQEu/75VCVB4SjBgV/ =s59y -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 2/5/13 8:29 PM, Howard W. Smith, Jr. wrote: Chris, On Sun, Feb 3, 2013 at 12:39 PM, Christopher Schultz ch...@christopherschultz.net wrote: If your requirements allow for users to have to re-authenticate when you have a failover-event, then I highly recommend that you use sticky-sessions /without/ session replication: performance will be much better that way (and it's easier to configure). this is very interesting but somewhat of a concern as well. does failover occur when one tomcat-and-web-app instance is taking a long time serving one request? there are a few places/pages in my web app that does some heavy lifting (reports that execute SQL SELECT, and yes, I am using indexes along with statement caching, etc...), and i've seen my app failover with a 404 error (i currently only have one instance). It depends on how you configure things. It's usually the lb that makes that decision, so you configure it there. I would imagine that a good lb would be able to do things like guess the health of the backend-server and take it out of rotation if it's not healthy. mod_jk has a variety of health-checks that you can perform to decide when a Tomcat needs to be abandoned (perhaps temporarily). Your app fails-over to a 404? Something must be misconfigured. Or were you saying that your webapp returns a 404 if a query runs too long or something... that doesn't sound optimal. Well, if you can tolerate re-auth on failover, then there's nothing to do at all: simply enable sticky-sessions and let it happen. Failovers should be rare, anyway, right? sticky sessions sound nice and I would love to give it a try, but honestly, i don't think the endusers want to have to re-auth on failover. just today, I read on the atmosphere list that sticky sessions are required for using atmosphere with/in clusters. i'm definitely using atmosphere, but not using sticky sessions (or clusters) yet. :( I think if you use async replication, you must use sticky sessions anyway, because otherwise you run the risk of having a user hit a different server before it has been updated from a previous request (on a different server). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlESjtAACgkQ9CaO5/Lv0PAQtwCgvzniTJG11eoJeOJcV3Gpnz6Y +UoAnRph0Qh6c8D5f9Mk0vHis3E1iMSy =GDk7 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, If you are interested in clustering, you should try to attend ApacheCon this month in Portland, OR in the US: Mark Thomas will be giving a talk on Tomcat clustering that I'm sure will be enlightening. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlESjwcACgkQ9CaO5/Lv0PC+0gCfaXt66fDFLzClnfzLAsZcLzXH 044Anj1e6NSZaUFk5MbhC92SLzVXwkWi =P9K0 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [Seriously OT] Help in diagnosing server unresponsiveness
-Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Wednesday, February 06, 2013 11:02 AM To: Tomcat Users List Subject: Re: [Seriously OT] Help in diagnosing server unresponsiveness -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jeffrey, On 2/6/13 10:26 AM, Jeffrey Janner wrote: -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, February 05, 2013 4:59 PM To: Tomcat Users List Subject: Re: [Seriously OT] Help in diagnosing server unresponsiveness -BEGIN PGP SIGNED MESSAGE- IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris Chris, I'd like to differ with you on this last point. As someone who's been a developer, support person, and admin, I've got a pretty good perspective on this subject. While servers may be cheap, they will never be cheap enough to overcome poor programming practices. I've worked with systems so poorly designed that we couldn't purchase a system big enough to run the software adequately, once you got above a handful of users. Yes, it's gotten to the point where systems are much cheaper than they used to be, while developer salaries are only increasing (supposedly), so wasting time on some minor performance improvement may not be cost-effective. However, when you aggregate the time that hundreds of users spend waiting on a response from a poorly designed, unresponsive system, I think you'll find that it trumps the cost of having the developer spending a few extra minutes to get it right the first time. How about servers are a lot cheaper than competent software developers? ;) - -chris +1 Can't tell how tired I am of hearing some version of the following: I don't care to learn how a library/API is supposed to work, I just want to use it.
RE: [Seriously OT] Help in diagnosing server unresponsiveness
-Original Message- From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] Sent: Wednesday, February 06, 2013 12:06 PM To: 'Tomcat Users List' Subject: RE: [Seriously OT] Help in diagnosing server unresponsiveness -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Wednesday, February 06, 2013 11:02 AM To: Tomcat Users List Subject: Re: [Seriously OT] Help in diagnosing server unresponsiveness -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jeffrey, On 2/6/13 10:26 AM, Jeffrey Janner wrote: -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, February 05, 2013 4:59 PM To: Tomcat Users List Subject: Re: [Seriously OT] Help in diagnosing server unresponsiveness -BEGIN PGP SIGNED MESSAGE- IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris Chris, I'd like to differ with you on this last point. As someone who's been a developer, support person, and admin, I've got a pretty good perspective on this subject. While servers may be cheap, they will never be cheap enough to overcome poor programming practices. I've worked with systems so poorly designed that we couldn't purchase a system big enough to run the software adequately, once you got above a handful of users. Yes, it's gotten to the point where systems are much cheaper than they used to be, while developer salaries are only increasing (supposedly), so wasting time on some minor performance improvement may not be cost-effective. However, when you aggregate the time that hundreds of users spend waiting on a response from a poorly designed, unresponsive system, I think you'll find that it trumps the cost of having the developer spending a few extra minutes to get it right the first time. How about servers are a lot cheaper than competent software developers? ;) - -chris +1 Can't tell how tired I am of hearing some version of the following: I don't care to learn how a library/API is supposed to work, I just want to use it. (replying to own post == bad form) Actually, that line usually begins: I don't have time to - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Seriously OT] Help in diagnosing server unresponsiveness
Jeffrey Janner wrote: -Original Message- From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] Sent: Wednesday, February 06, 2013 12:06 PM To: 'Tomcat Users List' Subject: RE: [Seriously OT] Help in diagnosing server unresponsiveness -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Wednesday, February 06, 2013 11:02 AM To: Tomcat Users List Subject: Re: [Seriously OT] Help in diagnosing server unresponsiveness -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jeffrey, On 2/6/13 10:26 AM, Jeffrey Janner wrote: -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, February 05, 2013 4:59 PM To: Tomcat Users List Subject: Re: [Seriously OT] Help in diagnosing server unresponsiveness -BEGIN PGP SIGNED MESSAGE- IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris Chris, I'd like to differ with you on this last point. As someone who's been a developer, support person, and admin, I've got a pretty good perspective on this subject. While servers may be cheap, they will never be cheap enough to overcome poor programming practices. I've worked with systems so poorly designed that we couldn't purchase a system big enough to run the software adequately, once you got above a handful of users. Yes, it's gotten to the point where systems are much cheaper than they used to be, while developer salaries are only increasing (supposedly), so wasting time on some minor performance improvement may not be cost-effective. However, when you aggregate the time that hundreds of users spend waiting on a response from a poorly designed, unresponsive system, I think you'll find that it trumps the cost of having the developer spending a few extra minutes to get it right the first time. How about servers are a lot cheaper than competent software developers? ;) - -chris +1 Can't tell how tired I am of hearing some version of the following: I don't care to learn how a library/API is supposed to work, I just want to use it. (replying to own post == bad form) Actually, that line usually begins: I don't have time to As maybe one more rant to add to this series, and as a reply to the just throw more hardware at it, it's cheap line. For a small service company offering ASP kind of services to customers, buying hardware to run apps is for us a relatively frequent occurrence. And while it is true that basic hardware is relatively cheap nowadays, one quickly notices that a bit more hardware can involve a significant step up the cost ladder. For example, let's say that a good-quality basic rack-mounted server costs 2000€, and has 4 slots for RAM. And let's say that 1 RAM unit nowadays can hold 8 GB and costs 50€. So in total you can fit 4x8=32 GB of RAM in that basic server, at a cost of 2200€ for the total. If you need 33 GB (thus say only 1x8 GB more, at a theoretical additional cost of 50€), you have to switch to the next server category, which may have 8 slots for RAM, but costs 3000€. So the additional 1 GB of Tomcat heap (just as an example) ends up costing 850€, not 50€. Additionally, this bigger server could be 2U in height instead of 1U, and cost 75% more per month to host in a rack. And so it goes for other server resources (disk, network bandwidth, etc.). I am not even counting the time that will be needed to move the applications from server 1 to server 2. (And the cloud turns out not to be much cheaper, once you seriously factor in the various cost elements). So there is still something to say about *how* one is writing applications. Of course in many cases such low-level materialistic considerations do not affect the programmer directly and immediately in his pay packet. But ultimately, someone pays for it, and at the end of the day it does reflect on the bottom-line and on the menu at the cantina. That is also why I sometimes look in wonder as someone on this forum will tell someone else something like 4 GB of heap is really not very much to run your webapp; you should at least double that for any serious work. What about instead Dude, 4 GB is four thousand million bytes (plus spares). Would you not have a quick look at your webapp, to see if it really needs that much workspace to send back a single html page ?. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Chris, On Wed, Feb 6, 2013 at 12:11 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, It depends on how you configure things. It's usually the lb that makes that decision, so you configure it there. I would imagine that a good lb would be able to do things like guess the health of the backend-server and take it out of rotation if it's not healthy. mod_jk has a variety of health-checks that you can perform to decide when a Tomcat needs to be abandoned (perhaps temporarily). good to know, thanks. Your app fails-over to a 404? Something must be misconfigured. Or were you saying that your webapp returns a 404 if a query runs too long or something... that doesn't sound optimal. Misconfigured? Maybe not configured quite yet as I only have 2 months experience with Tomcat/TomEE. I think the 404 is caused by a query (or update logic) that runs too long, and agreed, I'm sure some optimizing is in order, but the endusers don't report this to me at all, as they are a bit new to using a web application (as they have been using an MS-DOS dBase IV app ever since 1994/1995; that app just recently got migrated to a JSF web application, they have been using it since July 2012, and they are quite pleased with the web app), and I do my best to monitor the logs for exceptions that need to be fixed, and poll the users about their experience with the app. I may be the only one that experience the 404 at times. In reading some of today's responses, I am in full agreement that some queries/logic may need to be reworked, and definitely endusers will probably never view/read 35,000 objects on a single http request. As the developer of the web app, I am probably the only person selecting-and-viewing 35,000 objects in a single http request. The data is primarily filtered and/or selected by date, and most-case scenario, they are selecting data for one date in time, but sometimes, they may select data via a small range of dates. I think the user-requested database updates may be the cause, as google calendar API is used to strategically synchronize certain/selected data in the database with the organization's google calendar...so certain employees can view 'the schedule' on their mobile phones and tablets. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlESjtAACgkQ9CaO5/Lv0PAQtwCgvzniTJG11eoJeOJcV3Gpnz6Y +UoAnRph0Qh6c8D5f9Mk0vHis3E1iMSy =GDk7 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 06/02/2013 19:09, Howard W. Smith, Jr. escreveu: Chris, On Wed, Feb 6, 2013 at 12:11 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, It depends on how you configure things. It's usually the lb that makes that decision, so you configure it there. I would imagine that a good lb would be able to do things like guess the health of the backend-server and take it out of rotation if it's not healthy. mod_jk has a variety of health-checks that you can perform to decide when a Tomcat needs to be abandoned (perhaps temporarily). good to know, thanks. Your app fails-over to a 404? Something must be misconfigured. Or were you saying that your webapp returns a 404 if a query runs too long or something... that doesn't sound optimal. Misconfigured? Maybe not configured quite yet as I only have 2 months experience with Tomcat/TomEE. I think the 404 is caused by a query (or update logic) that runs too long, and agreed, I'm sure some optimizing is in order, but the endusers don't report this to me at all, as they are a bit new to using a web application (as they have been using an MS-DOS dBase IV app ever since 1994/1995; that app just recently got migrated to a JSF web application, they have been using it since July 2012, and they are quite pleased with the web app), and I do my best to monitor the logs for exceptions that need to be fixed, and poll the users about their experience with the app. Hi! 404 is page not found. If query runs too long, you will get timeout or Error 500. Check your web app... the only scenario I can think on 404 due timeout is if your app is querying the name of the next page, and then get an error as response (instead of the name of the page). Regards, Edson I may be the only one that experience the 404 at times. In reading some of today's responses, I am in full agreement that some queries/logic may need to be reworked, and definitely endusers will probably never view/read 35,000 objects on a single http request. As the developer of the web app, I am probably the only person selecting-and-viewing 35,000 objects in a single http request. The data is primarily filtered and/or selected by date, and most-case scenario, they are selecting data for one date in time, but sometimes, they may select data via a small range of dates. I think the user-requested database updates may be the cause, as google calendar API is used to strategically synchronize certain/selected data in the database with the organization's google calendar...so certain employees can view 'the schedule' on their mobile phones and tablets. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlESjtAACgkQ9CaO5/Lv0PAQtwCgvzniTJG11eoJeOJcV3Gpnz6Y +UoAnRph0Qh6c8D5f9Mk0vHis3E1iMSy =GDk7 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On Wed, Feb 6, 2013 at 6:37 PM, Edson Richter edsonrich...@hotmail.com wrote:
Hi!
404 is page not found.
If query runs too long, you will get timeout or Error 500.
I need to confirm which one it is, as I only use google Chrome, Google
Chrome says something like, could not connect to page. I migrated from
glassfish 3.1.2.2 to tomEE (tomcat7), and I really like the different
log files available in /logs, so I will have to check localhost access
log the next time I see it happens.
Check your web app... the only scenario I can think on 404 due timeout is if
your app is querying the name of the next page, and then get an error as
response (instead of the name of the page).
well, i am using dynamic ui:include src=#{bean.page} and now, even
more ui:include src=#{condition ? '/folder/pageName.xhtml' :
'/folder/anotherPageName.xhtml'}, and some of those page filenames
are rather long. Honestly, i do see 404 error quite often in localhost
access log, but it is usually related to some appleIpad gift/jpg file,
most likely provided by PrimeFaces (or should I say, probably not
packaged in the primefaces JAR file). My pages render very well and
quite fast... i did the ui:include src=#{...} to replace many
rendered=#{...}, as I heard/learned and was told that rendered=...
can be a performance hit.
Regards,
Edson
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Seriously OT] Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 2/1/13 9:57 PM, Howard W. Smith, Jr. wrote: On Fri, Feb 1, 2013 at 2:20 PM, Christopher Schultz ch...@christopherschultz.net wrote: If you want to improve performance even more, ditch EJB altogether. Wow, that hurt. ditch EJB altogether? I 'grew up' on EJB. When I was migrating from JSF managed beans to CDI managed beans, I recognized that I could reference EJBs via @Inject or @EJB via TomEE/OpenWebBeans, but I stuck with @EJB...'just because'. when I first read your recommendation, earlier on, i thought to myself, okay, that should be easy, but now, I don't know how I would 'ditch EJB altogether'. All of my EJBs are accessing database via JPA; there are a few native SQL statements, lot of dynamic SQL statements that I redefined as named queries. If you are referring to CDI managed beans performing database access... I think I read one blog that stated that CDI can do anything that EJB can do, but still, I guess I have not seen any examples where CDI bean is accessing database. So, why do you recommend to 'ditch EJB altogether'... to improve performance? please explain. EJB takes normal classes and interfaces and wraps them in remotable proxies that can be used across huge clusters of loosely-related servers. Most apps don't need all that, or, if they do, can do it in a smarter way because the author understands the use cases and can beat a generalized system handily. I'm not suggesting that you ditch JPA or J2EE in general... it's just that managed beans are dirt slow. I haven't used Hibernate or even JPA but older O-R mappers (Torque, etc.) all sucked and ended up pulling way more data than necessary from the database. Years ago, we ripped-out all that crap and wrote our own SQL ourselves. Not only were we able to see a dramatic improvement in performance (sorry, I don't have numbers for you... it's been a while) but we now have full-control over transactions, timeouts, complex where clauses, etc. with out having to fight against an API that wants to hide all that complexity from you. IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEUEAREIAAYFAlERjrMACgkQ9CaO5/Lv0PDaVgCTBdUHBpXKHZgl074pzojY1SZu 6ACfdhTJLLtK7GkKdra3E8gc4mntdGI= =Ruvy -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Seriously OT] Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chris, On 2/2/13 11:47 AM, chris derham wrote: In the Java world, most people would only call it a consider it a cluster if the app servers actually know about each other -- for instance, if you are using session replication. IMO session replication is a dog, and there are better ways to achieve similar goals that yield much higher performance. I am curious to hear your thoughts - care to elaborate? We've totally hijacked Zoran's thread. After this, let's move to a new OT thread (or threads). Tomcat's session replication basically blasts everything around to all other Tomcat instances (or you can choose one and use BackupManager). Every time you modify your session, you get a burst of traffic between all your cluster members. I dunno if Tomcat bothers updating the last-touched-time on the session if that's the only thing that changed during the request, but if so it means that there is a burst of replication traffic after /every single request/. Most web applications I've seen use HttpSession for a variety of things, and not all of them need to be replicated. If you want to hand-roll some wrapper objects to allow you to stash things into the session that are not serializable (or shouldn't be serialized, like a transient wrapper) you can avoid some of this stuff, but you really shouldn't have to do this kind of thing. Instead, rely on some other data-sharing mechanism such as a database, memcached, etc. You can create a token that you pass to the client just like a JSESSIONID cookie and if they fail-over to a different node, you can still fetch your stuff when you need it. The problem with using HttpSession + session replication is that lazy programmers will stash just about anything into the session without thinking about it, and it will get replicated all over the place. If you use an external data storage mechanism, it's much more obvious that something special is happening, and (hopefully) your devs will take more care with dipping into that data store all the time. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlERkVcACgkQ9CaO5/Lv0PD0YACgg0mbr02sf0YHp1u+TeXT0oo/ eEsAoJyirF+Vz5YsGa3t7WbTXuq8uAuc =c0H2 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Seriously OT] Help in diagnosing server unresponsiveness
Chris, On Tue, Feb 5, 2013 at 5:58 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 2/1/13 9:57 PM, Howard W. Smith, Jr. wrote: On Fri, Feb 1, 2013 at 2:20 PM, Christopher Schultz ch...@christopherschultz.net wrote: If you want to improve performance even more, ditch EJB altogether. Wow, that hurt. ditch EJB altogether? I 'grew up' on EJB. When I was migrating from JSF managed beans to CDI managed beans, I recognized that I could reference EJBs via @Inject or @EJB via TomEE/OpenWebBeans, but I stuck with @EJB...'just because'. when I first read your recommendation, earlier on, i thought to myself, okay, that should be easy, but now, I don't know how I would 'ditch EJB altogether'. All of my EJBs are accessing database via JPA; there are a few native SQL statements, lot of dynamic SQL statements that I redefined as named queries. If you are referring to CDI managed beans performing database access... I think I read one blog that stated that CDI can do anything that EJB can do, but still, I guess I have not seen any examples where CDI bean is accessing database. So, why do you recommend to 'ditch EJB altogether'... to improve performance? please explain. EJB takes normal classes and interfaces and wraps them in remotable proxies that can be used across huge clusters of loosely-related servers. Most apps don't need all that, or, if they do, can do it in a smarter way because the author understands the use cases and can beat a generalized system handily. Understood. I'm not suggesting that you ditch JPA or J2EE in general... it's just that managed beans are dirt slow. I can understand why this is true; the JVM is working with a lot more code to meet your business requirements. I haven't used Hibernate or even JPA but older O-R mappers (Torque, etc.) all sucked and ended up pulling way more data than necessary from the database. Years ago, we ripped-out all that crap and wrote our own SQL ourselves. Not only were we able to see a dramatic improvement in performance (sorry, I don't have numbers for you... it's been a while) but we now have full-control over transactions, timeouts, complex where clauses, etc. with out having to fight against an API that wants to hide all that complexity from you. I was about to say that I write all my own SQL, but understood. Everytime I use JPA to do CRUD, JPA is generating the SQL for me. I thought I could refer to the java app that I wrote to transfer data from legacy MS-DOS dBase IV app/database (i developed in 1994/1995) to Apache Derby (or Java DB), where the SELECT statements from the dBase IV database were pure JDBC calls, but the inserts and updates into the Apache Derby database were all JPA. I developed that prior to developing the JSF2/EJB/JPA/HTML5 web app that accesses the Apache Derby database (today). IMO, developer performance trumps runtime performance most of the time. So, if you can create a more maintainable system in less time by using EJB (or whatever), then you go ahead and do it: servers are cheap, while developer time is expensive. agreed. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEUEAREIAAYFAlERjrMACgkQ9CaO5/Lv0PDaVgCTBdUHBpXKHZgl074pzojY1SZu 6ACfdhTJLLtK7GkKdra3E8gc4mntdGI= =Ruvy -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [Seriously OT] Help in diagnosing server unresponsiveness
On Tue, Feb 5, 2013 at 6:10 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chris, On 2/2/13 11:47 AM, chris derham wrote: In the Java world, most people would only call it a consider it a cluster if the app servers actually know about each other -- for instance, if you are using session replication. IMO session replication is a dog, and there are better ways to achieve similar goals that yield much higher performance. I am curious to hear your thoughts - care to elaborate? We've totally hijacked Zoran's thread. After this, let's move to a new OT thread (or threads). Tomcat's session replication basically blasts everything around to all other Tomcat instances (or you can choose one and use BackupManager). Every time you modify your session, you get a burst of traffic between all your cluster members. I dunno if Tomcat bothers updating the last-touched-time on the session if that's the only thing that changed during the request, but if so it means that there is a burst of replication traffic after /every single request/. Interesting! Most web applications I've seen use HttpSession for a variety of things, and not all of them need to be replicated. If you want to hand-roll some wrapper objects to allow you to stash things into the session that are not serializable (or shouldn't be serialized, like a transient wrapper) you can avoid some of this stuff, but you really shouldn't have to do this kind of thing. Instead, rely on some other data-sharing mechanism such as a database, memcached, etc. You can create a token that you pass to the client just like a JSESSIONID cookie and if they fail-over to a different node, you can still fetch your stuff when you need it. interesting... i was reading about a memcached (open source) project that works really well with tomcat, and i was going to ask about it on this list. I liked the blog that I saw about this, because 'honestly' it seemed easier to read and easier to implement than reading tomcat's 'cluster/session-replication documentation page'. :( an example or blog is always a great read. :) The problem with using HttpSession + session replication is that lazy programmers will stash just about anything into the session without thinking about it, and it will get replicated all over the place. Ouch, that hurt again! I think I did a lot of this when I first began Java EE development, but have learned that it is not necessary any longer, and honestly, I may still be guilty of stashing stuff into the session. If you use an external data storage mechanism, it's much more obvious that something special is happening, and (hopefully) your devs will take more care with dipping into that data store all the time. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlERkVcACgkQ9CaO5/Lv0PD0YACgg0mbr02sf0YHp1u+TeXT0oo/ eEsAoJyirF+Vz5YsGa3t7WbTXuq8uAuc =c0H2 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Chris, On Sun, Feb 3, 2013 at 12:39 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, If your requirements allow for users to have to re-authenticate when you have a failover-event, then I highly recommend that you use sticky-sessions /without/ session replication: performance will be much better that way (and it's easier to configure). this is very interesting but somewhat of a concern as well. does failover occur when one tomcat-and-web-app instance is taking a long time serving one request? there are a few places/pages in my web app that does some heavy lifting (reports that execute SQL SELECT, and yes, I am using indexes along with statement caching, etc...), and i've seen my app failover with a 404 error (i currently only have one instance). Well, if you can tolerate re-auth on failover, then there's nothing to do at all: simply enable sticky-sessions and let it happen. Failovers should be rare, anyway, right? sticky sessions sound nice and I would love to give it a try, but honestly, i don't think the endusers want to have to re-auth on failover. just today, I read on the atmosphere list that sticky sessions are required for using atmosphere with/in clusters. i'm definitely using atmosphere, but not using sticky sessions (or clusters) yet. :( - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEOoM0ACgkQ9CaO5/Lv0PAlOwCffVnMaz9jFn+hg7pSVBZGGLCv yYEAoIqunDmRHiQvzLrtR+5UcmwTKIt0 =L0xm -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Here's some updated observations after a not quite incident (CPU and memory spiked but the app is still running): 1. Yesterday we had a 90% CPU spike at a time where there was absolutely no server traffic. Verified through both the HTTP logs and the mod_jk logs. The CPU spiked and recovered back to average levels. 2. Used memory spiked at 10GB from a pre incident average of 500MB throughout 2 busy days without incident 3. Used memory has only gone back down to 4GB and is holding at this level 4. The Used physical memory went up from 2GB to 14GB and has stayed there 5. Garbage collector time spikes to 24.0. I think with JavaMelody it means that GC took 24% of of the CPU?? So I think our issues are related to GC. Is there a way to trigger more frequent GC which will hopefully be less resource intensive? And why have the memory usage levels not recovered? Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On Wed, Feb 6, 2013 at 1:15 PM, Zoran Avtarovski zo...@sparecreative.comwrote: Here's some updated observations after a not quite incident (CPU and memory spiked but the app is still running): 1. Yesterday we had a 90% CPU spike at a time where there was absolutely no server traffic. Verified through both the HTTP logs and the mod_jk logs. The CPU spiked and recovered back to average levels. 2. Used memory spiked at 10GB from a pre incident average of 500MB throughout 2 busy days without incident 3. Used memory has only gone back down to 4GB and is holding at this level 4. The Used physical memory went up from 2GB to 14GB and has stayed there 5. Garbage collector time spikes to 24.0. I think with JavaMelody it means that GC took 24% of of the CPU?? So I think our issues are related to GC. Is there a way to trigger more frequent GC which will hopefully be less resource intensive? And why have the memory usage levels not recovered? Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Zoran, First I would like to recommend the following document for reading: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms It explains the GC in JVM 1.6 including the Concurrent Collector settings which is the one you are using. The values for the GC in the log file are the time the particular collector spent for the operation so the 24.0 would probably mean 24 seconds, which might mean that for 24 seconds your applications might be unresponsive during the CMS GC. As explained in the document from the above link the GC has minor and major collecting phases. During the minor collection the objects are GC'ed from the so called young generation or promoted in the old (tenured) generation. More of this objects pile up in the old generation more frequent the major GC needs to run. The major ones take usually much longer (they have to clean much bigger space) time than the minor ones but the minor ones have to run more frequently. Now, what you need to find is what is causing your problem really? If your application creates lot of new objects then you'll have lots of minor GC running. More of this objects survive, they get moved to the old generation space and then you'll have lots of major GC running as well. The danger here is that you might end up with constantly running GC which will render your application unusable due to pauses. So basically badly written application can cause lots of problems, not closing connections and freeing objects etc etc, and in that case even the best GC tunning in the world will not help you, your application(s) will eventually get to halt. So read the document carefully and decide which user case is best for you. If you are creating lots of new objects then maybe increasing the minor space (default new/old ratio is 1:3) can help. Also paste here the results of the GC logs. The link I provided has some more useful settings and recommendations for the CMS collector. This collector stops the application threads twice during the operation so you need to check those times too. Cheers Pozdrav, Igor
Re: Help in diagnosing server unresponsiveness
Thanks Igor, I just stumbled upon that same document. I think you may be on to something here. I have a feeling that the GC may not be configured well. Z. On 6/02/13 2:15 PM, Igor Cicimov icici...@gmail.com wrote: On Wed, Feb 6, 2013 at 1:15 PM, Zoran Avtarovski zo...@sparecreative.comwrote: Here's some updated observations after a not quite incident (CPU and memory spiked but the app is still running): 1. Yesterday we had a 90% CPU spike at a time where there was absolutely no server traffic. Verified through both the HTTP logs and the mod_jk logs. The CPU spiked and recovered back to average levels. 2. Used memory spiked at 10GB from a pre incident average of 500MB throughout 2 busy days without incident 3. Used memory has only gone back down to 4GB and is holding at this level 4. The Used physical memory went up from 2GB to 14GB and has stayed there 5. Garbage collector time spikes to 24.0. I think with JavaMelody it means that GC took 24% of of the CPU?? So I think our issues are related to GC. Is there a way to trigger more frequent GC which will hopefully be less resource intensive? And why have the memory usage levels not recovered? Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Zoran, First I would like to recommend the following document for reading: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms It explains the GC in JVM 1.6 including the Concurrent Collector settings which is the one you are using. The values for the GC in the log file are the time the particular collector spent for the operation so the 24.0 would probably mean 24 seconds, which might mean that for 24 seconds your applications might be unresponsive during the CMS GC. As explained in the document from the above link the GC has minor and major collecting phases. During the minor collection the objects are GC'ed from the so called young generation or promoted in the old (tenured) generation. More of this objects pile up in the old generation more frequent the major GC needs to run. The major ones take usually much longer (they have to clean much bigger space) time than the minor ones but the minor ones have to run more frequently. Now, what you need to find is what is causing your problem really? If your application creates lot of new objects then you'll have lots of minor GC running. More of this objects survive, they get moved to the old generation space and then you'll have lots of major GC running as well. The danger here is that you might end up with constantly running GC which will render your application unusable due to pauses. So basically badly written application can cause lots of problems, not closing connections and freeing objects etc etc, and in that case even the best GC tunning in the world will not help you, your application(s) will eventually get to halt. So read the document carefully and decide which user case is best for you. If you are creating lots of new objects then maybe increasing the minor space (default new/old ratio is 1:3) can help. Also paste here the results of the GC logs. The link I provided has some more useful settings and recommendations for the CMS collector. This collector stops the application threads twice during the operation so you need to check those times too. Cheers Pozdrav, Igor - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Help in diagnosing server unresponsiveness
From: Zoran Avtarovski [mailto:zo...@sparecreative.com] Subject: Re: Help in diagnosing server unresponsiveness In addition to Igor's excellent advice, try the following. 3. Used memory has only gone back down to 4GB and is holding at this level Take a heap dump and find out what's consuming the space. Look here for some ways to do so: http://wiki.eclipse.org/index.php/MemoryAnalyzer#Getting_a_Heap_Dump 4. The Used physical memory went up from 2GB to 14GB and has stayed there That may be just allocated but now unused Java heap - but it could also be something outside of Java that's eating up the memory. The Java heap is easier to diagnose first, but if that's not sufficient you may need a C++ heap analyzer, but using one is rather non-trivial with the JVM. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On Sun, Feb 3, 2013 at 12:39 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 2/1/13 6:27 PM, Howard W. Smith, Jr. wrote: For now, I want a cluster of at least 2 or 3 tomcat servers for fault-tolerance and load balancing. If your requirements allow for users to have to re-authenticate when you have a failover-event, then I highly recommend that you use sticky-sessions /without/ session replication: performance will be much better that way (and it's easier to configure). Yes, I read that it is not good for web apps to start (Java) threads to do background work, and per that advise, I have avoided that for now. so far, I have used @Asynchronous and @Schedule, very minimally. The container in that case will be managing a thread pool on your behalf. This is the recommended way to do things, but it's not terrible if you want to manage your own thread pool for some particular reason. But, if the container will do it for you in a standards-compliant way, there doesn't seem to be a reason not to take advantage of it. Chris, I'm glad you mentioned, IMO session replication is a dog, because honestly, I would love to avoid some of the pre-work required to prepare my app for session replication. I'm definitely interested in the 'better ways to achieve similar goals. I would love to have 2 or 3 instances of my web app accessing one database (Derby, at the present), and all 2 or 3 instances actually knowing about each other. :) Well, if you can tolerate re-auth on failover, then there's nothing to do at all: simply enable sticky-sessions and let it happen. Failovers should be rare, anyway, right? - -chris Wow, thanks Chris for the response! I am going to have to take a look at sticky sessions. If I have any more questions, it's best for me to respond in a separate post/thread. Zoran, my apologies for hijacking the thread. I hope that you will be able to resolve the issue that you're having and hope you will keep us posted. thanks, Howard -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEOoM0ACgkQ9CaO5/Lv0PAlOwCffVnMaz9jFn+hg7pSVBZGGLCv yYEAoIqunDmRHiQvzLrtR+5UcmwTKIt0 =L0xm -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
2013/2/3 Zoran Avtarovski zo...@sparecreative.com: Thanks for the reply Chris, The crash is more a hang than a crash as we don't get OOME. The app just stops responding and the monitoring data shows spikes in memory and CPU to 100% and then nothing. I can't find anything in the linux system logs or in the tomcat or java logs. My hope is that once I can work out what's happening I can then fix the cause, but at the moment I'm flying blind on this. 0. It seems that you have not mentioned what exact version of Tomcat 7.0.x you are using. 1. Using JAVA_OPTS to specify memory settings is wrong. The correct variable is CATALINA_OPTS. See the comments at the top of bin/catalina.sh 2. Beware of PermGen OOMs. Note that if you exhaust the PermGen memory - An OutOfMemory happens - Runtime.freeMemory() still displays as if you had a lot of free memory available. - Your app cannot load any more classes - The thread where OOM happened usually dies. In this case Thread.uncaughtExceptionHandler is used and the OOM is printed to stdout (catalina.out) instead of being logged through the logging system. Since 7.0.33 the Status page of the Manager webapp displays detailed memory statistics. That page can also be retrieved as XML by adding XML=true argument. 3. Beware of web bots that do not honor cookies and can create too many new sessions in a short period of time. Those can be countered with a CrawlerSessionManagerValve http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
I definitely do not want to take/lead away from Edson and Mark's recommendations and responses related to linux, but as someone that has found success with his first-and-only JAVA/JSF web application, running on Windows Server 2003 32-bit 4GB, and now Windows Server 2008 R2 64-bit 32GB RAM (preparing for future development), one thing I love is ...debugging, as that has been my major strength throughout my (almost 20 year) career, prior to learning java via (Oracle's) java ee 6 tutorial (summer 2011) and beginning java development immediately afterwards. Just wanted to share that tidbit, for any/all following my responses from this point on... as Edson mentioned earlier, I am definitely 'junior' java developer, now, that is loving java/jsf, and learning so much by listening in on topics in this mail list. :) Anyway, this CPU spike reminds me of when I migrated from JSF-managed-beans to CDI-managed-beans and I experienced the cyclic references that is an absolute no-no with CDI, and even recently, add some new code to the app, which resulted in the same...cyclic reference between 2 existing CDI @SessionScoped beans. I had to resolve that on both occasions by a patch which included use/reference of Boolean variable, which was initialized in @Postconstruct, of course. Is it possible that your app has CDI involved and any cyclic references...that could be causing this CPU spike? On Sat, Feb 2, 2013 at 11:17 PM, Zoran Avtarovski zo...@sparecreative.comwrote: Hi Howard, The move to linux was part of a move in-house for our client as the web services are only accessible behind the firewall. My gut feeling is that the issue isn't related to the WS as they run on a scheduled task 3 times a day. I think the issue lies in our app and struggling with not being able to see exactly what's happening during the crash. JavaMelody provides some insight but just not enough. I'm quite happy to post the charts for others to see. Just not sure what the best way to do it is. Z. On 3/02/13 3:11 PM, Howard W. Smith, Jr. smithh032...@gmail.com wrote: I know this is asking for too much or might be impossible to do but process of elimination.. If it was possible to eliminate or prevent web services from executing or being accessed, and no spikes occur, then problem is there. I think you said earlier that system was stable on Windows and migration to Linux was driven by the web services requirement. I wonder what kind of processing in those web services which may be causing this. A lot of database access, even more database access now because of web services? Did some developer try to add a manual call to gc, somewhere in the app to free resources. Maybe you can poll any / all developers or search code accordingly. Does the spike occur at certain time of day, maybe some code executed on schedule, or does it occur after certain activity occur in the app either by endusers or background processing? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 03/02/2013 07:20, Howard W. Smith, Jr. escreveu: I definitely do not want to take/lead away from Edson and Mark's recommendations and responses related to linux, but as someone that has found success with his first-and-only JAVA/JSF web application, running on Windows Server 2003 32-bit 4GB, and now Windows Server 2008 R2 64-bit 32GB RAM (preparing for future development), one thing I love is ...debugging, as that has been my major strength throughout my (almost 20 year) career, prior to learning java via (Oracle's) java ee 6 tutorial (summer 2011) and beginning java development immediately afterwards. Just wanted to share that tidbit, for any/all following my responses from this point on... as Edson mentioned earlier, I am definitely 'junior' java developer, now, that is loving java/jsf, and learning so much by listening in on topics in this mail list. :) I hope you did not get offended :-) That wasn't my intention - I was mentioning I have some juniors here (programmers with less than a year of experience), and they adopt bad practices all the time (that's why used pair programming and code review meetings to get rid of). Actually, I am learning here as well. The purpose to help is really to learn and avoid same problems in future. By sharing our collective experience we can confirm our hypothesis or not - and then construct new knowledge from there. Anyway, this CPU spike reminds me of when I migrated from JSF-managed-beans to CDI-managed-beans and I experienced the cyclic references that is an absolute no-no with CDI, and even recently, add some new code to the app, which resulted in the same...cyclic reference between 2 existing CDI @SessionScoped beans. I had to resolve that on both occasions by a patch which included use/reference of Boolean variable, which was initialized in @Postconstruct, of course. Is it possible that your app has CDI involved and any cyclic references...that could be causing this CPU spike? So I learned something new here! Thanks, I'll keep an eye on it :-) Edson On Sat, Feb 2, 2013 at 11:17 PM, Zoran Avtarovski zo...@sparecreative.comwrote: Hi Howard, The move to linux was part of a move in-house for our client as the web services are only accessible behind the firewall. My gut feeling is that the issue isn't related to the WS as they run on a scheduled task 3 times a day. I think the issue lies in our app and struggling with not being able to see exactly what's happening during the crash. JavaMelody provides some insight but just not enough. I'm quite happy to post the charts for others to see. Just not sure what the best way to do it is. Z. On 3/02/13 3:11 PM, Howard W. Smith, Jr. smithh032...@gmail.com wrote: I know this is asking for too much or might be impossible to do but process of elimination.. If it was possible to eliminate or prevent web services from executing or being accessed, and no spikes occur, then problem is there. I think you said earlier that system was stable on Windows and migration to Linux was driven by the web services requirement. I wonder what kind of processing in those web services which may be causing this. A lot of database access, even more database access now because of web services? Did some developer try to add a manual call to gc, somewhere in the app to free resources. Maybe you can poll any / all developers or search code accordingly. Does the spike occur at certain time of day, maybe some code executed on schedule, or does it occur after certain activity occur in the app either by endusers or background processing? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On Sun, Feb 3, 2013 at 6:50 AM, Edson Richter edsonrich...@hotmail.com wrote: Em 03/02/2013 07:20, Howard W. Smith, Jr. escreveu: I hope you did not get offended :-) That wasn't my intention - I was mentioning I have some juniors here (programmers with less than a year of experience), and they adopt bad practices all the time (that's why used pair programming and code review meetings to get rid of). Actually, I am learning here as well. The purpose to help is really to learn and avoid same problems in future. By sharing our collective experience we can confirm our hypothesis or not - and then construct new knowledge from there. Definitely was not offended, thanks. TomEE user list (committers/etc) helped me get up and running with TomEE and CDI, and since TomEE = tomcat, I decided to mosey my way over here to this list for the tomcat side of tomee. So, i'm enjoying and benefiting all the while. :) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Hello Zoran, in a thread with so many replies, its partially hard to follow all replies. Have you resolved your problem already? Have you been able to exclude some of the most typical scenarios? I've read that you were using javamelody, have you tried something more insightful? regards Leon On Sun, Feb 3, 2013 at 5:17 AM, Zoran Avtarovski zo...@sparecreative.comwrote: Hi Howard, The move to linux was part of a move in-house for our client as the web services are only accessible behind the firewall. My gut feeling is that the issue isn't related to the WS as they run on a scheduled task 3 times a day. I think the issue lies in our app and struggling with not being able to see exactly what's happening during the crash. JavaMelody provides some insight but just not enough. I'm quite happy to post the charts for others to see. Just not sure what the best way to do it is. Z. On 3/02/13 3:11 PM, Howard W. Smith, Jr. smithh032...@gmail.com wrote: I know this is asking for too much or might be impossible to do but process of elimination.. If it was possible to eliminate or prevent web services from executing or being accessed, and no spikes occur, then problem is there. I think you said earlier that system was stable on Windows and migration to Linux was driven by the web services requirement. I wonder what kind of processing in those web services which may be causing this. A lot of database access, even more database access now because of web services? Did some developer try to add a manual call to gc, somewhere in the app to free resources. Maybe you can poll any / all developers or search code accordingly. Does the spike occur at certain time of day, maybe some code executed on schedule, or does it occur after certain activity occur in the app either by endusers or background processing? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 2/1/13 6:27 PM, Howard W. Smith, Jr. wrote: For now, I want a cluster of at least 2 or 3 tomcat servers for fault-tolerance and load balancing. If your requirements allow for users to have to re-authenticate when you have a failover-event, then I highly recommend that you use sticky-sessions /without/ session replication: performance will be much better that way (and it's easier to configure). Yes, I read that it is not good for web apps to start (Java) threads to do background work, and per that advise, I have avoided that for now. so far, I have used @Asynchronous and @Schedule, very minimally. The container in that case will be managing a thread pool on your behalf. This is the recommended way to do things, but it's not terrible if you want to manage your own thread pool for some particular reason. But, if the container will do it for you in a standards-compliant way, there doesn't seem to be a reason not to take advantage of it. Chris, I'm glad you mentioned, IMO session replication is a dog, because honestly, I would love to avoid some of the pre-work required to prepare my app for session replication. I'm definitely interested in the 'better ways to achieve similar goals. I would love to have 2 or 3 instances of my web app accessing one database (Derby, at the present), and all 2 or 3 instances actually knowing about each other. :) Well, if you can tolerate re-auth on failover, then there's nothing to do at all: simply enable sticky-sessions and let it happen. Failovers should be rare, anyway, right? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEOoM0ACgkQ9CaO5/Lv0PAlOwCffVnMaz9jFn+hg7pSVBZGGLCv yYEAoIqunDmRHiQvzLrtR+5UcmwTKIt0 =L0xm -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
hello my name is tommy gunzelman i signed up because sending a message i noticed tomcat sent i was curious however thing are more than meets the eye is im informing you please check if im receiving any my than just mail if so its without my consent and i need to be informed please call me directly 505-554-7878 thank you From my Android phone on T-Mobile. The first nationwide 4G network. Original message From: Howard W. Smith, Jr. smithh032...@gmail.com Date: To: Tomcat Users List users@tomcat.apache.org Subject: Re: Help in diagnosing server unresponsiveness If you want to improve performance even more, ditch EJB altogether. Moving from APR to NIO may be a good move, but it really depends upon your requirements. For instance, APR provides superior SSL performance but if you don't need it, NIO will probably give you better results. Understood, thanks Chris. For now, only office personnel is accessing the web app behind a firewall, and others (including myself) access via laptop and mobile devices outside of the office via HTTP, so NIO is sufficient for now, until I'm ready to go HTTPS/SSL via APR. That depends upon what you want to accomplish. You can get messages about JDBC resource management problems without writing any interceptors at all. Okay, well, I'm using JPA to access JTA managed datasource (Apache Derby), and I really don't think I have any JDBC resource management issues. That depends upon what you mean by clustering. If you want to serve more clients (or have fault-tolerance), then clustering is a good idea. Clustering means different things to different people. If you just want to scale horizontally (more app servers) and use simple load-balancing, that can be considered a cluster. For now, I want a cluster of at least 2 or 3 tomcat servers for fault-tolerance and load balancing. When I mention thread programming, I mean WebApps that start (Java) threads to do background work, like import data, send automatica notifications, and so on. I know it is a (almost) bad practice to have web apps creating threads, but this is so common in real world enterprise apps! Yes, I read that it is not good for web apps to start (Java) threads to do background work, and per that advise, I have avoided that for now. so far, I have used @Asynchronous and @Schedule, very minimally. In the Java world, most people would only call it a consider it a cluster if the app servers actually know about each other -- for instance, if you are using session replication. IMO session replication is a dog, and there are better ways to achieve similar goals that yield much higher performance. Chris, I'm glad you mentioned, IMO session replication is a dog, because honestly, I would love to avoid some of the pre-work required to prepare my app for session replication. I'm definitely interested in the 'better ways to achieve similar goals. I would love to have 2 or 3 instances of my web app accessing one database (Derby, at the present), and all 2 or 3 instances actually knowing about each other. :) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
In the Java world, most people would only call it a consider it a cluster if the app servers actually know about each other -- for instance, if you are using session replication. IMO session replication is a dog, and there are better ways to achieve similar goals that yield much higher performance. I am curious to hear your thoughts - care to elaborate? Thanks Chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On 01/02/2013 20:08, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? I would go in that direction too. Enable logs and core or stack dumps and analyze them. Be sure you are not restarting Tomcat in your crontab (i had a server which was restarted once a week and masked some memory starvation). In my case I can tell you I had to end up disabling JaveMelody (it was provoking some side effects in our webapp as not managing international chars in the right way). If reports from Javamelody are not giving you any clue, beware that Javamelody has its own memory overhead (not much in your case but in my case it was around 200 Mb of heap in a 1 Gb virtual server). I followed Chris directions, I got stack dumps after a server crash and analyzed it with eclipse analyzer. I realized our programmer decided to load too many objects in memory than the server could cope with. So no memory leak but bad memory management was the root cause. Regards, Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Thanks for the advice. All libraries are within the apps WEB-INF folder. It also doesn't appear to be a memory leak. Typically I would expect memory use to increase over time with a memory leak, but our app shows no increase just a sharp spike to max allocated memory prior to becoming unresponsive. For persistence we use Mybatis ORM which we have no issues with on other apps. I've set the GC logging up, but because there are no OOME no thread dumps are generated. Z. On 2/02/13 4:09 AM, Edson Richter edsonrich...@hotmail.com wrote: Em 01/02/2013 15:03, Edson Richter escreveu: Removing the hardware issues (faulty memory or disk), that you obviously already tested, I'll try to give some directions for testing: a) Main cause of memory leaks are hard references in main class loader. This happens when you put all your libraries into $TOMCAT_HOME/lib. Try to move your libraries into WEB-INF/lib and check how does your app behave. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). c) Also, we see incorrect thread programming... When I say (we see..) I want to mean: I see lots of junior programmers doing the same mistake over and over... I don't know if this is your case, but I feel that worth to mention here. remember to have a way to signalize to your threads that the application is closing or reloading. In my apps, I have a LifeCicly listener that will notify all threads that they should shutdown immediately. If a thread is stuck using resources, it will remain with that forever... d) If your app is JDK6 compatible, give a try on JRockit VM (from Oracle), and the excellent JRockit Mission Control that helps you to identify problems in real time. e) Never store content in static classes. The references stay forever. If you are using JPA, let JPA implementation to handle the cache. Use Soft Weak or Hard Weak if using EclipseLink. f) Never ever use OneToMany just because it is easy. If you have one object that has OneToMany to other 100, that these has One to many to another 100, you will have 10001 objects in memory with 1 query that is supposed to returns 1 record (the first object). If your query returns 10 records of the first object, you would have 11 object in memory... If you have 20 users using different objects... well, you got the point, right? By using good server hardware (ECC memory, SCSI disks, etc), a stable linux distro (my preference is for CentOS 64bit), and following the rules above, I manage to have web apps that run withing 2Gb of memory (on 8Gb of hardware), hundred of users in databases with 20Gb, 24x7. I hope this helps, Edson Richter Em 31/01/2013 23:36, Zoran Avtarovski escreveu: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 03/02/2013 01:35, Zoran Avtarovski escreveu: Thanks for the advice. All libraries are within the apps WEB-INF folder. It also doesn't appear to be a memory leak. Typically I would expect memory use to increase over time with a memory leak, but our app shows no increase just a sharp spike to max allocated memory prior to becoming unresponsive. Which VM are you using? I had similar problems (even Kernel Panic!) running OpenJDK (default in some Linux distros). Regards, Edson For persistence we use Mybatis ORM which we have no issues with on other apps. I've set the GC logging up, but because there are no OOME no thread dumps are generated. Z. On 2/02/13 4:09 AM, Edson Richter edsonrich...@hotmail.com wrote: Em 01/02/2013 15:03, Edson Richter escreveu: Removing the hardware issues (faulty memory or disk), that you obviously already tested, I'll try to give some directions for testing: a) Main cause of memory leaks are hard references in main class loader. This happens when you put all your libraries into $TOMCAT_HOME/lib. Try to move your libraries into WEB-INF/lib and check how does your app behave. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). c) Also, we see incorrect thread programming... When I say (we see..) I want to mean: I see lots of junior programmers doing the same mistake over and over... I don't know if this is your case, but I feel that worth to mention here. remember to have a way to signalize to your threads that the application is closing or reloading. In my apps, I have a LifeCicly listener that will notify all threads that they should shutdown immediately. If a thread is stuck using resources, it will remain with that forever... d) If your app is JDK6 compatible, give a try on JRockit VM (from Oracle), and the excellent JRockit Mission Control that helps you to identify problems in real time. e) Never store content in static classes. The references stay forever. If you are using JPA, let JPA implementation to handle the cache. Use Soft Weak or Hard Weak if using EclipseLink. f) Never ever use OneToMany just because it is easy. If you have one object that has OneToMany to other 100, that these has One to many to another 100, you will have 10001 objects in memory with 1 query that is supposed to returns 1 record (the first object). If your query returns 10 records of the first object, you would have 11 object in memory... If you have 20 users using different objects... well, you got the point, right? By using good server hardware (ECC memory, SCSI disks, etc), a stable linux distro (my preference is for CentOS 64bit), and following the rules above, I manage to have web apps that run withing 2Gb of memory (on 8Gb of hardware), hundred of users in databases with 20Gb, 24x7. I hope this helps, Edson Richter Em 31/01/2013 23:36, Zoran Avtarovski escreveu: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
I'm using the Sun JVM 1.6.0_31-b04. We had other performance issues with openJDK. Z. On 3/02/13 2:41 PM, Edson Richter edsonrich...@hotmail.com wrote: Em 03/02/2013 01:35, Zoran Avtarovski escreveu: Thanks for the advice. All libraries are within the apps WEB-INF folder. It also doesn't appear to be a memory leak. Typically I would expect memory use to increase over time with a memory leak, but our app shows no increase just a sharp spike to max allocated memory prior to becoming unresponsive. Which VM are you using? I had similar problems (even Kernel Panic!) running OpenJDK (default in some Linux distros). Regards, Edson For persistence we use Mybatis ORM which we have no issues with on other apps. I've set the GC logging up, but because there are no OOME no thread dumps are generated. Z. On 2/02/13 4:09 AM, Edson Richter edsonrich...@hotmail.com wrote: Em 01/02/2013 15:03, Edson Richter escreveu: Removing the hardware issues (faulty memory or disk), that you obviously already tested, I'll try to give some directions for testing: a) Main cause of memory leaks are hard references in main class loader. This happens when you put all your libraries into $TOMCAT_HOME/lib. Try to move your libraries into WEB-INF/lib and check how does your app behave. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). c) Also, we see incorrect thread programming... When I say (we see..) I want to mean: I see lots of junior programmers doing the same mistake over and over... I don't know if this is your case, but I feel that worth to mention here. remember to have a way to signalize to your threads that the application is closing or reloading. In my apps, I have a LifeCicly listener that will notify all threads that they should shutdown immediately. If a thread is stuck using resources, it will remain with that forever... d) If your app is JDK6 compatible, give a try on JRockit VM (from Oracle), and the excellent JRockit Mission Control that helps you to identify problems in real time. e) Never store content in static classes. The references stay forever. If you are using JPA, let JPA implementation to handle the cache. Use Soft Weak or Hard Weak if using EclipseLink. f) Never ever use OneToMany just because it is easy. If you have one object that has OneToMany to other 100, that these has One to many to another 100, you will have 10001 objects in memory with 1 query that is supposed to returns 1 record (the first object). If your query returns 10 records of the first object, you would have 11 object in memory... If you have 20 users using different objects... well, you got the point, right? By using good server hardware (ECC memory, SCSI disks, etc), a stable linux distro (my preference is for CentOS 64bit), and following the rules above, I manage to have web apps that run withing 2Gb of memory (on 8Gb of hardware), hundred of users in databases with 20Gb, 24x7. I hope this helps, Edson Richter Em 31/01/2013 23:36, Zoran Avtarovski escreveu: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional
Re: Help in diagnosing server unresponsiveness
Thanks for the reply Chris, The crash is more a hang than a crash as we don't get OOME. The app just stops responding and the monitoring data shows spikes in memory and CPU to 100% and then nothing. I can't find anything in the linux system logs or in the tomcat or java logs. My hope is that once I can work out what's happening I can then fix the cause, but at the moment I'm flying blind on this. Z. On 2/02/13 6:08 AM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEMEpAACgkQ9CaO5/Lv0PDmvgCeL+F9onCbvkWL5+1BEhAGcoY5 mxYAnif8os4/JiNaH2U5OmiWYB/GXMKm =2pXj -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Hi Edson, We do have some background threads as we use Quartz for scheduling tasks but we haven't had any issues with it in the past. I also checked the monitoring and I'm seeing anything strange during the execution of the scheduled tasks. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 03/02/2013 02:01, Zoran Avtarovski escreveu: Hi Edson, We do have some background threads as we use Quartz for scheduling tasks but we haven't had any issues with it in the past. I also checked the monitoring and I'm seeing anything strange during the execution of the scheduled tasks. Z. My best advice is to put your application in executing using some monitoring tool (like the JRockit I proposed before, with the Mission Control) to try to discover the root of the problem. Regards, Edson - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Thanks Miguel, This is what I also suspect, but I can't see any evidence. The server has gone 10 days under heavy loads without a glitch and then it will hang a couple of times in the next few days with no apparent rhyme or reason. Z. On 3/02/13 5:56 AM, Miguel González Castaños miguel_3_gonza...@yahoo.es wrote: On 01/02/2013 20:08, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? I would go in that direction too. Enable logs and core or stack dumps and analyze them. Be sure you are not restarting Tomcat in your crontab (i had a server which was restarted once a week and masked some memory starvation). In my case I can tell you I had to end up disabling JaveMelody (it was provoking some side effects in our webapp as not managing international chars in the right way). If reports from Javamelody are not giving you any clue, beware that Javamelody has its own memory overhead (not much in your case but in my case it was around 200 Mb of heap in a 1 Gb virtual server). I followed Chris directions, I got stack dumps after a server crash and analyzed it with eclipse analyzer. I realized our programmer decided to load too many objects in memory than the server could cope with. So no memory leak but bad memory management was the root cause. Regards, Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Hi Howard, The move to linux was part of a move in-house for our client as the web services are only accessible behind the firewall. My gut feeling is that the issue isn't related to the WS as they run on a scheduled task 3 times a day. I think the issue lies in our app and struggling with not being able to see exactly what's happening during the crash. JavaMelody provides some insight but just not enough. I'm quite happy to post the charts for others to see. Just not sure what the best way to do it is. Z. On 3/02/13 3:11 PM, Howard W. Smith, Jr. smithh032...@gmail.com wrote: I know this is asking for too much or might be impossible to do but process of elimination.. If it was possible to eliminate or prevent web services from executing or being accessed, and no spikes occur, then problem is there. I think you said earlier that system was stable on Windows and migration to Linux was driven by the web services requirement. I wonder what kind of processing in those web services which may be causing this. A lot of database access, even more database access now because of web services? Did some developer try to add a manual call to gc, somewhere in the app to free resources. Maybe you can poll any / all developers or search code accordingly. Does the spike occur at certain time of day, maybe some code executed on schedule, or does it occur after certain activity occur in the app either by endusers or background processing? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Just occurred to me: Linux has one feature that Windows doesn't: the OOM killer. When happens, normally you get a message in log. Have you checked that? If not, there is a little introduction here: http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html?page=1 Also, I had some issues with OOM killer in virtual machines running over qemu (actually, the OOM was happening in the host machine, not in guest - then in guest, had no error message). Regards, Edson Em 03/02/2013 02:07, Zoran Avtarovski escreveu: Thanks Miguel, This is what I also suspect, but I can't see any evidence. The server has gone 10 days under heavy loads without a glitch and then it will hang a couple of times in the next few days with no apparent rhyme or reason. Z. On 3/02/13 5:56 AM, Miguel González Castaños miguel_3_gonza...@yahoo.es wrote: On 01/02/2013 20:08, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? I would go in that direction too. Enable logs and core or stack dumps and analyze them. Be sure you are not restarting Tomcat in your crontab (i had a server which was restarted once a week and masked some memory starvation). In my case I can tell you I had to end up disabling JaveMelody (it was provoking some side effects in our webapp as not managing international chars in the right way). If reports from Javamelody are not giving you any clue, beware that Javamelody has its own memory overhead (not much in your case but in my case it was around 200 Mb of heap in a 1 Gb virtual server). I followed Chris directions, I got stack dumps after a server crash and analyzed it with eclipse analyzer. I realized our programmer decided to load too many objects in memory than the server could cope with. So no memory leak but bad memory management was the root cause. Regards, Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On 2/2/2013 8:07 PM, Zoran Avtarovski wrote: Thanks Miguel, This is what I also suspect, but I can't see any evidence. The server has gone 10 days under heavy loads without a glitch and then it will hang a couple of times in the next few days with no apparent rhyme or reason. Z. On 3/02/13 5:56 AM, Miguel González Castaños miguel_3_gonza...@yahoo.es wrote: On 01/02/2013 20:08, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? I would go in that direction too. Enable logs and core or stack dumps and analyze them. Be sure you are not restarting Tomcat in your crontab (i had a server which was restarted once a week and masked some memory starvation). In my case I can tell you I had to end up disabling JaveMelody (it was provoking some side effects in our webapp as not managing international chars in the right way). If reports from Javamelody are not giving you any clue, beware that Javamelody has its own memory overhead (not much in your case but in my case it was around 200 Mb of heap in a 1 Gb virtual server). I followed Chris directions, I got stack dumps after a server crash and analyzed it with eclipse analyzer. I realized our programmer decided to load too many objects in memory than the server could cope with. So no memory leak but bad memory management was the root cause. Regards, Miguel I've sort of followed this thread. If I remember correctly, you've recently moved to Linux. Here's an approach that might tell you what's going on at the time of the problem. When you're experiencing the problem, if you can, get a full thread dump. For Linux, that means you send a -3 signal to the PID (kill -3 PID). PID is the process ID of the Tomcat in trouble. At the same time, either run ps -FL PID or run top -p PID and switch to watching threads (H). See if you can see which thread is using 100% of the CPU from either PS or top. Sadly, the thread ID in the thread dump is hex, while the thread ID in either of the above two methods is decimal. So you'll have to do a bit of digging. The number in the thread dump is nid=0xNNN. In top, the thread ID will be displayed as the process ID. In ps, the thread ID will be displayed as the LWP ID. Hopefully, if you catch the Tomcat process at the right time, you'll be able to see which thread is consuming all the CPU, and from the thread dump see what the thread is doing. . . . . just my (Saturday night) two cents /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On 03/02/2013 3:17 PM, Zoran Avtarovski zo...@sparecreative.com wrote: Hi Howard, The move to linux was part of a move in-house for our client as the web services are only accessible behind the firewall. My gut feeling is that the issue isn't related to the WS as they run on a scheduled task 3 times a day. I think the issue lies in our app and struggling with not being able to see exactly what's happening during the crash. JavaMelody provides some insight but just not enough. Norhing in the catalina.out or app specific logs? I'm quite happy to post the charts for others to see. Just not sure what the best way to do it is. Z. On 3/02/13 3:11 PM, Howard W. Smith, Jr. smithh032...@gmail.com wrote: I know this is asking for too much or might be impossible to do but process of elimination.. If it was possible to eliminate or prevent web services from executing or being accessed, and no spikes occur, then problem is there. I think you said earlier that system was stable on Windows and migration to Linux was driven by the web services requirement. I wonder what kind of processing in those web services which may be causing this. A lot of database access, even more database access now because of web services? Did some developer try to add a manual call to gc, somewhere in the app to free resources. Maybe you can poll any / all developers or search code accordingly. Does the spike occur at certain time of day, maybe some code executed on schedule, or does it occur after certain activity occur in the app either by endusers or background processing? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
From our experience with a Spring app under Tomcat, the JVM memory configuration looks low. You might try doubling it and seeing what it does. The virtual memory of the OS is not as much of an issue since the Tomcat JVM memory limit will stop it from growing to use the Linux address space. The GC only affects the JVM cache. You should make sure that you VM has lots of swap space and then monitor Linux's use of the swap space to make sure that it is not a problem outside the JVM. I am not sure how 6Gb of virtual OS RAM help when the Tomcat app is restricted to 1Gb. We found that the examples of Tomcat JVM memory configurations were all designed for trivial apps. We had about 50 war files and lots of heavy libraries(JasperReports, WebServices, Faces, etc.) so we had lots of troubles until we gave Tomcat some real JVM room to run. On 31/01/2013 10:58 PM, Howard W. Smith, Jr. wrote: On Thu, Jan 31, 2013 at 10:32 PM, Zoran Avtarovski zo...@sparecreative.com wrote: 1.The app has been running on Tomcat 7 in a virtualised linux environment for about 4 months and the issue has been persistent through that time. Prior to that it was running on a physical Windows server with no issues. The move to linux coincided with the addition of a substantial amount of WebServices functionality - consuming. very interesting, stable on windows (without webservices footprint). what was your java -version details on windows, and what is your java -version details on linux? so the webservices functionality is 'new' software/code? was the code implemented well for GC? any use of threadlocals? i am only shooting from the hip, here, as I primarily listen in on topics here, and have limited experience with java and tomcat. 2.The tomcat instance was originally running with a max of 6Gb of allocated memory and it would crash almost daily. When we upped it to 11gb the crashes reduced to about twice a week. 3. Tomcat is configured with the following options export JAVA_OPTS=-Xms256m -Xmx11460m -Djava.awt.headless=true -XX:MaxPermSize=256M -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC MaxPermSize seems to me to be too low for a real Tomcat app. I don't see any point for such a small -Xms256 setting since that seems to be barely enough to get Tomcat started. -Xmx11460 seems very high. We did very well with 10% of that number. 4. Xms and MaxPermSize were just reduced from 512m 5. The app generally has about 50-100 concurrent users - the logs show that Tomcat can crash with as few as 20 users. I'm wondering if it could be GC related as the crashes often follow periods of heavy use. 6. The app is a struts2 based app using Spring 3.5 for dependancy injection/ioc. this is good to know. have you seen/heard of similar reports from others using struts2 and spring 3.5 for dependancy injection/ioc? The problem with getting a thread dump is that high CPU usage is only a spike and server freezes. CPU usage only goes above 5-10% when it's about to crash. Support then reboot the server instances and off it goes. I've tried to spend a few days just watching, but of course it didn't skip a beat while I was monitoring. Is there a way to trigger gather relevant data when Tomcat crashes? Z. I searched tomcat user list, please see below, and consider what was mentioned there for troubleshooting this issue, and report back here. http://tomcat.10.n6.nabble.com/Tomcat-suddenly-dies-td4518543.html - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Ron Wheeler President Artifact Software Inc email: rwhee...@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435, ext 102 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Help in diagnosing server unresponsiveness
Ron Wheeler wrote: .. I am not sure how 6Gb of virtual OS RAM help when the Tomcat app is restricted to 1Gb. Really ? what a poor repressed little app. Just think of it : it only has one thousand million bytes of RAM to play with. A real scandal, that is. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Removing the hardware issues (faulty memory or disk), that you obviously already tested, I'll try to give some directions for testing: a) Main cause of memory leaks are hard references in main class loader. This happens when you put all your libraries into $TOMCAT_HOME/lib. Try to move your libraries into WEB-INF/lib and check how does your app behave. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). c) Also, we see incorrect thread programming... remember to have a way to signalize to your threads that the application is closing or reloading. In my apps, I have a LifeCicly listener that will notify all threads that they should shutdown immediately. If a thread is stuck using resources, it will remain with that forever... d) If your app is JDK6 compatible, give a try on JRockit VM (from Oracle), and the excellent JRockit Mission Control that helps you to identify problems in real time. e) Never store content in static classes. The references stay forever. If you are using JPA, let JPA implementation to handle the cache. Use Soft Weak or Hard Weak if using EclipseLink. f) Never ever use OneToMany just because it is easy. If you have one object that has OneToMany to other 100, that these has One to many to another 100, you will have 10001 objects in memory with 1 query that is supposed to returns 1 record (the first object). If your query returns 10 records of the first object, you would have 11 object in memory... If you have 20 users using different objects... well, you got the point, right? By using good server hardware (ECC memory, SCSI disks, etc), a stable linux distro (my preference is for CentOS 64bit), and following the rules above, I manage to have web apps that run withing 2Gb of memory (on 8Gb of hardware), hundred of users in databases with 20Gb, 24x7. I hope this helps, Edson Richter Em 31/01/2013 23:36, Zoran Avtarovski escreveu: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 01/02/2013 15:03, Edson Richter escreveu: Removing the hardware issues (faulty memory or disk), that you obviously already tested, I'll try to give some directions for testing: a) Main cause of memory leaks are hard references in main class loader. This happens when you put all your libraries into $TOMCAT_HOME/lib. Try to move your libraries into WEB-INF/lib and check how does your app behave. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). c) Also, we see incorrect thread programming... When I say (we see..) I want to mean: I see lots of junior programmers doing the same mistake over and over... I don't know if this is your case, but I feel that worth to mention here. remember to have a way to signalize to your threads that the application is closing or reloading. In my apps, I have a LifeCicly listener that will notify all threads that they should shutdown immediately. If a thread is stuck using resources, it will remain with that forever... d) If your app is JDK6 compatible, give a try on JRockit VM (from Oracle), and the excellent JRockit Mission Control that helps you to identify problems in real time. e) Never store content in static classes. The references stay forever. If you are using JPA, let JPA implementation to handle the cache. Use Soft Weak or Hard Weak if using EclipseLink. f) Never ever use OneToMany just because it is easy. If you have one object that has OneToMany to other 100, that these has One to many to another 100, you will have 10001 objects in memory with 1 query that is supposed to returns 1 record (the first object). If your query returns 10 records of the first object, you would have 11 object in memory... If you have 20 users using different objects... well, you got the point, right? By using good server hardware (ECC memory, SCSI disks, etc), a stable linux distro (my preference is for CentOS 64bit), and following the rules above, I manage to have web apps that run withing 2Gb of memory (on 8Gb of hardware), hundred of users in databases with 20Gb, 24x7. I hope this helps, Edson Richter Em 31/01/2013 23:36, Zoran Avtarovski escreveu: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On Fri, Feb 1, 2013 at 12:09 PM, Edson Richter edsonrich...@hotmail.com wrote: Em 01/02/2013 15:03, Edson Richter escreveu: When I say (we see..) I want to mean: I see lots of junior programmers doing the same mistake over and over... I don't know if this is your case, but I feel that worth to mention here. wow, this is all good to know! Thanks! I definitely consider myself junior 'java' developer even though i've been developing software since 1995. I'm loving java/jsf and tomee/tomcat right now. my app is running fine, but i'm always striving for perfection and performance, and that is why I made my way from mojarra to myfaces, glassfish to tomee/tomcat, and jsf-managed-beans to cdi-managed-beans, and just early this morning from APR to NIO connector. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). backtracking... is this a chance/time to use jdbc interceptors? i've seen some chatter about jdbc interceptors, but have not really dug into it quite yet. c) Also, we see incorrect thread programming... this sounds good for clusters, right? i'm hoping to use clustering for the app that i've developed. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEMEpAACgkQ9CaO5/Lv0PDmvgCeL+F9onCbvkWL5+1BEhAGcoY5 mxYAnif8os4/JiNaH2U5OmiWYB/GXMKm =2pXj -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ron, On 2/1/13 8:14 AM, Ron Wheeler wrote: From our experience with a Spring app under Tomcat, the JVM memory configuration looks low. You might try doubling it and seeing what it does. That's a big jump for not knowing anything about the OP's environment, requirements, etc. We found that the examples of Tomcat JVM memory configurations were all designed for trivial apps. We had about 50 war files and lots of heavy libraries(JasperReports, WebServices, Faces, etc.) so we had lots of troubles until we gave Tomcat some real JVM room to run. That's better advice than above. MaxPermSize seems to me to be too low for a real Tomcat app. We run real apps under default PermGen all the time. They run just fine. I don't see any point for such a small -Xms256 setting since that seems to be barely enough to get Tomcat started. In my testing, Tomcat takes roughly 12MiB of heap space after startup. I haven't actually tried to restrict Tomcat's JVM to that little amount of memory because it's impractical for adding any of our webapps to it. But saying that Tomcat needs a quarter gig just to get started is wildly out of step with reality. -Xmx11460 seems very high. We did very well with 10% of that number. Again, this is very environment-specific. Perhaps you have a webapp that doesn't use a lot of memory, while the OP may have a ton of stuff in his sessions, caches, etc. Maybe he needs 10GiB of heap space. Maybe he doesn't. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEUEAREIAAYFAlEME80ACgkQ9CaO5/Lv0PC3mwCYgURV99Bx/H5dq88PcsHd4Taa 8ACgoGUdHfl1JOh5Z6kn/reuPXSHtQY= =+9Jd -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 2/1/13 12:41 PM, Howard W. Smith, Jr. wrote: my app is running fine, but i'm always striving for perfection and performance, and that is why I made my way from mojarra to myfaces, glassfish to tomee/tomcat, and jsf-managed-beans to cdi-managed-beans, and just early this morning from APR to NIO connector. If you want to improve performance even more, ditch EJB altogether. Moving from APR to NIO may be a good move, but it really depends upon your requirements. For instance, APR provides superior SSL performance but if you don't need it, NIO will probably give you better results. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). backtracking... is this a chance/time to use jdbc interceptors? i've seen some chatter about jdbc interceptors, but have not really dug into it quite yet. That depends upon what you want to accomplish. You can get messages about JDBC resource management problems without writing any interceptors at all. c) Also, we see incorrect thread programming... this sounds good for clusters, right? i'm hoping to use clustering for the app that i've developed. That depends upon what you mean by clustering. If you want to serve more clients (or have fault-tolerance), then clustering is a good idea. Clustering means different things to different people. If you just want to scale horizontally (more app servers) and use simple load-balancing, that can be considered a cluster. In the Java world, most people would only call it a consider it a cluster if the app servers actually know about each other -- for instance, if you are using session replication. IMO session replication is a dog, and there are better ways to achieve similar goals that yield much higher performance. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEMFXwACgkQ9CaO5/Lv0PC0rACgw7bfEDyVZhHr4V5IWPndxM8Z bAEAnjhZlD0T6tK4jEI1XkszWmVZ4R85 =FwGs -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 01/02/2013 17:20, Christopher Schultz escreveu: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 2/1/13 12:41 PM, Howard W. Smith, Jr. wrote: my app is running fine, but i'm always striving for perfection and performance, and that is why I made my way from mojarra to myfaces, glassfish to tomee/tomcat, and jsf-managed-beans to cdi-managed-beans, and just early this morning from APR to NIO connector. If you want to improve performance even more, ditch EJB altogether. Moving from APR to NIO may be a good move, but it really depends upon your requirements. For instance, APR provides superior SSL performance but if you don't need it, NIO will probably give you better results. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). backtracking... is this a chance/time to use jdbc interceptors? i've seen some chatter about jdbc interceptors, but have not really dug into it quite yet. That depends upon what you want to accomplish. You can get messages about JDBC resource management problems without writing any interceptors at all. c) Also, we see incorrect thread programming... this sounds good for clusters, right? i'm hoping to use clustering for the app that i've developed. That depends upon what you mean by clustering. If you want to serve more clients (or have fault-tolerance), then clustering is a good idea. Clustering means different things to different people. If you just want to scale horizontally (more app servers) and use simple load-balancing, that can be considered a cluster. When I mention thread programming, I mean WebApps that start (Java) threads to do background work, like import data, send automatica notifications, and so on. I know it is a (almost) bad practice to have web apps creating threads, but this is so common in real world enterprise apps! Edson In the Java world, most people would only call it a consider it a cluster if the app servers actually know about each other -- for instance, if you are using session replication. IMO session replication is a dog, and there are better ways to achieve similar goals that yield much higher performance. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEMFXwACgkQ9CaO5/Lv0PC0rACgw7bfEDyVZhHr4V5IWPndxM8Z bAEAnjhZlD0T6tK4jEI1XkszWmVZ4R85 =FwGs -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
If you want to improve performance even more, ditch EJB altogether. Moving from APR to NIO may be a good move, but it really depends upon your requirements. For instance, APR provides superior SSL performance but if you don't need it, NIO will probably give you better results. Understood, thanks Chris. For now, only office personnel is accessing the web app behind a firewall, and others (including myself) access via laptop and mobile devices outside of the office via HTTP, so NIO is sufficient for now, until I'm ready to go HTTPS/SSL via APR. That depends upon what you want to accomplish. You can get messages about JDBC resource management problems without writing any interceptors at all. Okay, well, I'm using JPA to access JTA managed datasource (Apache Derby), and I really don't think I have any JDBC resource management issues. That depends upon what you mean by clustering. If you want to serve more clients (or have fault-tolerance), then clustering is a good idea. Clustering means different things to different people. If you just want to scale horizontally (more app servers) and use simple load-balancing, that can be considered a cluster. For now, I want a cluster of at least 2 or 3 tomcat servers for fault-tolerance and load balancing. When I mention thread programming, I mean WebApps that start (Java) threads to do background work, like import data, send automatica notifications, and so on. I know it is a (almost) bad practice to have web apps creating threads, but this is so common in real world enterprise apps! Yes, I read that it is not good for web apps to start (Java) threads to do background work, and per that advise, I have avoided that for now. so far, I have used @Asynchronous and @Schedule, very minimally. In the Java world, most people would only call it a consider it a cluster if the app servers actually know about each other -- for instance, if you are using session replication. IMO session replication is a dog, and there are better ways to achieve similar goals that yield much higher performance. Chris, I'm glad you mentioned, IMO session replication is a dog, because honestly, I would love to avoid some of the pre-work required to prepare my app for session replication. I'm definitely interested in the 'better ways to achieve similar goals. I would love to have 2 or 3 instances of my web app accessing one database (Derby, at the present), and all 2 or 3 instances actually knowing about each other. :) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 01/02/2013 21:27, Howard W. Smith, Jr. escreveu: If you want to improve performance even more, ditch EJB altogether. Moving from APR to NIO may be a good move, but it really depends upon your requirements. For instance, APR provides superior SSL performance but if you don't need it, NIO will probably give you better results. Understood, thanks Chris. For now, only office personnel is accessing the web app behind a firewall, and others (including myself) access via laptop and mobile devices outside of the office via HTTP, so NIO is sufficient for now, until I'm ready to go HTTPS/SSL via APR. That depends upon what you want to accomplish. You can get messages about JDBC resource management problems without writing any interceptors at all. Okay, well, I'm using JPA to access JTA managed datasource (Apache Derby), and I really don't think I have any JDBC resource management issues. Have you look into your JPA graph and check if you are not loading millions of objects in memory? While JPA is loading data from database, you will experience unresponsiveness as you said. I do use PostgresSQL (that has superb performance, IMHO), and to discover the bloated references, I just had to set log to record all queries sent by the application. Regards, Edson That depends upon what you mean by clustering. If you want to serve more clients (or have fault-tolerance), then clustering is a good idea. Clustering means different things to different people. If you just want to scale horizontally (more app servers) and use simple load-balancing, that can be considered a cluster. For now, I want a cluster of at least 2 or 3 tomcat servers for fault-tolerance and load balancing. When I mention thread programming, I mean WebApps that start (Java) threads to do background work, like import data, send automatica notifications, and so on. I know it is a (almost) bad practice to have web apps creating threads, but this is so common in real world enterprise apps! Yes, I read that it is not good for web apps to start (Java) threads to do background work, and per that advise, I have avoided that for now. so far, I have used @Asynchronous and @Schedule, very minimally. In the Java world, most people would only call it a consider it a cluster if the app servers actually know about each other -- for instance, if you are using session replication. IMO session replication is a dog, and there are better ways to achieve similar goals that yield much higher performance. Chris, I'm glad you mentioned, IMO session replication is a dog, because honestly, I would love to avoid some of the pre-work required to prepare my app for session replication. I'm definitely interested in the 'better ways to achieve similar goals. I would love to have 2 or 3 instances of my web app accessing one database (Derby, at the present), and all 2 or 3 instances actually knowing about each other. :) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 01/02/2013 21:57, Howard W. Smith, Jr. escreveu: Okay, well, I'm using JPA to access JTA managed datasource (Apache Derby), and I really don't think I have any JDBC resource management issues. Have you look into your JPA graph and check if you are not loading millions of objects in memory? While JPA is loading data from database, you will experience unresponsiveness as you said. I do use PostgresSQL (that has superb performance, IMHO), and to discover the bloated references, I just had to set log to record all queries sent by the application. Well, Zoran reported unresponsiveness in the OP. :) Sorry, I mixed up things in my head... I'll pay more attention. I am quite pleased with the responsiveness of my app... using EclipseLink and Derby. Good. What is the best way to look into the JPA? via JMX? In development, I do use NetBeans with Profiler (memory or cpu profile are good start point). In test environment I do use JRockit Mission Control remote monitoring. I do not use any of these in production. Probably would be possible to monitor using JMX (via JMX Proxy Servlet in Tomcat?) There is one interesting blog entry here: http://blog.vennster.nl/2012/09/managing-eclipselink-using-jmx.html But I never tried that, just had in my favorits just in case. Regards, Edson - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
What is the best way to look into the JPA? via JMX? In development, I do use NetBeans with Profiler (memory or cpu profile are good start point). In test environment I do use JRockit Mission Control remote monitoring. I do not use any of these in production. Probably would be possible to monitor using JMX (via JMX Proxy Servlet in Tomcat?) There is one interesting blog entry here: http://blog.vennster.nl/2012/09/managing-eclipselink-using-jmx.html But I never tried that, just had in my favorits just in case. Earlier, I think I saw you (or someone else) mention/recommend JRockit if you're using JDK6 (I could be wrong though). If that is the case, I migrated most of my source code from JDK6 to JDK7 (switch, ArrayList, ...), so I may not be able to use that. I like how you favorite/bookmark for 'just in case'... i do the same. :) Thanks for sharing. I briefly looked at the page you referenced and saw 'weblogic'... I hope i would be able to configure the same for 'tomcat'. :) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Chris, On Fri, Feb 1, 2013 at 2:20 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 2/1/13 12:41 PM, Howard W. Smith, Jr. wrote: my app is running fine, but i'm always striving for perfection and performance, and that is why I made my way from mojarra to myfaces, glassfish to tomee/tomcat, and jsf-managed-beans to cdi-managed-beans, and just early this morning from APR to NIO connector. If you want to improve performance even more, ditch EJB altogether. Wow, that hurt. ditch EJB altogether? I 'grew up' on EJB. When I was migrating from JSF managed beans to CDI managed beans, I recognized that I could reference EJBs via @Inject or @EJB via TomEE/OpenWebBeans, but I stuck with @EJB...'just because'. when I first read your recommendation, earlier on, i thought to myself, okay, that should be easy, but now, I don't know how I would 'ditch EJB altogether'. All of my EJBs are accessing database via JPA; there are a few native SQL statements, lot of dynamic SQL statements that I redefined as named queries. If you are referring to CDI managed beans performing database access... I think I read one blog that stated that CDI can do anything that EJB can do, but still, I guess I have not seen any examples where CDI bean is accessing database. So, why do you recommend to 'ditch EJB altogether'... to improve performance? please explain. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Help in diagnosing server unresponsiveness
Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z.
Re: Help in diagnosing server unresponsiveness
On 01/02/2013 12:37 PM, Zoran Avtarovski zo...@sparecreative.com wrote: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. Take couple of thread dumps during the high cpu usage.
Re: Help in diagnosing server unresponsiveness
We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. How long have you seen this behavior? Was it running more stable on an earlier tomcat7 version? what version of tomcat7 was more stable? it may be best to provide your version/environment info/details, and java options and any other config details. i recently started using tomcat7.0.35, too, since I am using tomee 1.5.2-snapshot, and i recognized similar behavior (app was crashing unexpectedly due to atmosphere and tomcat APR/native), but I corrected my 'tomee' configuration (used entired tomee stack instead of tomcat7 stack running my tomee stack) and reverted to earlier-more-stable version of atmosphere (1.0.6), and i did not experience any downtime today (but the app was not under much load today), so i'm glad i got that fixed; i am keeping an eye on things though. also, you may want to share what 'all' changed between now and the last time that the app/tomcat7 was stable. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. you may want to share some numbers I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. i'm wondering if GC is causing the CPU spikes. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Howard, Igor, Thanks for your responses. In relation to more detail: 1.The app has been running on Tomcat 7 in a virtualised linux environment for about 4 months and the issue has been persistent through that time. Prior to that it was running on a physical Windows server with no issues. The move to linux coincided with the addition of a substantial amount of WebServices functionality - consuming. 2.The tomcat instance was originally running with a max of 6Gb of allocated memory and it would crash almost daily. When we upped it to 11gb the crashes reduced to about twice a week. 3. Tomcat is configured with the following options export JAVA_OPTS=-Xms256m -Xmx11460m -Djava.awt.headless=true -XX:MaxPermSize=256M -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC 4. Xms and MaxPermSize were just reduced from 512m 5. The app generally has about 50-100 concurrent users - the logs show that Tomcat can crash with as few as 20 users. I'm wondering if it could be GC related as the crashes often follow periods of heavy use. 6. The app is a struts2 based app using Spring 3.5 for dependancy injection/ioc. The problem with getting a thread dump is that high CPU usage is only a spike and server freezes. CPU usage only goes above 5-10% when it's about to crash. Support then reboot the server instances and off it goes. I've tried to spend a few days just watching, but of course it didn't skip a beat while I was monitoring. Is there a way to trigger gather relevant data when Tomcat crashes? Z. On 1/02/13 12:36 PM, Zoran Avtarovski zo...@sparecreative.com wrote: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On Thu, Jan 31, 2013 at 10:32 PM, Zoran Avtarovski zo...@sparecreative.com wrote: 1.The app has been running on Tomcat 7 in a virtualised linux environment for about 4 months and the issue has been persistent through that time. Prior to that it was running on a physical Windows server with no issues. The move to linux coincided with the addition of a substantial amount of WebServices functionality - consuming. very interesting, stable on windows (without webservices footprint). what was your java -version details on windows, and what is your java -version details on linux? so the webservices functionality is 'new' software/code? was the code implemented well for GC? any use of threadlocals? i am only shooting from the hip, here, as I primarily listen in on topics here, and have limited experience with java and tomcat. 2.The tomcat instance was originally running with a max of 6Gb of allocated memory and it would crash almost daily. When we upped it to 11gb the crashes reduced to about twice a week. 3. Tomcat is configured with the following options export JAVA_OPTS=-Xms256m -Xmx11460m -Djava.awt.headless=true -XX:MaxPermSize=256M -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC 4. Xms and MaxPermSize were just reduced from 512m 5. The app generally has about 50-100 concurrent users - the logs show that Tomcat can crash with as few as 20 users. I'm wondering if it could be GC related as the crashes often follow periods of heavy use. 6. The app is a struts2 based app using Spring 3.5 for dependancy injection/ioc. this is good to know. have you seen/heard of similar reports from others using struts2 and spring 3.5 for dependancy injection/ioc? The problem with getting a thread dump is that high CPU usage is only a spike and server freezes. CPU usage only goes above 5-10% when it's about to crash. Support then reboot the server instances and off it goes. I've tried to spend a few days just watching, but of course it didn't skip a beat while I was monitoring. Is there a way to trigger gather relevant data when Tomcat crashes? Z. I searched tomcat user list, please see below, and consider what was mentioned there for troubleshooting this issue, and report back here. http://tomcat.10.n6.nabble.com/Tomcat-suddenly-dies-td4518543.html - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Current Java Version is 1.6.0_31-b04 Sun MicroSystems The old Windows version was the Sun 1.6 as well but I can't remember which specific release. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Help in diagnosing server unresponsiveness
From: Zoran Avtarovski [mailto:zo...@sparecreative.com] Subject: Re: Help in diagnosing server unresponsiveness I'm wondering if it could be GC related as the crashes often follow periods of heavy use. Is there a way to trigger gather relevant data when Tomcat crashes? Typically it's too late at that point. You might want to try these command line options to see what's happening to the heap (assuming that heap exhaustion is a factor): -Xloggc:filename -XX:+PrintGCDetails You can also use these to capture the heap if and when an OOME occurs: -XX:HeapDumpPath=filename -XX:+HeapDumpOnOutOfMemoryError - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
