Also one more thing we are facing issues when the request is not going to jetty server, there is no entry in access-log and application keeps loading.
Any pointers would be helpful On Fri, Jan 12, 2018 at 3:45 PM, kapil gupta <[email protected]> wrote: > Sure Greg, > > I will check out for the points suggested. But one thing, why my VisualVM > doesn show heap size hike, it hardly consumes 50-60% of allocated memory. > > On Fri, Jan 12, 2018 at 3:34 PM, Greg Wilkins <[email protected]> wrote: > >> Kapil, >> >> if your server is running out of memory, then it is most likely the >> application rather than jetty itself. >> I would advise: >> + increasing the memory allocated to your production JVMs >> + running a test system with realistic load and profiling the >> applications memory usage both over time (for leaks) or just for memory >> hungry operations. >> >> regards >> >> >> On 12 January 2018 at 05:38, kapil gupta <[email protected]> wrote: >> >>> Thanks everyone for your replies. >>> >>> Some more points into it: >>> >>> 1. @Silvio - As you mentioned, yes sometimes OOM killer invokes >>> jetty process to stop, but many of times it doesnt. >>> 2. @Greg - We are using cloud, and running on CentOS 7. We have >>> stopped updates due to some reasons. >>> 3. @Joakim - Yes, checked the thread dump and some times OOM killer >>> invokes jetty crash, sometimes I see below properties in dump file: >>> 1. >>> 2. Current thread (0x00007f18e01b9800): VMThread [stack: >>> 0x00007f18d33f4000,0x00007f18d34f5000] [id=11584] >>> 3. >>> 4. Stack: [0x00007f18d33f4000,0x00007f18d34f5000], >>> sp=0x00007f18d34f3140, free space=1020k >>> 5. Native frames: (J=compiled Java code, j=interpreted, Vv=VM >>> code, C=native code) >>> 6. V [libjvm.so+0x9a320a] VMError::report_and_die()+0x2ea >>> 7. V [libjvm.so+0x498d3b] report_vm_out_of_memory(char const*, >>> int, unsigned long, char const*)+0x9b >>> 8. V [libjvm.so+0x82191e] os::Linux::commit_memory_impl(char*, >>> unsigned long, bool)+0xfe >>> 9. V [libjvm.so+0x821e69] os::pd_commit_memory(char*, unsigned >>> long, unsigned long, bool)+0x29 >>> 10. V [libjvm.so+0x81bb6a] os::commit_memory(char*, unsigned >>> long, unsigned long, bool)+0x2a >>> 11. V [libjvm.so+0x88d623] PSVirtualSpace::expand_by(unsigned >>> long)+0x53 >>> 12. V [libjvm.so+0x88e9f8] PSYoungGen::resize_generation(unsigned >>> long, unsigned long)+0xf8 >>> 13. V [libjvm.so+0x88db62] PSYoungGen::resize(unsigned long, >>> unsigned long)+0x22 >>> 14. V [libjvm.so+0x88af1d] PSScavenge::invoke_no_policy()+0xf3d >>> 15. V [libjvm.so+0x88b761] PSScavenge::invoke()+0x41 >>> 16. V [libjvm.so+0x843f40] >>> ParallelScavengeHeap::failed_mem_allocate(unsigned >>> long)+0x70 >>> 17. V [libjvm.so+0x9a4a97] VM_ParallelGCFailedAllocation >>> ::doit()+0x97 >>> 18. V [libjvm.so+0x9abf35] VM_Operation::evaluate()+0x55 >>> 19. V [libjvm.so+0x9aa2fa] VMThread::evaluate_operation( >>> VM_Operation*)+0xba >>> 20. V [libjvm.so+0x9aa67e] VMThread::loop()+0x1ce >>> 21. V [libjvm.so+0x9aaaf0] VMThread::run()+0x70 >>> 22. V [libjvm.so+0x8238c8] java_start(Thread*)+0x108 >>> >>> I do not have much of knowledge in to these configuration, need you guys >>> help. >>> Is this because of my server goes OutOfMemory, if yes then why it doesnt >>> show the memory occupied more in VisualVM when we check heap size >>> >>> >>> On Thu, Jan 11, 2018 at 3:00 PM, Silvio Bierman < >>> [email protected]> wrote: >>> >>>> Kapil, >>>> >>>> If you are running on Linux check the system log for kernel panics due >>>> to lack of memory. Linux will ruthlessly kill the process with highest >>>> memory load in such a situation. That has happened to me plenty of times >>>> and also left me wondering where my JVM process had gone. >>>> >>>> And to add to what Greg said: the System.exit thing can also happen in >>>> a library. I used a library once that would do a System.exit in some error >>>> situations killing my server now and then. That was a real puzzler. >>>> >>>> Cheers, >>>> >>>> Silvio >>>> >>>> >>>> >>>> On 10-01-18 12:50, kapil gupta wrote: >>>> >>>> Greg, >>>> >>>> Actually below things are observed: >>>> >>>> 1. No process of jetty server running >>>> 2. MySQL still runs >>>> 3. CPU usage shows data fine >>>> 4. RAM is also fine >>>> >>>> Not sure how can we isolate if issue could be because of JVM, can you >>>> please guide on setting some parameters or moniotring them. >>>> >>>> Thanks >>>> >>>> On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <[email protected]> >>>> wrote: >>>> >>>>> >>>>> Kapil, >>>>> >>>>> I'm sorry but you've not given us enough information to say anything. >>>>> What do you mean by "crashed"? If it is the JVM that stops, then that is >>>>> not a Jetty problem. Is there a stack trace? Is there any logging? Is >>>>> the >>>>> process still running? Can any requests be served? Is the connector still >>>>> listening? Can you run with debug? Why do you think it is jetty and not >>>>> some other component? >>>>> >>>>> regards >>>>> >>>>> >>>>> >>>>> On 10 January 2018 at 05:37, kapil gupta <[email protected]> >>>>> wrote: >>>>> >>>>>> We are using jetty server and when it is on load then the server is >>>>>> crashed without showing much of information. >>>>>> We have also put it on JMX. The CPU usage and memory looks fine, but >>>>>> still Jetty server crashes. Please let me know what parameters we should >>>>>> look for and how can we isolate issue which is causing jetty server >>>>>> crash. >>>>>> >>>>>> _______________________________________________ >>>>>> jetty-users mailing list >>>>>> [email protected] >>>>>> To change your delivery options, retrieve your password, or >>>>>> unsubscribe from this list, visit >>>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Greg Wilkins <[email protected]> CTO http://webtide.com >>>>> >>>>> _______________________________________________ >>>>> jetty-users mailing list >>>>> [email protected] >>>>> To change your delivery options, retrieve your password, or >>>>> unsubscribe from this list, visit >>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> jetty-users mailing [email protected] >>>> To change your delivery options, retrieve your password, or unsubscribe >>>> from this list, visithttps://dev.eclipse.org/mailman/listinfo/jetty-users >>>> >>>> >>>> >>>> _______________________________________________ >>>> jetty-users mailing list >>>> [email protected] >>>> To change your delivery options, retrieve your password, or unsubscribe >>>> from this list, visit >>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>> >>> >>> >>> _______________________________________________ >>> jetty-users mailing list >>> [email protected] >>> To change your delivery options, retrieve your password, or unsubscribe >>> from this list, visit >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>> >> >> >> >> -- >> Greg Wilkins <[email protected]> CTO http://webtide.com >> >> _______________________________________________ >> jetty-users mailing list >> [email protected] >> To change your delivery options, retrieve your password, or unsubscribe >> from this list, visit >> https://dev.eclipse.org/mailman/listinfo/jetty-users >> > >
_______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
