On Thu, Dec 21, 2017 at 4:42 PM, Kirk Pepperdine <[email protected]> wrote:
> Hi, > > If GC logging is giving you grief, you’ve got bigger problems because GC > logs a very small amount of data. If this is an issue there will be a hint > of it in the GC logs. At any rate, the problem is with GC and turning off > GC logging leaves you blind to any possible solutions. > I was in no way suggesting turning off GC logging. When we encountered this problem, the solution was to fix the drives, not turn off GC logging. (I can't remember if our HWOPS folks just replaced the drives or if there was a firmware upgrade they could apply). Also, I mis-spoke when I suggested that GC logging uses sync I/O. But there are paths where an async write can still block in the kernel under heavy I/O or (in our case) there are problems with the I/O device. LinkedIn had a very detailed write-up: https://engineering.linkedin.com/blog/2016/02/ eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic > > Kind regards, > Kirk Pepperdine > > On Dec 21, 2017, at 9:40 PM, Josh Humphries <[email protected]> wrote: > > Like Erik said, check that the host isn't swapping. An old-gen GC can end > up scanning parts of the heap that the OS may have paged out if it > encountered memory pressure. > > Another issue I've seen that causes this sort of thing is if you have GC > logging turned on and encounter issues in I/O. GC logging is not async. > I've seen the GC logging itself, to a drive that had a SCSI firmware issue, > cause silly-long GC times. You can possibly corroborate by checking kernel > logs to see if there are other signals of disk/driver issues. > > > ---- > *Josh Humphries* > [email protected] > > On Thu, Dec 21, 2017 at 3:36 PM, Erik Svensson <[email protected]> wrote: > >> If it’s on a virtual host then I’ve seen the same behaviour when the >> underlying hw started swapping. I’ve also seen the same when the other >> hosts on the hw had high cpu demands (i.e. my host was cpu starved). >> But, as martin said, gc logging always. >> >> cheer >> >> Erik Svensson >> D >> >> On 21 Dec 2017, at 18:25, Abhishek Agarwal <[email protected]> wrote: >> >> Hello everyone, >> Recently, I had one of the JVMs going into this weird state wherein minor >> gc start kicking almost every minute but what is most surprising that it is >> one minor GC is taking as much as 100 seconds. The heap size is around 12 >> gigs and young gen is about 30% of that. As soon as gc starts, >> understandably CPU usage started increasing as well. I am not able to >> figure out how would one minor GC take this much time. The machine does not >> have swapping enabled. I also ran sar and confirmed there was no swapping >> during the issue. We are using parallel GC for young gen and CMS for old >> gen GC. How can I go about troubleshooting this further? Any pointers will >> be appreciated. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "mechanical-sympathy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "mechanical-sympathy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > You received this message because you are subscribed to the Google Groups > "mechanical-sympathy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "mechanical-sympathy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
