Hi,
Yes. I did try increasing to the value of number of cores. It did not
work as expected. I know System.gc is not a proper way. But, since this is
a batch application, I'm okay if it spends more time in doing GC and takes
considerably more cpu. Frequent system.gc calls with lower xmx (6 GB) gave
us more throughout as compared to no manual gc calls with higher xmx (8
GB). If I don't do manual gc calls when xmx is 6 GB the executor would go
oom. I was just wondering what Could be the best tuning parameters that
would give similar effect.
Thanks,
Akshay
On Tue, 25 Dec 2018, 11:28 pm Ramandeep Singh Nanda <ramannan...@gmail.com
wrote:
> Hi,
>
> Did you try increasing concurrentgcthreads for the marking?
>
> System.gc is not a good way to handle this, as it is not guaranteed and is
> a high pause,full gc.
>
> Regards,
> Ramandeep Singh
>
> On Tue, Dec 25, 2018, 07:09 Akshay Mendole <akshaymend...@gmail.com wrote:
>
>> Hi,
>>
>> My JVM is basically a spark executor which is running tasks one after
>> another. A task is memory hungry and requires significant memory during its
>> lifecycle.
>>
>> [image: Screen Shot 2018-12-25 at 3.48.51 PM.png]
>> The above JVM is running on G1GC with default params. As you can see in
>> the VisualVM report on the right hand side between 4:25 to 4:32 PM, the
>> spikes are due to each tasks run by the executor (essentially, each spike
>> is due to the executor picking up new task after the previous one is
>> finished). When I triggered a manual GC at 4:35, I saw a sharp decline the
>> heap usage. Also, as you can see on the left hand side in the JConsole
>> report, the old gen space is never collected by G1GC (the sharp decline in
>> old gen space just before 16:35 is due to the manual GC).
>>
>> As my application is a spark batch job application, I am ok if the JVMs
>> spent good amount of time doing GC. But, I am running bit short on memory.
>> So, I wanted to know how I can tune my JVM G1GC params so that there are
>> more frequent GC (with the old gen space also getting collected) and I can
>> get the work done with considerably lesses heap space (XMX).
>>
>> I tried changing -XX:InitiatingHeapOccupancyPercent . I tried with 0, 5,
>> 10. Lower the value, more frequent was the GC and more CPU it would
>> consume, but the behaviour would not be consistent. After sometime, the
>> heap usage would spike up and if we have set the XMX value lesser (6 GB)
>> than what's there in the picture above, it would throw OOM.
>>
>> *Something that is really working for us is attaching a daemon thread
>> that's calling System.gc() after an interval of 1 minute in each executor.*
>> But, I wanted to know how we can gracefully tune our GC for the same that
>> gives the same effect as the manual System.GC() calls.
>>
>> Thanks,
>> Akshay
>>
>>
