Re: How to verify if checkpoints are asynchronous or sync

I evaluated performance by looking at the number of input records processed over 10min and 30mins duration. Thanks, R On Wed, Jan 8, 2020 at 2:21 AM Congxian Qiu wrote: > If you want to figure out the performance problem, maybe async-profile[1] > can be helpful > [1] https://github.com/jvm-prof

Re: How to verify if checkpoints are asynchronous or sync

If you want to figure out the performance problem, maybe async-profile[1] can be helpful [1] https://github.com/jvm-profiling-tools/async-profiler Best, Congxian William C 于2020年1月8日周三 上午11:37写道： > Hallo > > on 2020/1/8 11:31, RKandoji wrote: > > I'm running my job on a EC2 instance with 32 cor

Re: How to verify if checkpoints are asynchronous or sync

Hallo on 2020/1/8 11:31, RKandoji wrote: I'm running my job on a EC2 instance with 32 cores and according to the documentation I tried to use as many task slots the number of cores, numOfTaskSlots=32 and parallelism=32. But I noticed that the performance is slightly degrading when I'm using 32

Re: How to verify if checkpoints are asynchronous or sync

Sorry for not providing the context earlier. I'm running my job on a EC2 instance with 32 cores and according to the documentation I tried to use as many task slots the number of cores, numOfTaskSlots=32 and parallelism=32. But I noticed that the performance is slightly degrading when I'm using 32

Re: How to verify if checkpoints are asynchronous or sync

Hi RocksDB supports Incremental and full snapshot, both are async, do you want to verify whether it's incremental or full snapshot? but I don't know an easy to get this information currently Best, Congxian Zhijiang 于2020年1月8日周三 上午10:56写道： > The log way is simple for tracing and you can also g

Re: How to verify if checkpoints are asynchronous or sync

The log way is simple for tracing and you can also grep some keywords to find your requirement messages to avoid skimming through the whole large logs. I am not quite sure what's your specific motivation for doing this. Besides the log way, you can also monitor the thread stack for confirming whe

Re: How to verify if checkpoints are asynchronous or sync

Thanks for the reply. I will check and enable debug logs specifically for the class that contains this log. But in general logs are already too huge and I'm trying to suppress some of them, so wondering if there is any other way? Thanks, RKandoji On Tue, Jan 7, 2020 at 7:50 PM William C wrote:

Re: How to verify if checkpoints are asynchronous or sync

Can you enable debug log to check with that? regards. on 2020/1/8 6:36, RKandoji wrote: But I'm curious if there is way to verify if the checkpoints are happening asynchronously or synchronously.