What is the max heap size configured? Does the gc log show any major pauses around that time?
On Thu, Mar 10, 2016 at 7:42 AM, Tommy Becker <tobec...@tivo.com> wrote: > We had one of our Samza jobs restart overnight recently and noticed that > restoration from the changelog took much longer than I would expect (well > over an hour). Looking through the logs, the throughput initially seems > reasonable if not stellar. But nearly every container seems to encounter > one or more long pauses during the restoration process: > > 2016-03-09 03:50:09,940 (default) [main] INFO > [org.apache.samza.storage.kv.KeyValueStorageEngine] - 56000000 entries > restored... > 2016-03-09 03:50:19,895 (default) [main] INFO > [org.apache.samza.storage.kv.KeyValueStorageEngine] - 57000000 entries > restored... > 2016-03-09 03:51:41,310 (default) [main] INFO > [org.apache.samza.storage.kv.KeyValueStorageEngine] - 58000000 entries > restored... > 2016-03-09 04:22:13,003 (default) [main] INFO > [org.apache.samza.storage.kv.KeyValueStorageEngine] - 59000000 entries > restored... > > > > Here we see a nearly 30 minute span with no logs. So far as we can tell, > Kafka is healthy during this period and other containers are making > progress restoring their partitions around this time, so the "gaps" are not > happening at the same time across containers. We are running Samza 0.9.1 on > a YARN cluster in AWS so some variance in performance is to be expected, > but this seems pretty extreme. Is anyone else seeing this behavior? > > -- > Tommy Becker > Senior Software Engineer > > Digitalsmiths > A TiVo Company > > www.digitalsmiths.com<http://www.digitalsmiths.com> > tobec...@tivo.com<mailto:tobec...@tivo.com> > > ________________________________ > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, copying, > or distribution of this email (or any attachments) by others is prohibited. > If you are not the intended recipient, please contact the sender > immediately and permanently delete this email and any attachments. No > employee or agent of TiVo Inc. is authorized to conclude any binding > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > Inc. may only be made by a signed written agreement. >
