Hi, Jae, That's correct. I mentioned that just to confirm that checkpoint topic should be log-compacted. Then, the next question is: what's the size of the data in the checkpoint topic and how many input topic partitions you have in the job?
It would also be helpful if you can share which version of samza you are using. Thanks! -Yi On Tue, Nov 3, 2015 at 11:03 PM, Bae, Jae Hyeon <metac...@gmail.com> wrote: > Hi Yi > > My colleague found that samza automatically set log compaction when > creating the checkpointing topic. > > Topic:__samza_checkpoint_ver_1_for_xxx_1 PartitionCount:1 > ReplicationFactor:3 Configs:segment.bytes=26214400,cleanup.policy=compact > > Topic: __samza_checkpoint_ver_1_for_xxx_1 Partition: 0 Leader: 66 Replicas: > 66,24,65 Isr: 24,65,66 > > So, the problem is not log-compaction. > > > > On Tue, Nov 3, 2015 at 8:33 PM, Yi Pan <nickpa...@gmail.com> wrote: > > > Hi, Bae, > > > > Where did you see this log? Is it in JobRunner? Or AppMaster? Or > > SamzaContainer? > > > > There are a few factors that may have the impact: > > 1. How many system stream partitions you have as the input? And how many > > tasks are there? > > 2. Did you set your checkpoint topic as log-compact topic in Kafka? The > > topic size would be much smaller if log compaction is turned on. > > > > Regards > > > > -Yi > > > > On Tue, Nov 3, 2015 at 3:59 PM, Bae, Jae Hyeon <metac...@gmail.com> > wrote: > > > > > Hi Samza Dev > > > > > > Do you know why the following job is taking too long? > > > > > > 2015-11-03 23:58:17 KafkaCheckpointManager [INFO] Get latest offset > > 3386930 > > > for topic __samza_checkpoint_ver_1_for_xxx_1 and partition 0. > > > > > > This is seriously slowing down development. How can I fix this problem? > > > > > > Thank you > > > Best, Jae > > > > > >
