ading checkpoints from Kafka:
> > >> >
> > >> > 2017-09-20 03:21:02.060 INFO o.a.s.c.kafka.KafkaCheckpointManager
> > >> [main]
> > >> > -
> > >> > Got offset 0 for topic __samza_checkpoint_ver_1_for_test-job_1 and
> > >> > partition 0. Attempting to fetch messages for checkpoint log.
> > >> > 2017-09-20 03:21:02.072 INFO o.a.s.c.kafka.KafkaCheckpointManager
> > >> [main]
> > >> > -
> > >> > Get latest offset 42890599 for topic
> > >> > __samza_checkpoint_ver_1_for_test-job_1 and partition 0.
> > >> >
> > >> > Looking at this line in KafkaCheckpointManager
> > >> > <https://github.com/apache/samza/blob/0.9.1/samza-kafka/
> > >> > src/main/scala/org/apache/samza/checkpoint/kafka/
> > >> > KafkaCheckpointManager.scala#L275>,
> > >> > it seems to indicate that the loop iterates from 0 to 42890599 and
> > make
> > >> > requests for each.
> > >> >
> > >> > Questions:
> > >> > 1. What does that loop do exactly?
> > >> > 2. Is this an expected behaviour? Is "Got offset 0 for topic ..."
> > >> normal?
> > >> > 3. Any ideas on how to fix this?
> > >> >
> > >> > Thanks,
> > >> > Xiaochuan Yu
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Jagadish V,
> > >> Graduate Student,
> > >> Department of Computer Science,
> > >> Stanford University
> > >>
> > >
> >
>
--
All the best
Liu Bo
ModelManager$.apply(
> JobCoordinator.scala:109)
> at org.apache.samza.coordinator.JobModelManager$.apply(
> JobCoordinator.scala:115)
> at org.apache.samza.job.local.ThreadJobFactory.getJob(
> ThreadJobFactory.scala:40)
> at org.apache.samza.job.JobRunner.run(JobRunner.scala:125)
> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:65)
> at org.apache.samza.job.JobRunner.main(JobRunner.scala)
>
>
--
All the best
Liu Bo
>
> You can see the log “Begin to close files” are not printed out and of
> course the logic is not executed.
>
> 2. The hadoop cluster I use is “HDP-2.5.0”,the log aggregation is also
> enabled, but logs of containers can not be collected, only the log of am
> can be seen.
>
the close and stop method in
> > BaseMessageChooser and SystemProducer will not be called and the
> container
> > log will be removed automatically, how can resolve this?
> >
> > Thanks.
> >
> >
> > ShuQi
>
--
All the best
Liu Bo
23:03:50.289 [main] SamzaContainer [INFO] Shutting down consumer
multiplexer.
--
All the best
Liu Bo
n the email list, I guess I have to wait for a
while.
On 10 May 2016 at 01:24, Yi Pan <nickpa...@gmail.com> wrote:
> Hi, Bo,
>
> I embedded my answers in-between:
>
> On Sun, May 8, 2016 at 9:00 PM, Liu Bo <diabl...@gmail.com> wrote:
>
> > The other thing i
, Jagadish Venkatraman <jagadish1...@gmail.com> wrote:
> Hi,
>
> Is your checkpoint topic log compacted? That may help in reducing the size
> of the log.
>
> On Sat, May 7, 2016 at 2:35 AM, Liu Bo <diabl...@gmail.com> wrote:
>
> > Hi group
> >
> > I've go
of my appreciation.
BTW: I think the checkpoint manager could start from the end of the
checkpoint, and try to "look backward" for the offset for current partition
instead of reading all of them.
--
All the best
Liu Bo
; On Tuesday, March 15, 2016, David Yu <david...@optimizely.com> wrote:
> >
> > > Our samza job is consuming from a Kafka topic. AFAIU, samza will auto
> > > assign the job a consumer group id and client id. However, I'm not able
> > to
> > > see that showing up under zookeeper. Am I missing something?
> > >
> >
>
--
All the best
Liu Bo
jobs, we
> really don't have a "recommended" GC for the users. It would probably also
> depend on the application's usage pattern as well. Our perf partner Tao
> Feng @LinkedIn may have some more insights.
>
> @Tao, do you have any comments on this?
>
> -Yi
>
> On Sun, Jan
:
systems.kafka.streams.metrics.samza.msg.serde=metrics
ref: http://samza.apache.org/learn/documentation/0.9/container/metrics.html
Metrics are written to kafka correctly after adding this to the metrics
config.
Hope it's correct for 0.10.
--
All the best
Liu Bo
11 matches
Mail list logo
