Hi Bharath,
Thank you for the detailed explanation. My code does not use samza
directly. We use Beam's Samza Runner. And I use Beam's KafkaIO. When I
check the internal of Beam KafkaIO they use kafka consumer client and they
use Assign mode. So even though Kafka does not assign partitions, Beam
Hi Talat,
It is expected behavior since Samza uses low level Kafka consumer instead
of the high level consumer. As a result, offset management is done by Samza
and doesn't leverage the offset management that Kafka consumer has by
default.
Additionally, kafka consumer group doesn't apply to samza
Thank you rayman. But my question is when i check kafka consumer group of
the job. I dont see any offset movement. I chose to store checkpoints on
file system. Do you think because of that i dont see my job's consumer
group does not move offset ?
On Tue, Aug 10, 2021, 9:32 PM rayman preet
Hi Talat,
Since in the job.properties the task.checkpoint.factory is set to
FileSystemCheckpointManagerFactory
and not org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory.
That is why its writing checkpoints to the filesystem (with its. location
controlled by task.checkpoint.path).
