I tried this numerous times (regardless of PLAINTEXT/SSL connnections). The
setup is to do with a single node (1 Zeek, 1 Broker) as mentioned in the
startup docs on Kafka site.

1) Follow kafka documentation to initialise all Zk and Kafka props.
2) Enable automatic topic creation with a minimum partition 1 in broker
3) Start Zk and Kafka.
4) Use the standard shipped producer.properties file with
bootstrap.servers, key.serializer, and value.serializer properties values.
5) Use the standard shipped consumer.properties file with the following

#consumer group id

Since we are using console consumer, I kept the enable.auto.commit=true (i
think every 5 secs is the default interval for committing auto)

4) Run following command:

kafka-console-producer.bat --topic test --broker-list localhost:9092
--producer.config ..\..\config\producer.properties
> ^ C (keyboard close producer)

5) Run the following command
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test
--new-consumer --consumer.config ..\..\config\consumer.properties
(wait for 5 seconds or more)
^C (keyboard close consumer)

6) Now repeat #4 with some more messages and close producer
7) Repeat #5.- confirm that all messages (old + new ones) are being

>From the help documentation for console-consumer.

> --from-beginning      If the conumer doesn not already have an established
> offset to consume from, start from the earliest message present in the log
> rather than the latest message.

The consumer group is present so any offset committed correctly should have
been used by a newly added consumer to retrieve offset details.
`auto.offset.reset=earliest` but that's not relevant since offsets have
been committed and should have been found.

When I did the following:
kafka-consumer-groups.bat --bootstrap.server localhost:9092 --describe
--group group1 --new-consumer

I see "Error group1 does not exist".

I also tried to use kafka-consumer-offset-checker - but ran into the error
" org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /consumers/group1/offsets/test/0"

Which seems to be a bug as mentioned by -

My question is - why does the console consumer fail to retrieve the offset
information and loses data? Or is this expected? I didn't have any consumer
offset expiry default configs altered. So any info. is helpful.

I don't believe this is relevant to Windows/Linux but if anyone has this
example (with the same setup) working in Linux ONLY - it's good to know. I
can confirm that I can see all the consumer offsets created in the specific
log directory in broker prop file.

Kindest Regards,

Reply via email to