[
https://issues.apache.org/jira/browse/KAFKA-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905510#comment-16905510
]
Raman Gupta edited comment on KAFKA-8789 at 8/12/19 7:49 PM:
-------------------------------------------------------------
I'm reopening this. The problem does not appear to be with the SR tooling at
all, but rather that the console consumer for the Kafka version included in
Confluent 5.3.0 is a lot slower than the console consumer in 5.0.3. Using a
timeout of 15s is consistently enough to read all messages on the topic in
5.0.3 but has to be at least 60s in 5.3.0, against the same brokers and with
the same parameters.
Also interesting is that the total time for the command to run is pretty much
the same:
{code:java}
confluent-5.0.3 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
2.60user 0.22system 0:32.15elapsed 8%CPU (0avgtext+0avgdata 145764maxresident)k
0inputs+0outputs (0major+33989minor)pagefaults 0swaps
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
3.09user 0.28system 0:32.43elapsed 10%CPU (0avgtext+0avgdata 176440maxresident)k
0inputs+0outputs (0major+40773minor)pagefaults 0swaps
confluent-5.0.3 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[...]
Processed a total of 1 messages
2.58user 0.24system 0:32.29elapsed 8%CPU (0avgtext+0avgdata 144780maxresident)k
0inputs+0outputs (0major+33562minor)pagefaults 0swaps
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages
2.09user 0.17system 0:31.47elapsed 7%CPU (0avgtext+0avgdata 149300maxresident)k
0inputs+8outputs (0major+33949minor)pagefaults 0swaps
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 45000
Processed a total of 1 messages
3.12user 0.27system 0:32.55elapsed 10%CPU (0avgtext+0avgdata 178252maxresident)k
0inputs+0outputs (0major+41263minor)pagefaults 0swaps
{code}
so perhaps the behavior of the `--timeout-ms` parameter has changed?
As an aside, the only reason I need the timeout here is because this command is
part of a unix pipeline, and I need it to exit when there are no more messages
to read. Unfortunately, there doesn't appear to be any way to do that except to
set a timeout.
was (Author: rocketraman):
I'm reopening this. The problem does not appear to be with the SR tooling at
all, but rather that the console consumer for the Kafka version included in
Confluent 5.3.0 is a lot slower than the console consumer in 5.0.3. Using a
timeout of 15s is consistently enough to read all messages on the topic in
5.0.3 but has to be at least 60s in 5.3.0, against the same brokers and with
the same parameters.
Also interesting is that the total time for the command to run is pretty much
the same:
{code:java}
confluent-5.0.3 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
2.60user 0.22system 0:32.15elapsed 8%CPU (0avgtext+0avgdata 145764maxresident)k
0inputs+0outputs (0major+33989minor)pagefaults 0swaps
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
3.09user 0.28system 0:32.43elapsed 10%CPU (0avgtext+0avgdata 176440maxresident)k
0inputs+0outputs (0major+40773minor)pagefaults 0swaps
confluent-5.0.3 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[...]
Processed a total of 1 messages
2.58user 0.24system 0:32.29elapsed 8%CPU (0avgtext+0avgdata 144780maxresident)k
0inputs+0outputs (0major+33562minor)pagefaults 0swaps
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages
2.09user 0.17system 0:31.47elapsed 7%CPU (0avgtext+0avgdata 149300maxresident)k
0inputs+8outputs (0major+33949minor)pagefaults 0swaps{code}
so perhaps the behavior of the `--timeout-ms` parameter has changed?
As an aside, the only reason I need the timeout here is because this command is
part of a unix pipeline, and I need it to exit when there are no more messages
to read. Unfortunately, there doesn't appear to be any way to do that except to
set a timeout.
> kafka-console-consumer needs bigger timeout-ms setting in order to work
> -----------------------------------------------------------------------
>
> Key: KAFKA-8789
> URL: https://issues.apache.org/jira/browse/KAFKA-8789
> Project: Kafka
> Issue Type: Bug
> Components: tools
> Affects Versions: 2.3.0
> Reporter: Raman Gupta
> Priority: Major
>
> I have a topic with about 20,000 events in it. When I run the following tools
> command using Kafka 2.
> bin/kafka-avro-console-consumer \
> --bootstrap-server $KAFKA --property schema.registry.url=$SCHEMAREGISTRY \
> --topic $TOPICPREFIX-user-clickstream-events-ui-v2 \
> --from-beginning --max-messages 100 \
> --isolation-level read_committed --skip-message-on-error \
> --timeout-ms 15000
> I get 100 messages as expected.
> However, when running the exact same command using Kafka 2.3.0 I get
> org.apache.kafka.common.errors.TimeoutException, and 0 messages processed.
> The version of Kafka on the server is 2.3.0.
> NOTE: I am using the Confluent distribution of Kafka for the client side
> tools, specifically Confluent 5.0.3 and Confluent 5.3.0. I can certainly try
> to replicate with a vanilla Kafka if necessary.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
