M. Manna created KAFKA-8759:
-------------------------------
Summary: Message Order is reversed when client run behind a VPN
Key: KAFKA-8759
URL: https://issues.apache.org/jira/browse/KAFKA-8759
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 2.2.0
Reporter: M. Manna
We have noticed this behaviour whilst testing console producer against a kafka
service installed on GCP. We have been using a fork from confluent Helm Chart.
[https://github.com/helm/charts/tree/master/incubator/kafka]
FYI - we've used cp 5.3.0 with Apache Kafka 2.3.0
Our VPN connection throughput was 1 mbps. Upon connecting to VPN, we opened a
console producer client (2.2.0) with the following command:
{code:java}
kafka-console-producer.bat --topic some_topic --broker-list
gcp_broker1:19092,gcp_broker2:19092,gcp_broker3:19092{code}
Similarly, we ran a consumer with the following command before publishing
messages
{code:java}
kafka-console-consumer.bat --topic some_topic --bootstrap-server
gcp_broker1:19092,gcp_broker2:19092,gcp_broker3:19092{code}
For producer console, we did receive a carat (>) prompt for publishing, so we
entered messages:
{code:java}
>one
>two
>three
>{code}
After a while, it responded with NETWORK_EXCEPTION
{code:java}
[2019-08-02 11:17:19,690] WARN [Producer clientId=console-producer] Got error
produce response with correlation id 8 on topic-partition some_topic-0,
retrying (2 attempts left). Error: NETWORK_EXCEPTION
(org.apache.kafka.clients.producer.internals.Sender){code}
We then hit "Enter" and received a carat (>) back
{code:java}
[2019-08-02 11:17:19,690] WARN [Producer clientId=console-producer] Got error
produce response with correlation id 8 on topic-partition some_topic-0,
retrying (2 attempts left). Error: NETWORK_EXCEPTION
(org.apache.kafka.clients.producer.internals.Sender)
>{code}
Immediately after that, on consumer window, we received the following:
{code:java}
three
two
one{code}
We ran the same exercise from a regular network (wifi/lan) and didn't see this
issue (i.e. works as described on Quickstart).
This is slightly concerning for us since tunneling into a VPN shouldn't have
any impact (or, should it) how kafka message protocol works over tcp. It seems
that Kafka couldn't guarantee order of messages when network latency is
involved.
FYI
1) We tried on VPN with --request_timeout_ms 120000 and still same results.
2) Our setup was 3 node (3 br, 3 zk) with every topic having 1 partition only
(RF - 3).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
