[jira] [Comment Edited] (KAFKA-6189) Loosing messages on OFFSET_OUT_OF_RANGE error in consumer
[
https://issues.apache.org/jira/browse/KAFKA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249419#comment-16249419
]
Xinyang Gao edited comment on KAFKA-6189 at 11/13/17 2:25 PM:
--
I had seen similar behaviors which also led to message loss, setup is as
follows:
topic has 3 partitions and replication factor 3, named gao31
min.insync.replicas=2
consumer has default "auto.offset.reset=latest"
consumer manually commitSync offsets after handling messages with
consumer.commitSync()
unclean leader election = false
kafka cluster has 3 brokers, kafka-foo-0, kafka-foo-1 and kafka-foo-2
consumer group ID is GROUPID
session.timeout.ms and max.poll.interval.ms are using default values
Initially kafka-foo-1 is selected as group coordinator
To reproduce:
1. kill kafka-foo-1
2. See following logs which implies that consumer disconnect to kafka-foo-1 and
try to discover a new group coordinator
{code:java}
2017-11-10 11:27:45,097 DEBUG org.apache.kafka.clients.NetworkClient
- Node 2147483646 disconnected. [kafka-consumer]
2017-11-10 11:27:45,097 INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the
coordinator kafka-foo-1.***.com:9092 (id: 2147483646 rack: null) dead for group
GROUPID [kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-1.***.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-2.***.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node kafka-foo-0.***.com:9092 (id: 0 rack:
null) [kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group GROUPID to broker kafka-foo-0.***.com:9092
(id: 0 rack: null) [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-1.***.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-2.***.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node kafka-foo-0.***.com:9092 (id: 0 rack:
null) [kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group GROUPID to broker kafka-foo-0.***.com:9092
(id: 0 rack: null) [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-1.***.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-2.***.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node kafka-foo-0.***.com:9092 (id: 0 rack:
null) [kafka-consumer]
2017-11-10 11:27:45,099 DEBUG org.apache.kafka.clients.NetworkClient
- Sending metadata request (type=MetadataRequest, topics=gao31) to
node 0 [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Sending METADATA {topics=[gao31],allow_auto_topic_creation=true}
to node 0. [kafka-consumer]
2017-11-10 11:27:45,100 TRACE org.apache.kafka.clients.NetworkClient
- Sending FIND_COORDINATOR
{coordinator_key=GROUPID,coordinator_type=0} to node 0. [kafka-consumer]
2017-11-10 11:27:45,276 TRACE org.apache.kafka.clients.NetworkClient
- Completed receive from node 2, for key 1, received
{throttle_time_ms=0,responses=[{topic=gao31,partition_responses=[{partition_header={partition=2,error_code=0,high_watermark=1631,last_stable_offset=-1,log_start_offset=0,aborted_transactions=null},record_set=[(record=DefaultRecord(offset=1630,
timestamp=1510313264997, key=1 bytes, value=102
[jira] [Comment Edited] (KAFKA-6189) Loosing messages on OFFSET_OUT_OF_RANGE error in consumer
[
https://issues.apache.org/jira/browse/KAFKA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249419#comment-16249419
]
Xinyang Gao edited comment on KAFKA-6189 at 11/13/17 2:23 PM:
--
I had seen similar behaviors which also led to message loss, setup is as
follows:
topic has 3 partitions and replication factor 3, named gao31
min.insync.replicas=2
consumer has default "auto.offset.reset=latest"
consumer manually commitSync offsets after handling messages with
consumer.commitSync()
unclean leader election = false
kafka cluster has 3 brokers, kafka-foo-0, kafka-foo-1 and kafka-foo-2
consumer group ID is uklonvd826214
session.timeout.ms and max.poll.interval.ms are using default values
Initially kafka-foo-1 is selected as group coordinator
To reproduce:
1. kill kafka-foo-1
2. See following logs which implies that consumer disconnect to kafka-foo-1 and
try to discover a new group coordinator
{code:java}
2017-11-10 11:27:45,097 DEBUG org.apache.kafka.clients.NetworkClient
- Node 2147483646 disconnected. [kafka-consumer]
2017-11-10 11:27:45,097 INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the
coordinator kafka-foo-1.***.com:9092 (id: 2147483646 rack: null) dead for group
uklonvd826214 [kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-1.***.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-2.***.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node kafka-foo-0.***.com:9092 (id: 0 rack:
null) [kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.***.com:9092 (id: 0 rack: null) [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-1.***.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-2.***.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node kafka-foo-0.***.com:9092 (id: 0 rack:
null) [kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.***.com:9092 (id: 0 rack: null) [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-1.***.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-2.***.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node kafka-foo-0.***.com:9092 (id: 0 rack:
null) [kafka-consumer]
2017-11-10 11:27:45,099 DEBUG org.apache.kafka.clients.NetworkClient
- Sending metadata request (type=MetadataRequest, topics=gao31) to
node 0 [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Sending METADATA {topics=[gao31],allow_auto_topic_creation=true}
to node 0. [kafka-consumer]
2017-11-10 11:27:45,100 TRACE org.apache.kafka.clients.NetworkClient
- Sending FIND_COORDINATOR
{coordinator_key=uklonvd826214,coordinator_type=0} to node 0. [kafka-consumer]
2017-11-10 11:27:45,276 TRACE org.apache.kafka.clients.NetworkClient
- Completed receive from node 2, for key 1, received
{throttle_time_ms=0,responses=[{topic=gao31,partition_responses=[{partition_header={partition=2,error_code=0,high_watermark=1631,last_stable_offset=-1,log_start_offset=0,aborted_transactions=null},record_set=[(record=DefaultRecord(offset=1630,
timestamp=1510313264997, key=1 bytes, value=102
[jira] [Comment Edited] (KAFKA-6189) Loosing messages on OFFSET_OUT_OF_RANGE error in consumer
[
https://issues.apache.org/jira/browse/KAFKA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249419#comment-16249419
]
Xinyang Gao edited comment on KAFKA-6189 at 11/13/17 2:23 PM:
--
I had seen similar behaviors which also led to message loss, setup is as
follows:
topic has 3 partitions and replication factor 3, named gao31
min.insync.replicas=2
consumer has default "auto.offset.reset=latest"
consumer manually commitSync offsets after handling messages with
consumer.commitSync()
unclean leader election = false
kafka cluster has 3 brokers, kafka-foo-0, kafka-foo-1 and kafka-foo-2
consumer group ID is uklonvd826214
session.timeout.ms and max.poll.interval.ms are using default values
Initially kafka-foo-1 is selected as group coordinator
To reproduce:
1. kill kafka-foo-1
2. See following logs which implies that consumer disconnect to kafka-foo-1 and
try to discover a new group coordinator
{code:java}
2017-11-10 11:27:45,097 DEBUG org.apache.kafka.clients.NetworkClient
- Node 2147483646 disconnected. [kafka-consumer]
2017-11-10 11:27:45,097 INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the
coordinator kafka-foo-1.***.com:9092 (id: 2147483646 rack: null) dead for group
uklonvd826214 [kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-1.***.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-2.***.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node kafka-foo-0.***.com:9092 (id: 0 rack:
null) [kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.***.com:9092 (id: 0 rack: null) [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-1.***.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-2.***.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node kafka-foo-0.***.com:9092 (id: 0 rack:
null) [kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.***.com:9092 (id: 0 rack: null) [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-1.***.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node kafka-foo-2.***.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node kafka-foo-0.***.com:9092 (id: 0 rack:
null) [kafka-consumer]
2017-11-10 11:27:45,099 DEBUG org.apache.kafka.clients.NetworkClient
- Sending metadata request (type=MetadataRequest, topics=gao31) to
node 0 [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Sending METADATA {topics=[gao31],allow_auto_topic_creation=true}
to node 0. [kafka-consumer]
2017-11-10 11:27:45,100 TRACE org.apache.kafka.clients.NetworkClient
- Sending FIND_COORDINATOR
{coordinator_key=uklonvd826214,coordinator_type=0} to node 0. [kafka-consumer]
2017-11-10 11:27:45,276 TRACE org.apache.kafka.clients.NetworkClient
- Completed receive from node 2, for key 1, received
{throttle_time_ms=0,responses=[{topic=gao31,partition_responses=[{partition_header={partition=2,error_code=0,high_watermark=1631,last_stable_offset=-1,log_start_offset=0,aborted_transactions=null},record_set=[(record=DefaultRecord(offset=1630,
timestamp=1510313264997, key=1 bytes, value=102
[jira] [Comment Edited] (KAFKA-6189) Loosing messages on OFFSET_OUT_OF_RANGE error in consumer
[
https://issues.apache.org/jira/browse/KAFKA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249419#comment-16249419
]
Xinyang Gao edited comment on KAFKA-6189 at 11/13/17 11:58 AM:
---
I had seen similar behaviors which also led to message loss, setup is as
follows:
topic has 3 partitions and replication factor 3, named gao31
min.insync.replicas=2
consumer has default "auto.offset.reset=latest"
consumer manually commitSync offsets after handling messages with
consumer.commitSync()
unclean leader election = false
kafka cluster has 3 brokers, kafka-foo-0, kafka-foo-1 and kafka-foo-2
consumer group ID is uklonvd826214
session.timeout.ms and max.poll.interval.ms are using default values
Initially kafka-foo-1 is selected as group coordinator
To reproduce:
1. kill kafka-foo-1
2. See following logs which implies that consumer disconnect to kafka-foo-1 and
try to discover a new group coordinator
{code:java}
2017-11-10 11:27:45,097 DEBUG org.apache.kafka.clients.NetworkClient
- Node 2147483646 disconnected. [kafka-consumer]
2017-11-10 11:27:45,097 INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the
coordinator kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id:
2147483646 rack: null) dead for group uklonvd826214 [kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 DEBUG org.apache.kafka.clients.NetworkClient
- Sending metadata request (type=MetadataRequest, topics=gao31) to
node 0 [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Sending METADATA {topics=[gao31],allow_auto_topic_creation=true}
to node 0. [kafka-consumer]
2017-11-10 11:27:45,100 TRACE org.apache.kafka.clients.NetworkClient
- Sending FIND_COORDINATOR
{coordinator_key=uklonvd826214,coordinator_type=0} to node 0. [kafka-consumer]
2017-11-10 11:27:45,276 TRACE org.apache.kafka.clients.NetworkClient
- Completed receive from node 2, for key 1, received
[jira] [Comment Edited] (KAFKA-6189) Loosing messages on OFFSET_OUT_OF_RANGE error in consumer
[
https://issues.apache.org/jira/browse/KAFKA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249419#comment-16249419
]
Xinyang Gao edited comment on KAFKA-6189 at 11/13/17 11:57 AM:
---
I had seen similar behaviors which also led to message loss, setup is as
follows:
topic has 3 partitions and replication factor 3, named gao31
min.insync.replicas=2
consumer has default "auto.offset.reset=latest"
consumer manually commitSync offsets after handling messages with
consumer.commitSync()
unclean leader election = false
kafka cluster has 3 brokers, kafka-foo-0, kafka-foo-1 and kafka-foo-2
consumer group ID is uklonvd826214
session.timeout.ms and max.poll.interval.ms are using default values
Initially kafka-foo-1 is selected as group coordinator
To reproduce:
1. kill kafka-foo-1
2. See following logs which implies that consumer disconnect to kafka-foo-1 and
try to discover a new group coordinator
{code:java}
2017-11-10 11:27:45,097 DEBUG org.apache.kafka.clients.NetworkClient
- Node 2147483646 disconnected. [kafka-consumer]
2017-11-10 11:27:45,097 INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the
coordinator kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id:
2147483646 rack: null) dead for group uklonvd826214 [kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 DEBUG org.apache.kafka.clients.NetworkClient
- Sending metadata request (type=MetadataRequest, topics=gao31) to
node 0 [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Sending METADATA {topics=[gao31],allow_auto_topic_creation=true}
to node 0. [kafka-consumer]
2017-11-10 11:27:45,100 TRACE org.apache.kafka.clients.NetworkClient
- Sending FIND_COORDINATOR
{coordinator_key=uklonvd826214,coordinator_type=0} to node 0. [kafka-consumer]
2017-11-10 11:27:45,276 TRACE org.apache.kafka.clients.NetworkClient
- Completed receive from node 2, for key 1, received
[jira] [Comment Edited] (KAFKA-6189) Loosing messages on OFFSET_OUT_OF_RANGE error in consumer
[
https://issues.apache.org/jira/browse/KAFKA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249419#comment-16249419
]
Xinyang Gao edited comment on KAFKA-6189 at 11/13/17 11:55 AM:
---
I had seen similar behaviors which also led to message loss, setup is as
follows:
topic has 3 partitions and replication factor 3, named gao31
min.insync.replicas=2
consumer has default "auto.offset.reset=latest"
consumer manually commitSync offsets after handling messages with
consumer.commitSync()
unclean leader election = false
kafka cluster has 3 brokers, kafka-foo-0, kafka-foo-1 and kafka-foo-2
consumer group ID is uklonvd826214
session.timeout.ms and max.poll.interval.ms are using default values
Initially kafka-foo-1 is selected as group coordinator
To reproduce:
1. kill kafka-foo-1
2. See following logs which implies that consumer disconnect to kafka-foo-1 and
try to discover a new group coordinator
{code:java}
2017-11-10 11:27:45,097 DEBUG org.apache.kafka.clients.NetworkClient
- Node 2147483646 disconnected. [kafka-consumer]
2017-11-10 11:27:45,097 INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the
coordinator kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id:
2147483646 rack: null) dead for group uklonvd826214 [kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 DEBUG org.apache.kafka.clients.NetworkClient
- Sending metadata request (type=MetadataRequest, topics=gao31) to
node 0 [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Sending METADATA {topics=[gao31],allow_auto_topic_creation=true}
to node 0. [kafka-consumer]
2017-11-10 11:27:45,100 TRACE org.apache.kafka.clients.NetworkClient
- Sending FIND_COORDINATOR
{coordinator_key=uklonvd826214,coordinator_type=0} to node 0. [kafka-consumer]
2017-11-10 11:27:45,276 TRACE org.apache.kafka.clients.NetworkClient
- Completed receive from node 2, for key 1, received
[jira] [Comment Edited] (KAFKA-6189) Loosing messages on OFFSET_OUT_OF_RANGE error in consumer
[
https://issues.apache.org/jira/browse/KAFKA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249419#comment-16249419
]
Xinyang Gao edited comment on KAFKA-6189 at 11/13/17 11:30 AM:
---
I had seen similar behaviors which also lead to message loss, setup is as
follows:
topic has 3 partitions and replication factor 3, named gao31
min.insync.replicas=2
consumer has default "auto.offset.reset=latest"
consumer manually commitSync offsets after handling messages with
consumer.commitSync()
unclean leader election = false
kafka cluster has 3 brokers, kafka-foo-0, kafka-foo-1 and kafka-foo-2
consumer group ID is uklonvd826214
session.timeout.ms and max.poll.interval.ms are using default values
Initially kafka-foo-1 is selected as group coordinator
To reproduce:
1. kill kafka-foo-1
2. See following logs which implies that consumer disconnect to kafka-foo-1 and
try to discover a new group coordinator
{code:java}
2017-11-10 11:27:45,097 DEBUG org.apache.kafka.clients.NetworkClient
- Node 2147483646 disconnected. [kafka-consumer]
2017-11-10 11:27:45,097 INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the
coordinator kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id:
2147483646 rack: null) dead for group uklonvd826214 [kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 DEBUG org.apache.kafka.clients.NetworkClient
- Sending metadata request (type=MetadataRequest, topics=gao31) to
node 0 [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Sending METADATA {topics=[gao31],allow_auto_topic_creation=true}
to node 0. [kafka-consumer]
2017-11-10 11:27:45,100 TRACE org.apache.kafka.clients.NetworkClient
- Sending FIND_COORDINATOR
{coordinator_key=uklonvd826214,coordinator_type=0} to node 0. [kafka-consumer]
2017-11-10 11:27:45,276 TRACE org.apache.kafka.clients.NetworkClient
- Completed receive from node 2, for key 1, received
[jira] [Comment Edited] (KAFKA-6189) Loosing messages on OFFSET_OUT_OF_RANGE error in consumer
[
https://issues.apache.org/jira/browse/KAFKA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249419#comment-16249419
]
Xinyang Gao edited comment on KAFKA-6189 at 11/13/17 11:29 AM:
---
I had seen similar behaviors which also lead to message loss, setup is as
follows:
topic has 3 partitions and replication factor 3, named gao31
min.insync.replicas=2
consumer has default "auto.offset.reset=latest"
consumer manually commitSync offsets after handling messages with
consumer.commitSync()
unclean leader election = false
kafka cluster has 3 brokers, kafka-foo-0, kafka-foo-1 and kafka-foo-2
consumer group ID is uklonvd826214
session.timeout.ms and max.poll.interval.ms are using default values
Initially kafka-foo-1 is selected as group coordinator
To reproduce:
1. kill kafka-foo-1
2. See following logs which implies that consumer disconnect to kafka-foo-1 and
try to discover a new group coordinator
{code:java}
2017-11-10 11:27:45,097 DEBUG org.apache.kafka.clients.NetworkClient
- Node 2147483646 disconnected. [kafka-consumer]
2017-11-10 11:27:45,097 INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the
coordinator kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id:
2147483646 rack: null) dead for group uklonvd826214 [kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,098 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,098 DEBUG
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending
GroupCoordinator request for group uklonvd826214 to broker
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-1.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 1 rack: null) from
least loaded node selection: is-blacked-out: true, in-flight-requests: 0
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Removing node
kafka-foo-2.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 2 rack: null) from
least loaded node selection: is-blacked-out: false, in-flight-requests: 1
[kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Found least loaded node
kafka-foo-0.dk0095-a.int.uk.paas.intranet.db.com:9092 (id: 0 rack: null)
[kafka-consumer]
2017-11-10 11:27:45,099 DEBUG org.apache.kafka.clients.NetworkClient
- Sending metadata request (type=MetadataRequest, topics=gao31) to
node 0 [kafka-consumer]
2017-11-10 11:27:45,099 TRACE org.apache.kafka.clients.NetworkClient
- Sending METADATA {topics=[gao31],allow_auto_topic_creation=true}
to node 0. [kafka-consumer]
2017-11-10 11:27:45,100 TRACE org.apache.kafka.clients.NetworkClient
- Sending FIND_COORDINATOR
{coordinator_key=uklonvd826214,coordinator_type=0} to node 0. [kafka-consumer]
2017-11-10 11:27:45,276 TRACE org.apache.kafka.clients.NetworkClient
- Completed receive from node 2, for key 1, received
