[jira] [Commented] (KAFKA-6990) CommitFailedException; this task may be no longer owned by the thread
[
https://issues.apache.org/jira/browse/KAFKA-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640256#comment-16640256
]
Ryanne Dolan commented on KAFKA-6990:
-
Usually this is due to records being processed slowly and sequentially, which
seems to be the case here also. Consider processing records asynchronously / in
parallel if you can, so that you can process the batch of records faster. If
you can't do that, e.g. order matters, then reduce max.poll.records to
something small s.t. poll() returns only a few records for you to process at a
time.
> CommitFailedException; this task may be no longer owned by the thread
> -
>
> Key: KAFKA-6990
> URL: https://issues.apache.org/jira/browse/KAFKA-6990
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.1
>Reporter: Eugen Feller
>Priority: Blocker
>
> We are seeing a lot of CommitFailedExceptions on one of our Kafka stream
> apps. Running Kafka Streams 0.11.0.1 and Kafka Broker 0.10.2.1. Full error
> message:
> {code:java}
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] INFO
> org.apache.kafka.streams.KafkaStreams - stream-client
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d] State transition from
> REBALANCING to RUNNING.
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_0] Failed
> offset commits
> {sightings_sighting_byclientmac_0-0=OffsetAndMetadata{offset=11,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-0=OffsetAndMetadata{offset=26,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_0 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_1 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_1] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-1=OffsetAndMetadata{offset=38,
> metadata=''}, sightings_sighting_byclientmac_0-1=OffsetAndMetadata{offset=1,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_2] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-2=OffsetAndMetadata{offset=8,
> metadata=''},
> sightings_sighting_byclientmac_0-2=OffsetAndMetadata{offset=24, metadata=''}}
> due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_2 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_3] Failed
> offset commits
> {sightings_sighting_byclientmac_0-3=OffsetAndMetadata{offset=21,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-3=OffsetAndMetadata{offset=102,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_3 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_4] Failed
> offset commits
> {sightings_sighting_byclientmac_0-4=OffsetAndMetadata{offset=5, metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-4=OffsetAndMetadata{offset=20,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_4
[jira] [Commented] (KAFKA-6990) CommitFailedException; this task may be no longer owned by the thread
[
https://issues.apache.org/jira/browse/KAFKA-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508843#comment-16508843
]
Eugen Feller commented on KAFKA-6990:
-
Thanks a lot. For sake of completeness. Logs sent via Slack. :)
> CommitFailedException; this task may be no longer owned by the thread
> -
>
> Key: KAFKA-6990
> URL: https://issues.apache.org/jira/browse/KAFKA-6990
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.1
>Reporter: Eugen Feller
>Priority: Blocker
>
> We are seeing a lot of CommitFailedExceptions on one of our Kafka stream
> apps. Running Kafka Streams 0.11.0.1 and Kafka Broker 0.10.2.1. Full error
> message:
> {code:java}
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] INFO
> org.apache.kafka.streams.KafkaStreams - stream-client
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d] State transition from
> REBALANCING to RUNNING.
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_0] Failed
> offset commits
> {sightings_sighting_byclientmac_0-0=OffsetAndMetadata{offset=11,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-0=OffsetAndMetadata{offset=26,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_0 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_1 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_1] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-1=OffsetAndMetadata{offset=38,
> metadata=''}, sightings_sighting_byclientmac_0-1=OffsetAndMetadata{offset=1,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_2] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-2=OffsetAndMetadata{offset=8,
> metadata=''},
> sightings_sighting_byclientmac_0-2=OffsetAndMetadata{offset=24, metadata=''}}
> due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_2 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_3] Failed
> offset commits
> {sightings_sighting_byclientmac_0-3=OffsetAndMetadata{offset=21,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-3=OffsetAndMetadata{offset=102,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_3 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_4] Failed
> offset commits
> {sightings_sighting_byclientmac_0-4=OffsetAndMetadata{offset=5, metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-4=OffsetAndMetadata{offset=20,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_4 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_5] Failed
> offset commits
>
[jira] [Commented] (KAFKA-6990) CommitFailedException; this task may be no longer owned by the thread
[
https://issues.apache.org/jira/browse/KAFKA-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508693#comment-16508693
]
Matthias J. Sax commented on KAFKA-6990:
There are two thread one for heartbeats and one for processing. The heartbeat
thread is subject to heartbeat interval and session.timeout.ms – as you have
session timeout with 30 seconds and heartbeat are sent every 5 second, you
would need to miss 6 consecutive heartbeat to drop out of the group. For the
main processing thread, max.poll.interval.ms is the timeout, ie, poll() must be
called every 20 minutes in your case – this should actually be sufficient. Even
if you wait max 5 seconds for MongoDB to response, you have maximum 50 records
to process and thus, it should not take longer than 250 seconds until poll() is
called again... Really hard to say what it happening. Let's see if Kafka
Streams/Consumer logs reveal the issue.
> CommitFailedException; this task may be no longer owned by the thread
> -
>
> Key: KAFKA-6990
> URL: https://issues.apache.org/jira/browse/KAFKA-6990
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.1
>Reporter: Eugen Feller
>Priority: Blocker
>
> We are seeing a lot of CommitFailedExceptions on one of our Kafka stream
> apps. Running Kafka Streams 0.11.0.1 and Kafka Broker 0.10.2.1. Full error
> message:
> {code:java}
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] INFO
> org.apache.kafka.streams.KafkaStreams - stream-client
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d] State transition from
> REBALANCING to RUNNING.
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_0] Failed
> offset commits
> {sightings_sighting_byclientmac_0-0=OffsetAndMetadata{offset=11,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-0=OffsetAndMetadata{offset=26,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_0 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_1 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_1] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-1=OffsetAndMetadata{offset=38,
> metadata=''}, sightings_sighting_byclientmac_0-1=OffsetAndMetadata{offset=1,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_2] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-2=OffsetAndMetadata{offset=8,
> metadata=''},
> sightings_sighting_byclientmac_0-2=OffsetAndMetadata{offset=24, metadata=''}}
> due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_2 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_3] Failed
> offset commits
> {sightings_sighting_byclientmac_0-3=OffsetAndMetadata{offset=21,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-3=OffsetAndMetadata{offset=102,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_3 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_4] Failed
> offset commits
> {sightings_sighting_byclientmac_0-4=OffsetAndMetadata{offset=5, metadata=''},
>
[jira] [Commented] (KAFKA-6990) CommitFailedException; this task may be no longer owned by the thread
[
https://issues.apache.org/jira/browse/KAFKA-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508566#comment-16508566
]
Eugen Feller commented on KAFKA-6990:
-
Hi [~mjsax]. Thank you very much for your inputs. That makes total sense to me.
My current defaults are as follows:
{code:java}
requestTimeout: Duration = 30 minutes,
maxPollInterval: Duration = 20 minutes,
maxPollRecords: Long = 1000,
fetchMaxBytes: Long = ConsumerConfig.DEFAULT_FETCH_MAX_BYTES,
fetchMaxWait: Duration = 10 minutes,
sessionTimeout: Duration = 30 seconds,
heartbeatInterval: Duration = 5 seconds,
stateDirCleanupDelay: Long = Long.MaxValue,
numStreamThreads: Long = 1,
numStandbyReplicas: Long = 0,
cacheMaxBytesBuffering: Long = 128 * 1024L * 1024L{code}
Should I further increase the max.poll.interval.ms (currently 20 minutes)? This
particular job does the following logic:
{code:java}
stream().foreach(record => {
// Buffered write to MongoDB
// Await.result(5 seconds)
}){code}
Can it be that writes to Mongo take too long and that leads to problems? I was
under the impression that by setting heartbeat interval to 5 seconds in Kafka
0.11.0.1, we should have two threads, one that does heartbeat sending (to
considered alive) and one that actually calls poll(). In that case blocking too
long inside foreach should not kick us out of the consumer group?
I will try to collect DEBUG level logs today.
> CommitFailedException; this task may be no longer owned by the thread
> -
>
> Key: KAFKA-6990
> URL: https://issues.apache.org/jira/browse/KAFKA-6990
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.1
>Reporter: Eugen Feller
>Priority: Blocker
>
> We are seeing a lot of CommitFailedExceptions on one of our Kafka stream
> apps. Running Kafka Streams 0.11.0.1 and Kafka Broker 0.10.2.1. Full error
> message:
> {code:java}
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] INFO
> org.apache.kafka.streams.KafkaStreams - stream-client
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d] State transition from
> REBALANCING to RUNNING.
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_0] Failed
> offset commits
> {sightings_sighting_byclientmac_0-0=OffsetAndMetadata{offset=11,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-0=OffsetAndMetadata{offset=26,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_0 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_1 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_1] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-1=OffsetAndMetadata{offset=38,
> metadata=''}, sightings_sighting_byclientmac_0-1=OffsetAndMetadata{offset=1,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_2] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-2=OffsetAndMetadata{offset=8,
> metadata=''},
> sightings_sighting_byclientmac_0-2=OffsetAndMetadata{offset=24, metadata=''}}
> due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_2 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_3] Failed
> offset commits
> {sightings_sighting_byclientmac_0-3=OffsetAndMetadata{offset=21,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-3=OffsetAndMetadata{offset=102,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
>
[jira] [Commented] (KAFKA-6990) CommitFailedException; this task may be no longer owned by the thread
[
https://issues.apache.org/jira/browse/KAFKA-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507611#comment-16507611
]
Matthias J. Sax commented on KAFKA-6990:
I think that the `Fetcher` TRACE level log are not relevant. However, the
constantly rebalancing is an issue and explain why commits fail. If a instance
misses a rebalance and drops out of the group, it gets fenced because it does
not own any partitions anymore – if a fenced instance tried to commit, the
observed error occurs. After that, Kafka Streams will try to rejoin the group.
It seems that an instance is repeatedly missing a rebalance, triggering a new
rebalance, missing it and so on. You might want to increase some timeout to get
the application more time to join the group and to avoid that it drops out of
the group. Try increasing `max.poll.interval.ms`. Let us know if it resolved
the issue. If not, and you can still reproduce, it would be great to get DEBUG
level logs of both instanced to dig into it in more detail.
> CommitFailedException; this task may be no longer owned by the thread
> -
>
> Key: KAFKA-6990
> URL: https://issues.apache.org/jira/browse/KAFKA-6990
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.1
>Reporter: Eugen Feller
>Priority: Blocker
>
> We are seeing a lot of CommitFailedExceptions on one of our Kafka stream
> apps. Running Kafka Streams 0.11.0.1 and Kafka Broker 0.10.2.1. Full error
> message:
> {code:java}
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] INFO
> org.apache.kafka.streams.KafkaStreams - stream-client
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d] State transition from
> REBALANCING to RUNNING.
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_0] Failed
> offset commits
> {sightings_sighting_byclientmac_0-0=OffsetAndMetadata{offset=11,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-0=OffsetAndMetadata{offset=26,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_0 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_1 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_1] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-1=OffsetAndMetadata{offset=38,
> metadata=''}, sightings_sighting_byclientmac_0-1=OffsetAndMetadata{offset=1,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_2] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-2=OffsetAndMetadata{offset=8,
> metadata=''},
> sightings_sighting_byclientmac_0-2=OffsetAndMetadata{offset=24, metadata=''}}
> due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_2 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_3] Failed
> offset commits
> {sightings_sighting_byclientmac_0-3=OffsetAndMetadata{offset=21,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-3=OffsetAndMetadata{offset=102,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_3 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_4] Failed
> offset commits
[jira] [Commented] (KAFKA-6990) CommitFailedException; this task may be no longer owned by the thread
[
https://issues.apache.org/jira/browse/KAFKA-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501100#comment-16501100
]
Eugen Feller commented on KAFKA-6990:
-
I have also noticed a lot messages like this in that job:
{code:java}
00:56:28.894 [visits-e80b42d2-6c8c-4228-aaca-f7b15fdd410b-StreamThread-1] TRACE
o.a.k.c.consumer.internals.Fetcher - Skipping fetch for partition
visits_enrichedsighting_byclientmac_0-7 because there is an in-flight request
to 10.16.210.152:9092 (id: 3 rack: us-west-2c)
00:56:28.894 [visits-e80b42d2-6c8c-4228-aaca-f7b15fdd410b-StreamThread-1] TRACE
o.a.k.c.consumer.internals.Fetcher - Skipping fetch for partition
visits_enrichedsighting_byclientmac_0-9 because there is an in-flight request
to 10.16.208.24:9092 (id: 9 rack: us-west-2a)
{code}
> CommitFailedException; this task may be no longer owned by the thread
> -
>
> Key: KAFKA-6990
> URL: https://issues.apache.org/jira/browse/KAFKA-6990
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.1
>Reporter: Eugen Feller
>Priority: Blocker
>
> We are seeing a lot of CommitFailedExceptions on one of our Kafka stream
> apps. Running Kafka Streams 0.11.0.1 and Kafka Broker 0.10.2.1. Full error
> message:
> {code:java}
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] INFO
> org.apache.kafka.streams.KafkaStreams - stream-client
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d] State transition from
> REBALANCING to RUNNING.
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_0] Failed
> offset commits
> {sightings_sighting_byclientmac_0-0=OffsetAndMetadata{offset=11,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-0=OffsetAndMetadata{offset=26,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_0 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_1 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_1] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-1=OffsetAndMetadata{offset=38,
> metadata=''}, sightings_sighting_byclientmac_0-1=OffsetAndMetadata{offset=1,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_2] Failed
> offset commits
> {mactocontactmappings_contactmapping_byclientmac_0-2=OffsetAndMetadata{offset=8,
> metadata=''},
> sightings_sighting_byclientmac_0-2=OffsetAndMetadata{offset=24, metadata=''}}
> due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_2 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_3] Failed
> offset commits
> {sightings_sighting_byclientmac_0-3=OffsetAndMetadata{offset=21,
> metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-3=OffsetAndMetadata{offset=102,
> metadata=''}} due to CommitFailedException
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.AssignedTasks - stream-thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] Failed to commit
> stream task 0_3 during commit state due to CommitFailedException; this task
> may be no longer owned by the thread
> [visits-bdb209f1-c9dd-404b-9233-064f2fb4227d-StreamThread-1] WARN
> org.apache.kafka.streams.processor.internals.StreamTask - task [0_4] Failed
> offset commits
> {sightings_sighting_byclientmac_0-4=OffsetAndMetadata{offset=5, metadata=''},
> mactocontactmappings_contactmapping_byclientmac_0-4=OffsetAndMetadata{offset=20,
> metadata=''}} due to CommitFailedException
>
