[
https://issues.apache.org/jira/browse/KAFKA-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259522#comment-16259522
]
ASF GitHub Bot commented on KAFKA-6214:
---------------------------------------
GitHub user dguy opened a pull request:
https://github.com/apache/kafka/pull/4239
KAFKA-6214: enable use of in-memory store for standby tasks
Remove the flag in `ProcessorStateManager` that checks if a store is
persistent when registering it as a standby task.
Updated the smoke test to use an in-memory store.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dguy/kafka kafka-6214
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/4239.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4239
----
commit a9cc7b3bbf2bc42462d0876c6c13fdc8173f47ec
Author: Damian Guy <[email protected]>
Date: 2017-11-20T13:30:35Z
enable use of in-memory store for standby tasks
----
> Using standby replicas with an in memory state store causes Streams to crash
> ----------------------------------------------------------------------------
>
> Key: KAFKA-6214
> URL: https://issues.apache.org/jira/browse/KAFKA-6214
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.11.0.1
> Reporter: Matt Farmer
> Assignee: Damian Guy
> Labels: statestore
>
> We decided to start experimenting with Standby Replicas of our State Stores
> by setting the following configuration setting:
> {code}
> num.standby.replicas=1
> {code}
> Most applications did okay with this except for one that used an in memory
> state store instead of a persistent state store. With the new configuration,
> the first instance of this application booted fine. When the second instance
> came up, both instances crashed with the following exception:
> {code}
> java.lang.IllegalStateException: Consumer is not subscribed to any topics or
> assigned any partitions
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1037)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:752)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:524)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
> {code}
> Monit attempted to restart both instances but they would just continue to
> crash over and over again. The state store in our problematic application is
> declared like so:
> {code}
> Stores
> .create("TheStateStore")
> .withStringKeys()
> .withStringValues()
> .inMemory()
> .build()
> {code}
> Luckily we had a config switch in place that could turn on an alternate,
> persistent state store. As soon as we flipped to the persistent state store,
> things started working as we expected.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
