[jira] [Commented] (KAFKA-6106) Postpone normal processing of tasks within a thread until restoration of all tasks have completed
[
https://issues.apache.org/jira/browse/KAFKA-6106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360308#comment-16360308
]
Guozhang Wang commented on KAFKA-6106:
--
Thanks!
> Postpone normal processing of tasks within a thread until restoration of all
> tasks have completed
> -
>
> Key: KAFKA-6106
> URL: https://issues.apache.org/jira/browse/KAFKA-6106
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.11.0.1, 1.0.0
>Reporter: Guozhang Wang
>Assignee: Kamal Chandraprakash
>Priority: Major
> Labels: newbie++
>
> Let's say a stream thread hosts multiple tasks, A and B. At the very
> beginning when A and B are assigned to the thread, the thread state is
> {{TASKS_ASSIGNED}}, and the thread start restoring these two tasks during
> this state using the restore consumer while using normal consumer for
> heartbeating.
> If task A's restoration has completed earlier than task B, then the thread
> will start processing A immediately even when it is still in the
> {{TASKS_ASSIGNED}} phase. But processing task A will slow down restoration of
> task B since it is single-thread. So the thread's transition to {{RUNNING}}
> when all of its assigned tasks have completed restoring and now can be
> processed will be delayed.
> Note that the streams instance's state will only transit to {{RUNNING}} when
> all of its threads have transit to {{RUNNING}}, so the instance's transition
> will also be delayed by this scenario.
> We'd better to not start processing ready tasks immediately, but instead
> focus on restoration during the {{TASKS_ASSIGNED}} state to shorten the
> overall time of the instance's state transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6397) Consumer should not block setting initial positions of unavailable partitions
[
https://issues.apache.org/jira/browse/KAFKA-6397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360154#comment-16360154
]
ASF GitHub Bot commented on KAFKA-6397:
---
hachikuji opened a new pull request #4557: KAFKA-6397: Consumer should not
block setting positions of unavailable partitions
URL: https://github.com/apache/kafka/pull/4557
Prior to this patch, the consumer always blocks in poll() if there are any
partitions which are awaiting their initial positions. This behavior was
inconsistent with normal fetch behavior since we allow fetching on available
partitions even if one or more of the assigned partitions becomes unavailable
_after_ initial offset lookup. With this patch, the consumer will do offset
resets asynchronously, which allows other partitions to make progress even if
the initial positions for some partitions cannot be found.
I have added several new unit tests in `FetcherTest` and `KafkaConsumerTest`
to verify the new behavior. One minor compatibility implication worth
mentioning is apparent from the change I made in
`DynamicBrokerReconfigurationTest`. Previously it was possible to assume that
all partitions had a fetch position after `poll()` completed with a non-empty
assignment. This assumption is no longer generally true, but you can force the
positions to be updated using the `position()` API which still blocks
indefinitely until a position is available.
Note that this this patch also removes the logic to cache committed offsets
in `SubscriptionState` since it was no longer needed (the consumer's
`committed()` API always does an offset lookup anyway). In addition to avoiding
the complexity of maintaining the cache, this avoids wasteful offset lookups to
refresh the cache when `commitAsync()` is used.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Consumer should not block setting initial positions of unavailable partitions
> -
>
> Key: KAFKA-6397
> URL: https://issues.apache.org/jira/browse/KAFKA-6397
> Project: Kafka
> Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Labels: consumer
> Fix For: 1.2.0
>
>
> Currently the consumer will block in poll() after receiving its assignment in
> order to set the starting offset for every assigned partition. If the topic
> is deleted or if a partition is unavailable, the consumer can be stuck
> indefinitely. Most of the time this is not a problem since the starting
> offset is obtained from the committed offsets, which does not depend on
> partition availability. However, if there are no committed offsets or if the
> user has manually called {{seekToBeginning}} or {{seekToEnd}}, then we will
> need to do a lookup for the starting offset from the partition leader, which
> will stall the consumer until the partition is available or recreated. It
> would be better to let the consumer fetch on partitions which are available
> and periodically check availability for the rest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6552) “entity_type” not exactly in description of kafka-configs.sh
[ https://issues.apache.org/jira/browse/KAFKA-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359860#comment-16359860 ] ASF GitHub Bot commented on KAFKA-6552: --- auroraxlh opened a new pull request #4556: KAFKA-6552:“entity_type” not exactly in description of kafka-configs.sh URL: https://github.com/apache/kafka/pull/4556 *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > “entity_type” not exactly in description of kafka-configs.sh > - > > Key: KAFKA-6552 > URL: https://issues.apache.org/jira/browse/KAFKA-6552 > Project: Kafka > Issue Type: Bug > Components: admin >Affects Versions: 0.11.0.2 >Reporter: Xin >Priority: Trivial > > There are some “entity_type” in the description of the command option > “--add-config”, but the “--entity-type ” is the command option of > kafka-configs.sh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6552) “entity_type” not exactly in description of kafka-configs.sh
Xin created KAFKA-6552: -- Summary: “entity_type” not exactly in description of kafka-configs.sh Key: KAFKA-6552 URL: https://issues.apache.org/jira/browse/KAFKA-6552 Project: Kafka Issue Type: Bug Components: admin Affects Versions: 0.11.0.2 Reporter: Xin There are some “entity_type” in the description of the command option “--add-config”, but the “--entity-type ” is the command option of kafka-configs.sh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
