[jira] [Updated] (KAFKA-2350) Add KafkaConsumer pause capability
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-2350:
-
Fix Version/s: (was: 0.9.0)
0.8.3
Add KafkaConsumer pause capability
--
Key: KAFKA-2350
URL: https://issues.apache.org/jira/browse/KAFKA-2350
Project: Kafka
Issue Type: Improvement
Components: consumer
Reporter: Jason Gustafson
Assignee: Jason Gustafson
Fix For: 0.8.3
There are some use cases in stream processing where it is helpful to be able
to pause consumption of a topic. For example, when joining two topics, you
may need to delay processing of one topic while you wait for the consumer of
the other topic to catch up. The new consumer currently doesn't provide a
nice way to do this. If you skip calls to poll() or if you unsubscribe, then
a rebalance will be triggered and your partitions will be reassigned to
another consumer. The desired behavior is instead that you keep the partition
assigned and simply
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(TopicPartition... partitions);
void resume(TopicPartition... partitions);
{code}
Here is the expected behavior of pause/resume:
* When a partition is paused, calls to KafkaConsumer.poll will not initiate
any new fetches for that partition.
* After the partition is resumed, fetches will begin again.
* While a partition is paused, seek() and position() can still be used to
advance or query the current position.
* Rebalance does not preserve pause/resume state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2350) Add KafkaConsumer pause capability
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewen Cheslack-Postava updated KAFKA-2350:
-
Component/s: consumer
Add KafkaConsumer pause capability
--
Key: KAFKA-2350
URL: https://issues.apache.org/jira/browse/KAFKA-2350
Project: Kafka
Issue Type: Improvement
Components: consumer
Reporter: Jason Gustafson
Assignee: Jason Gustafson
There are some use cases in stream processing where it is helpful to be able
to pause consumption of a topic. For example, when joining two topics, you
may need to delay processing of one topic while you wait for the consumer of
the other topic to catch up. The new consumer currently doesn't provide a
nice way to do this. If you skip calls to poll() or if you unsubscribe, then
a rebalance will be triggered and your partitions will be reassigned to
another consumer. The desired behavior is instead that you keep the partition
assigned and simply
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(TopicPartition... partitions);
void resume(TopicPartition... partitions);
{code}
Here is the expected behavior of pause/resume:
* When a partition is paused, calls to KafkaConsumer.poll will not initiate
any new fetches for that partition.
* After the partition is resumed, fetches will begin again.
* While a partition is paused, seek() and position() can still be used to
advance or query the current position.
* Rebalance does not preserve pause/resume state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2350) Add KafkaConsumer pause capability
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson updated KAFKA-2350:
---
Description:
There are some use cases in stream processing where it is helpful to be able to
pause consumption of a topic. For example, when joining two topics, you may
need to delay processing of one topic while you wait for the consumer of the
other topic to catch up. The new consumer currently doesn't provide a nice way
to do this. If you skip calls to poll() or if you unsubscribe, then a rebalance
will be triggered and your partitions will be reassigned to another consumer.
The desired behavior is instead that you keep the partition assigned and simply
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(TopicPartition... partitions);
void resume(TopicPartition... partitions);
{code}
Here is the expected behavior of pause/resume:
* When a partition is paused, calls to KafkaConsumer.poll will not initiate any
new fetches for that partition.
* After the partition is resumed, fetches will begin again.
* While a partition is paused, seek() and position() can still be used to
advance or query the current position.
* Rebalance does not preserve pause/resume state.
was:
There are some use cases in stream processing where it is helpful to be able to
pause consumption of a topic. For example, when joining two topics, you may
need to delay processing of one topic while you wait for the consumer of the
other topic to catch up. The new consumer currently doesn't provide a nice way
to do this. If you skip poll() or if you unsubscribe, then a rebalance will be
triggered and your partitions will be reassigned.
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(String... topics);
void unpause(String... topics);
{code}
When a topic is paused, a call to KafkaConsumer.poll will not initiate any new
fetches for that topic. After it is unpaused, fetches will begin again.
Add KafkaConsumer pause capability
--
Key: KAFKA-2350
URL: https://issues.apache.org/jira/browse/KAFKA-2350
Project: Kafka
Issue Type: Improvement
Reporter: Jason Gustafson
Assignee: Jason Gustafson
There are some use cases in stream processing where it is helpful to be able
to pause consumption of a topic. For example, when joining two topics, you
may need to delay processing of one topic while you wait for the consumer of
the other topic to catch up. The new consumer currently doesn't provide a
nice way to do this. If you skip calls to poll() or if you unsubscribe, then
a rebalance will be triggered and your partitions will be reassigned to
another consumer. The desired behavior is instead that you keep the partition
assigned and simply
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(TopicPartition... partitions);
void resume(TopicPartition... partitions);
{code}
Here is the expected behavior of pause/resume:
* When a partition is paused, calls to KafkaConsumer.poll will not initiate
any new fetches for that partition.
* After the partition is resumed, fetches will begin again.
* While a partition is paused, seek() and position() can still be used to
advance or query the current position.
* Rebalance does not preserve pause/resume state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2350) Add KafkaConsumer pause capability
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson updated KAFKA-2350:
---
Description:
There are some use cases in stream processing where it is helpful to be able to
pause consumption of a topic. For example, when joining two topics, you may
need to delay processing of one topic while you wait for the consumer of the
other topic to catch up. The new consumer currently doesn't provide a nice way
to do this. If you skip poll() or if you unsubscribe, then a rebalance will be
triggered and your partitions will be reassigned.
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(String... topics);
void unpause(String... topics);
{code}
When a topic is paused, a call to KafkaConsumer.poll will not initiate any new
fetches for that topic. After it is unpaused, fetches will begin again.
was:
There are some use cases in stream processing where it is helpful to be able to
pause consumption of a topic. For example, when joining two topics, you may
need to delay processing of one topic while you wait for the consumer of the
other topic to catch up. The new consumer currently doesn't provide a nice way
to do this. If you skip poll() or if you unsubscribe, then a rebalance will be
triggered and your partitions will be reassigned.
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(String... partitions);
void unpause(String... partitions);
{code}
When a topic is paused, a call to KafkaConsumer.poll will not initiate any new
fetches for that topic. After it is unpaused, fetches will begin again.
Add KafkaConsumer pause capability
--
Key: KAFKA-2350
URL: https://issues.apache.org/jira/browse/KAFKA-2350
Project: Kafka
Issue Type: Improvement
Reporter: Jason Gustafson
Assignee: Jason Gustafson
There are some use cases in stream processing where it is helpful to be able
to pause consumption of a topic. For example, when joining two topics, you
may need to delay processing of one topic while you wait for the consumer of
the other topic to catch up. The new consumer currently doesn't provide a
nice way to do this. If you skip poll() or if you unsubscribe, then a
rebalance will be triggered and your partitions will be reassigned.
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(String... topics);
void unpause(String... topics);
{code}
When a topic is paused, a call to KafkaConsumer.poll will not initiate any
new fetches for that topic. After it is unpaused, fetches will begin again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
