Re: serveral questions about auto.offset.reset
I think of it as default.offset :) Realistically though, changing the configuration name will probably cause more confusion than leaving a bad name as is. On Tue, Apr 14, 2015 at 11:10 AM, James Cheng jch...@tivo.com wrote: What to do when there is no initial offset in ZooKeeper or if an offset is out of range I personally find the name auto.offset.reset to be somewhat confusing. That's mostly because I only knew of it as the no initial offset setting. An alternate name could be auto.offset.initial, to handle the no initial offset in Zookeeper case. But that doesn't describe the offset out of range case. -James On Apr 13, 2015, at 10:28 PM, Ewen Cheslack-Postava e...@confluent.io wrote: On Mon, Apr 13, 2015 at 10:10 PM, bit1...@163.com bit1...@163.com wrote: Hi, Kafka experts: I got serveral questions about auto.offset.reset. This configuration parameter governs how consumer read the message from Kafka when there is no initial offset in ZooKeeper or if an offset is out of range. Q1. no initial offset in zookeeper means that there isn't any consumer to consume the message yet(The offset is set once the consumer starts to consume)? Yes, or if you consumed messages, but auto offset commit is disabled and you haven't explicitly committed any offsets. Q2: What does offset is out of range mean? Can you eleborate one scenario when offset is out of range could happen? Kafka uses a retention policy for topics to expire data and clean it up. If some messages expire and your consumer hasn't run in awhile, the last committed offset may no longer exist. auto.offset.reset has two values:smallest and largest. Assume one scenario: A producer has produced 10 messages to kafka, and there is no consumer yet to consume it. Q3: If auto.offset.reset is set to smallest, does it mean that the consumer will read the message from the offset 0?(0 is smallest here) Yes. Q4: If auto.offset.reset is set to largest, does it mean that the consumer will not read any message but wait until new messages come? Also correct. This is why in the quickstart you need to use the --from-beginning flag on the console consumer. Since the consumer is executed after the console producer it wouldn't see any messages unless it set auto.offset.reset to smallest, which is what --from-beginning does. bit1...@163.com -- Thanks, Ewen
Re: serveral questions about auto.offset.reset
What to do when there is no initial offset in ZooKeeper or if an offset is out of range I personally find the name auto.offset.reset to be somewhat confusing. That's mostly because I only knew of it as the no initial offset setting. An alternate name could be auto.offset.initial, to handle the no initial offset in Zookeeper case. But that doesn't describe the offset out of range case. -James On Apr 13, 2015, at 10:28 PM, Ewen Cheslack-Postava e...@confluent.io wrote: On Mon, Apr 13, 2015 at 10:10 PM, bit1...@163.com bit1...@163.com wrote: Hi, Kafka experts: I got serveral questions about auto.offset.reset. This configuration parameter governs how consumer read the message from Kafka when there is no initial offset in ZooKeeper or if an offset is out of range. Q1. no initial offset in zookeeper means that there isn't any consumer to consume the message yet(The offset is set once the consumer starts to consume)? Yes, or if you consumed messages, but auto offset commit is disabled and you haven't explicitly committed any offsets. Q2: What does offset is out of range mean? Can you eleborate one scenario when offset is out of range could happen? Kafka uses a retention policy for topics to expire data and clean it up. If some messages expire and your consumer hasn't run in awhile, the last committed offset may no longer exist. auto.offset.reset has two values:smallest and largest. Assume one scenario: A producer has produced 10 messages to kafka, and there is no consumer yet to consume it. Q3: If auto.offset.reset is set to smallest, does it mean that the consumer will read the message from the offset 0?(0 is smallest here) Yes. Q4: If auto.offset.reset is set to largest, does it mean that the consumer will not read any message but wait until new messages come? Also correct. This is why in the quickstart you need to use the --from-beginning flag on the console consumer. Since the consumer is executed after the console producer it wouldn't see any messages unless it set auto.offset.reset to smallest, which is what --from-beginning does. bit1...@163.com -- Thanks, Ewen
Re: serveral questions about auto.offset.reset
On Mon, Apr 13, 2015 at 10:10 PM, bit1...@163.com bit1...@163.com wrote: Hi, Kafka experts: I got serveral questions about auto.offset.reset. This configuration parameter governs how consumer read the message from Kafka when there is no initial offset in ZooKeeper or if an offset is out of range. Q1. no initial offset in zookeeper means that there isn't any consumer to consume the message yet(The offset is set once the consumer starts to consume)? Yes, or if you consumed messages, but auto offset commit is disabled and you haven't explicitly committed any offsets. Q2: What does offset is out of range mean? Can you eleborate one scenario when offset is out of range could happen? Kafka uses a retention policy for topics to expire data and clean it up. If some messages expire and your consumer hasn't run in awhile, the last committed offset may no longer exist. auto.offset.reset has two values:smallest and largest. Assume one scenario: A producer has produced 10 messages to kafka, and there is no consumer yet to consume it. Q3: If auto.offset.reset is set to smallest, does it mean that the consumer will read the message from the offset 0?(0 is smallest here) Yes. Q4: If auto.offset.reset is set to largest, does it mean that the consumer will not read any message but wait until new messages come? Also correct. This is why in the quickstart you need to use the --from-beginning flag on the console consumer. Since the consumer is executed after the console producer it wouldn't see any messages unless it set auto.offset.reset to smallest, which is what --from-beginning does. bit1...@163.com -- Thanks, Ewen
