[jira] [Commented] (KAFKA-6547) group offset reset and begin_offset ignored/no effect
[
https://issues.apache.org/jira/browse/KAFKA-6547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358415#comment-16358415
]
Dan commented on KAFKA-6547:
I think I've eliminated my client from the problem definition. I start the
client first to get the group started, then I stop it. Describe shows:
{code:java}
kafka@kafka-server-1:~$ KAFKA_OPTS="-Xmx128M
-Djava.security.auth.login.config=./config/kafka_client_jaas.conf"
kafka-consumer-groups.sh --command-config ./config/consumer.properties
--bootstrap-server kafka-server-1:9092 --group meta-printer --describe
Note: This will not show information about old Zookeeper-based consumers.
Consumer group 'meta-printer' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST
CLIENT-ID
payload-meta 0 836464 1060485
224021 - -
-
{code}
Then I run my client for about 20s, with an offset commit frequency of 10s.
Here is a describe after that:
{code:java}
kafka@kafka-server-1:~$ KAFKA_OPTS="-Xmx128M
-Djava.security.auth.login.config=./config/kafka_client_jaas.conf"
kafka-consumer-groups.sh --command-config ./config/consumer.properties
--bootstrap-server kafka-server-1:9092 --group meta-printer --describe
Note: This will not show information about old Zookeeper-based consumers.
Consumer group 'meta-printer' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST
CLIENT-ID
payload-meta 0 974424 1060485 86061
- -
-
{code}
Now I'm going to try to reset the group to the earliest available:
{code:java}
kafka@kafka-server-1:~$ KAFKA_OPTS="-Xmx128M
-Djava.security.auth.login.config=./config/kafka_client_jaas.conf"
kafka-consumer-groups.sh --command-config ./config/consumer.properties
--bootstrap-server kafka-server-1:9092 --group meta-printer --reset-offsets
--execute --to-earliest --topic payload-meta
Note: This will not show information about old Zookeeper-based consumers.
TOPIC PARTITION NEW-OFFSET
payload-meta 0 0
{code}
And then a subsequent describe:
{code:java}
kafka@kafka-server-1:~$ KAFKA_OPTS="-Xmx128M
-Djava.security.auth.login.config=./config/kafka_client_jaas.conf"
kafka-consumer-groups.sh --command-config ./config/consumer.properties
--bootstrap-server kafka-server-1:9092 --group meta-printer --describe
Note: This will not show information about old Zookeeper-based consumers.
Consumer group 'meta-printer' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST
CLIENT-ID
payload-meta 0 974424 1060485 86061
- -
-
{code}
Unchanged. The default server retention is forever, so no log should be
pruned. But even if there was pruning, a reset to earliest should be able to at
least get the offset back to the first offset shown above. The problem is not
present in 0.11. I you want output of that, I'll have to set it up.
Am I misunderstanding what should be happening here?
Many thanks.
> group offset reset and begin_offset ignored/no effect
> -
>
> Key: KAFKA-6547
> URL: https://issues.apache.org/jira/browse/KAFKA-6547
> Project: Kafka
> Issue Type: Bug
> Components: offset manager
>Affects Versions: 1.0.0
> Environment: ubuntu 16, java 1.8
>Reporter: Dan
>Priority: Major
> Fix For: 0.11.0.2
>
>
> Use of kafka-consumer-group.sh with --reset-offsets --execute <--to-earliest
> or anything> has no effect in 1.0. When my group client connects and requests
> a specific offset or an earliest there's no effect and the consumer is unable
> to poll, so no messages, even new ones are ignored.
> I installed 0.11 and these problems are not manifest.
> I'm unfamiliar with the internals and put the offset manager as the possible
> component, but that's a guess.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-6547) group offset reset and begin_offset ignored/no effect
Dan created KAFKA-6547: -- Summary: group offset reset and begin_offset ignored/no effect Key: KAFKA-6547 URL: https://issues.apache.org/jira/browse/KAFKA-6547 Project: Kafka Issue Type: Bug Components: offset manager Affects Versions: 1.0.0 Environment: ubuntu 16, java 1.8 Reporter: Dan Fix For: 0.11.0.2 Use of kafka-consumer-group.sh with --reset-offsets --execute <--to-earliest or anything> has no effect in 1.0. When my group client connects and requests a specific offset or an earliest there's no effect and the consumer is unable to poll, so no messages, even new ones are ignored. I installed 0.11 and these problems are not manifest. I'm unfamiliar with the internals and put the offset manager as the possible component, but that's a guess. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.
[
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106965#comment-16106965
]
Dan commented on KAFKA-2729:
Happened in 0.11.0.0 as well. Had to restart the broker to bring it back to
operational state.
> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other,
> we started seeing a large number of undereplicated partitions. The zookeeper
> cluster recovered, however we continued to see a large number of
> undereplicated partitions. Two brokers in the kafka cluster were showing this
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66]
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered
> after a restart. Our own investigation yielded nothing, I was hoping you
> could shed some light on this issue. Possibly if it's related to:
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using
> 0.8.2.1.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5587) Processor got uncaught exception: NullPointerException
Dan created KAFKA-5587: -- Summary: Processor got uncaught exception: NullPointerException Key: KAFKA-5587 URL: https://issues.apache.org/jira/browse/KAFKA-5587 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.10.1.1 Reporter: Dan [2017-07-12 21:56:39,964] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.NullPointerException at kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:490) at kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:487) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.network.Processor.processCompletedReceives(SocketServer.scala:487) at kafka.network.Processor.run(SocketServer.scala:417) at java.lang.Thread.run(Thread.java:745) Anyone knows the cause of this exception? What's the effect of it? When this exception occurred, the log also showed that the broker was frequently shrinking ISR to itself. Are these two things interrelated? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
