AndrewJSchofield commented on PR #22197:
URL: https://github.com/apache/kafka/pull/22197#issuecomment-4706101183
> Thanks for the PR and feature @AndrewJSchofield. I tested it and have some
questions. I ll test once again with more scenarios once you help with below
questions
>
> 1. Should we have share groups output for reset-offsets on consumer-group
cli as well? Also, I see a difference when the offsets are fetched from
consumer-groups.sh vs share-groups.sh
>
> ```
> kafka git:(KAFKA-20524) bin/kafka-consumer-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --to-current --export
--dry-run --topic T1
> T1,0,10
>
> ➜ kafka git:(KAFKA-20524) bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --to-current --export
--dry-run --topic T1
> T1,0,9
> ```
>
> When there was only a share group and no consumer group.
>
> ```
> kafka git:(KAFKA-20524) ./bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --list
> grp_1_share
> ➜ kafka git:(KAFKA-20524) ./bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --list
> ```
>
> 2. I reset offset to 2 and started console share consumer again which
consumed till offset 10 but start offset didn't move while describing and
neither when I exported the offsets. I stopped consumer and restarted but it
didn't re-consumed messages from offset 2 rather keep waiting for new data,
which was correct. But the outputs were incorrect.
>
> ```
> kafka git:(KAFKA-20524) bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --topic T1:0 --to-offset 2
--execute
>
> GROUP TOPIC PARTITION NEW-OFFSET
> grp_1_share T1 0 2
> ```
>
> Consumed from offset 2:
>
> ```
> kafka git:(KAFKA-20524) ./bin/kafka-console-share-consumer.sh
--bootstrap-server localhost:9092 --topic T1 --group grp_1_share
> 3
> 4
> 5
> 6
> 7
> 8
> 9
> 10
> ^CProcessed a total of 8 messages
> ```
>
> But listed start offset incorrectly:
>
> ```
> kafka git:(KAFKA-20524) ./bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --group grp_1_share --describe
>
> GROUP TOPIC PARTITION START-OFFSET LAG
> grp_1_share T1 0 2
0
> ```
>
> Output of export was incorrect as well:
>
> ```
> kafka git:(KAFKA-20524) bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --to-current --export
--dry-run --topic T1
> T1,0,2
> ```
>
> Restarted and it didn't consume any old message from offset 2.
>
> ```
> ➜ kafka git:(KAFKA-20524) ./bin/kafka-console-share-consumer.sh
--bootstrap-server localhost:9092 --topic T1 --group grp_1_share
> ```
>
> Once produced another message and checked the output as new message was
consumed by console consumer then the start offset and export displayed correct
data.
>
> 3. Resetting offsets from unknown file provides complete stack trace with
correct error. However do we need the stack trace or just the right error as
string?
>
> ```
> kafka git:(KAFKA-20524) bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --from-file a --execute
>
> Error: Executing share group command failed due to java.io.IOException:
Unable to read file a
> java.lang.RuntimeException: java.io.IOException: Unable to read file a
> at
org.apache.kafka.tools.GroupOffsetsResetter.resetPlanFromFile(GroupOffsetsResetter.java:105)
> at
org.apache.kafka.tools.GroupOffsetsResetter.resetFromFile(GroupOffsetsResetter.java:389)
> at
org.apache.kafka.tools.consumer.group.ShareGroupCommand$ShareGroupService.prepareOffsetsToReset(ShareGroupCommand.java:535)
> at
org.apache.kafka.tools.consumer.group.ShareGroupCommand$ShareGroupService.resetOffsetsForInactiveGroup(ShareGroupCommand.java:465)
> at
org.apache.kafka.tools.consumer.group.ShareGroupCommand$ShareGroupService.resetOffsets(ShareGroupCommand.java:439)
> at
org.apache.kafka.tools.consumer.group.ShareGroupCommand.run(ShareGroupCommand.java:104)
> at
org.apache.kafka.tools.consumer.group.ShareGroupCommand.main(ShareGroupCommand.java:89)
> Caused by: java.io.IOException: Unable to read file a
> at org.apache.kafka.common.utils.Utils.readFileAsString(Utils.java:771)
> at
org.apache.kafka.tools.GroupOffsetsResetter.resetPlanFromFile(GroupOffsetsResetter.java:101)
> ... 6 more
> Caused by: java.nio.file.NoSuchFileException: a
> at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
> at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:262)
> at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
> at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
> at java.base/java.nio.file.Files.readAllBytes(Files.java:3292)
> at org.apache.kafka.common.utils.Utils.readFileAsString(Utils.java:768)
> ... 7 more
> ```
>
> 4. What's the difference between `--dry-run` vs `--execute` when we are
exporting the offsets with `--export`. Both seems to do the same thing, do we
need both? I somewhere remember we discussed it before but was not sure hence
thought to write.
>
> ```
> ➜ kafka git:(KAFKA-20524) bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --to-current --export
--dry-run --topic T1
> T1,0,10
>
> ➜ kafka git:(KAFKA-20524) bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --to-current --export
--execute --topic T1
> T1,0,10
> ```
>
> 5. While running the same commands from 4 using kafka-consumer-groups.sh
but with share group then for --execute there is the correct error but with
--dry-run it succeeds:
>
> ```
> ➜ kafka git:(KAFKA-20524) bin/kafka-consumer-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --to-current --export
--execute --topic T1
>
> Error: Executing consumer group command failed due to Failed altering
group offsets for the following partitions: [T1-0]
> org.apache.kafka.common.errors.GroupIdNotFoundException: Failed altering
group offsets for the following partitions: [T1-0]
>
> ➜ kafka git:(KAFKA-20524) bin/kafka-consumer-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --to-current --export
--dry-run --topic T1
> T1,0,11
> ```
>
> 6. After resetting the offset there is a difference in output from
different clis:
>
> ```
> ➜ kafka git:(KAFKA-20524) ✗ bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --from-file a --execute
>
> GROUP TOPIC PARTITION NEW-OFFSET
> grp_1_share T1 0 2
> ➜ kafka git:(KAFKA-20524) ✗ bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --reset-offsets --group grp_1_share
--to-current --export --dry-run --topic T1
> T1,0,11
>
> ➜ kafka git:(KAFKA-20524) ✗ bin/kafka-share-groups.sh --bootstrap-server
localhost:9092 --reset-offsets --group grp_1_share --to-current --export
--dry-run --topic T1
> T1,0,2
> ```
Thanks for the review @apoorvmittal10.
1. When you use `--dry-run`, the tools do not actually make changes to the
group. It is only when you attempt the `--execute` on a consumer group when
that group ID is already taken by a share group that the
`bin/kafka-consumer-groups.sh` tool complains.
2. This is because share groups lazily update the start offset. That's all
you are seeing here.
3. This code is shared with the `bin/kafka-consumer-groups.sh` tool. If
you'd like me to change the exception handling for them both, let me know.
Personally, I don't mind that much because it is easy enough to tell what's
wrong and I prefer not to update existing code unnecessarily because of the
risk of introducing unforeseen problems.
4. The `--export` option just tells the tools (both of them) how to output
the offset information. If you do `--reset-offsets --dry-run --export`, the
group will not be reset, the offsets information for the reset operation is
displayed, and the output is made as a comma-separated list which can be
redirected to a file. If you do `--reset-offsets --execute --export`, the group
will be reset, the offsets information for the reset operation is displayed,
and the output is made as a comma-separated list which can be redirected to a
file.
What I'd like to do here is open a new KIP which changes how the export
works for all group types. It would:
* Introduce `--export-offsets` as a new top-level option. This just obtains
the offsets and outputs as a comma-separated list.
* Deprecate `--export` from `--reset-offsets` in AK 4.5, to be removed in AK
5.0.
5. That's because `--dry-run` checks whether there is a non-empty group of
the same type as the command being used. This means you can export offsets
using the consumer group tool when there is an active share group, and you can
export offsets using the share group tool when there is an active consumer
group. The first of these is a pre-existing bug.
6. I think this related to the lack of a group type check. The consumer
group command looks for a consumer group with the provided group ID, doesn't
find one and obtains the information for the topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
