I am writing an exactly-once Kafka sink for Apache Beam.
In order to avoid duplicates due to retries, it stores a sequential id and
producer signature in consumer group metadata, and commits it atomically
with messages (using sendOffsetsToTransaction()).
I have a couple of clarification questions on partition metadata associated
with a consumer group.
- Looks like a partition number could be larger than number of
partitions for a topic. Is this formally supported (which is great!)?
- E.g. I was able to set and read partition metadata for partition
#20 for a topic with 4 partitions (though `console-consumer-group` didn't
quite work.).
- This is critical information for a sink. But looks like the metadata
gets purged in 24 hours (server config) if it is inactive. Is there a way
to set a longer TTL programatically?
Thanks.
Raghu.
