Scenario: A kafka + zookeeper setup with topics that span partitions. Obviously high level consumer is desirable here.
But let's say we have a need to consume by offset, and we don't want the earliest offset as that has extra messages we'd have to filter out, and from latest offset doesn't have the messages we desire. And for simplicity, we don't want to muck with changing high level consumer group offset (e.g. reset it to start from some offset, etc.). Is there a way to query (figure out) based on kafka/zookeeper metadata and/or the produced message (response) what given partition and offset the message is at? Such that I can then create a simple consumer to fetch message(s) off that partition from that offset. And script this to work all dynamically, so that as I send messages, I know where to consume from with simple consumer, even if say partitions change. Initially, I'm looking at this from just the kafka shell script utilities but later migrating to Ruby and the kafka libraries available there. Ideally of course, I'd just wish for a high level consumer that could consume by some offset (besides earliest/latest and without mucking with anything).
