Hi. You should be able to do this with the storm-kafka-client spout and a sufficiently new Kafka cluster (0.11, see the documentation at https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling). Kafka ships with the kafka/bin/kafka-consumer-groups.sh script. You can use this script with the --reset-offsets option to reset offsets for a consumer group based on a bunch of conditions (timestamp, offset and others).
Note that I haven't tried to do this, but I'd imagine you'd need to do something like this: * Ensure your kafka spout is configured to use the UNCOMMITTED_EARLIEST or UNCOMMITTED_LATEST as its FirstPollOffsetStrategy. * Find the consumer group your spout is running as. * Stop (kill) the topology containing the spout you want to reset. You need to do this to avoid having the spout commit offsets while you're working with the kafka-consumer-groups tool. If you reset the offset while the spout is running, it will ignore the reset. * Use kafka-consumer-groups.sh to reset the offsets for your consumer group. Reference the KIP link to see how to do this, or just run kafka-consumer-groups.sh to get it to print usage. * Redeploy your topology 2017-07-26 11:24 GMT+02:00 Elyahou Ittah <[email protected]>: > Is there an option to set the spout kafka offset to a specific offset ? > > It can be very usefull for replaying kafka log from specific time or > offset. > > Thanks >
