Hi!
I have a job that I would like to run in batch that reads all contents of a
Kafka log.

To do this I need to use something like

```
KafkaIO.read()
    .updateConsumerProperties(ImmutableMap.of(
        ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"))
    .withMaxReadTime(Duration.standardMinutes(10))
```

which requires me to on a hunch know how long it takes to read the whole
log. It feels a bit flakey as this might change over time.

The other alternative as I understand it it to use `withMaxNumRecords(...)`
but that makes even less sense in my case as I do not know the number of
records in the log.

So my question is, is there any way I can read up to the current offsets
that the log is on when the job start or similar instead?

/ Vilhelm

Reply via email to