+ Raghu Angadi Raghu, I guess you did a review of the PR with this feature. Could you shed some light on this, please?
Alexey > On 25 Jan 2021, at 13:59, Dan Mason <daniel.ma...@king.com> wrote: > > Hi Beam Community, > > While preparing a few dataflow streaming pipelines, with Kafka as a source, I > have come across a bit of an issue. Some of the topics I am reading from have > very low throughput, but I hope to utilise the withStartReadTime option to > help control the offset at start up. > > The issue I am facing is related to the hard failure which arises when there > is no data present to consume after setting the withStartReadTime option as > documented here [1]. Draining is blocked while this hard error is occurring > this gives false alerts in our monitoring to detect failing jobs. The use of > multiple topics is also problematic as the job will not read from any topic > as long as any one is producing this error. > > I would like to understand why has this been made such a hard error when it > feels a situation pipelines can easily be in, and would there be any > possibility of reducing it to a softer error allowing features such as > draining and multiple topics on these jobs. > > Thanks for any help understanding this issue, > > Dan > > [1] > https://beam.apache.org/releases/javadoc/2.27.0/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html#withStartReadTime-org.joda.time.Instant- > > <https://beam.apache.org/releases/javadoc/2.27.0/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html#withStartReadTime-org.joda.time.Instant->
