I guess that is the question. [2] and [3] above make me think that this is experimental and just not labeled as such.
It doesn't seem reasonable to have both an open feature request for bounded KafkaIO (BEAM-2185), and a bug report regarding bounded KafkaIO (BEAM-6466). On Fri, Apr 30, 2021 at 11:26 AM Pablo Estrada <[email protected]> wrote: > Are they experimental? I suppose this is a valid use case, right? I am in > favor of adding a warning, but I don't know if I would call them > experimental. > > I suppose a repeated-batch use case may do this repeatedly (though then > users would need to recover the latest offsets for each partition, which I > guess is not possible at the moment?) > > On Thu, Apr 29, 2021 at 4:17 PM Brian Hulette <[email protected]> wrote: > >> Our oldest open P1 issue is BEAM-6466 - "KafkaIO doesn't commit offsets >> while being used as bounded source" [1]. I'm not sure this is an actual >> issue since KafkaIO doesn't seem to officially support this use-case. The >> relevant parameters indicate they are "mainly used for tests and demo >> applications" [2], and BEAM-2185 - "KafkaIO bounded source" [3] is still >> open. >> >> I think we should close out BEAM-6466 by more clearly indicating that >> withMaxReadTime() and withMaxRecords() are experimental, and/or logging a >> warning when they are used. >> >> I'm happy to make such a change, but I wanted to check if there are any >> objections to this first. >> >> Thanks, >> Brian >> >> [1] https://issues.apache.org/jira/browse/BEAM-6466 >> [2] >> https://github.com/apache/beam/blob/3d4db26cfa4ace0a0f2fbb602f422fe30670c35f/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L960 >> [3] https://issues.apache.org/jira/browse/BEAM-2185 >> >
