This would be a great contribution, Mingmin! As a general rule, we'd like the connector to work with as many versions as possible, with as little code duplication as possible.
Slightly orthogonal -- BigtableIO [1] is an example of a connector that separates API portions from the underlying service. With a similar layering, perhaps KafkaIO can support multiple versions of Kafka, while maintaining as much of the common code as possible. KafkaIO case is a bit more complicated, given that it needs changes on the API side as well. [1] https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable On Mon, Feb 6, 2017 at 12:21 PM, Raghu Angadi <[email protected]> wrote: > I see. kafka-clients dependency could also be in 'provided' scope so that > is simpler to use different versions at runtime. > > On Mon, Feb 6, 2017 at 12:05 PM, Xu Mingmin <[email protected]> wrote: > > > The one I meet is external authentication added in 0.10, we use a > > standalone token-based security service. In 0.9 the SASL-based > > implementation is fixed with Kerberos. > > Kafka client 0.10 cannot connect to Kafka server 0.9, that's why I > mention > > a separated project. > > > > Mingmin > > > > On Mon, Feb 6, 2017 at 11:45 AM, Raghu Angadi <[email protected] > > > > wrote: > > > > > Current KafkaIO works just fine with Kafka 0.10. I don't know of any > > > incompatibilities or regressions. > > > > > > It does not take advantage of message timestamps, of course. It would > be > > > good to take handle tme in in a backward compatible way.. it might be > > > required anyway if they are optional in 0.10. > > > > > > Not sure of scope of (1) below. I don't think it needs to be a new > > > implementation. > > > > > > On Mon, Feb 6, 2017 at 11:35 AM, Xu Mingmin <[email protected]> > wrote: > > > > > > > Hello, > > > > > > > > Is there anybody working on https://issues.apache.org/ > > > jira/browse/BEAM-307 > > > > ? > > > > The existing KafkaIO is implemented with Kafka 0.9, and not > compatible > > > well > > > > with Kafka 0.10. > > > > > > > > I'd like to take this task if not duplicated: > > > > 1). a new KafkaIO based on Kafka 0.10, suggest a separated project > for > > > > easy-to-build; > > > > 2). use timestamp of Kafka message as default event-timestamp > > > > > > > > Thanks! > > > > Mingmin > > > > > > > > > >
