There're some tricky API changes from Kafka client 0.9 to 0.10. Some are
easy to keep compatible,
public void assign(List<TopicPartition> partitions) --> public void
assign(Collection<TopicPartition> partitions) )
while some are not,
public void seekToBeginning(TopicPartition... partitions); -> public
void seekToBeginning(Collection<TopicPartition> partitions);.
public void seekToEnd(TopicPartition... partitions); -> public void
seekToEnd(Collection<TopicPartition> partitions);
Agree that one single IO is preferred, with provided client libs for
difference versions. Thanks @Bonaci, let me read BigtableIO to find some
help.
Mingmin
On Mon, Feb 6, 2017 at 12:54 PM, Davor Bonaci <[email protected]> wrote:
> This would be a great contribution, Mingmin!
>
> As a general rule, we'd like the connector to work with as many versions as
> possible, with as little code duplication as possible.
>
> Slightly orthogonal -- BigtableIO [1] is an example of a connector that
> separates API portions from the underlying service. With a similar
> layering, perhaps KafkaIO can support multiple versions of Kafka, while
> maintaining as much of the common code as possible. KafkaIO case is a bit
> more complicated, given that it needs changes on the API side as well.
>
> [1]
> https://github.com/apache/beam/tree/master/sdks/java/io/
> google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable
>
> On Mon, Feb 6, 2017 at 12:21 PM, Raghu Angadi <[email protected]>
> wrote:
>
> > I see. kafka-clients dependency could also be in 'provided' scope so that
> > is simpler to use different versions at runtime.
> >
> > On Mon, Feb 6, 2017 at 12:05 PM, Xu Mingmin <[email protected]> wrote:
> >
> > > The one I meet is external authentication added in 0.10, we use a
> > > standalone token-based security service. In 0.9 the SASL-based
> > > implementation is fixed with Kerberos.
> > > Kafka client 0.10 cannot connect to Kafka server 0.9, that's why I
> > mention
> > > a separated project.
> > >
> > > Mingmin
> > >
> > > On Mon, Feb 6, 2017 at 11:45 AM, Raghu Angadi
> <[email protected]
> > >
> > > wrote:
> > >
> > > > Current KafkaIO works just fine with Kafka 0.10. I don't know of any
> > > > incompatibilities or regressions.
> > > >
> > > > It does not take advantage of message timestamps, of course. It
> would
> > be
> > > > good to take handle tme in in a backward compatible way.. it might be
> > > > required anyway if they are optional in 0.10.
> > > >
> > > > Not sure of scope of (1) below. I don't think it needs to be a new
> > > > implementation.
> > > >
> > > > On Mon, Feb 6, 2017 at 11:35 AM, Xu Mingmin <[email protected]>
> > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > Is there anybody working on https://issues.apache.org/
> > > > jira/browse/BEAM-307
> > > > > ?
> > > > > The existing KafkaIO is implemented with Kafka 0.9, and not
> > compatible
> > > > well
> > > > > with Kafka 0.10.
> > > > >
> > > > > I'd like to take this task if not duplicated:
> > > > > 1). a new KafkaIO based on Kafka 0.10, suggest a separated project
> > for
> > > > > easy-to-build;
> > > > > 2). use timestamp of Kafka message as default event-timestamp
> > > > >
> > > > > Thanks!
> > > > > Mingmin
> > > > >
> > > >
> > >
> >
>
