Re: Spark Streaming Kafka - DirectKafkaInputDStream: Using the new Kafka Consumer API

[Mario Ds Briggs]

sounds sane for a first cut.

Since all creation methods take a KafkaParams, i was thinking along lines
of maybe  a temp property in there which trigger usage of new consumer.

thanks
Mario



From:   Cody Koeninger <c...@koeninger.org>
To:     Mario Ds Briggs/India/IBM@IBMIN
Cc:     "dev@spark.apache.org" <dev@spark.apache.org>
Date:   04/12/2015 08:45 pm
Subject:        Re: Spark Streaming Kafka - DirectKafkaInputDStream: Using the
            new Kafka Consumer API



Brute force way to do it might be to just have a separate
streaming-kafka-new-consumer subproject, or something along those lines.

On Fri, Dec 4, 2015 at 3:12 AM, Mario Ds Briggs <mario.bri...@in.ibm.com>
wrote:
  >>
  forcing people on kafka 8.x to upgrade their brokers is questionable.
  <<

  I agree and i was more thinking maybe there is a way to support both for
  a period of time (of course means some more code to maintain :-)).


  thanks
  Mario

  Inactive hide details for Cody Koeninger ---04/12/2015 12:15:55
  am---Honestly my feeling on any new API is to wait for a point Cody
  Koeninger ---04/12/2015 12:15:55 am---Honestly my feeling on any new API
  is to wait for a point release before taking it seriously :)

  From: Cody Koeninger <c...@koeninger.org>
  To: Mario Ds Briggs/India/IBM@IBMIN
  Cc: "dev@spark.apache.org" <dev@spark.apache.org>
  Date: 04/12/2015 12:15 am
  Subject: Re: Spark Streaming Kafka - DirectKafkaInputDStream: Using the
  new Kafka Consumer API



  Honestly my feeling on any new API is to wait for a point release before
  taking it seriously :)

  Auth and encryption seem like the only compelling reason to move, but
  forcing people on kafka 8.x to upgrade their brokers is questionable.

  On Thu, Dec 3, 2015 at 11:30 AM, Mario Ds Briggs <mario.bri...@in.ibm.com
  > wrote:
        Hi,

        Wanted to pick Cody's mind on what he thinks about
        DirectKafkaInputDStream/KafkaRDD internally using the new Kafka
        consumer API. I know the latter is documented as beta-quality, but
        yet wanted to know if he sees any blockers as to why shouldn't go
        there shortly. On my side the consideration is that kafka 0.9.0.0
        introduced Authentication and Encryption (beta again) between
        clients & brokers, but this is available only newer Consumer API's
        and not in the older Low-level/High-level API's.

        From briefly studying the implementation of
        DirectKafkaInputDStream/KafkaRDD and new Consumer API, my thinking
        is that it is possible to support the exact current implementation
        you have using the new API's.
        One area that isnt so straightforward was the ctor of KafkaRDD
        fixes the offsetRange (I did read about the deterministic feature
        you were after) and i couldnt find a direct method in the new
        Consumer API to get the current 'latest' offset - however one can
        do a consumer.seekToEnd() and then call a consumer.position().
        Of course one other benefit is that the new Consumer API's
        abstracts away having to deal with finding the leader for a
        partition, so can get rid of that code

        Would be great to get your thoughts.

        thanks in advance
        Mario