Yes, spark-streaming-kafka-0-10 uses the new consumer.   Besides
pre-fetching messages, the big reason for that is that security
features are only available with the new consumer.

The Kafka project is at release 0.10.0.1 now, they think most of the
issues with the new consumer have been ironed out.  You can track the
progress as to when they'll remove the "beta" label at
https://issues.apache.org/jira/browse/KAFKA-3283

As far as I know, Kafka in general can't achieve sub-millisecond
end-to-end stream processing, so my guess is you need to be more
specific about your terms there.

I promise I'm not trying to start a pissing contest :)  just wanted to
check if you were aware of the current state of the other consumers.
Collaboration is always welcome.


On Tue, Aug 23, 2016 at 10:18 PM, Jeoffrey Lim <[email protected]> wrote:
> Apologies, I was not aware that Spark 2.0 has Kafka Consumer caching/pooling
> now.
> What I have checked is the latest Kafka Consumer, and I believe it is still
> in beta quality.
>
> https://kafka.apache.org/documentation.html#newconsumerconfigs
>
>> Since 0.9.0.0 we have been working on a replacement for our existing
>> simple and high-level consumers.
>> The code is considered beta quality.
>
> Not sure about this, does Spark 2.0 Kafka 0.10 integration already uses this
> one? Is it now stable?
> With this caching feature in Spark 2,.0 could it achieve sub-milliseconds
> stream processing now?
>
>
> Maelstrom still uses the old Kafka Simple Consumer, this library was made
> open source so that I
> could continue working on it for future updates & improvements like when the
> latest Kafka Consumer
> gets a stable release.
>
> We have been using Maelstrom "caching concept" for a long time now, as
> Receiver based Spark Kafka integration
> does not work for us. There were thoughts about using Direct Kafka APIs,
> however Maelstrom has
> very simple APIs and just "simply works" even under unstable scenarios (e.g.
> advertised hostname failures on EMR).
>
> Maelstrom will work I believe even for Spark 1.3 and Kafka 0.8.2.1 (and of
> course with the latest Kafka 0.10 as well)
>
>
> On Wed, Aug 24, 2016 at 9:49 AM, Cody Koeninger <[email protected]> wrote:
>>
>> Were you aware that the spark 2.0 / kafka 0.10 integration also reuses
>> kafka consumer instances on the executors?
>>
>> On Tue, Aug 23, 2016 at 3:19 PM, Jeoffrey Lim <[email protected]> wrote:
>> > Hi,
>> >
>> > I have released the first version of a new Kafka integration with Spark
>> > that we use in the company I work for: open sourced and named Maelstrom.
>> >
>> > It is unique compared to other solutions out there as it reuses the
>> > Kafka Consumer connection to achieve sub-milliseconds latency.
>> >
>> > This library has been running stable in production environment and has
>> > been proven to be resilient to numerous production issues.
>> >
>> >
>> > Please check out the project's page in github:
>> >
>> > https://github.com/jeoffreylim/maelstrom
>> >
>> >
>> > Contributors welcome!
>> >
>> >
>> > Cheers!
>> >
>> > Jeoffrey Lim
>> >
>> >
>> > P.S. I am also looking for a job opportunity, please look me up at
>> > Linked In
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]

Reply via email to