Re: When will Spark Streaming supports Kafka-simple consumer API?

2015-02-05 Thread Xuelin Cao.2015
Hi, Tathagata Thanks for the information, I'm trying to build 1.3 snapshot and make another try. There are 2 reasons for why we use Kafka SimpleConsumer API 1. Previously, in our company, all of the real time processing system were build on Apache Storm. So, the kafka environment i

Re: When will spark support "push" style shuffle?

2015-01-07 Thread Xuelin Cao.2015
Got it. The explain makes sense. Thank you. On Thu, Jan 8, 2015 at 1:06 PM, Patrick Wendell [via Apache Spark Developers List] wrote: > This question is conflating a few different concepts. I think the main > question is whether Spark will have a shuffle implementation that > streams data rathe