Not sure what you mean by "no documentation regarding ways to achieve effective communication between the 2", but the docs on integrating with kafka are at http://spark.apache.org/docs/latest/streaming-kafka-integration.html
As far as custom partitioners go, Learning Spark from O'Reilly has a good section on partitioning, or you can grep the spark code for "extends Partitioner" to see examples On Thu, Nov 19, 2015 at 8:58 AM, Sachin Mousli <samou...@gmx.com> wrote: > Hi, > > I would like to implement a custom partitioning algorithm in a streaming > environment, preferably in a generic manner using a single pass. The > only sources I could find mention Apache Kafka which adds a complexity I > would like to avoid since there seem to be no documentation regarding > ways to achieve effective communication between the 2. > > Could you give me pointers in regards to implementing a custom > partitioner with streaming support or redirect me to related sources? > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
