Re: Kafka 0.8.x / 0.9.x support in structured streaming
I haven't done Structured Streaming in Spark 2.1 with Kafka 0.9.x, but I did do stream processing with Spark 2.0.1 and Kafka 0.10. Here's the official documenation that I used for Spark Streaming with Kafka 0.10: https://spark.apache.org/docs/2.1.0/streaming-kafka-integration.html. It looks like you might need to use the 0.8.x connector, but I found that connector to be inadequate for some use cases, such as using a regex to subscribe to multiple topics. Your results may vary, and I wouldn't be surprised if there was another source of official documentation on the topic. On Mon, May 15, 2017 at 5:25 AM Swapnil Chougulewrote: > Hello > > I am new to structured streaming. Wanted to learn if there is support for > Kafka 0.8.x or Kafka 0.9.x in structured streaming ? > My Kafka source is of version 0.9.x & want get have structured streaming > solution on top of it. > I checked documentation for Spark release 2.1.0 but didn't get exact info. > Any help is most welcome. Thanks in advance. > > --Swapnil >
Re: running spark program on intellij connecting to remote master for cluster
Do you have Spark installed locally on your laptop with IntelliJ? Are you using the SparkLauncher class or your local spark-submit script? A while back, I was trying to submit a spark job from my local workstation to a remote cluster using the SparkLauncher class, but I didn't actually have SPARK_HOME set or the spark-submit script on my local machine yet, so the submit was failing. I think the error I was getting was that SPARK_HOME environment variable was not set, though. On Wed, May 10, 2017 at 5:51 AM s twrote: > Hello, > > I am trying to run spark code from my laptop with intellij. I have cluster > of 2 nodes and a master. When i start the program from intellij it gets > error of some missing classes. > > I am aware that some jars need to be distributed to the workers but do not > know if it is possible programatically. spark submit or jupyter notebook > handles the issue but intellij does not. > > can any one give some advices to me ? > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >