Maven dependecy problem about spark-streaming-kafka_2.11:1.6.3

Hi, our project includes this dependency by: org.apache.spark spark-streaming-kafka_2.11 1.6.3 From dependency tree, we can see it dependency kafka_2.11:0.8.2.1 verson. [cid:part1.8B915977.629F799E@outlook.com] But when we move this dependency to parent pom file, the dependency

Re: Having problem with Spark streaming with Kinesis

Thanks Aniket , clears a lot of confusion.  On Dec 14, 2014 7:11 PM, Aniket Bhatnagar aniket.bhatna...@gmail.com wrote: The reason is because of the following code: val numStreams = numShards val kinesisStreams = (0 until numStreams).map { i = KinesisUtils.createStream(ssc, streamName,

Re: Having problem with Spark streaming with Kinesis

The reason is because of the following code: val numStreams = numShards val kinesisStreams = (0 until numStreams).map { i = KinesisUtils.createStream(ssc, streamName, endpointUrl, kinesisCheckpointInterval, InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2) } In the above

Re: Having problem with Spark streaming with Kinesis

Thanks Aniket, The trick is to have the #workers = #shards + 1. But I don’t know why is that. http://spark.apache.org/docs/latest/streaming-kinesis-integration.html Here in the figure[spark streaming kinesis architecture], it seems like one node should be able to take on more than one shards.

Re: Having problem with Spark streaming with Kinesis

Guys, In my local machine it consumes a stream of Kinesis with 3 shards. But in EC2 it does not consume from the stream. Later we found that the EC2 machine was of 2 cores and my local machine was of 4 cores. I am using a single machine and in spark standalone mode. And we got a larger machine

Having problem with Spark streaming with Kinesis

Hi guys, When we are using Kinesis with 1 shard then it works fine. But when we use more that 1 then it falls into an infinite loop and no data is processed by the spark streaming. In the kinesis dynamo DB, I can see that it keeps increasing the leaseCounter. But it do start processing. I am

Re: Having problem with Spark streaming with Kinesis

I have it working without any issues (tried with 5 shrads), except my java version was 1.7. Here's the piece of code that i used. System.setProperty(AWS_ACCESS_KEY_ID, this.kConf.getOrElse(access_key, )) System.setProperty(AWS_SECRET_KEY, this.kConf.getOrElse(secret, )) val streamName

Re: Having problem with Spark streaming with Kinesis

What's your cluster size? For streamig to work, it needs shards + 1 executors. On Wed, Nov 26, 2014, 5:53 PM A.K.M. Ashrafuzzaman ashrafuzzaman...@gmail.com wrote: Hi guys, When we are using Kinesis with 1 shard then it works fine. But when we use more that 1 then it falls into an infinite

Re: Having problem with Spark streaming with Kinesis

Did you set spark master as local[*]? If so, then it means that nunber of executors is equal to number of cores of the machine. Perhaps your mac machine has more cores (certainly more than number of kinesis shards +1). Try explicitly setting master as local[N] where N is number of kinesis shards

Re: Problem in Spark Streaming

] [Times: user=0.12 sys=0.01, real=0.02 secs] Can anyone help me to understand these messgaes related to GC ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7384.html Sent from the Apache Spark User List mailing list archive

Re: Problem in Spark Streaming

: user=0.12 sys=0.01, real=0.02 secs] Can anyone help me to understand these messgaes related to GC ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7384.html Sent from the Apache Spark User List mailing list archive

Problem in Spark Streaming

.nabble.com/Problem-in-Spark-Streaming-tp7310.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Problem in Spark Streaming

Hi Nilmish, I confront the same problem. I am wondering how do you measure the latency? Regards, Yingjun -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7311.html Sent from the Apache Spark User List mailing list archive

Re: Problem in Spark Streaming

You can measure the latency from the logs. Search for words like Total delay in the logs. This denotes the total end to end delay for a particular query. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7312.html Sent from

Re: Problem in Spark Streaming

-in-Spark-Streaming-tp7310p7321.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Problem in Spark Streaming

for some instant and then it drops down again to the normal level. I want to get away with these spikes in between. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7325.html Sent from the Apache Spark User List mailing list

Re: Problem in Spark Streaming

can't explain it either. I'm not sure if the cause is the same as yours. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7327.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Problem in Spark Streaming

in 14 ms on (progress: 1/6) --what is TID? and what is the progress? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7329.html Sent from the Apache Spark User List mailing list archive at Nabble.com.