SPARK_SUBMIT_CLASSPATH Windows PYSPARK

2015-04-28 Thread art1i
Hi, So I was trying to get the Kafka Streaming working in a standalone python application. I needed to add the dependencies for this to work. The suggested way for this is doing --jars using spark-submit, which is not practical considering I wanted to launch and debug an application. Also you

How to deploy self-build spark source code on EC2

2015-04-28 Thread Bo Fu
Hi all, I have an issue. I added some timestamps in Spark source code and built it using: mvn package -DskipTests I checked the new version in my own computer and it works. However, when I ran spark on EC2, the spark code EC2 machines ran is the original version. Anyone knows how to deploy

Re: Spark SQL 1.3.1 saveAsParquetFile will output tachyon file with different block size

2015-04-28 Thread sara mustafa
Hi Zhang, How did you compile Spark 1.3.1 with Tachyon? when i changed Tachyon version to 0.6.3 in core/pom.xml, make-distribution.sh and try to compile again, many compilation errors raised. Thanks, -- View this message in context:

Re: Spark SQL 1.3.1 saveAsParquetFile will output tachyon file with different block size

2015-04-28 Thread Calvin Jia
Hi, You can apply this patch https://github.com/apache/spark/pull/5354 and recompile. Hope this helps, Calvin On Tue, Apr 28, 2015 at 1:19 PM, sara mustafa eng.sara.must...@gmail.com wrote: Hi Zhang, How did you compile Spark 1.3.1 with Tachyon? when i changed Tachyon version to 0.6.3 in

Re: creating hive packages for spark

2015-04-28 Thread Manku Timma
Yash, This is exactly what I wanted! Thanks a bunch. On 27 April 2015 at 15:39, yash datta sau...@gmail.com wrote: Hi, you can build spark-project hive from here : https://github.com/pwendell/hive/tree/0.13.1-shaded-protobuf Hope this helps. On Mon, Apr 27, 2015 at 3:23 PM, Manku Timma

Re:Re: java.lang.StackOverflowError when recovery from checkpoint in Streaming

2015-04-28 Thread wyphao.2007
Hi Akhil Das, Thank you for your reply. It is very similar to my problem, I will focus on it. Thanks Best Regards At 2015-04-28 18:08:32,Akhil Das ak...@sigmoidanalytics.com wrote: There's a similar issue reported over here https://issues.apache.org/jira/browse/SPARK-6847 Thanks Best Regards On

Re: java.lang.StackOverflowError when recovery from checkpoint in Streaming

2015-04-28 Thread Akhil Das
There's a similar issue reported over here https://issues.apache.org/jira/browse/SPARK-6847 Thanks Best Regards On Tue, Apr 28, 2015 at 7:35 AM, wyphao.2007 wyphao.2...@163.com wrote: Hi everyone, I am using val messages = KafkaUtils.createDirectStream[String, String, StringDecoder,