Re: python/run-tests fails at spark master branch

2015-04-21 Thread Saisai Shao 
Hi Hrishikesh,

Seems the behavior of Kafka-assembly is a little different when using Maven
to sbt. The assembly jar name and location is different while using `mvn
package`. This is a actually bug, I'm fixing this now.

Thanks
Jerry


2015-04-22 13:37 GMT+08:00 Hrishikesh Subramonian <
hrishikesh.subramon...@flytxt.com>:

>  Hi,
>
> The *python/run-tests* executes successfully after I ran *'build/sbt
> assembly*' command. But the tests fail if I run it after *'mvn
> -Dskiptests clean package'* command. Why does it run in *sbt assembly*
> and not in* mvn package*?
>
> --
> Hrishikesh
>
> On Wednesday 22 April 2015 07:38 AM, Saisai Shao wrote:
>
> Hi Hrishikesh,
>
>  Now we add Kafka unit test for python which relies on Kafka assembly
> jar, so you need to run `sbt assembly` or mvn `package` at first to get an
> assemble jar.
>
>
>
> 2015-04-22 1:15 GMT+08:00 Marcelo Vanzin :
>
>> On Tue, Apr 21, 2015 at 1:30 AM, Hrishikesh Subramonian
>>  wrote:
>>
>> > Run streaming tests ...
>> > Failed to find Spark Streaming Kafka assembly jar in
>> > /home/xyz/spark/external/kafka-assembly
>> > You need to build Spark with  'build/sbt assembly/assembly
>> > streaming-kafka-assembly/assembly' or 'build/mvn package' before running
>> > this program
>> >
>> >
>> > Is anybody facing the same problem?
>>
>> Have you built the assemblies before running the tests? (mvn package
>> -DskipTests, or sbt assembly)
>>
>>
>> --
>> Marcelo
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>>
>
>

Re: python/run-tests fails at spark master branch

2015-04-21 Thread Hrishikesh Subramonian 

Hi,

The /python/run-tests/ executes successfully after I ran /'build/sbt 
assembly/' command. But the tests fail if I run it after /'mvn 
-Dskiptests clean package'/ command. Why does it run in /sbt assembly/ 
and not in/mvn package/?


--
Hrishikesh

On Wednesday 22 April 2015 07:38 AM, Saisai Shao wrote:

Hi Hrishikesh,

Now we add Kafka unit test for python which relies on Kafka assembly 
jar, so you need to run `sbt assembly` or mvn `package` at first to 
get an assemble jar.




2015-04-22 1:15 GMT+08:00 Marcelo Vanzin >:


On Tue, Apr 21, 2015 at 1:30 AM, Hrishikesh Subramonian
mailto:hrishikesh.subramon...@flytxt.com>> wrote:

> Run streaming tests ...
> Failed to find Spark Streaming Kafka assembly jar in
> /home/xyz/spark/external/kafka-assembly
> You need to build Spark with  'build/sbt assembly/assembly
> streaming-kafka-assembly/assembly' or 'build/mvn package' before
running
> this program
>
>
> Is anybody facing the same problem?

Have you built the assemblies before running the tests? (mvn package
-DskipTests, or sbt assembly)


--
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org

For additional commands, e-mail: dev-h...@spark.apache.org

Re: python/run-tests fails at spark master branch

2015-04-21 Thread Saisai Shao 
Hi Hrishikesh,

Now we add Kafka unit test for python which relies on Kafka assembly jar,
so you need to run `sbt assembly` or mvn `package` at first to get an
assemble jar.



2015-04-22 1:15 GMT+08:00 Marcelo Vanzin :

> On Tue, Apr 21, 2015 at 1:30 AM, Hrishikesh Subramonian
>  wrote:
>
> > Run streaming tests ...
> > Failed to find Spark Streaming Kafka assembly jar in
> > /home/xyz/spark/external/kafka-assembly
> > You need to build Spark with  'build/sbt assembly/assembly
> > streaming-kafka-assembly/assembly' or 'build/mvn package' before running
> > this program
> >
> >
> > Is anybody facing the same problem?
>
> Have you built the assemblies before running the tests? (mvn package
> -DskipTests, or sbt assembly)
>
>
> --
> Marcelo
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: python/run-tests fails at spark master branch

2015-04-21 Thread Marcelo Vanzin 
On Tue, Apr 21, 2015 at 1:30 AM, Hrishikesh Subramonian
 wrote:

> Run streaming tests ...
> Failed to find Spark Streaming Kafka assembly jar in
> /home/xyz/spark/external/kafka-assembly
> You need to build Spark with  'build/sbt assembly/assembly
> streaming-kafka-assembly/assembly' or 'build/mvn package' before running
> this program
>
>
> Is anybody facing the same problem?

Have you built the assemblies before running the tests? (mvn package
-DskipTests, or sbt assembly)


-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

python/run-tests fails at spark master branch

2015-04-21 Thread hrishikesh91 
Hi,

I cloned spark master branch from github and was built successfully using
the mvn -DskipTests clean package command.
But the python/run-tests command fails. Please see the log below:

/Running PySpark tests. Output is in python/unit-tests.log.
Testing with Python version:
Python 2.7.3
Run core tests ...
Running test: pyspark/rdd.py
Running test: pyspark/context.py
Running test: pyspark/conf.py
Running test: pyspark/broadcast.py
Running test: pyspark/accumulators.py
Running test: pyspark/serializers.py
Running test: pyspark/profiler.py
Running test: pyspark/shuffle.py
Running test: pyspark/tests.py
Run sql tests ...
Running test: pyspark/sql/_types.py
Running test: pyspark/sql/context.py
Running test: pyspark/sql/dataframe.py
Running test: pyspark/sql/functions.py
Running test: pyspark/sql/tests.py
Run mllib tests ...
Running test: pyspark/mllib/classification.py
Running test: pyspark/mllib/clustering.py
Running test: pyspark/mllib/evaluation.py
Running test: pyspark/mllib/feature.py
Running test: pyspark/mllib/fpm.py
Running test: pyspark/mllib/linalg.py
Running test: pyspark/mllib/rand.py
Running test: pyspark/mllib/recommendation.py
Running test: pyspark/mllib/regression.py
Running test: pyspark/mllib/stat/_statistics.py
Running test: pyspark/mllib/tree.py
Running test: pyspark/mllib/util.py
Running test: pyspark/mllib/tests.py
Run ml tests ...
Running test: pyspark/ml/feature.py
Running test: pyspark/ml/classification.py
Running test: pyspark/ml/tests.py
Run streaming tests ...
Failed to find Spark Streaming Kafka assembly jar in
/home/xyz/spark/external/kafka-assembly
You need to build Spark with  'build/sbt assembly/assembly
streaming-kafka-assembly/assembly' or 'build/mvn package' before running
this program/


Is anybody facing the same problem?

Regards,

Hrishikesh




--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/python-run-tests-fails-at-spark-master-branch-tp11691.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

python/run-tests fails at spark master branch

2015-04-21 Thread Hrishikesh Subramonian 

Hi,

I cloned spark master branch from github and was built successfully 
using the mvn -DskipTests clean package command.

But the python/run-tests command fails. Please see the log below:

Running PySpark tests. Output is in python/unit-tests.log.
Testing with Python version:
Python 2.7.3
Run core tests ...
Running test: pyspark/rdd.py
Running test: pyspark/context.py
Running test: pyspark/conf.py
Running test: pyspark/broadcast.py
Running test: pyspark/accumulators.py
Running test: pyspark/serializers.py
Running test: pyspark/profiler.py
Running test: pyspark/shuffle.py
Running test: pyspark/tests.py
Run sql tests ...
Running test: pyspark/sql/_types.py
Running test: pyspark/sql/context.py
Running test: pyspark/sql/dataframe.py
Running test: pyspark/sql/functions.py
Running test: pyspark/sql/tests.py
Run mllib tests ...
Running test: pyspark/mllib/classification.py
Running test: pyspark/mllib/clustering.py
Running test: pyspark/mllib/evaluation.py
Running test: pyspark/mllib/feature.py
Running test: pyspark/mllib/fpm.py
Running test: pyspark/mllib/linalg.py
Running test: pyspark/mllib/rand.py
Running test: pyspark/mllib/recommendation.py
Running test: pyspark/mllib/regression.py
Running test: pyspark/mllib/stat/_statistics.py
Running test: pyspark/mllib/tree.py
Running test: pyspark/mllib/util.py
Running test: pyspark/mllib/tests.py
Run ml tests ...
Running test: pyspark/ml/feature.py
Running test: pyspark/ml/classification.py
Running test: pyspark/ml/tests.py
Run streaming tests ...
Failed to find Spark Streaming Kafka assembly jar in 
/home/xyz/spark/external/kafka-assembly
You need to build Spark with  'build/sbt assembly/assembly 
streaming-kafka-assembly/assembly' or 'build/mvn package' before running 
this program



Is anybody facing the same problem?

Regards,

Hrishikesh