Re: what should I do when spark ut hang?

2016-04-27 Thread Ted Yu
Did you have a chance to take jstack when VersionsSuite was running ?

You can use the following command to run the test:

sbt/sbt "test-only org.apache.spark.sql.hive.client.VersionsSuite"

On Wed, Apr 27, 2016 at 9:01 PM, Demon King  wrote:

> Hi, all:
>I compile spark-1.6.1  in redhat 5.7(I have installed
> spark-1.6.0-cdh5.7.0 hive-1.1.0+cdh5.7.0 and hadoop-2.6.0+cdh5.7.0 in this
> machine). my compile cmd is:
>
> build/mvn --force -Psparkr -Pyarn -Phadoop-2.6
> -Dhadoop.version=2.6.0-cdh5.7.0 -Phive -Phive-thriftserver
>
>  when I use this cmd to run unit test. test hanged for a whole
> night(nearly 10 hour):
>
>  build/mvn --force -Psparkr -Pyarn -Phadoop-2.6
> -Dhadoop.version=2.6.0-cdh5.7.0 -Phive -Phive-thriftserver  test
>
>  That is the last log:
>
> Discovery completed in 25 seconds, 803 milliseconds.
> Run starting. Expected test count is: 1716
> StatisticsSuite:
> - parse analyze commands
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> log4j:WARN No appenders could be found for logger (hive.log).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> Warning: fs.defaultFS is not set when running "chgrp" command.
> Warning: fs.defaultFS is not set when running "chmod" command.
> - analyze MetastoreRelations
> - estimates the size of a test MetastoreRelation
> - auto converts to broadcast hash join, by size estimate of a relation
> - auto converts to broadcast left semi join, by size estimate of a relation
> VersionsSuite:
>
>
>  I
> modify 
> ./sql/hive/target/scala-2.10/test-classes/org/apache/spark/sql/hive/client/VersionsSuite.class
> and add print statement in each beginning of function, but no message
> print. Now what should I do to solve this problem?
>
>  Thank you!
>
>


what should I do when spark ut hang?

2016-04-27 Thread Demon King
Hi, all:
   I compile spark-1.6.1  in redhat 5.7(I have installed
spark-1.6.0-cdh5.7.0 hive-1.1.0+cdh5.7.0 and hadoop-2.6.0+cdh5.7.0 in this
machine). my compile cmd is:

build/mvn --force -Psparkr -Pyarn -Phadoop-2.6
-Dhadoop.version=2.6.0-cdh5.7.0 -Phive -Phive-thriftserver

 when I use this cmd to run unit test. test hanged for a whole
night(nearly 10 hour):

 build/mvn --force -Psparkr -Pyarn -Phadoop-2.6
-Dhadoop.version=2.6.0-cdh5.7.0 -Phive -Phive-thriftserver  test

 That is the last log:

Discovery completed in 25 seconds, 803 milliseconds.
Run starting. Expected test count is: 1716
StatisticsSuite:
- parse analyze commands
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
log4j:WARN No appenders could be found for logger (hive.log).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
Warning: fs.defaultFS is not set when running "chgrp" command.
Warning: fs.defaultFS is not set when running "chmod" command.
- analyze MetastoreRelations
- estimates the size of a test MetastoreRelation
- auto converts to broadcast hash join, by size estimate of a relation
- auto converts to broadcast left semi join, by size estimate of a relation
VersionsSuite:


 I
modify 
./sql/hive/target/scala-2.10/test-classes/org/apache/spark/sql/hive/client/VersionsSuite.class
and add print statement in each beginning of function, but no message
print. Now what should I do to solve this problem?

 Thank you!