GitHub user gaborgsomogyi opened a pull request:

    [SPARK-23422][Core] YarnShuffleIntegrationSuite fix when SPARK_PREPEN…

    …D_CLASSES set to 1
    ## What changes were proposed in this pull request?
    YarnShuffleIntegrationSuite fails when SPARK_PREPEND_CLASSES set to 1.
    Normally mllib built before yarn module. When SPARK_PREPEND_CLASSES used 
mllib classes are on yarn test classpath.
    Before 2.3 that did not cause issues. But 2.3 has SPARK-22450, which 
registered some mllib classes with the kryo serializer. Now it dies with the 
following error:
    18/02/13 07:33:29 INFO SparkContext: Starting job: collect at 
    Exception in thread "dag-scheduler-event-loop" 
java.lang.NoClassDefFoundError: breeze/linalg/DenseMatrix
    In this PR NoClassDefFoundError caught only in case of testing and then do 
    ## How was this patch tested?
    Automated: Pass the Jenkins.

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-23422

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20608
commit 9c1c415e4c58b334bc1339754fae75f2e991dfa2
Author: Gabor Somogyi <gabor.g.somogyi@...>
Date:   2018-02-14T12:55:05Z

    [SPARK-23422][Core] YarnShuffleIntegrationSuite fix when 



To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to