I agree, we should not work around the testcase but rather understand and fix the root cause. Closure cleaner should have null'ed out the references and allowed it to be serialized.
Regards, Mridul On Sun, Aug 5, 2018 at 8:38 PM Wenchen Fan <cloud0...@gmail.com> wrote: > > It seems to me that the closure cleaner fails to clean up something. The > failed test case defines a serializable class inside the test case, and the > class doesn't refer to anything in the outer class. Ideally it can be > serialized after cleaning up the closure. > > This is somehow a very weird way to define a class, so I'm not sure how > serious the problem is. > > On Mon, Aug 6, 2018 at 3:41 AM Stavros Kontopoulos > <stavros.kontopou...@lightbend.com> wrote: >> >> Makes sense, not sure if closure cleaning is related to the last one for >> example or others. The last one is a bit weird, unless I am missing >> something about the LegacyAccumulatorWrapper logic. >> >> Stavros >> >> On Sun, Aug 5, 2018 at 10:23 PM, Sean Owen <sro...@gmail.com> wrote: >>> >>> Yep that's what I did. There are more failures with different resolutions. >>> I'll open a JIRA and PR and ping you, to make sure that the changes are all >>> reasonable, and not an artifact of missing something about closure cleaning >>> in 2.12. >>> >>> In the meantime having a 2.12 build up and running for master will just >>> help catch these things. >>> >>> On Sun, Aug 5, 2018 at 2:16 PM Stavros Kontopoulos >>> <stavros.kontopou...@lightbend.com> wrote: >>>> >>>> Hi Sean, >>>> >>>> I run a quick build so the failing tests seem to be: >>>> >>>> - SPARK-17644: After one stage is aborted for too many failed attempts, >>>> subsequent stagesstill behave correctly on fetch failures *** FAILED *** >>>> A job with one fetch failure should eventually succeed >>>> (DAGSchedulerSuite.scala:2422) >>>> >>>> >>>> - LegacyAccumulatorWrapper with AccumulatorParam that has no >>>> equals/hashCode *** FAILED *** >>>> java.io.NotSerializableException: >>>> org.scalatest.Assertions$AssertionsHelper >>>> Serialization stack: >>>> - object not serializable (class: >>>> org.scalatest.Assertions$AssertionsHelper, value: >>>> org.scalatest.Assertions$AssertionsHelper@3bc5fc8f) >>>> >>>> >>>> The last one can be fixed easily if you set class `MyData(val i: Int) >>>> extends Serializable `outside of the test suite. For some reason outers >>>> (not removed) are capturing >>>> the Scalatest stuff in 2.12. >>>> >>>> Let me know if we see the same failures. >>>> >>>> Stavros >>>> >>>> On Sun, Aug 5, 2018 at 5:10 PM, Sean Owen <sro...@gmail.com> wrote: >>>>> >>>>> Shane et al - could we get a test job in Jenkins to test the Scala 2.12 >>>>> build? I don't think I have the access or expertise for it, though I >>>>> could probably copy and paste a job. I think we just need to clone the, >>>>> say, master Maven Hadoop 2.7 job, and add two steps: run >>>>> "./dev/change-scala-version.sh 2.12" first, then add "-Pscala-2.12" to >>>>> the profiles that are enabled. >>>>> >>>>> I can already see two test failures for the 2.12 build right now and will >>>>> try to fix those, but this should help verify whether the failures are >>>>> 'real' and detect them going forward. >>>>> >>>>> >>>> >> >> >> --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
