I switched Scala 2.13 to Scala 2.12 today. The test is still in progress and it has not been hung.
Yang Jie 发件人: Dongjoon Hyun <dongjoon.h...@gmail.com> 日期: 2022年11月16日 星期三 01:17 收件人: "Yang,Jie(INF)" <yangji...@baidu.com> 抄送: huaxin gao <huaxin.ga...@gmail.com>, "L. C. Hsieh" <vii...@gmail.com>, Chao Sun <sunc...@apache.org>, dev <dev@spark.apache.org> 主题: Re: [VOTE] Release Spark 3.2.3 (RC1) Did you hit that in Scala 2.12, too? Dongjoon. On Tue, Nov 15, 2022 at 4:36 AM Yang,Jie(INF) <yangji...@baidu.com<mailto:yangji...@baidu.com>> wrote: Hi, all I test v3.2.3 with following command: ``` dev/change-scala-version.sh 2.13 build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -fn ``` The testing environment is: OS: CentOS 6u3 Final Java: zulu 11.0.17 Python: 3.9.7 Scala: 2.13 The above test command has been executed twice, and all times hang in the following stack: ``` "ScalaTest-main-running-JoinSuite" #1 prio=5 os_prio=0 cpu=312870.06ms elapsed=1552.65s tid=0x00007f2ddc02d000 nid=0x7132 waiting on condition [0x00007f2de3929000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method) - parking to wait for <0x0000000790d00050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.17/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.17/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.17/LinkedBlockingQueue.java:433) at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:275) at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec$$Lambda$9429/0x0000000802269840.apply(Unknown Source) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:228) - locked <0x0000000790d00208> (a java.lang.Object) at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:370) at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:355) at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:185) at org.apache.spark.sql.execution.SparkPlan$$Lambda$8573/0x0000000801f99c40.apply(Unknown Source) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:223) at org.apache.spark.sql.execution.SparkPlan$$Lambda$8574/0x0000000801f9a040.apply(Unknown Source) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:220) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:181) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:172) - locked <0x0000000790d00218> (a org.apache.spark.sql.execution.QueryExecution) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:171) at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:3247) - locked <0x0000000790d002d8> (a org.apache.spark.sql.Dataset) at org.apache.spark.sql.Dataset.rdd(Dataset.scala:3245) at org.apache.spark.sql.QueryTest$.$anonfun$getErrorMessageInCheckAnswer$1(QueryTest.scala:265) at org.apache.spark.sql.QueryTest$$$Lambda$8564/0x0000000801f94440.apply$mcJ$sp(Unknown Source) at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) at org.apache.spark.sql.QueryTest$.getErrorMessageInCheckAnswer(QueryTest.scala:265) at org.apache.spark.sql.QueryTest$.checkAnswer(QueryTest.scala:242) at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:151) at org.apache.spark.sql.JoinSuite.checkAnswer(JoinSuite.scala:58) at org.apache.spark.sql.JoinSuite.$anonfun$new$138(JoinSuite.scala:1062) at org.apache.spark.sql.JoinSuite$$Lambda$2827/0x00000008013d5840.apply$mcV$sp(Unknown Source) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190) at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236) at org.scalatest.funsuite.AnyFunSuiteLike$$Lambda$8386/0x0000000801f0a840.apply(Unknown Source) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236) at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218) at org.apache.spark.SparkFunSuite.org<https://mailshield.baidu.com/check?q=g7Y99OToCMn4N2%2fkSCzdHsZrBdM%2bveoiGIScN4zUn0Njh7UjGZk7toykI%2bX5TsXD>$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62) at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234) at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227) at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269) at org.scalatest.funsuite.AnyFunSuiteLike$$Lambda$8382/0x0000000801f0e840.apply(Unknown Source) at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) at org.scalatest.SuperEngine$$Lambda$8383/0x0000000801f0d840.apply(Unknown Source) at scala.collection.immutable.List.foreach(List.scala:333) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475) at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269) at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268) at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1563) at org.scalatest.Suite.run(Suite.scala:1112) at org.scalatest.Suite.run$(Suite.scala:1094) at org.scalatest.funsuite.AnyFunSuite.org<https://mailshield.baidu.com/check?q=CyXfi1lNK3XEW4vrZ3%2bg3gQrijHc2pVnzFcFMt1SPMOwpGuNU2lkrpNxUMa%2br1jUF7fqtA%3d%3d>$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1563) at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) at org.scalatest.funsuite.AnyFunSuiteLike$$Lambda$8376/0x0000000801f07840.apply(Unknown Source) at org.scalatest.SuperEngine.runImpl(Engine.scala:535) at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) at org.apache.spark.SparkFunSuite.org<https://mailshield.baidu.com/check?q=g7Y99OToCMn4N2%2fkSCzdHsZrBdM%2bveoiGIScN4zUn0Njh7UjGZk7toykI%2bX5TsXD>$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:62) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:62) at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1175) at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1222) at org.scalatest.Suite$$Lambda$7247/0x000000080193d040.apply(Unknown Source) at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323) at org.scalatest.Suite.runNestedSuites(Suite.scala:1220) at org.scalatest.Suite.runNestedSuites$(Suite.scala:1154) at org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:30) at org.scalatest.Suite.run(Suite.scala:1109) at org.scalatest.Suite.run$(Suite.scala:1094) at org.scalatest.tools.DiscoverySuite.run(DiscoverySuite.scala:30) at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45) at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1322) at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1316) at org.scalatest.tools.Runner$$$Lambda$7245/0x000000080193e840.apply(Unknown Source) at scala.collection.immutable.List.foreach(List.scala:333) at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1316) at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:993) at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:971) at org.scalatest.tools.Runner$$$Lambda$60/0x0000000800148040.apply(Unknown Source) at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1482) at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:971) at org.scalatest.tools.Runner$.main(Runner.scala:775) at org.scalatest.tools.Runner.main(Runner.scala) ``` I think the test case being executed is `SPARK-28323: PythonUDF should be able to use in join condition`, does anyone have the same problem? Yang Jie 发件人: huaxin gao <huaxin.ga...@gmail.com<mailto:huaxin.ga...@gmail.com>> 日期: 2022年11月15日 星期二 13:59 收件人: "L. C. Hsieh" <vii...@gmail.com<mailto:vii...@gmail.com>> 抄送: Dongjoon Hyun <dongjoon.h...@gmail.com<mailto:dongjoon.h...@gmail.com>>, Chao Sun <sunc...@apache.org<mailto:sunc...@apache.org>>, dev <dev@spark.apache.org<mailto:dev@spark.apache.org>> 主题: Re: [VOTE] Release Spark 3.2.3 (RC1) +1 Thanks Chao! On Mon, Nov 14, 2022 at 9:37 PM L. C. Hsieh <vii...@gmail.com<mailto:vii...@gmail.com>> wrote: +1 Thanks Chao. On Mon, Nov 14, 2022 at 6:55 PM Dongjoon Hyun <dongjoon.h...@gmail.com<mailto:dongjoon.h...@gmail.com>> wrote: > > +1 > > Thank you, Chao. > > On Mon, Nov 14, 2022 at 4:12 PM Chao Sun > <sunc...@apache.org<mailto:sunc...@apache.org>> wrote: >> >> Please vote on releasing the following candidate as Apache Spark version >> 3.2.3. >> >> The vote is open until 11:59pm Pacific time Nov 17th and passes if a >> majority +1 PMC votes are cast, with a minimum of 3 +1 votes. >> >> [ ] +1 Release this package as Apache Spark 3.2.3 >> [ ] -1 Do not release this package because ... >> >> To learn more about Apache Spark, please see >> http://spark.apache.org/<https://mailshield.baidu.com/check?q=eJcUboQ1HRRomPZKEwRzpl69wA8DbI%2fNIiRNsQ%3d%3d> >> >> The tag to be voted on is v3.2.3-rc1 (commit >> b53c341e0fefbb33d115ab630369a18765b7763d): >> https://github.com/apache/spark/tree/v3.2.3-rc1<https://mailshield.baidu.com/check?q=1l6n2dPAt62Hg3fIHnm%2bZiYaoxYOpGmWVjYnQimr29zfPU2uzgcQkLawzjK1HF%2bdD1yLsQ%3d%3d> >> >> The release files, including signatures, digests, etc. can be found at: >> https://dist.apache.org/repos/dist/dev/spark/v3.2.3-rc1-bin/<https://mailshield.baidu.com/check?q=%2bhYCySw46HEVZlrXDDlrMzy6tGHW57fkaPIHyKgUBg1o83YtunwZU58LlZ4ZuxMWCmqht4FYN7WaSUyqTl4vFw%3d%3d> >> >> Signatures used for Spark RCs can be found in this file: >> https://dist.apache.org/repos/dist/dev/spark/KEYS<https://mailshield.baidu.com/check?q=E6fHbSXEWw02TTJBpc3bfA9mi7ea0YiWcNHkm%2fDJxwlaWinGnMdaoO1PahHhgj00vKwcbElpuHA%3d> >> >> The staging repository for this release can be found at: >> https://repository.apache.org/content/repositories/orgapachespark-1431/<https://mailshield.baidu.com/check?q=wDoqNv5amlDmohUUV9yoBzlZBzBNp%2boFTrpIiSJ1wVxVYO3fQIzfMzclIsTAx8QnfdrepoSTkqsCu6xyMsjDBjXEXXyCYM501j8CMA%3d%3d> >> >> The documentation corresponding to this release can be found at: >> https://dist.apache.org/repos/dist/dev/spark/v3.2.3-rc1-docs/<https://mailshield.baidu.com/check?q=bkrbaqzrftuBwkOZuV2GMw%2b3MANrfqFNBDTqjFATKt3KVRHB2cw5JRGvO5UfSg%2bchMROqtBVhqX2%2bDX0hO0azrRJ%2fI4%3d> >> >> The list of bug fixes going into 3.2.3 can be found at the following URL: >> https://issues.apache.org/jira/projects/SPARK/versions/12352105<https://mailshield.baidu.com/check?q=InWtXpCRA7pNM4sgk90HUJRIE0cQByBdwj%2bfh%2bAu9JfLFQp7oZSMs0uliYo08CplvvNo3kRvPzALmuBJNTz39yYCyUw%3d> >> >> This release is using the release script of the tag v3.2.3-rc1. >> >> >> FAQ >> >> ========================= >> How can I help test this release? >> ========================= >> If you are a Spark user, you can help us test this release by taking >> an existing Spark workload and running on this release candidate, then >> reporting any regressions. >> >> If you're working in PySpark you can set up a virtual env and install >> the current RC and see if anything important breaks, in the Java/Scala >> you can add the staging repository to your projects resolvers and test >> with the RC (make sure to clean up the artifact cache before/after so >> you don't end up building with a out of date RC going forward). >> >> =========================================== >> What should happen to JIRA tickets still targeting 3.2.3? >> =========================================== >> The current list of open tickets targeted at 3.2.3 can be found at: >> https://issues.apache.org/jira/projects/SPARK<https://mailshield.baidu.com/check?q=4UUpJqq41y71Gnuj0qTUYo6hTjqzT7oytN6x%2fvgC5XUtQUC8MfJ77tj7K70O%2f1QMmNoa1A%3d%3d> >> and search for "Target >> Version/s" = 3.2.3 >> >> Committers should look at those and triage. Extremely important bug >> fixes, documentation, and API tweaks that impact compatibility should >> be worked on immediately. Everything else please retarget to an >> appropriate release. >> >> ================== >> But my bug isn't fixed? >> ================== >> In order to make timely releases, we will typically not hold the >> release unless the bug in question is a regression from the previous >> release. That being said, if there is something which is a regression >> that has not been correctly targeted please ping me or a committer to >> help target the issue. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: >> dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org> >> --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org>
