Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-15 Thread Yang,Jie(INF) 
I switched Scala 2.13 to Scala 2.12 today. The test is still in progress and it 
has not been hung.

Yang Jie

发件人: Dongjoon Hyun 
日期: 2022年11月16日 星期三 01:17
收件人: "Yang,Jie(INF)" 
抄送: huaxin gao , "L. C. Hsieh" , Chao 
Sun , dev 
主题: Re: [VOTE] Release Spark 3.2.3 (RC1)

Did you hit that in Scala 2.12, too?

Dongjoon.

On Tue, Nov 15, 2022 at 4:36 AM Yang,Jie(INF) 
mailto:yangji...@baidu.com>> wrote:
Hi, all

I test v3.2.3 with following command:

```

dev/change-scala-version.sh 2.13
build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl 
-Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive  -Pscala-2.13 -fn
```

The testing environment is:

OS: CentOS 6u3 Final
Java: zulu 11.0.17
Python: 3.9.7
Scala: 2.13

The above test command has been executed twice, and all times hang in the 
following stack:

```
"ScalaTest-main-running-JoinSuite" #1 prio=5 os_prio=0 cpu=312870.06ms 
elapsed=1552.65s tid=0x7f2ddc02d000 nid=0x7132 waiting on condition  
[0x7f2de3929000]
   java.lang.Thread.State: WAITING (parking)
   at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method)
   - parking to wait for  <0x000790d00050> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at 
java.util.concurrent.locks.LockSupport.park(java.base@11.0.17/LockSupport.java:194)
   at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.17/AbstractQueuedSynchronizer.java:2081)
   at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.17/LinkedBlockingQueue.java:433)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:275)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec$$Lambda$9429/0x000802269840.apply(Unknown
 Source)
   at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:228)
   - locked <0x000790d00208> (a java.lang.Object)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:370)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:355)
   at 
org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:185)
   at 
org.apache.spark.sql.execution.SparkPlan$$Lambda$8573/0x000801f99c40.apply(Unknown
 Source)
   at 
org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:223)
   at 
org.apache.spark.sql.execution.SparkPlan$$Lambda$8574/0x000801f9a040.apply(Unknown
 Source)
   at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   at 
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:220)
   at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:181)
   at 
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:172)
   - locked <0x000790d00218> (a 
org.apache.spark.sql.execution.QueryExecution)
   at 
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:171)
   at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:3247)
   - locked <0x000790d002d8> (a org.apache.spark.sql.Dataset)
   at org.apache.spark.sql.Dataset.rdd(Dataset.scala:3245)
   at 
org.apache.spark.sql.QueryTest$.$anonfun$getErrorMessageInCheckAnswer$1(QueryTest.scala:265)
   at 
org.apache.spark.sql.QueryTest$$$Lambda$8564/0x000801f94440.apply$mcJ$sp(Unknown
 Source)
   at 
scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
   at 
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
   at 
org.apache.spark.sql.QueryTest$.getErrorMessageInCheckAnswer(QueryTest.scala:265)
   at org.apache.spark.sql.QueryTest$.checkAnswer(QueryTest.scala:242)
   at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:151)
   at org.apache.spark.sql.JoinSuite.checkAnswer(JoinSuite.scala:58)
   at org.apache.spark.sql.JoinSuite.$anonfun$new$138(JoinSuite.scala:1062)
   at 
org.apache.spark.sql.JoinSuite$$Lambda$2827/0x0008013d5840.apply$mcV$sp(Unknown
 Source)
   at 
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
   at org.scalatest.Transformer.apply(Transformer.scala:22)
   at org.scalatest.Transformer.apply(Transformer.scala:20)
   at 
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
   at 
org.s

Re: [DISCUSS] SPIP: Better Spark UI scalability and Driver stability for large applications

2022-11-15 Thread Wenchen Fan 
This looks great! UI stability/scalability has been a pain point for a long
time.

On Sat, Nov 12, 2022 at 5:24 AM Gengliang Wang  wrote:

> Hi Everyone,
>
> I want to discuss the "Better Spark UI scalability and Driver stability
> for large applications" proposal. Please find the links below:
>
> *JIRA* - https://issues.apache.org/jira/browse/SPARK-41053
> *SPIP Document* -
> https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing
>
> *Excerpt from the document: *
>
> After SPARK-18085 ,
> the Spark history server(SHS) becomes more scalable for processing large
> applications by supporting a persistent KV-store(LevelDB/RocksDB) as the
> storage layer.
>
> As for the live Spark UI, all the data is still stored in memory, which
> can bring memory pressures to the Spark driver for large applications.
>
> For better Spark UI scalability and Driver stability, I propose to
>
>-
>
>Support storing all the UI data in a persistent KV store.
>RocksDB/LevelDB provides low memory overhead. Their write/read performance
>is fast enough to serve the workloads of live UI. Spark UI can retain more
>data with the new backend, while SHS can leverage it to fasten its startup.
>- Support a new Protobuf serializer for all the UI data. The new
>serializer is supposed to be faster, according to benchmarks. It will be
>the default serializer for the persistent KV store of live UI.
>
>
>
>
> I appreciate any suggestions you can provide,
> Gengliang
>

Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-15 Thread Mridul Muralidharan 
+1

Signatures, digests, etc check out fine.
Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes

Regards,
Mridul


On Tue, Nov 15, 2022 at 1:00 PM kazuyuki tanimura
 wrote:

> +1 (non-binding)
>
> Thank you Chao
>
> Kazu
>
>
>  | Kazuyuki Tanimura | ktanim...@apple.com | +1-408-207-7176
>
> Apple Confidential and Proprietary Information
>
> This email and any attachments is privileged and contains confidential
> information intended only for the recipient(s) named above. Any
> other distribution, forwarding, copying or disclosure of this message is
> strictly prohibited. If you have received this email in error, please
> notify me immediately by telephone or return email, and delete this message
> from your system.
>
> On Nov 15, 2022, at 10:04 AM, Sean Owen  wrote:
>
> +1 from me, at least from my testing. Java 8 + Scala 2.12 and Java 8 +
> Scala 2.13 worked for me, and I didn't see a test hang. I am testing with
> Python 3.10 FWIW.
>
> On Tue, Nov 15, 2022 at 6:37 AM Yang,Jie(INF)  wrote:
>
>> Hi, all
>>
>>
>>
>> I test v3.2.3 with following command:
>>
>>
>>
>> ```
>>
>> dev/change-scala-version.sh 2.13
>>
>> build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn
>> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
>> -Pscala-2.13 -fn
>>
>> ```
>>
>>
>>
>> The testing environment is:
>>
>>
>>
>> OS: CentOS 6u3 Final
>>
>> Java: zulu 11.0.17
>>
>> Python: 3.9.7
>>
>> Scala: 2.13
>>
>>
>>
>> The above test command has been executed twice, and all times hang in the
>> following stack:
>>
>>
>>
>> ```
>>
>> "ScalaTest-main-running-JoinSuite" #1 prio=5 os_prio=0 cpu=312870.06ms
>> elapsed=1552.65s tid=0x7f2ddc02d000 nid=0x7132 waiting on condition
>> [0x7f2de3929000]
>>
>>java.lang.Thread.State: WAITING (parking)
>>
>>at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method)
>>
>>- parking to wait for  <0x000790d00050> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>at java.util.concurrent.locks.LockSupport.park(java.base@11.0.17
>> /LockSupport.java:194)
>>
>>at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.17
>> /AbstractQueuedSynchronizer.java:2081)
>>
>>at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.17
>> /LinkedBlockingQueue.java:433)
>>
>>at
>> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:275)
>>
>>at
>> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec$$Lambda$9429/0x000802269840.apply(Unknown
>> Source)
>>
>>at
>> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
>>
>>at
>> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:228)
>>
>>- locked <0x000790d00208> (a java.lang.Object)
>>
>>at
>> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:370)
>>
>>at
>> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:355)
>>
>>at
>> org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:185)
>>
>>at
>> org.apache.spark.sql.execution.SparkPlan$$Lambda$8573/0x000801f99c40.apply(Unknown
>> Source)
>>
>>at
>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:223)
>>
>>at
>> org.apache.spark.sql.execution.SparkPlan$$Lambda$8574/0x000801f9a040.apply(Unknown
>> Source)
>>
>>at
>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>>
>>at
>> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:220)
>>
>>at
>> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:181)
>>
>>at
>> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:172)
>>
>>- locked <0x000790d00218> (a
>> org.apache.spark.sql.execution.QueryExecution)
>>
>>at
>> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:171)
>>
>>at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:3247)
>>
>>- locked <0x000790d002d8> (a org.apache.spark.sql.Dataset)
>>
>>at org.apache.spark.sql.Dataset.rdd(Dataset.scala:3245)
>>
>>at
>> org.apache.spark.sql.QueryTest$.$anonfun$getErrorMessageInCheckAnswer$1(QueryTest.scala:265)
>>
>>at
>> org.apache.spark.sql.QueryTest$$$Lambda$8564/0x000801f94440.apply$mcJ$sp(Unknown
>> Source)
>>
>>at
>> scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
>>
>>at
>> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
>>
>>at
>> org.apache.spark.sql.QueryTest$.getErrorMessageInCheckAnswer(QueryTest.scala:265)
>>
>>

Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-15 Thread kazuyuki tanimura 
+1 (non-binding)

Thank you Chao

Kazu


 | Kazuyuki Tanimura | ktanim...@apple.com | +1-408-207-7176

Apple Confidential and Proprietary Information

This email and any attachments is privileged and contains confidential 
information intended only for the recipient(s) named above. Any other 
distribution, forwarding, copying or disclosure of this message is strictly 
prohibited. If you have received this email in error, please notify me 
immediately by telephone or return email, and delete this message from your 
system.

> On Nov 15, 2022, at 10:04 AM, Sean Owen  wrote:
> 
> +1 from me, at least from my testing. Java 8 + Scala 2.12 and Java 8 + Scala 
> 2.13 worked for me, and I didn't see a test hang. I am testing with Python 
> 3.10 FWIW.
> 
> On Tue, Nov 15, 2022 at 6:37 AM Yang,Jie(INF)  > wrote:
> Hi, all
> 
>  
> 
> I test v3.2.3 with following command:
> 
>  
> 
> ```
> 
> dev/change-scala-version.sh 2.13
> 
> build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn 
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive  
> -Pscala-2.13 -fn
> 
> ```
> 
>  
> 
> The testing environment is:
> 
>  
> 
> OS: CentOS 6u3 Final
> 
> Java: zulu 11.0.17
> 
> Python: 3.9.7
> 
> Scala: 2.13
> 
>  
> 
> The above test command has been executed twice, and all times hang in the 
> following stack:
> 
>  
> 
> ```
> 
> "ScalaTest-main-running-JoinSuite" #1 prio=5 os_prio=0 cpu=312870.06ms 
> elapsed=1552.65s tid=0x7f2ddc02d000 nid=0x7132 waiting on condition  
> [0x7f2de3929000]
> 
>java.lang.Thread.State: WAITING (parking)
> 
>at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method)
> 
>- parking to wait for  <0x000790d00050> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 
>at 
> java.util.concurrent.locks.LockSupport.park(java.base@11.0.17/LockSupport.java:194)
> 
>at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.17/AbstractQueuedSynchronizer.java:2081)
> 
>at 
> java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.17/LinkedBlockingQueue.java:433)
> 
>at 
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:275)
> 
>at 
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec$$Lambda$9429/0x000802269840.apply(Unknown
>  Source)
> 
>at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
> 
>at 
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:228)
> 
>- locked <0x000790d00208> (a java.lang.Object)
> 
>at 
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:370)
> 
>at 
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:355)
> 
>at 
> org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:185)
> 
>at 
> org.apache.spark.sql.execution.SparkPlan$$Lambda$8573/0x000801f99c40.apply(Unknown
>  Source)
> 
>at 
> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:223)
> 
>at 
> org.apache.spark.sql.execution.SparkPlan$$Lambda$8574/0x000801f9a040.apply(Unknown
>  Source)
> 
>at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> 
>at 
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:220)
> 
>at 
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:181)
> 
>at 
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:172)
> 
>- locked <0x000790d00218> (a 
> org.apache.spark.sql.execution.QueryExecution)
> 
>at 
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:171)
> 
>at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:3247)
> 
>- locked <0x000790d002d8> (a org.apache.spark.sql.Dataset)
> 
>at org.apache.spark.sql.Dataset.rdd(Dataset.scala:3245)
> 
>at 
> org.apache.spark.sql.QueryTest$.$anonfun$getErrorMessageInCheckAnswer$1(QueryTest.scala:265)
> 
>at 
> org.apache.spark.sql.QueryTest$$$Lambda$8564/0x000801f94440.apply$mcJ$sp(Unknown
>  Source)
> 
>at 
> scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
> 
>at 
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
> 
>at 
> org.apache.spark.sql.QueryTest$.getErrorMessageInCheckAnswer(QueryTest.scala:265)
> 
>at org.apache.spark.sql.QueryTest$.checkAnswer(QueryTest.scala:242)
> 
>at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:151)
> 
>at org.apache.spark.sql.JoinSuite.checkAnswer(JoinSuite.scala

Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-15 Thread Sean Owen 
+1 from me, at least from my testing. Java 8 + Scala 2.12 and Java 8 +
Scala 2.13 worked for me, and I didn't see a test hang. I am testing with
Python 3.10 FWIW.

On Tue, Nov 15, 2022 at 6:37 AM Yang,Jie(INF)  wrote:

> Hi, all
>
>
>
> I test v3.2.3 with following command:
>
>
>
> ```
>
> dev/change-scala-version.sh 2.13
>
> build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> -Pscala-2.13 -fn
>
> ```
>
>
>
> The testing environment is:
>
>
>
> OS: CentOS 6u3 Final
>
> Java: zulu 11.0.17
>
> Python: 3.9.7
>
> Scala: 2.13
>
>
>
> The above test command has been executed twice, and all times hang in the
> following stack:
>
>
>
> ```
>
> "ScalaTest-main-running-JoinSuite" #1 prio=5 os_prio=0 cpu=312870.06ms
> elapsed=1552.65s tid=0x7f2ddc02d000 nid=0x7132 waiting on condition
> [0x7f2de3929000]
>
>java.lang.Thread.State: WAITING (parking)
>
>at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method)
>
>- parking to wait for  <0x000790d00050> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>at java.util.concurrent.locks.LockSupport.park(java.base@11.0.17
> /LockSupport.java:194)
>
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.17
> /AbstractQueuedSynchronizer.java:2081)
>
>at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.17
> /LinkedBlockingQueue.java:433)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:275)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec$$Lambda$9429/0x000802269840.apply(Unknown
> Source)
>
>at
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:228)
>
>- locked <0x000790d00208> (a java.lang.Object)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:370)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:355)
>
>at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:185)
>
>at
> org.apache.spark.sql.execution.SparkPlan$$Lambda$8573/0x000801f99c40.apply(Unknown
> Source)
>
>at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:223)
>
>at
> org.apache.spark.sql.execution.SparkPlan$$Lambda$8574/0x000801f9a040.apply(Unknown
> Source)
>
>at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>
>at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:220)
>
>at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:181)
>
>at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:172)
>
>- locked <0x000790d00218> (a
> org.apache.spark.sql.execution.QueryExecution)
>
>at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:171)
>
>at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:3247)
>
>- locked <0x000790d002d8> (a org.apache.spark.sql.Dataset)
>
>at org.apache.spark.sql.Dataset.rdd(Dataset.scala:3245)
>
>at
> org.apache.spark.sql.QueryTest$.$anonfun$getErrorMessageInCheckAnswer$1(QueryTest.scala:265)
>
>at
> org.apache.spark.sql.QueryTest$$$Lambda$8564/0x000801f94440.apply$mcJ$sp(Unknown
> Source)
>
>at
> scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
>
>at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
>
>at
> org.apache.spark.sql.QueryTest$.getErrorMessageInCheckAnswer(QueryTest.scala:265)
>
>at org.apache.spark.sql.QueryTest$.checkAnswer(QueryTest.scala:242)
>
>at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:151)
>
>at org.apache.spark.sql.JoinSuite.checkAnswer(JoinSuite.scala:58)
>
>at
> org.apache.spark.sql.JoinSuite.$anonfun$new$138(JoinSuite.scala:1062)
>
>at
> org.apache.spark.sql.JoinSuite$$Lambda$2827/0x0008013d5840.apply$mcV$sp(Unknown
> Source)
>
>at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
>
>at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
>
>at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
>
>at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>
>at org.scalatest.Transformer.apply(Transformer.scala:22)
>
>at org.scalatest.Transformer.apply(Transformer.scala:20)
>
>at
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.

Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-15 Thread Dongjoon Hyun 
Did you hit that in Scala 2.12, too?

Dongjoon.

On Tue, Nov 15, 2022 at 4:36 AM Yang,Jie(INF)  wrote:

> Hi, all
>
>
>
> I test v3.2.3 with following command:
>
>
>
> ```
>
> dev/change-scala-version.sh 2.13
>
> build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> -Pscala-2.13 -fn
>
> ```
>
>
>
> The testing environment is:
>
>
>
> OS: CentOS 6u3 Final
>
> Java: zulu 11.0.17
>
> Python: 3.9.7
>
> Scala: 2.13
>
>
>
> The above test command has been executed twice, and all times hang in the
> following stack:
>
>
>
> ```
>
> "ScalaTest-main-running-JoinSuite" #1 prio=5 os_prio=0 cpu=312870.06ms
> elapsed=1552.65s tid=0x7f2ddc02d000 nid=0x7132 waiting on condition
> [0x7f2de3929000]
>
>java.lang.Thread.State: WAITING (parking)
>
>at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method)
>
>- parking to wait for  <0x000790d00050> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>at java.util.concurrent.locks.LockSupport.park(java.base@11.0.17
> /LockSupport.java:194)
>
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.17
> /AbstractQueuedSynchronizer.java:2081)
>
>at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.17
> /LinkedBlockingQueue.java:433)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:275)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec$$Lambda$9429/0x000802269840.apply(Unknown
> Source)
>
>at
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:228)
>
>- locked <0x000790d00208> (a java.lang.Object)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:370)
>
>at
> org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:355)
>
>at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:185)
>
>at
> org.apache.spark.sql.execution.SparkPlan$$Lambda$8573/0x000801f99c40.apply(Unknown
> Source)
>
>at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:223)
>
>at
> org.apache.spark.sql.execution.SparkPlan$$Lambda$8574/0x000801f9a040.apply(Unknown
> Source)
>
>at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>
>at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:220)
>
>at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:181)
>
>at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:172)
>
>- locked <0x000790d00218> (a
> org.apache.spark.sql.execution.QueryExecution)
>
>at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:171)
>
>at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:3247)
>
>- locked <0x000790d002d8> (a org.apache.spark.sql.Dataset)
>
>at org.apache.spark.sql.Dataset.rdd(Dataset.scala:3245)
>
>at
> org.apache.spark.sql.QueryTest$.$anonfun$getErrorMessageInCheckAnswer$1(QueryTest.scala:265)
>
>at
> org.apache.spark.sql.QueryTest$$$Lambda$8564/0x000801f94440.apply$mcJ$sp(Unknown
> Source)
>
>at
> scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
>
>at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
>
>at
> org.apache.spark.sql.QueryTest$.getErrorMessageInCheckAnswer(QueryTest.scala:265)
>
>at org.apache.spark.sql.QueryTest$.checkAnswer(QueryTest.scala:242)
>
>at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:151)
>
>at org.apache.spark.sql.JoinSuite.checkAnswer(JoinSuite.scala:58)
>
>at
> org.apache.spark.sql.JoinSuite.$anonfun$new$138(JoinSuite.scala:1062)
>
>at
> org.apache.spark.sql.JoinSuite$$Lambda$2827/0x0008013d5840.apply$mcV$sp(Unknown
> Source)
>
>at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
>
>at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
>
>at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
>
>at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>
>at org.scalatest.Transformer.apply(Transformer.scala:22)
>
>at org.scalatest.Transformer.apply(Transformer.scala:20)
>
>at
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
>
>at
> org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
>
>at
> org.

Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-15 Thread Yang,Jie(INF) 
Hi, all

I test v3.2.3 with following command:

```

dev/change-scala-version.sh 2.13
build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl 
-Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive  -Pscala-2.13 -fn
```

The testing environment is:

OS: CentOS 6u3 Final
Java: zulu 11.0.17
Python: 3.9.7
Scala: 2.13

The above test command has been executed twice, and all times hang in the 
following stack:

```
"ScalaTest-main-running-JoinSuite" #1 prio=5 os_prio=0 cpu=312870.06ms 
elapsed=1552.65s tid=0x7f2ddc02d000 nid=0x7132 waiting on condition  
[0x7f2de3929000]
   java.lang.Thread.State: WAITING (parking)
   at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method)
   - parking to wait for  <0x000790d00050> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at 
java.util.concurrent.locks.LockSupport.park(java.base@11.0.17/LockSupport.java:194)
   at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.17/AbstractQueuedSynchronizer.java:2081)
   at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.17/LinkedBlockingQueue.java:433)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:275)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec$$Lambda$9429/0x000802269840.apply(Unknown
 Source)
   at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:228)
   - locked <0x000790d00208> (a java.lang.Object)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:370)
   at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:355)
   at 
org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:185)
   at 
org.apache.spark.sql.execution.SparkPlan$$Lambda$8573/0x000801f99c40.apply(Unknown
 Source)
   at 
org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:223)
   at 
org.apache.spark.sql.execution.SparkPlan$$Lambda$8574/0x000801f9a040.apply(Unknown
 Source)
   at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   at 
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:220)
   at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:181)
   at 
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:172)
   - locked <0x000790d00218> (a 
org.apache.spark.sql.execution.QueryExecution)
   at 
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:171)
   at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:3247)
   - locked <0x000790d002d8> (a org.apache.spark.sql.Dataset)
   at org.apache.spark.sql.Dataset.rdd(Dataset.scala:3245)
   at 
org.apache.spark.sql.QueryTest$.$anonfun$getErrorMessageInCheckAnswer$1(QueryTest.scala:265)
   at 
org.apache.spark.sql.QueryTest$$$Lambda$8564/0x000801f94440.apply$mcJ$sp(Unknown
 Source)
   at 
scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
   at 
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
   at 
org.apache.spark.sql.QueryTest$.getErrorMessageInCheckAnswer(QueryTest.scala:265)
   at org.apache.spark.sql.QueryTest$.checkAnswer(QueryTest.scala:242)
   at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:151)
   at org.apache.spark.sql.JoinSuite.checkAnswer(JoinSuite.scala:58)
   at org.apache.spark.sql.JoinSuite.$anonfun$new$138(JoinSuite.scala:1062)
   at 
org.apache.spark.sql.JoinSuite$$Lambda$2827/0x0008013d5840.apply$mcV$sp(Unknown
 Source)
   at 
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
   at org.scalatest.Transformer.apply(Transformer.scala:22)
   at org.scalatest.Transformer.apply(Transformer.scala:20)
   at 
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
   at 
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
   at 
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
   at 
org.scalatest.funsuite.AnyFunSuiteLike$$Lambda$8386/0x000801f0a840.apply(Unknown
 Source)
   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
   at 
org.scalatest.funsuite.AnyFunSuiteLike.runTe