[
https://issues.apache.org/jira/browse/SPARK-51365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17932038#comment-17932038
]
Dongjoon Hyun edited comment on SPARK-51365 at 3/3/25 6:05 PM:
---------------------------------------------------------------
Thank you for pinging me, [~LuciferYang]. However, the root cause of OOM is a
well-known known issue of GitHub Action runners. FYI, macOS runner has only 7GB
while linux runner has 16GB.
[Standard GitHub-hosted runners for public
repositories|https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories]
{code:java}
Virtual Machine Processor (CPU) Memory (RAM) Storage (SSD) Architecture
Linux 4 16 GB 14 GB x64
macOS 3 (M1) 7 GB 14 GB arm64
{code}
was (Author: dongjoon):
Thank you for pinging me, [~LuciferYang]. However, the root cause of OOM is a
well-known known issue of GitHub Action runners. FYI, macOS runner has only 7GB
while linux runner has 16GB.
[Standard GitHub-hosted runners for public
repositories](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories)
{code}
Virtual Machine Processor (CPU) Memory (RAM) Storage (SSD) Architecture
Linux 4 16 GB 14 GB x64
macOS 3 (M1) 7 GB 14 GB arm64
{code}
> OOM occurred during macOS daily tests
> -------------------------------------
>
> Key: SPARK-51365
> URL: https://issues.apache.org/jira/browse/SPARK-51365
> Project: Spark
> Issue Type: Bug
> Components: Tests
> Affects Versions: 4.1.0
> Reporter: Yang Jie
> Priority: Major
>
> * [https://github.com/apache/spark/actions/runs/13316147273/job/37299839380]
> {code:java}
> Warning: [343.044s][warning][os,thread] Failed to start thread "Unknown
> thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 4096k,
> guardsize: 16k, detached.
> 11372Warning: [343.044s][warning][os,thread] Failed to start the native
> thread for java.lang.Thread "shuffle-exchange-1529"
> 11373*** RUN ABORTED ***
> 11374An exception or error caused a run to abort: unable to create native
> thread: possibly out of memory or process/resource limits reached
> 11375 java.lang.OutOfMemoryError: unable to create native thread: possibly
> out of memory or process/resource limits reached
> 11376 at java.base/java.lang.Thread.start0(Native Method)
> 11377 at java.base/java.lang.Thread.start(Thread.java:1553)
> 11378 at java.base/java.lang.System$2.start(System.java:2577)
> 11379 at
> java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
> 11380 at
> java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
> 11381 at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> 11382 at
> scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21)
> 11383 at
> java.base/java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1782)
> 11384 at
> java.base/java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:2005)
> 11385 at
> org.apache.spark.sql.execution.SQLExecution$.withThreadLocalCaptured(SQLExecution.scala:329)
> 11386 ...
> 11387Warning: The requested profile "volcano" could not be activated because
> it does not exist.
> 11388Warning: The requested profile "hive" could not be activated because it
> does not exist.
> 11389Error: Failed to execute goal
> org.scalatest:scalatest-maven-plugin:2.2.0:test (test) on project
> spark-sql_2.13: There are test failures -> [Help 1]
> 11390Error:
> 11391Error: To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> 11392Error: Re-run Maven using the -X switch to enable full debug logging.
> 11393Error:
> 11394Error: For more information about the errors and possible solutions,
> please read the following articles:
> 11395Error: [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> 11396Error: Process completed with exit code 1. {code}
>
> * [https://github.com/apache/spark/actions/runs/13316147273/job/37299839259]
>
> {code:java}
> - group-by-ordinal.sql
> 10493- group-by-ordinal.sql_analyzer_test
> 10494Warning: [495.950s][warning][os,thread] Failed to start thread "Unknown
> thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 4096k,
> guardsize: 16k, detached.
> 10495Warning: [495.950s][warning][os,thread] Failed to start the native
> thread for java.lang.Thread "shuffle-exchange-1799"
> 1049616:17:17.464 ERROR org.apache.spark.sql.SQLQueryTestSuite: Error using
> configs:
> spark.sql.codegen.wholeStage=false,spark.sql.codegen.factoryMode=CODEGEN_ONLY
> 10497
> 10498*** RUN ABORTED ***
> 10499An exception or error caused a run to abort: unable to create native
> thread: possibly out of memory or process/resource limits reached
> 10500 java.lang.OutOfMemoryError: unable to create native thread: possibly
> out of memory or process/resource limits reached
> 10501 at java.base/java.lang.Thread.start0(Native Method)
> 10502 at java.base/java.lang.Thread.start(Thread.java:1553)
> 10503 at java.base/java.lang.System$2.start(System.java:2577)
> 10504 at
> java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
> 10505 at
> java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
> 10506 at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> 10507 at
> scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21)
> 10508 at
> java.base/java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1782)
> 10509 at
> java.base/java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:2005)
> 10510 at
> org.apache.spark.sql.execution.SQLExecution$.withThreadLocalCaptured(SQLExecution.scala:329)
> 10511 ... {code}
>
>
> The root cause is unknown for now, and we need to investigate it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
