zhengruifeng opened a new pull request, #43739:
URL: https://github.com/apache/spark/pull/43739
### What changes were proposed in this pull request?
Since JVM runs static codes only once, if loading functions$ fails, it will
always report java.lang.NoClassDefFoundError: Could not initialize class
```
23/11/01 23:06:21 WARN TaskSetManager: Lost task 136.0 in stage 9565.0 (TID
4557384) (10.4.35.209 executor 16): TaskKilled (Stage cancelled: Job aborted
due to stage failure: Task 2 in stage 9565.0 failed 4 times, most recent
failure: Lost task 2.3 in stage 9565.0 (TID 4558369) (10.4.56.6 executor 71):
java.io.IOException: unexpected exception type
at
java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1750)
at
java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1280)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222)
…
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:900)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at
com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:795)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor520.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
at sun.reflect.GeneratedMethodAccessor224.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1274)
... 388 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.ml.functions$
... 396 more
```
This PR just changes `functions.*` as lazy avoid hitting this issue because
the initialization codes of a lazy val is not in static codes.
### Why are the changes needed?
to fix a intermittent bug
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
manually checked
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
