xiaoyanxie commented on code in PR #2395:
URL: https://github.com/apache/auron/pull/2395#discussion_r3818799517
##########
.github/workflows/tpcds-reusable.yml:
##########
@@ -64,6 +64,23 @@ on:
required: false
type: string
default: ''
+ assert-no-classcastexception:
+ description: >-
+ Whether to fail the job if the TPC-DS run logs any
ClassCastException. Spark retries
+ absorb task-level deserialization failures, so queries can report
PASS while throwing
+ hundreds of exceptions; this asserts on the log instead.
+ required: false
+ type: string
+ default: 'false'
+ jar-on-system-classpath:
+ description: >-
+ Whether to also copy the Auron jar into $SPARK_HOME/jars. When true
the jar is
+ loaded by the application class loader; when false it reaches the
JVM only through
+ spark-submit --jars, i.e. Spark's MutableURLClassLoader. Some
class-loading defects
+ only reproduce in the latter configuration.
Review Comment:
The auron-it jar does contain Auron classes. I checked locally, and Auron is
actually loaded from that jar, not from --jars:
> Auron Class Loader: org.apache.spark.util.MutableURLClassLoader
Auron Code Source:
file:/.../auron-it-spark-4.1_2.13-...-jar-with-dependencies.jar
Auron On System Classpath: false
So my old comment was wrong. But the application jar is also loaded by
`MutableURLClassLoader`, which is the real condition the reproduction needs, so
it still reproduced the bug before the fix.
I tried your first suggestion but reverted it: the uber jar's
dependency-reduced pom re-declares every Auron module, so excluding only the
uber jar still pulls them in transitively, and `org.apache.auron:*` also
removes our own integration test classes.
So I used your second suggestion. auron-it now prints the class loader and
code source at startup, and the job asserts `Auron On System Classpath: false`.
I verified it works: with the jar in `$SPARK_HOME/jars` it prints `Auron On
System Classpath: true` and the step fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
