xiaoyanxie commented on code in PR #2395:
URL: https://github.com/apache/auron/pull/2395#discussion_r3818843428
##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala:
##########
@@ -1505,24 +1507,63 @@ object NativeConverters extends Logging {
}
}
+ /**
+ * ObjectInputStream that resolves classes against an explicit class loader.
+ *
+ * The default ObjectInputStream.resolveClass resolves each class through
+ * VM.latestUserDefinedLoader(), which selects a loader from the live call
stack rather than the
+ * context class loader. During a nested read the most recent user-defined
frame is often a
+ * Spark or Scala class, whose loader cannot see Auron classes when Auron is
supplied through
+ * spark.jars and therefore loaded by MutableURLClassLoader. The expression
graph then resolves
+ * only partially and an un-readResolve'd DefaultSerializationProxy is
assigned into
+ * RDD.dependencies_, raising a ClassCastException. Pinning the loader keeps
resolution
+ * independent of the call stack. Spark's own JavaDeserializationStream does
the same.
+ */
+ private class AuronObjectInputStream(in: InputStream, loader: ClassLoader)
+ extends ObjectInputStream(in) {
+
+ // scalastyle:off classforname
+ private def load(name: String, cl: ClassLoader): Class[_] =
Class.forName(name, false, cl)
+ // scalastyle:on classforname
+
+ // resolveProxyClass is deliberately not overridden: the only
non-deprecated way to obtain a
+ // proxy Class is Proxy.getProxyClass, and serialized expressions contain
no dynamic proxies.
+ override def resolveClass(desc: ObjectStreamClass): Class[_] = {
Review Comment:
Correction to my comment above: I attributed the primitive descriptors to
`ScalaUDF`'s `ExpressionEncoder`s, and that attribution is wrong.
I checked it properly this time. A `ScalaUDF` over `(Int, Long, Double)`
taken from an analyzed plan carries three `inputEncoders`, and on Spark 4.1
they serialize as `AgnosticEncoders$PrimitiveIntEncoder$`,
`PrimitiveLongEncoder$` and `PrimitiveDoubleEncoder$` — singleton module
objects written through `ModuleSerializationProxy`. Deserializing that payload
through a class loader that records every name requested gives 67 names, and
not one primitive among them. `scala.reflect.ClassTag.Int` behaves the same
way: it serializes as `ManifestFactory$IntManifest`, and `runtimeClass` is
restored by `readResolve` rather than written to the stream.
So the 578µs number is real, but it was measured on a payload I built by
hand with eight primitive descriptors in it, not on one that arises from this
path. I have not found a real payload that carries them.
Unless you know of one, I would rather drop the primitive lookup table
altogether than add an optimization I cannot justify. Sorry for the noise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
