[
https://issues.apache.org/jira/browse/SPARK-26963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-26963.
-------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 23866
[https://github.com/apache/spark/pull/23866]
> SizeEstimator can't make some JDK fields accessible in Java 9+
> --------------------------------------------------------------
>
> Key: SPARK-26963
> URL: https://issues.apache.org/jira/browse/SPARK-26963
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Sean Owen
> Assignee: Sean Owen
> Priority: Minor
> Fix For: 3.0.0
>
>
> Copied from https://github.com/apache/spark/pull/23804#issuecomment-466198782
> Under Java 11, tests fail with:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.935
> s <<< FAILURE! - in org.apache.spark.ml.regression.JavaGBTRegressorSuite
> [ERROR] runDT(org.apache.spark.ml.regression.JavaGBTRegressorSuite) Time
> elapsed: 0.933 s <<< ERROR!
> java.lang.reflect.InaccessibleObjectException: Unable to make field
> jdk.internal.ref.PhantomCleanable jdk.internal.ref.PhantomCleanable.prev
> accessible: module java.base does not "opens jdk.internal.ref" to unnamed
> module @4212a0c8
> at
> org.apache.spark.ml.regression.JavaGBTRegressorSuite.runDT(JavaGBTRegressorSuite.java:65)
> [INFO] Running org.apache.spark.ml.regression.JavaDecisionTreeRegressorSuite
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.43 s
> - in org.apache.spark.ml.regression.JavaDecisionTreeRegressorSuite
> [INFO] Running org.apache.spark.ml.regression.JavaRandomForestRegressorSuite
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.507
> s <<< FAILURE! - in
> org.apache.spark.ml.regression.JavaRandomForestRegressorSuite
> [ERROR] runDT(org.apache.spark.ml.regression.JavaRandomForestRegressorSuite)
> Time elapsed: 0.506 s <<< ERROR!
> java.lang.reflect.InaccessibleObjectException: Unable to make field
> jdk.internal.ref.PhantomCleanable jdk.internal.ref.PhantomCleanable.prev
> accessible: module java.base does not "opens jdk.internal.ref" to unnamed
> module @4212a0c8
> at
> org.apache.spark.ml.regression.JavaRandomForestRegressorSuite.runDT(JavaRandomForestRegressorSuite.java:88)
> {code}
> Stack trace:
> {code}
> at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:337)
> at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:281)
> at
> java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:176)
> at java.base/java.lang.reflect.Field.setAccessible(Field.java:170)
> at
> org.apache.spark.util.SizeEstimator$.$anonfun$getClassInfo$2(SizeEstimator.scala:337)
> at
> org.apache.spark.util.SizeEstimator$.$anonfun$getClassInfo$2$adapted(SizeEstimator.scala:331)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at
> org.apache.spark.util.SizeEstimator$.getClassInfo(SizeEstimator.scala:331)
> at
> org.apache.spark.util.SizeEstimator$.getClassInfo(SizeEstimator.scala:325)
> at
> org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:223)
> at
> org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:202)
> at
> org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:70)
> at
> org.apache.spark.util.collection.SizeTracker.takeSample(SizeTracker.scala:78)
> at
> org.apache.spark.util.collection.SizeTracker.afterUpdate(SizeTracker.scala:70)
> at
> org.apache.spark.util.collection.SizeTracker.afterUpdate$(SizeTracker.scala:67)
> at
> org.apache.spark.util.collection.SizeTrackingVector.$plus$eq(SizeTrackingVector.scala:31)
> at
> org.apache.spark.storage.memory.DeserializedValuesHolder.storeValue(MemoryStore.scala:665)
> at
> org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
> at
> org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
> at
> org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1166)
> at
> org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1092)
> at
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1157)
> at
> org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:915)
> at
> org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1482)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:133)
> at
> org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:91)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1427)
> at
> org.apache.spark.ml.regression.GBTRegressionModel.transformImpl(GBTRegressor.scala:249)
> {code}
> The fix is to not include fields in the estimate that can't be accessed
> anyway, for this or really any SecurityException on setAccessible. The impact
> on the estimate should be fairly limited, as it didn't cause failures in but
> a few tests.
> One other open question is, why did only this come up in a very few cases
> like broadcasting GBTRegressionModel? and why PhantomReference, which isn't
> used in (non-test) Spark code? I can't see anything odd about this model
> class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
