srowen commented on a change in pull request #34368:
URL: https://github.com/apache/spark/pull/34368#discussion_r734914860
##########
File path: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala
##########
@@ -407,6 +417,24 @@ private[spark] object ClosureCleaner extends Logging {
}
}
+ /**
+ * This method is used to get the final modifier field when use Java 17.
+ */
+ private def getFinalModifiersFieldForJava17(field: Field): Option[Field] = {
+ if (SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17) &&
+ Modifier.isFinal(field.getModifiers)) {
+ val methodGetDeclaredFields0 = classOf[Class[_]]
+ .getDeclaredMethod("getDeclaredFields0", classOf[Boolean])
+ methodGetDeclaredFields0.setAccessible(true)
+ val fields = methodGetDeclaredFields0.invoke(classOf[Field],
false.asInstanceOf[Object])
+ .asInstanceOf[Array[Field]]
+ val modifiersFieldOption = fields.find(field =>
"modifiers".equals(field.getName))
+ assert(modifiersFieldOption.isDefined)
Review comment:
require, not assert
##########
File path: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala
##########
@@ -394,8 +395,17 @@ private[spark] object ClosureCleaner extends Logging {
parent = null, outerThis, capturingClass, accessedFields)
val outerField = func.getClass.getDeclaredField("arg$1")
+ // SPARK-37072: When Java 17 is used and `outerField` is read-only,
+ // the content of `outerField` cannot be set by reflect api directly.
+ // But We can remove the `final` modifier of `outerField` before set
value
Review comment:
I wonder what happens if we don't clear this field in the closure in
this case - seems kind of risky to do this. That said, who knows what behavior
differences arise if we don't
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
