LuciferYang commented on a change in pull request #32536:
URL: https://github.com/apache/spark/pull/32536#discussion_r632263291
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1434,15 +1432,7 @@ object CodeGenerator extends Logging {
*/
private def updateAndGetCompilationStats(evaluator: ClassBodyEvaluator):
ByteCodeStats = {
// First retrieve the generated classes.
- val classes = {
- val scField = classOf[ClassBodyEvaluator].getDeclaredField("sc")
- scField.setAccessible(true)
- val compiler = scField.get(evaluator).asInstanceOf[SimpleCompiler]
- val loader = compiler.getClassLoader.asInstanceOf[ByteArrayClassLoader]
- val classesField = loader.getClass.getDeclaredField("classes")
- classesField.setAccessible(true)
- classesField.get(loader).asInstanceOf[JavaMap[String,
Array[Byte]]].asScala
- }
+ val classes = evaluator.getBytecodes.asScala
Review comment:
@maropu If we want to do some code checking, maybe we can can be
enhanced the case `metrics are recorded on compile` in `CodeGenerationSuite` as
follows:
```
test("metrics are recorded on compile") {
...
val metricGeneratedClassByteCodeSizeSnapshot =
CodegenMetrics.METRIC_GENERATED_CLASS_BYTECODE_SIZE.getSnapshot.getValues
val metricGeneratedMethodByteCodeSizeSnapshot =
CodegenMetrics.METRIC_GENERATED_METHOD_BYTECODE_SIZE.getSnapshot.getValues
GenerateOrdering.generate(Add(Literal(123), Literal(1)).asc :: Nil)
...
// Make sure the stat content doesn't change before/after SPARK-35398
assert(CodegenMetrics.METRIC_GENERATED_CLASS_BYTECODE_SIZE.getSnapshot.getValues
.diff(metricGeneratedClassByteCodeSizeSnapshot)
.sameElements(Array(740, 1293)))
assert(CodegenMetrics.METRIC_GENERATED_METHOD_BYTECODE_SIZE.getSnapshot.getValues
.diff(metricGeneratedMethodByteCodeSizeSnapshot)
.sameElements(Array(5, 5, 6, 7, 10, 15, 46)))
}
```
The new assertion can be passed before and after this pr, however, if we
update the version of janino or change the codegen of `Add`, we may need to
update the content of the assertion because the size of the generated code may
change.
For example `CodegenMetrics.METRIC_GENERATED_CLASS_BYTECODE_SIZE` with
janino 3.1.4 are `Array(740, 1293)`, but withe janino 3.0.16 are `Array(687,
1036)`, so I'm not sure if we need to add these assertions in this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
