Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19480#discussion_r146859947
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -277,13 +292,25 @@ class CodegenContext {
funcName: String,
funcCode: String,
inlineToOuterClass: Boolean = false): String = {
+ val newFunction = addNewFunctionInternal(funcName, funcCode,
inlineToOuterClass)
+ newFunction match {
+ case NewFunction(functionName, None, None) => functionName
+ case NewFunction(functionName, Some(_), Some(subclassInstance)) =>
+ subclassInstance + "." + functionName
+ }
+ }
+
+ private[this] def addNewFunctionInternal(
+ funcName: String,
+ funcCode: String,
+ inlineToOuterClass: Boolean): NewFunction = {
// The number of named constants that can exist in the class is
limited by the Constant Pool
// limit, 65,536. We cannot know how many constants will be inserted
for a class, so we use a
- // threshold of 1600k bytes to determine when a function should be
inlined to a private, nested
+ // threshold of 1000k bytes to determine when a function should be
inlined to a private, nested
// sub-class.
val (className, classInstance) = if (inlineToOuterClass) {
outerClassName -> ""
- } else if (currClassSize > 1600000) {
+ } else if (currClassSize > 1000000) {
--- End diff --
I can do that for this PR. But what do you think about introducing an
internal configuration for this as I proposed in the other PR? If this should
not be done, I can close the other PR: I think that the discussion here and
there showed the reasons for its creation. If they are not enough, then I will
close the PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
