Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21619#discussion_r200114007
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/javaCode.scala
---
@@ -190,26 +194,29 @@ object Block {
while (strings.hasNext) {
val input = inputs.next
input match {
- case _: ExprValue | _: Block =>
+ case _: ExprValue | _: CodeBlock =>
codeParts += buf.toString
buf.clear
blockInputs += input.asInstanceOf[JavaCode]
+ case EmptyBlock =>
case _ =>
buf.append(input)
}
buf.append(strings.next)
}
- if (buf.nonEmpty) {
- codeParts += buf.toString
- }
+ codeParts += buf.toString
(codeParts.toSeq, blockInputs.toSeq)
}
}
/**
* A block of java code. Including a sequence of code parts and some
inputs to this block.
- * The actual java code is generated by embedding the inputs into the code
parts.
+ * The actual java code is generated by embedding the inputs into the code
parts. Here we keep
+ * inputs of `JavaCode` instead of simply folding them as a string of
code, because we need to
+ * track expressions (`ExprValue`) in this code block. We need to be able
to manipulate the
+ * expressions later without changing the behavior of this code block in
some applications, e.g.,
+ * method splitting.
*/
case class CodeBlock(codeParts: Seq[String], blockInputs: Seq[JavaCode])
extends Block {
override lazy val exprValues: Set[ExprValue] = {
--- End diff --
not related to this PR, but we should think about it in the future. If we
treat `CodeBlock` as a tree of generated code, then this method doesn't make a
lot of sense: it collects all references from its children and put them into a
set, which means every time we transform a `CodeBlock` and create a new copy,
we need to build this set again.
It's unclear how `exprValues` would be used, but I'd image we can provide a
`contains` method which recursively check the children.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
