Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/11301#discussion_r56623654
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
---
@@ -418,6 +419,13 @@ abstract class TreeNode[BaseType <:
TreeNode[BaseType]] extends Product {
override def toString: String = treeString
+ def toOriginString: String =
+ if (this.origin.callSite.isDefined &&
!this.isInstanceOf[BoundReference]) {
--- End diff --
@sarutak , the latest code showorigin (if defined) in toOriginString. As a
result, the following program can generate ```/* input[1, int] @ selectExpr ...
*/``` in ```GenerateSafeProjection``` while the whole stage code generation
does not generate comment for this expression.
Is this OK with you?
WholeStageCodeGen
```
/* 005 */ /** Codegened pipeline for:
/* 006 */ * Project [_1#0 AS i#3]
/* 007 */ +- Filter (_1#0 > 1)
/* 008 */ +- INPUT
/* 009 */ */
/* 010 */ class GeneratedIterator extends
org.apache.spark.sql.execution.BufferedRowIterator {
...
/* 032 */ protected void processNext() throws java.io.IOException {
/* 033 */ /*** PRODUCE: Project [_1#0 AS i#3] */
/* 034 */
/* 035 */ /*** PRODUCE: Filter (_1#0 > 1) */
/* 036 */
/* 037 */ /*** PRODUCE: INPUT */
/* 038 */
/* 039 */ while (!shouldStop() && inputadapter_input.hasNext()) {
/* 040 */ InternalRow inputadapter_row = (InternalRow)
inputadapter_input.next();
/* 041 */ /*** CONSUME: Filter (_1#0 > 1) */
/* 042 */ /* input[0, int] @ filter at Dataset1.scala:25 */
/* 043 */ int filter_value = inputadapter_row.getInt(0);
/* 044 */
/* 045 */ /* (input[0, int] > 1) @ filter at Dataset1.scala:25 */
/* 046 */ boolean filter_value2 = false;
/* 047 */ filter_value2 = filter_value > 1;
/* 048 */ if (!filter_value2) continue;
/* 049 */
/* 050 */ filter_metricValue.add(1);
/* 051 */
/* 052 */ /*** CONSUME: Project [_1#0 AS i#3] */
/* 053 */
/* 054 */ project_rowWriter.write(0, filter_value);
/* 055 */ append(project_result);
/* 056 */ }
/* 057 */ }
/* 058 */ }
```
GenerateSafeProjection:
```
/* 006 */ class SpecificSafeProjection extends
org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
...
/* 019 */ public java.lang.Object apply(java.lang.Object _i) {
/* 020 */ InternalRow i = (InternalRow) _i;
/* 021 */ /* createexternalrow(if (isnull(input[0, int])) null else
input[0, int], StructField(i,IntegerType,false)) @ selectExpr at
Dataset1.scala:24 */
/* 022 */ boolean isNull = false;
/* 023 */ final Object[] values = new Object[1];
/* 024 */ /* if (isnull(input[0, int])) null else input[0, int] @
selectExpr at Dataset1.scala:24 */
/* 025 */ /* isnull(input[0, int]) @ selectExpr at Dataset1.scala:24 */
/* 026 */ /* input[0, int] @ selectExpr at Dataset1.scala:24 */
/* 027 */ int value3 = i.getInt(0);
/* 028 */ boolean isNull1 = false;
/* 029 */ int value1 = -1;
/* 030 */ if (!false && false) {
/* 031 */ /* null @ selectExpr at Dataset1.scala:24 */
/* 032 */ final int value4 = -1;
/* 033 */ isNull1 = true;
/* 034 */ value1 = value4;
/* 035 */ } else {
/* 036 */ /* input[0, int] @ selectExpr at Dataset1.scala:24 */
/* 037 */ int value5 = i.getInt(0);
/* 038 */ isNull1 = false;
/* 039 */ value1 = value5;
/* 040 */ }
/* 041 */ if (isNull1) {
/* 042 */ values[0] = null;
/* 043 */ } else {
/* 044 */ values[0] = value1;
/* 045 */ }
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
