jonvex commented on PR #10826:
URL: https://github.com/apache/hudi/pull/10826#issuecomment-1981362523
@yihua @danny0405 @beyond1920:
Without adding the optimizer, we get the exception:
`java.lang.RuntimeException: After applying rule
org.apache.spark.sql.catalyst.optimizer.FoldablePropagation in batch Operator
Optimization before Inferring Filters, the structural integrity of the plan is
broken.`
Here is the plan before and after the FoldablePropagation step when running
the test "Test ignoring case for MOR table":
```
Before:
Project [ID#10, NAME#11, price#12, TS#13, dt#14]
+- Join LeftOuter, (id#20 = id#10)
:- Project [1 AS id#10, a1 AS NAME#11, 111 AS price#12, 1111 AS ts#13,
2021-05-05 AS DT#14]
: +- OneRowRelation
+- Project [ID#20]
+- Relation
default.h0[_hoodie_commit_time#15,_hoodie_commit_seqno#16,_hoodie_record_key#17,_hoodie_partition_path#18,_hoodie_file_name#19,ID#20,NAME#21,price#22,TS#23L,dt#24]
org.apache.hudi.EmptyRelation@67cd84f9
After:
Project [1 AS id#10, a1 AS NAME#11, 111 AS price#12, 1111 AS ts#13,
2021-05-05 AS DT#14]
+- Join LeftOuter, (id#20 = 1)
:- Project [1 AS id#10, a1 AS NAME#11, 111 AS price#12, 1111 AS ts#13,
2021-05-05 AS DT#14]
: +- OneRowRelation
+- Project [ID#20]
+- Relation
default.h0[_hoodie_commit_time#15,_hoodie_commit_seqno#16,_hoodie_record_key#17,_hoodie_partition_path#18,_hoodie_file_name#19,ID#20,NAME#21,price#22,TS#23L,dt#24]
org.apache.hudi.EmptyRelation@67cd84f9
```
@KnightChess can we avoid running the optimizer here if we find a way to
rename all id#10 and not just the outermost projection?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
