littleeleventhwolf commented on issue #7892: URL: https://github.com/apache/hudi/issues/7892#issuecomment-1424085535
I don't think [#7085](https://github.com/apache/hudi/pull/7085) can fixed this problem. If we query same table twice with different filter and projection in one sql, then join or union these two subquery part. Flink will produce different `TableScanSource` rel-node digest, for example (Case 1 I described above): ``` ### Left Part TableSourceScan(table=[[default_catalog, default_database, TABLENAME, filter=[equals(dim1, VARCHAR(xxx))], project=[field1, dim1, join_key, group_key]]], fields=[field1, dim1, join_key, group_key], changelogMode=[I]), rowType=[RecordType(VARCHAR(2147483647) field1, VARCHAR(2147483647) dim1, VARCHAR(2147483647) join_key, VARCHAR(2147483647) group_key)] ### Right Part TableSourceScan(table=[[default_catalog, default_database, TABLENAME, filter=[equals(dim2, VARCHAR(xxx))], project=[field1, dim2, join_key, group_key]]], fields=[field1, dim2, join_key, group_key], changelogMode=[I]), rowType=[RecordType(VARCHAR(2147483647) field1, VARCHAR(2147483647) dim2, VARCHAR(2147483647) join_key, VARCHAR(2147483647) group_key)] ``` Different digest eventually cause the Flink job to fail to reuse the source. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
