JingsongLi commented on PR #8782: URL: https://github.com/apache/paimon/pull/8782#issuecomment-5087487486
Could we avoid applying selected-key pushdown when the MAP field uses a map-specific merge aggregator, or perform the selected-key projection only after the merge? The temporary selected-key `ROW` currently becomes the projected value type passed into `MergeFileSplitRead`. For an aggregation table configured with `fields.metrics.aggregate-function=merge_map`, `AggregateMergeFunction.Factory` therefore tries to create `FieldMergeMapAgg` with `ROW<0: BIGINT, 1: BIGINT, ...>` instead of the original MAP type, and the scan fails with: `Data type for merge map column must be 'MAP' but was 'ROW<...>'` I reproduced this for both ORC and Parquet, with thin mode both enabled and disabled, by writing multiple files to a primary-key aggregation table and reading `metrics['key1']`. The existing tests only cover append-only selected-key reads. Please add a regression test for merge reads. Until selected-key aggregation semantics are implemented, the Spark rule should conservatively skip this pushdown for affected merge or aggregation configurations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
