lszskye opened a new pull request, #8478:
URL: https://github.com/apache/paimon/pull/8478
### Purpose
This PR introduces the preliminary Spark-side and metadata infrastructure
for MAP selected-key pushdown.
The target use case is to allow upper engines, such as Spark, to describe a
read schema like:
`SELECT
id,
attrs['key1'] AS key1_value,
attrs['key2'] AS key2_value
FROM T;`
as a temporary Paimon read type where the MAP field is rewritten to a ROW
field, and the selected MAP keys are recorded in DataField.description:
`__PAIMON_MAP_SELECTED_KEYS:key1;key2`
This metadata is only intended to exist in the temporary scan/read schema.
It must not be persisted into table schema or catalog schema.
#### Explicitly out of scope
This PR does not enable the full read path yet. The following parts are
intentionally left for a follow-up PR for #8392 is not merged:
- FormatReaderMapping support for consuming __PAIMON_MAP_SELECTED_KEYS.
- Core reader support for reading a MAP field as a rewritten ROW field.
- Shared-shredding MAP reader support for selected-key recall.
- End-to-end SQL execution for attrs['key1'] / element_at(attrs, 'key1')
selected-key pushdown.
Because the reader/core side is not implemented in this PR, the optimizer
rule is not registered in PaimonSparkSessionExtensions.
### Tests
MAP selected-key metadata construction and parsing.
Empty key handling.
Duplicate selected-key validation.
Spark read type rewrite behavior for selected MAP keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
