AHeise opened a new pull request, #29415:
URL: https://github.com/apache/flink/pull/29415
## What is the purpose of the change
For a source that implements `SupportsReadingMetadata`, projection push down
records a `ReadingMetadata` ability even if the query reads no metadata column:
```sql
CREATE TABLE src (a INT, b BIGINT) WITH ('connector' = 'values');
SELECT a FROM src WHERE b > 1;
```
```
TableSourceScan(table=[[default_catalog, default_database, src,
filter=[...], project=[a], metadata=[]]], fields=[a])
```
and the compiled plan carries a no-op `{"type" : "ReadingMetadata",
"metadataKeys" : [ ], ...}`.
## Brief change log
- `PushProjectIntoTableSourceScanRule` records the `ReadingMetadataSpec`
only if a metadata key is projected, or if the scan already has one that must
be narrowed (the table declares metadata columns, see
`DynamicSourceUtils#validateAndApplyMetadata`).
- `ScanReuser` skips the spec if the reused scans use no metadata key. It
rebuilds the source without the earlier projection and metadata specs, so there
is nothing to narrow.
| Table / query | Before | After |
|---|---|---|
| physical columns only | `metadata=[]` | no `ReadingMetadata` |
| metadata column declared, not projected | `metadata=[]` | `metadata=[]`
(narrows the earlier spec) |
| metadata column projected | `metadata=[m]` | `metadata=[m]` |
## Verifying this change
This change is already covered by existing tests:
- Expected plans of 53 plan test classes and the `*JsonPlanTest`s are
regenerated; the diff only drops `metadata=[]` from scan digests and empty
`ReadingMetadata` entries from JSON plans.
- Plans where an earlier spec is narrowed keep `metadata=[]`, e.g.
`AggregateTest#testCountStartWithNestedRow`,
`PushLocalAggIntoTableSourceScanRuleTest#testCanPushDownLocalAggWithAuxGrouping`.
- All restore tests pass unchanged: the stored compiled plans that contain
an empty `ReadingMetadata` entry still restore.
- `ExplainTest` golden files and the `EXPLAIN` examples in the docs are
updated the same way.
- Resolves the TODO in `DeltaJoinUtil`; `ReadingMetadataSpec` stays a
supported spec there, since narrowing still produces one with empty keys.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
- A source that supports metadata no longer receives an
`applyReadableMetadata` call with an empty key list when the table declares no
metadata columns.
- Plan tests of external connectors (e.g. Kafka) lose `metadata=[]` in the
same way.
- `ScanReuserUtils#containsRexNodeSpecAfterProjection` counts a
`ReadingMetadataSpec` as a projection, so a metadata source without projection
push down can now be reused in cases where it was not before. Without metadata
the field indices do not shift, so this is safe.
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
