Admaing commented on PR #28864: URL: https://github.com/apache/flink/pull/28864#issuecomment-6938709847
Looked into this while clearing the `TODO` in `DynamicPartitionPruningUtils`, and the blocker is structural rather than a missing check. Writing it down in case it saves someone time. **Why the optimizer cannot check the source** `DynamicPartitionPruningProgram` runs inside the optimizer, where only the `DynamicTableSource` / `BatchPhysicalScan` are available. The only way to see the actual transformation is `DataStreamScanProvider#produceDataStream(ProviderContext, StreamExecutionEnvironment)`, which needs a `StreamExecutionEnvironment` — the optimizer does not have one. That is why the TODO says the source should be taken from the translated DataStream: the translation phase has the env, the optimization phase does not. There is also no capability marker to ask instead. `org.apache.flink.table.connector.source` currently exposes only `SourceProvider`, `SourceFunctionProvider`, `InputFormatProvider` and `DataStreamScanProvider`; nothing lets a source declare whether it can actually participate in dynamic filtering. **Two directions, both needing a decision** 1. Add a default method to the `@PublicEvolving` `DataStreamScanProvider`, e.g. `default boolean supportsDynamicFiltering()`, and have the optimizer query it. Small change, but it touches a public interface, so it needs agreement first. 2. Move the DPP decision into the translation phase where the env exists. Cleaner conceptually, but it changes the boundary between optimizer and translation — far more than the TODO suggests. A smaller variant — degrading to a plain scan during `BatchTableSourceScan` translation — would still leave the DPP node in the plan, which is exactly the symptom this ticket describes. **Is it worth it?** The TODO itself notes the result is correct either way; the only effect is that the physical and exec node plans contain DPP nodes that do nothing at runtime. So the trade-off is "plan noise" against a public API change. Which direction would maintainers prefer? If the answer is (1) I am happy to prepare the patch; (2) is probably better done by someone closer to the optimizer/translation boundary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
