Admaing commented on PR #28864:
URL: https://github.com/apache/flink/pull/28864#issuecomment-6938709847

   Looked into this while clearing the `TODO` in 
`DynamicPartitionPruningUtils`, and the blocker is structural rather than a 
missing check. Writing it down in case it saves someone time.
   
   **Why the optimizer cannot check the source**
   
   `DynamicPartitionPruningProgram` runs inside the optimizer, where only the 
`DynamicTableSource` / `BatchPhysicalScan` are available. The only way to see 
the actual transformation is 
`DataStreamScanProvider#produceDataStream(ProviderContext, 
StreamExecutionEnvironment)`, which needs a `StreamExecutionEnvironment` — the 
optimizer does not have one. That is why the TODO says the source should be 
taken from the translated DataStream: the translation phase has the env, the 
optimization phase does not.
   
   There is also no capability marker to ask instead. 
`org.apache.flink.table.connector.source` currently exposes only 
`SourceProvider`, `SourceFunctionProvider`, `InputFormatProvider` and 
`DataStreamScanProvider`; nothing lets a source declare whether it can actually 
participate in dynamic filtering.
   
   **Two directions, both needing a decision**
   
   1. Add a default method to the `@PublicEvolving` `DataStreamScanProvider`, 
e.g. `default boolean supportsDynamicFiltering()`, and have the optimizer query 
it. Small change, but it touches a public interface, so it needs agreement 
first.
   2. Move the DPP decision into the translation phase where the env exists. 
Cleaner conceptually, but it changes the boundary between optimizer and 
translation — far more than the TODO suggests.
   
   A smaller variant — degrading to a plain scan during `BatchTableSourceScan` 
translation — would still leave the DPP node in the plan, which is exactly the 
symptom this ticket describes.
   
   **Is it worth it?**
   
   The TODO itself notes the result is correct either way; the only effect is 
that the physical and exec node plans contain DPP nodes that do nothing at 
runtime. So the trade-off is "plan noise" against a public API change.
   
   Which direction would maintainers prefer? If the answer is (1) I am happy to 
prepare the patch; (2) is probably better done by someone closer to the 
optimizer/translation boundary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to