schenksj opened a new issue, #27543:
URL: https://github.com/apache/datafusion/issues/27543

   ### Describe the bug
   
   A physical plan that has already been executed can still be serialized with 
`datafusion-proto`. When it is, every `DynamicFilterPhysicalExpr` in it (the 
TopK filter held by `SortExec`, and the copy pushed into the `DataSourceExec` 
predicate) is encoded with its **current** inner expression, `generation` and 
`is_complete`, that is, the final threshold of the run that just finished. 
`DynamicFilterPhysicalExpr::try_to_proto` reads `self.inner` as is.
   
   Decoding that plan and executing it again applies the stale threshold from 
the start. Because the TopK threshold is a strict bound that excludes the rows 
that set it, the re-run returns too few rows. No error is raised.
   
   For `SELECT * FROM t ORDER BY id LIMIT 1` over ids 1..=200 the decoded plan 
is
   
   ```
   SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST], 
preserve_partitioning=[false], filter=[id@0 < 1]
     DataSourceExec: ..., predicate=DynamicFilter [ id@0 < 1 ], ..., 
pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 < 1, ...
   ```
   
   and returns 0 rows instead of 1. `ORDER BY id DESC LIMIT 3` decodes with 
`filter=[id@0 IS NULL OR id@0 > 198]` and returns 2 rows instead of 3.
   
   We hit this when distributing plans: a coordinator plans and runs a query, 
then ships the same `Arc<dyn ExecutionPlan>` to workers.
   
   ### To Reproduce
   
   `Cargo.toml`:
   
   ```toml
   [dependencies]
   datafusion = "=55.3.0"
   datafusion-proto = "=55.3.0"
   tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
   ```
   
   `src/main.rs`:
   
   ```rust
   //! Minimal repro: a physical plan serialized with datafusion-proto *after* 
it has executed
   //! carries the TopK DynamicFilterPhysicalExpr at its final runtime value.
   use std::sync::Arc;
   
   use datafusion::arrow::array::{Int64Array, RecordBatch};
   use datafusion::arrow::datatypes::{DataType, Field, Schema};
   use datafusion::parquet::arrow::ArrowWriter;
   use datafusion::parquet::file::properties::WriterProperties;
   use datafusion::physical_plan::{collect, displayable, ExecutionPlan};
   use datafusion::prelude::*;
   use datafusion_proto::bytes::{physical_plan_from_bytes, 
physical_plan_to_bytes};
   
   /// Writes `files` Parquet files holding ids 1..=200 between them, 25 rows 
per row group.
   fn write_table(dir: &std::path::Path, files: usize) -> 
datafusion::error::Result<()> {
       let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, 
false)]));
       let per_file = 200 / files as i64;
       for f in 0..files as i64 {
           let file = 
std::fs::File::create(dir.join(format!("part-{f}.parquet")))?;
           let props = 
WriterProperties::builder().set_max_row_group_size(25).build();
           let mut w = ArrowWriter::try_new(file, schema.clone(), Some(props))?;
           let ids: Vec<i64> = (f * per_file + 1..=(f + 1) * 
per_file).collect();
           for chunk in ids.chunks(25) {
               let batch = RecordBatch::try_new(schema.clone(), 
vec![Arc::new(Int64Array::from(chunk.to_vec()))])?;
               w.write(&batch)?;
           }
           w.close()?;
       }
       Ok(())
   }
   
   async fn new_ctx(dir: &std::path::Path) -> 
datafusion::error::Result<SessionContext> {
       let ctx = SessionContext::new();
       ctx.register_parquet("t", dir.to_str().unwrap(), 
ParquetReadOptions::default()).await?;
       Ok(ctx)
   }
   
   fn show(label: &str, plan: &Arc<dyn ExecutionPlan>) {
       println!("--- {label} ---\n{}", displayable(plan.as_ref()).indent(true));
   }
   
   fn rows(batches: &[RecordBatch]) -> usize {
       batches.iter().map(|b| b.num_rows()).sum()
   }
   
   async fn run_case(sql: &str, files: usize) -> datafusion::error::Result<()> {
       println!("\n================ {sql}   ({files} file(s)) 
================");
       let dir = 
std::env::temp_dir().join(format!("df-dynfilter-repro-{files}"));
       let _ = std::fs::remove_dir_all(&dir);
       std::fs::create_dir_all(&dir)?;
       write_table(&dir, files)?;
   
       // Control: round-trip the plan BEFORE executing it.
       let ctx = new_ctx(&dir).await?;
       let plan = ctx.sql(sql).await?.create_physical_plan().await?;
       let bytes = physical_plan_to_bytes(plan.clone())?;
       let ctx2 = new_ctx(&dir).await?;
       let decoded = physical_plan_from_bytes(&bytes, &ctx2.task_ctx())?;
       let n = rows(&collect(decoded, ctx2.task_ctx()).await?);
       println!("control (encode before execute): decoded plan returned {n} 
row(s)");
   
       // Bug case: execute the plan, then encode the same (executed) plan.
       let ctx = new_ctx(&dir).await?;
       let plan = ctx.sql(sql).await?.create_physical_plan().await?;
       show("plan before execution", &plan);
       let n = rows(&collect(plan.clone(), ctx.task_ctx()).await?);
       println!("original plan returned {n} row(s)");
       show("same plan after execution", &plan);
       let bytes = physical_plan_to_bytes(plan.clone())?;
       let ctx2 = new_ctx(&dir).await?;
       let decoded = physical_plan_from_bytes(&bytes, &ctx2.task_ctx())?;
       show("decoded plan (encoded after execution)", &decoded);
       let n2 = rows(&collect(decoded, ctx2.task_ctx()).await?);
       println!("decoded plan returned {n2} row(s)   [expected {n}]  => {}",
           if n2 == n { "OK" } else { "WRONG RESULT" });
       Ok(())
   }
   
   #[tokio::main]
   async fn main() -> datafusion::error::Result<()> {
       run_case("SELECT * FROM t ORDER BY id LIMIT 1", 1).await?;
       run_case("SELECT * FROM t ORDER BY id LIMIT 1", 4).await?;
       run_case("SELECT * FROM t ORDER BY id DESC LIMIT 3", 4).await?;
       Ok(())
   }
   ```
   
   Output (55.3.0; 55.2.0 is identical):
   
   ```
   ================ SELECT * FROM t ORDER BY id LIMIT 1   (1 file(s)) 
================
   control (encode before execute): decoded plan returned 1 row(s)
   --- plan before execution ---
   SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST], 
preserve_partitioning=[false]
     DataSourceExec: file_groups={1 group: 
[[tmp/df-dynfilter-repro-1/part-0.parquet]]}, projection=[id], 
file_type=parquet, predicate=DynamicFilter [ empty ], 
sort_order_for_reorder=[id@0 ASC NULLS LAST], dynamic_rg_pruning=eligible
   
   original plan returned 1 row(s)
   --- same plan after execution ---
   SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST], 
preserve_partitioning=[false], filter=[id@0 < 1]
     DataSourceExec: file_groups={1 group: 
[[tmp/df-dynfilter-repro-1/part-0.parquet]]}, projection=[id], 
file_type=parquet, predicate=DynamicFilter [ id@0 < 1 ], 
sort_order_for_reorder=[id@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, 
pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 < 1, 
required_guarantees=[]
   
   --- decoded plan (encoded after execution) ---
   SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST], 
preserve_partitioning=[false], filter=[id@0 < 1]
     DataSourceExec: file_groups={1 group: 
[[tmp/df-dynfilter-repro-1/part-0.parquet]]}, projection=[id], 
file_type=parquet, predicate=DynamicFilter [ id@0 < 1 ], 
dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 
AND id_min@0 < 1, required_guarantees=[]
   
   decoded plan returned 0 row(s)   [expected 1]  => WRONG RESULT
   
   ================ SELECT * FROM t ORDER BY id LIMIT 1   (4 file(s)) 
================
   control (encode before execute): decoded plan returned 1 row(s)
   --- plan before execution ---
   SortPreservingMergeExec: [id@0 ASC NULLS LAST], fetch=1
     SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST], 
preserve_partitioning=[true]
       DataSourceExec: file_groups={4 groups: 
[[tmp/df-dynfilter-repro-4/part-0.parquet], 
[tmp/df-dynfilter-repro-4/part-1.parquet], 
[tmp/df-dynfilter-repro-4/part-2.parquet], 
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id], 
file_type=parquet, predicate=DynamicFilter [ empty ], 
sort_order_for_reorder=[id@0 ASC NULLS LAST], dynamic_rg_pruning=eligible
   
   original plan returned 1 row(s)
   --- same plan after execution ---
   SortPreservingMergeExec: [id@0 ASC NULLS LAST], fetch=1
     SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST], 
preserve_partitioning=[true], filter=[id@0 < 1]
       DataSourceExec: file_groups={4 groups: 
[[tmp/df-dynfilter-repro-4/part-0.parquet], 
[tmp/df-dynfilter-repro-4/part-1.parquet], 
[tmp/df-dynfilter-repro-4/part-2.parquet], 
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id], 
file_type=parquet, predicate=DynamicFilter [ id@0 < 1 ], 
sort_order_for_reorder=[id@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, 
pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 < 1, 
required_guarantees=[]
   
   --- decoded plan (encoded after execution) ---
   SortPreservingMergeExec: [id@0 ASC NULLS LAST], fetch=1
     SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST], 
preserve_partitioning=[true], filter=[id@0 < 1]
       DataSourceExec: file_groups={4 groups: 
[[tmp/df-dynfilter-repro-4/part-0.parquet], 
[tmp/df-dynfilter-repro-4/part-1.parquet], 
[tmp/df-dynfilter-repro-4/part-2.parquet], 
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id], 
file_type=parquet, predicate=DynamicFilter [ id@0 < 1 ], 
dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 
AND id_min@0 < 1, required_guarantees=[]
   
   decoded plan returned 0 row(s)   [expected 1]  => WRONG RESULT
   
   ================ SELECT * FROM t ORDER BY id DESC LIMIT 3   (4 file(s)) 
================
   control (encode before execute): decoded plan returned 3 row(s)
   --- plan before execution ---
   SortPreservingMergeExec: [id@0 DESC], fetch=3
     SortExec: TopK(fetch=3), expr=[id@0 DESC], preserve_partitioning=[true]
       DataSourceExec: file_groups={4 groups: 
[[tmp/df-dynfilter-repro-4/part-0.parquet], 
[tmp/df-dynfilter-repro-4/part-1.parquet], 
[tmp/df-dynfilter-repro-4/part-2.parquet], 
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id], 
file_type=parquet, predicate=DynamicFilter [ empty ], 
sort_order_for_reorder=[id@0 DESC], reverse_row_groups=true, 
dynamic_rg_pruning=eligible
   
   original plan returned 3 row(s)
   --- same plan after execution ---
   SortPreservingMergeExec: [id@0 DESC], fetch=3
     SortExec: TopK(fetch=3), expr=[id@0 DESC], preserve_partitioning=[true], 
filter=[id@0 IS NULL OR id@0 > 198]
       DataSourceExec: file_groups={4 groups: 
[[tmp/df-dynfilter-repro-4/part-0.parquet], 
[tmp/df-dynfilter-repro-4/part-1.parquet], 
[tmp/df-dynfilter-repro-4/part-2.parquet], 
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id], 
file_type=parquet, predicate=DynamicFilter [ id@0 IS NULL OR id@0 > 198 ], 
sort_order_for_reorder=[id@0 DESC], reverse_row_groups=true, 
dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@0 > 0 OR 
id_null_count@0 != row_count@2 AND id_max@1 > 198, required_guarantees=[]
   
   --- decoded plan (encoded after execution) ---
   SortPreservingMergeExec: [id@0 DESC], fetch=3
     SortExec: TopK(fetch=3), expr=[id@0 DESC], preserve_partitioning=[true], 
filter=[id@0 IS NULL OR id@0 > 198]
       DataSourceExec: file_groups={4 groups: 
[[tmp/df-dynfilter-repro-4/part-0.parquet], 
[tmp/df-dynfilter-repro-4/part-1.parquet], 
[tmp/df-dynfilter-repro-4/part-2.parquet], 
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id], 
file_type=parquet, predicate=DynamicFilter [ id@0 IS NULL OR id@0 > 198 ], 
dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@0 > 0 OR 
id_null_count@0 != row_count@2 AND id_max@1 > 198, required_guarantees=[]
   
   decoded plan returned 2 row(s)   [expected 3]  => WRONG RESULT
   ```
   
   The control (encode the plan before executing it) returns the right row 
count in every case.
   
   ### Expected behavior
   
   Serializing a plan should give the same result whether or not that plan 
object has run in this process. The decoded plan should return 1 row for `ORDER 
BY id LIMIT 1` (and 3 for `ORDER BY id DESC LIMIT 3`), exactly as when the plan 
is encoded before it runs.
   
   ### Additional context
   
   - `DynamicFilterPhysicalExpr::try_to_proto` 
(`datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs`) encodes 
`self.inner.read().clone()`, so `inner_expr`, `generation` and `is_complete` 
are the runtime values.
   - Calling `datafusion::physical_plan::execution_plan::reset_plan_states` 
before encoding works around it for the filter a node produces. As documented, 
it does not reset a dynamic filter that planning pushed into a child (for 
example a hash join's filter pushed into a Parquet scan), so it is not a full 
fix.
   - Possible fixes:
     1. Serialize dynamic filters in their initial state: the placeholder 
expression (`lit(true)`), generation 0 and `is_complete = false`. Runtime state 
is per execution and does not belong in a plan. This needs the initial 
expression to be kept on the `DynamicFilterPhysicalExpr`, or recoverable from 
it.
     2. Alternatively, expose the initial state, e.g. a public 
`DynamicFilterPhysicalExpr::reset()` or `initial_expr()`, so a 
`PhysicalProtoConverterExtension` or a `reset_plan_states`-style pass can 
restore it, including for copies pushed into children.
   - Possibly related: #19905 (dynamic filters are not reset by 
`reset_plan_states`), #21650 (shared mutable state on `ExecutionPlan`), #27130 
(reusing a `limit=1` plan returns an empty result), #25154 (dynamic filter 
identity across serialization). This may belong under #26672.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to