JingsongLi opened a new pull request, #1315:
URL: https://github.com/apache/paimon-rust/pull/1315

   ### Purpose
   
   Use DataFusion 55's native MERGE planner and Paimon's table provider for 
execution. Planning and EXPLAIN must not mutate tables; executing the DML plan 
applies the merge and returns its affected-row count.
   
   ### Brief change log
   
   - Implement TableProvider::merge_into with typed DataFusion expressions and 
a DataSinkExec, replacing the custom MERGE AST-to-SQL execution path.
   - Preserve data-evolution partial-column updates, copy-on-write rewrites, 
atomic publication, and five-attempt conflict retries. Capture the source once 
and rebuild the target scan and writers on retry, including structured row-ID 
existence conflicts.
   - Preserve positional inserts excluding the virtual _ROW_ID and 
source-scoped unqualified NOT MATCHED expressions. Honor SQLOptions and retain 
Paimon's existing table/action restrictions.
   - Keep target projection pushdown for partial updates, avoid collisions with 
internal tracking-column names, and reject duplicate target actions before 
publication.
   - Fix the two timestamp procedure tests: $snapshots.commit_time is a local 
TIMESTAMP, while procedure arguments are epoch milliseconds. Use deterministic 
snapshot metadata times and assert exact/in-between timestamp boundaries and 
the selected snapshots.
   
   ### Tests
   
   - RUST_MIN_STACK=16777216 cargo test --locked -p paimon-datafusion 
--all-targets (no skip filters).
   - Timestamp procedure tests with TZ=UTC and TZ=Asia/Shanghai.
   - Regression coverage for side-effect-free planning/EXPLAIN, direct 
native-session execution, conflict retries with one source capture, atomic 
failures, identifier/column mappings, BLOB projection, and internal 
tracking-column collisions.
   - cargo fmt --all -- --check
   - cargo clippy --locked -p paimon-datafusion --all-targets --no-deps -- -D 
warnings
   - git diff --check
   
   ### API and Format
   
   MERGE through SQLContext now returns a native lazy DML DataFrame; 
collect/execute the plan to perform the write. Storage formats and public Rust 
signatures are unchanged. The executor materializes the captured source and CoW 
target batches; large-input memory/performance behavior has not been 
benchmarked. Alias resolution follows the native planner; existing 
data-evolution action limitations remain.
   
   ### Documentation
   
   Update SQL support, MERGE execution and syntax limits, SQLOptions behavior, 
and the epoch-millisecond contract for timestamp procedures.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to