abnobdoss commented on issue #3554: URL: https://github.com/apache/iceberg-python/issues/3554#issuecomment-4861345525
Thank you for putting this together, this looks great! Personally, and from what I've seen across the community, the items on this list are things a lot of people are looking for. My 2 cents, for what it's worth: I think this might benefit from being split into three separate tracks, each of which is easier to land on its own. 1. **The v2/v3 semantics that are engine-independent.** Most items on the "Operations Unblocked" checklist are gated on table-format work that has to be built in pyiceberg under any engine: equality delete reads (a missing v2 feature tracked in #1210; #3285 already builds the DeleteFileIndex foundation, which this proposal cites), delete-file writes (v2 positional deletes, v3 deletion vectors), and the REPLACE API and commit retry work the proposal already lists as prerequisites for compaction. Whichever executor runs the data step, these are what actually unlock the operations. 2. **Streaming over materialization in the existing pyarrow path.** We are close to having workable end-to-end streaming paths in pyarrow that would mitigate the OOM concerns for delete and overwrite: #3335 (RecordBatchReader in append/overwrite) merged recently, and there's prior PR work on the eager file materialization read path (#3036) to build on. Meanwhile CoW delete still loads each affected file fully into memory (`Transaction.delete` calls `to_table` per file) even though the complementary filter is per-batch. Reviving and finishing that conversion gets us there with no new dependency, and it helps every pyiceberg user by default, no new engine to opt into. 3. **DataFusion for the operations that are genuinely blocking.** What's left after 1 and 2 is the ops where an operator has to hold more than memory: global sort for compaction, large join builds (huge equality delete files, upserts with a huge source). That's where spill-to-disk earns its place, and where I think this proposal is strongest. One thing worth designing in early: delegating these ops crosses a semantics boundary (for example, the spec requires a null delete value to match null rows, which needs `IS NOT DISTINCT FROM` in datafusion rather than `=`, and pyarrow's join can't express it at all), so spec fixtures that run identically against every execution path would keep divergences visible as test failures. Part of why I'd sequence it this way is that review bandwidth is the scarcest resource in any of these projects. Tracks 1 and 2 sit in code paths that are well established in pyiceberg, while a new engine is a long-term commitment (docs, CI, version compatibility, triage), so the narrower its scope when it arrives, the better its odds of landing and staying healthy. More broadly, this issue has surfaced how blurry the line is between the engine pyiceberg uses internally and the engines it serves externally. On that note, @kevinjqliu @Fokko I'd be keen to understand where the existing iceberg-core expansion tickets stand (#2396, #2303, apache/iceberg-rust#1144, apache/iceberg-rust#1694): are those still the desired end state, or deprioritized in favor of this direction? The community has been very active lately, so I think it would be awesome to have a fresh roadmap discussion along the lines of #1856 to recap the longer-term plans and 2026 goals. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
