nevzheng commented on issue #3702: URL: https://github.com/apache/iceberg-rust/issues/3702#issuecomment-6955782851
**TL;DR:** excited about this! Before diving into PR reviews, could we align on the end goal with a short alignment doc and a Rust PoC? Details and questions below. 1. **praise:** thanks for driving this, @comphead. The diagrams are clear and 809s vs. 5,912s is compelling. Happy to help review. 1. **suggestion:** could we start with a short alignment doc? Ideally it has use cases, the end-to-end flow and the PR order, plus a full Rust PoC with benchmark data. That would help reviewers and committers see the end goal and commit to the work. Spark's [SPIP template](https://spark.apache.org/improvement-proposals.html) could be a useful starting point. 1. **question:** how do this epic and the Comet PoC (apache/datafusion-comet#7282) relate? The PoC runs planning, bin-packing and the commit in Scala/Java and uses iceberg-rust only for the existing scan + write, so none of this epic's Rust PRs run in it yet. - **3a.** What's the goal, and what needs to change in iceberg-rust to get there? - **3b.** Grouping the issues into one epic helps, but I don't see how the pieces fit into a working Rust compaction path. Would a PoC or the alignment doc above help show that? The SPIP questions could be a good guide. - **3c.** Can non-Comet engines take advantage of this, and is that in scope? 1. **question:** which PRs are ready for review now? 1. **suggestion:** would you be able to join an upcoming Rust community sync and present this? A walkthrough would be a great way to get more eyes on it. 1. **thought:** this could fit into the Q4 roadmap (#3680), wdyt? cc @mbutrovich @liurenjie1024 @dannycjones Warmly, Nevin (🤖 claude assisted) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
