nevzheng commented on issue #3702:
URL: https://github.com/apache/iceberg-rust/issues/3702#issuecomment-6955782851

   **TL;DR:** excited about this! Before diving into PR reviews, could we align 
on the end goal with a short alignment doc and a Rust PoC? Details and 
questions below.
   
   1. **praise:** thanks for driving this, @comphead. The diagrams are clear 
and 809s vs. 5,912s is compelling. Happy to help review.
   
   1. **suggestion:** could we start with a short alignment doc? Ideally it has 
use cases, the end-to-end flow and the PR order, plus a full Rust PoC with 
benchmark data. That would help reviewers and committers see the end goal and 
commit to the work. Spark's [SPIP 
template](https://spark.apache.org/improvement-proposals.html) could be a 
useful starting point.
   
   1. **question:** how do this epic and the Comet PoC 
(apache/datafusion-comet#7282) relate? The PoC runs planning, bin-packing and 
the commit in Scala/Java and uses iceberg-rust only for the existing scan + 
write, so none of this epic's Rust PRs run in it yet.
       - **3a.** What's the goal, and what needs to change in iceberg-rust to 
get there?
       - **3b.** Grouping the issues into one epic helps, but I don't see how 
the pieces fit into a working Rust compaction path. Would a PoC or the 
alignment doc above help show that? The SPIP questions could be a good guide.
       - **3c.** Can non-Comet engines take advantage of this, and is that in 
scope?
   
   1. **question:** which PRs are ready for review now?
   
   1. **suggestion:** would you be able to join an upcoming Rust community sync 
and present this? A walkthrough would be a great way to get more eyes on it.
   
   1. **thought:** this could fit into the Q4 roadmap (#3680), wdyt? cc 
@mbutrovich @liurenjie1024 @dannycjones
   
   
   Warmly,
   Nevin (🤖 claude assisted)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to