Hi Kevin/Russell, Thank you for providing those details —they were very helpful.
I've reviewed the Pyiceberg code, but I was wondering if there is an integration test in the Iceberg repository that specifically covers Spark conversion and verifies the FileScanTask objects returned by the doPlanFiles method? Regards, Vaibhav On Tue, Jan 6, 2026 at 8:52 AM Frank Bertsch <[email protected]> wrote: > > Russell, do you perhaps have a link to your slides? The presentation is > great, but a few slides have the bottom right blocked out due to the > presenter view. > > Thanks! > -Frank > > On Mon, Dec 29, 2025 at 3:06 PM Kevin Liu <[email protected]> wrote: >> >> I'd loveeee to have a more technical explanation in the form of a >> project-level blog :) Inspired by Apache Datafusion [1] >> Amogh did a great presentation on the current state of "V4 Adaptive Metadata >> Tree" [2]. It's a good resource for some of the current challenges. >> >> Best, >> Kevin Liu >> >> [1] >> https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/ >> [2] https://www.youtube.com/watch?v=X3SQHWwE1fA >> >> >> >> On Mon, Dec 29, 2025 at 11:38 AM Russell Spitzer <[email protected]> >> wrote: >>> >>> I do want to make a higher level video about scan planning some day ... >>> although I may wait till we finish V4 so I can do it in that context >>> >>> On Mon, Dec 29, 2025 at 1:22 PM Kevin Liu <[email protected]> wrote: >>>> >>>> The video Russell posted is a great resource. >>>> I don't know of any docs or threads specifically explaining the internals >>>> behind scan planning. There might be some in the wild from community >>>> contribution. >>>> >>>> Most of the design is in the code itself. For example, this is the >>>> pyiceberg code for scan planning [1] >>>> >>>> Best, >>>> Kevin Liu >>>> >>>> >>>> [1] >>>> https://github.com/apache/iceberg-python/blob/59cdf33d75cc726c92ca4fa0c7b0ad1e16f8f30a/pyiceberg/table/__init__.py#L1973 >>>> >>>> >>>> On Sat, Dec 27, 2025 at 10:12 AM Vaibhav Kumar <[email protected]> >>>> wrote: >>>>> >>>>> This was a great video. I have been through this already last month. I >>>>> would really appreciate if you have some docs/thread handy for the same. >>>>> >>>>> On Sat, 27 Dec 2025 at 7:19 PM, Russell Spitzer >>>>> <[email protected]> wrote: >>>>>> >>>>>> https://youtu.be/kJaD0WuQ1Bg?si=woefIj2FJlFVMtmV >>>>>> >>>>>> I did a pretty long overview of it earlier this year , this may be more >>>>>> in depth than you are looking for >>>>>> >>>>>> On Sat, Dec 27, 2025 at 6:35 AM Vaibhav Kumar <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Hi Team >>>>>>> >>>>>>> I was trying to understand the internals behind scan planning. Can >>>>>>> someone please help me with relevant design doc/thread or any >>>>>>> community sync video where we would have it in detail? It would help >>>>>>> me to better understand iceberg as a project. >>>>>>> >>>>>>> Regards, >>>>>>> Vaibhav
