Hi Lei, Thanks for your input. We anticipate that the number of row groups within a single parquet file will not be too large. Therefore, we can first implement direct sorting by key to ensure data order within individual files. Later, we can further consider more complex sorting algorithms such as Z-Order based on query requirements and the number of row groups.
Best, FangYong On Thu, Jan 15, 2026 at 2:33 PM Yong Fang <[email protected]> wrote: > Hi Jingsong, > > I have separated PIP-41 into PIP-41: Introduce FilePath Global Index And > Optimizations For Lookup In Append Table [1] and PIP-42: Local Sort And > Parquet Lookup Optimizations For In Append Table [2], google docs are > https://docs.google.com/document/d/1qV7lUW5GsZ72IuF9FV96fX3wtJO_gzEWY6aLG97caxU/edit?tab=t.0 > and > https://docs.google.com/document/d/1M6R5V6zRwlj0LeuKnhZcUK4Ss-9JLs9pF42n822k-Bo/edit?tab=t.0 > > [1] > https://cwiki.apache.org/confluence/display/PAIMON/PIP-41%3A+Introduce+FilePath+Global+Index+And+Optimizations+For+Lookup+In+Append+Table > [2] > https://cwiki.apache.org/confluence/display/PAIMON/PIP-42%3A+Local+Sort+And+Parquet+Lookup+Optimizations+For+In+Append+Table > > Best, > FangYong > > On Wed, Jan 14, 2026 at 8:31 PM lei li <[email protected]> wrote: > >> Hi Yong, >> >> >> Thank you for sharing this fascinating PIP proposal! >> >> >> I'm particularly intrigued by the sort buffer concept for Append tables. >> I'd like to ask: does the current sorting process only support multi-field >> sequential sorting, or would it be possible to introduce more advanced >> sorting strategies, such as Z-order sorting or other space-filling curve >> algorithms? >> >> >> The reason I'm curious about this is that in high-volume batch write >> scenarios, having the data already sorted upon completion of the batch >> write could significantly improve query efficiency. Z-order sorting, for >> instance, could provide better data locality for multi-dimensional range >> queries, which is quite common in analytical workloads. >> >> >> Would love to hear your thoughts on whether such sorting enhancements >> align with the current design goals! Looking forward to your insights and >> the continued development of this proposal! >> >> >> Best regards, >> >> Lei Li >> >> 2026年1月13日 17:32，Yong Fang <[email protected]> 写道： >> >> Hi devs, >> >> I'd like to initiate a discussion on PIP-41: Introduce FilePath Global >> Index And Optimizations For Lookup In Append Table [1]. >> >> We use Paimon as cold storage for sample data of businesses such as >> search, >> recommendation, and advertising. Given the extremely large volume of >> sample >> data, we adopt Append Tables for data storage. >> >> Batch jobs read and process this sample data, while lookup by key >> capability for historical data is also required during the sample data >> processing. To support hybrid queries for such ultra-high-dimensional data >> in Paimon, we introduce the FilePath Global Index, along with >> optimizations >> for reading Parquet metadata and Parquet file data in Append Tables (Some >> of the major optimization designs come from our partner teams cc >> @lingpeng, >> @guanziyue, thx), aiming to enhance the lookup capability of Paimon Append >> Tables. >> >> Looking forward to hearing from you, thanks >> >> >> [1] >> >> https://cwiki.apache.org/confluence/display/PAIMON/PIP-41%3A+Introduce+FilePath+Global+Index+And+Optimizations+For+Lookup+In+Append+Table >> >> Best, >> Fang Yong >> >>
