Thanks Yong for driving this! Cwiki not good in my network... Can you send the google doc too?
Best, Jingsong On Tue, Jan 13, 2026 at 6:26 PM Aitozi <[email protected]> wrote: > > Hi Yong: > Thanks for bringing this discussion. File level local sort + Global > file path index looks attractive in the senior of point query. > I think it's common ability to paimon community, Nice to see this feature. > > Best, > Aitozi > > > > > Yong Fang <[email protected]> 于2026年1月13日周二 17:33写道： > > > Hi devs, > > > > I'd like to initiate a discussion on PIP-41: Introduce FilePath Global > > Index And Optimizations For Lookup In Append Table [1]. > > > > We use Paimon as cold storage for sample data of businesses such as search, > > recommendation, and advertising. Given the extremely large volume of sample > > data, we adopt Append Tables for data storage. > > > > Batch jobs read and process this sample data, while lookup by key > > capability for historical data is also required during the sample data > > processing. To support hybrid queries for such ultra-high-dimensional data > > in Paimon, we introduce the FilePath Global Index, along with optimizations > > for reading Parquet metadata and Parquet file data in Append Tables (Some > > of the major optimization designs come from our partner teams cc @lingpeng, > > @guanziyue, thx), aiming to enhance the lookup capability of Paimon Append > > Tables. > > > > Looking forward to hearing from you, thanks > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-41%3A+Introduce+FilePath+Global+Index+And+Optimizations+For+Lookup+In+Append+Table > > > > Best, > > Fang Yong > >
