Hi Yong, I think sort and hbase index are totally different stories, can we separate them to two PIPs?
Best, Jingsong On Tue, Jan 13, 2026 at 9:19 PM Jingsong Li <[email protected]> wrote: > > Thanks Yong! > > Best, > Jingsong > > On Tue, Jan 13, 2026 at 8:50 PM Yong Fang <[email protected]> wrote: > > > > Hi Jingsong, I have created a google doc > > https://docs.google.com/document/d/1qV7lUW5GsZ72IuF9FV96fX3wtJO_gzEWY6aLG97caxU/edit?usp=sharing > > from cwiki, please help to review it > > > > Best, > > FangYong > > > > On Tue, Jan 13, 2026 at 6:58 PM Jingsong Li <[email protected]> wrote: > > > > > Thanks Yong for driving this! > > > > > > Cwiki not good in my network... Can you send the google doc too? > > > > > > Best, > > > Jingsong > > > > > > On Tue, Jan 13, 2026 at 6:26 PM Aitozi <[email protected]> wrote: > > > > > > > > Hi Yong: > > > > Thanks for bringing this discussion. File level local sort + Global > > > > file path index looks attractive in the senior of point query. > > > > I think it's common ability to paimon community, Nice to see this > > > feature. > > > > > > > > Best, > > > > Aitozi > > > > > > > > > > > > > > > > > > > > Yong Fang <[email protected]> 于2026年1月13日周二 17:33写道： > > > > > > > > > Hi devs, > > > > > > > > > > I'd like to initiate a discussion on PIP-41: Introduce FilePath Global > > > > > Index And Optimizations For Lookup In Append Table [1]. > > > > > > > > > > We use Paimon as cold storage for sample data of businesses such as > > > search, > > > > > recommendation, and advertising. Given the extremely large volume of > > > sample > > > > > data, we adopt Append Tables for data storage. > > > > > > > > > > Batch jobs read and process this sample data, while lookup by key > > > > > capability for historical data is also required during the sample data > > > > > processing. To support hybrid queries for such ultra-high-dimensional > > > data > > > > > in Paimon, we introduce the FilePath Global Index, along with > > > optimizations > > > > > for reading Parquet metadata and Parquet file data in Append Tables > > > (Some > > > > > of the major optimization designs come from our partner teams cc > > > @lingpeng, > > > > > @guanziyue, thx), aiming to enhance the lookup capability of Paimon > > > Append > > > > > Tables. > > > > > > > > > > Looking forward to hearing from you, thanks > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-41%3A+Introduce+FilePath+Global+Index+And+Optimizations+For+Lookup+In+Append+Table > > > > > > > > > > Best, > > > > > Fang Yong > > > > > > > >
