Hi dian, Thanks for the information!
I have some questions: 1. Does it have Java SDK? 2. Lance format has no file group concept, so how to write it? Need to collect all records and write per column? Or we need a temporary disk storage? Best, Jingsong On Tue, Aug 27, 2024 at 7:56 PM dian <[email protected]> wrote: > > Hi, all > > Lance format provides better support for machine learning. It can better > support point lookup and wide columns to support vectorized data storage. > And it can be used to build Rag pipelines > > Do we have a plan to support this file format in Paimon? > > Reference: > https://lancedb.github.io/lance/format > https://blog.lancedb.com/lance-v2/ > https://www.databricks.com/dataaisummit/session/building-production-scale-totally-private-oss-rag-pipeline-dbrx-spark
