Hi Jingsong, Thanks for driving this, and for the thorough summary. +1 for cutting a 2.0.0 release.
The areas you listed — data evolution and row tracking, multimodal tables, the global index framework, and native vector / full-text search — together mark a real shift in what Paimon is. It's no longer just a lakehouse table engine, but a storage foundation serving streaming, analytics, and AI/multimodal workloads in one abstraction. A major version is the right way to communicate that, and it also gives us a clean point to consolidate the APIs and defaults accumulated across the 1.x line. A few areas I'd like us to align on before cutting the release: 1. Scope & maturity of the new modules. 2.0 pulls in a fairly large set of new components — paimon-mosaic for the columnar-bucket format, the vector index modules (paimon-vector / paimon-faiss / paimon-lumina, covering the IVF and DiskANN paths), paimon-full-text (Tantivy + BM25), and the globalindex framework in paimon-common/paimon-core, plus the newer format work (paimon-lance, paimon-vortex) and paimon-vfs. I think the first thing we should agree on is which of these ship as stable in 2.0.0 versus experimental, and document that per-module — it sets user expectations and tells us where the release bar actually sits. 2. Compatibility & upgrade path. Existing users' first question will be whether their 1.x tables keep working. I'd suggest a clear compatibility statement — what reads/writes in place under 2.0, what upgrades automatically, and what needs explicit migration — backed by a concise upgrade guide. We should also confirm the table-format and snapshot/manifest compatibility guarantees, so downstream engines (Flink, Spark, Trino, StarRocks) and connectors know what to rely on. 3. Documentation & end-to-end examples. The new surfaces — data evolution workflows, global index creation and reading, vector search across the IVF/DiskANN backends, and full-text search with BM25 and tokenizer config — each need user-facing docs and at least one runnable example. For the AI/multimodal side, a couple of realistic scenarios (RAG-style vector retrieval, and a hybrid scalar + vector + text query) would make them adoptable without reading the source. 4. Test & Java/Python interoperability coverage. Given the scope, it's worth taking stock of end-to-end coverage first — especially the Java/Python interop paths, global index read/write across the btree/bitmap/vector/full-text backends, and data evolution workflows across Flink, Spark, Python, Ray, and Daft. Agreeing on a minimum coverage bar for the headline 2.0 features would give us more confidence in the release. On the data evolution / append row-tracking side, I've been working on that path recently (batch scan and the row-id read/pushdown flow), so I'm happy to audit the remaining known issues there, get them tracked, and drive them to closure ahead of the release. It might also help to start a shared list of blocker / must-include items with a rough target timeline, so we can track 2.0.0 readiness together and see where help is needed. I'm glad to own the data evolution items and to help review the related areas. Best, Nicholas Jiang On 2026/07/07 03:11:44 Jingsong Li wrote: > Hi everyone, > > I would like to start a discussion about preparing an Apache Paimon > 2.0.0 release. > > Over the past release cycles, Paimon has evolved significantly. In > addition to continuing improvements to the core lakehouse table > engine, we have been building a broader storage foundation for > streaming, analytics, and AI/multimodal workloads. > > Some major areas that have landed or are being stabilized include: > > 1. Data Evolution and row tracking > > Paimon now has a much stronger foundation for append tables with row > tracking and data evolution. This enables efficient partial column > updates, schema evolution, dedicated storage for special column types, > and global-index-based retrieval without rewriting entire data files. > > 2. Multimodal table capabilities > > We have introduced multimodal table support, covering blob storage, > vector storage, full-text content, and global indexes in one table > abstraction. This allows Paimon tables to store and query structured > data together with images, videos, audio, embeddings, and text > content. > > 3. Global Index framework > > The global index framework has been extended to support multiple index > types, including BTree, Bitmap, vector indexes, full-text indexes, and > hybrid search. This provides a unified mechanism for scalar filtering, > vector similarity search, text retrieval, and combined retrieval > workflows. > > 4. Vector and full-text search > > We have added native vector index support, including paimon-vindex > IVF-based indexes and Lumina DiskANN-based indexes, as well as native > full-text search integration with BM25 scoring and tokenizer > configuration. These features are important for RAG, recommendation, > image retrieval, and other AI-native scenarios. > > 5. New storage formats and supporting components > > We have also made several supporting components available around > Paimon, including Mosaic, paimon-vindex, and paimon-full-text-index. > > Mosaic provides a columnar-bucket hybrid file format optimized for > wide tables and efficient column projection. paimon-vindex provides > native vector indexing capabilities for Paimon. paimon-full-text-index > provides the native full-text engine used by Paimon full-text global > indexes. > > Together, these components make Paimon more capable as a storage layer > for both traditional lakehouse workloads and emerging AI/multimodal > workloads. > > 6. Ecosystem improvements > > We have continued to improve Flink, Spark, Python, Ray, and Daft > integration, and have been adding more end-to-end coverage for > Java/Python interoperability, global index creation and reading, > vector data, full-text search, and data evolution workflows. > > Given the scope of these changes, I think it is a good time to discuss > whether we should cut a 2.0.0 release. A 2.0 release would help > communicate that Paimon has entered a new stage, with data evolution, > multimodal storage, and global indexes becoming first-class parts of > the project. > > Please share your thoughts, concerns, or any issues that you believe > should block or be included in the 2.0.0 release. > > Best, > Jingsong >
