This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new aeb377ddf01 [blog] Add September 2026 technical articles (#4144)
aeb377ddf01 is described below
commit aeb377ddf01e2f64690bbd3741123ef370adf97b
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Sep 16 11:41:41 2026 +0800
[blog] Add September 2026 technical articles (#4144)
## What this PR changes
Adds five first-party English blog posts for the September 2026
publication schedule:
- Apache Doris 5.0 Preview(1): A Unified Multimodal Lakehouse for
Real-Time Analytics
- Apache Doris x Lance for Autonomous Driving and Robotics Analytics
- Apache Doris x Paimon 2.0: Closing the Agentic AI Data Loop
- Apache Doris 5.0 Variant Across Internal and Open Lakehouse Tables
- Kwai Vector Search on Paimon with Apache Doris
Each post includes a compressed cover and local content images. The Blog
header now features four selected posts, with the Apache Doris 5.0
Preview as the lead story. The Blog authoring guide also documents the
reusable cover design and source-image compression workflow used for
these posts.
## Validation
- Verified required front matter and ASF license headers for all five
posts.
- Verified all 32 Markdown image references resolve to local files.
- Verified the featured Blog orders are unique and contiguous from 1
through 4.
- Checked the new posts for commercial trial, community-group, and
SelectDB references.
- Ran `git diff --check`.
- Did not run a local site build, as required by
`doc-tools/HOW-TO-ADD-BLOG.md`.
---
.../apache-doris-5-multimodal-lakehouse-preview.md | 179 +++++++++++
blog/apache-doris-lance-physical-ai-analytics.md | 346 ++++++++++++++++++++
blog/apache-doris-paimon-agentic-ai-data-loop.md | 231 ++++++++++++++
...n-udf-calling-the-python-ecosystem-from-sql.mdx | 2 -
blog/apache-doris-variant-open-lakehouse.md | 216 +++++++++++++
...ris-clickhouse-elasticsearch-and-opensearch.mdx | 2 +-
...de-a-b-testing-metrics-145x-faster-at-scale.mdx | 2 -
blog/kwai-paimon-vector-search-apache-doris.md | 351 +++++++++++++++++++++
...doris-real-time-sync-with-one-sql-statement.mdx | 2 -
doc-tools/HOW-TO-ADD-BLOG.md | 62 +++-
.../agent-data-loop.jpg | Bin 0 -> 145567 bytes
.../cover.jpg | Bin 0 -> 87006 bytes
.../physical-ai-episode.jpg | Bin 0 -> 143665 bytes
.../search-evolution.jpg | Bin 0 -> 122838 bytes
.../unified-execution.jpg | Bin 0 -> 161474 bytes
.../unified-query-architecture.jpg | Bin 0 -> 130822 bytes
.../autonomous-driving-hard-cases.jpg | Bin 0 -> 136801 bytes
.../cover.jpg | Bin 0 -> 62049 bytes
.../physical-ai-data-architecture.jpg | Bin 0 -> 137890 bytes
.../robot-episode-analysis.jpg | Bin 0 -> 136686 bytes
.../agent-vector-search.jpg | Bin 0 -> 172229 bytes
.../cover.jpg | Bin 0 -> 59881 bytes
.../kwai-vector-search.jpg | Bin 0 -> 190360 bytes
.../open-ai-lakehouse-architecture.jpg | Bin 0 -> 180159 bytes
.../vector-index-query-plan.jpg | Bin 0 -> 136421 bytes
.../agent-trace-dual-write.jpg | Bin 0 -> 83836 bytes
.../apache-doris-variant-open-lakehouse/cover.jpg | Bin 0 -> 83437 bytes
.../hot-cold-data.jpg | Bin 0 -> 112717 bytes
.../open-variant-standard.jpg | Bin 0 -> 96812 bytes
.../unified-variant-execution.jpg | Bin 0 -> 103477 bytes
.../benchmark-overview.jpg | Bin 0 -> 184312 bytes
.../bucket-parallelism.jpg | Bin 0 -> 168151 bytes
.../cover.jpg | Bin 0 -> 71357 bytes
.../ivf-rq-tradeoffs.jpg | Bin 0 -> 173561 bytes
.../late-materialization.jpg | Bin 0 -> 143335 bytes
.../paimon-data-index-manifests.jpg | Bin 0 -> 138273 bytes
.../rust-reader.jpg | Bin 0 -> 136718 bytes
.../top-k-pushdown.jpg | Bin 0 -> 143303 bytes
38 files changed, 1385 insertions(+), 8 deletions(-)
diff --git a/blog/apache-doris-5-multimodal-lakehouse-preview.md
b/blog/apache-doris-5-multimodal-lakehouse-preview.md
new file mode 100644
index 00000000000..b627fa1b4b9
--- /dev/null
+++ b/blog/apache-doris-5-multimodal-lakehouse-preview.md
@@ -0,0 +1,179 @@
+---
+title: 'Apache Doris 5.0 Preview(1): A Unified Multimodal Lakehouse for
Real-Time Analytics'
+summary: 'A look at how Doris plans to extend unified search and real-time
analytics across open lake formats and multimodal data.'
+description: 'Explore the Apache Doris 5.0 multimodal lakehouse preview,
including open-format integration, Variant and vector analytics, Physical AI,
and agent use cases.'
+picked: "true"
+order: "1"
+keywords:
+ - 'Apache Doris 5.0'
+ - 'multimodal lakehouse'
+ - 'open table formats'
+ - 'real-time analytics'
+date: '2026-09-04'
+author: 'Apache Doris · Mingyu Chen'
+tags:
+ - 'Tech Sharing'
+image: '/images/blogs/apache-doris-5-multimodal-lakehouse-preview/cover.jpg'
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+> Overview
+>
+> This article covers the core changes planned for Apache Doris 5.0 for the
multimodal lakehouse:
+>
+> 1. Core concept: Extend unified search to the open lakehouse and use Doris
to build a unified platform for multimodal data analytics.
+>
+> 2. Integration with open formats: Provide read and write integration with
mainstream open formats such as Iceberg, Paimon, and Lance.
+>
+> 3. Multimodal data analytics: Adapt Doris to multimodal data types such as
Variant, Vector, GEO, and Blob.
+>
+> 4. Use cases: Build a unified real-time multimodal data analytics platform
for more AI scenarios, including real-time context for agents and embodied
intelligence.
+
+---
+
+For more than a decade, most data platforms have been organized around
structured tables and SQL. Data is consolidated in a data warehouse, where
stable batch processing and OLAP queries support reporting and business
analysis.
+
+AI changes the assumptions behind this architecture.
+
+Enterprises now need to process business tables, JSON, logs, documents,
images, audio, video, embeddings, and model outputs together. The set of data
consumers has also grown beyond analysts to include applications and AI agents.
A real-world request often combines structured filtering with full-text and
vector retrieval, then joins the results with the latest business status,
permissions, and historical metrics.
+
+The traditional approach spreads these capabilities across separate systems.
Object storage holds raw content, lakehouses hold table data and metadata,
search engines handle full-text retrieval, vector databases provide semantic
retrieval, OLAP systems perform metric analysis, and the application layer
assembles the results. Each system solves a specific problem, but users are
responsible for data consistency, version alignment, access governance, and
result fusion across them.
+
+Open formats are also expanding beyond structured data. AI workloads now
include dynamic attributes, blobs, embeddings, and model results. Iceberg,
Paimon, and Lance serve long-term analytical facts, real-time updated data, and
AI datasets, respectively, so enterprises can select the appropriate format for
each workload. In this article, a multimodal lakehouse refers to structured,
semi-structured, and multimodal data stored collectively in these open formats.
+
+For the multimodal lakehouse, Doris 5.0 will focus unification on query and
execution instead of requiring users to move all data into one storage system.
Data can remain in the open format best suited to it, while Doris provides
unified search, analytics, and real-time serving.
+
+
+
+Figure 1: Overall architecture of an open-format multimodal lakehouse.
Iceberg, Paimon, Lance, and Fluss hold different types of open data assets. At
the unified query and compute layer, Doris understands format semantics and
connects hybrid search, cross-source joins, MPP OLAP, real-time serving, and
access control in one execution pipeline.
+
+> Availability: The Iceberg, Paimon, and Lance capabilities discussed here are
in the Doris 4.2 release branch and are scheduled for public release at the end
of September. The broader Doris 5.0 work remains a preview.
+
+## From 4.x to 5.0: Extending unified search to the open lakehouse
+
+Apache Doris 4.x already provides high-performance unified search and
analytics for structured, semi-structured, and vector data in internal tables.
Users can combine SQL analytics, full-text search, vector search, and real-time
updates in one system. For open lake formats such as Iceberg and Paimon, Doris
has primarily supported queries and analysis of structured data.
+
+Doris 5.0 will continue this work by extending capabilities already proven on
internal tables to the open lakehouse. Variant data in Iceberg and Paimon will
become part of a unified read and write pipeline. Paimon and Lance will gain
native full-text and vector search. A new Fluss Catalog will support joint
queries across real-time data in Fluss and historical data tiered to Paimon,
which will further improve data freshness for analytics.
+
+
+
+Figure 2: Capability evolution from 4.x to 5.0. Doris 4.x provides unified
search and OLAP on internal tables, along with structured SQL for Iceberg and
Paimon. Doris 5.0 extends unified querying to the open lakehouse with Variant
reads and writes, native vector search, and joint queries across real-time
Fluss data and historical Paimon data.
+
+Doris 5.0 will provide the following capabilities across open formats and the
unified execution layer:
+
+- Iceberg: Building on mature support for reads, writes, time travel,
row-level updates, and table lifecycle management, Doris will add reads and
writes for Iceberg V3 Variant data in both plain and shredded layouts.
+- Paimon: The Paimon Catalog will move from read-only access to full read and
write support, including `INSERT`, `INSERT OVERWRITE`, `UPDATE`, `DELETE`,
`MERGE`, DDL, and reads and writes for Variant, full-text indexes, and vector
indexes. It will provide a unified entry point for real-time lakehouses and
evolving multimodal data.
+- Lance: Doris will introduce native read and retrieval capabilities for
multimodal AI, including Catalog support, parallel scans, column pruning,
predicate pushdown, vector and full-text search, and vector index management.
+- Fluss: A new Fluss Catalog will support reads from log tables and
primary-key tables. Through Union Read, it will jointly query real-time
incremental data in Fluss and historical data tiered to Paimon.
+- Unified Doris execution: Doris will connect internal tables, open lake
tables, full-text and vector retrieval, cross-source joins, aggregations,
window calculations, and permission filtering in one SQL pipeline.
+
+## From multiple systems to a unified platform
+
+The difficulty of assembling multiple systems goes beyond the number of
systems involved. The same business entity may have different update times,
versions, and permission boundaries in each system. A product that has been
delisted, a device currently under maintenance, or a document that has been
withdrawn may still appear in a stale vector index and be passed to an agent.
+
+Placing unification at the query and execution layer makes the division of
responsibilities clearer. Open formats continue to store long-term data and
multimodal assets. Doris internal tables hold real-time business facts that
require high concurrency and low latency. The Doris MPP engine performs
retrieval, filtering, joins, and analytics in one query, so applications no
longer need to move and assemble results across multiple systems. The same
layer evaluates business state and applies [...]
+
+
+
+Figure 3: Evolution of the multimodal lakehouse data architecture. The
architecture moves from traditional multi-system assembly to unified execution
with Doris. Data remains in the appropriate open formats, while Doris unifies
querying, computing, and serving.
+
+### OLAP: From retrieving candidates to determining business outcomes
+
+Vector or full-text search discovers candidates. Answering the business
question usually requires joins, aggregations, window calculations, and
multidimensional ranking as well.
+
+For example, image-based product search cannot return products based only on
visual similarity. It must also account for inventory, regional availability,
price, conversion rate, and return rate. Equipment fault diagnosis cannot stop
at finding similar logs. It must compare models, firmware versions, sensor
metrics, and repair outcomes.
+
+Whether candidates come from Doris internal tables or open lake tables, Doris
can pass search results directly into its MPP OLAP execution pipeline. This
allows the query to turn a "similar" result into a business conclusion that is
currently valid, complies with the relevant rules, and is worth acting on.
+
+### Variant, full text, and vector: Combining different forms of evidence in
one SQL query
+
+Model-generated data rarely fits a stable wide-table schema. OCR results,
captions, labels, tool-call parameters, and evaluation results change as model
versions evolve. Variant provides a flexible way to store semi-structured data.
Full-text search supports precise keyword and rule matching, while vector
search broadens semantic recall. Structured conditions constrain time, status,
permissions, and business scope.
+
+Doris 5.0 will make Variant data, full-text indexes, and vector indexes in
open lake tables available within the same SQL query. Users will not need to
copy multimodal data into additional indexing systems solely to make it
searchable.
+
+## Physical AI: From a single record to a complete episode
+
+Data generated by autonomous driving systems, robots, and industrial equipment
records a continuous process of observation, decision, action, and feedback
rather than a collection of static documents. The smallest unit of analysis is
therefore no longer an image. It is an Episode that contains time, space,
device state, model version, and business outcome.
+
+An Episode involving a robotic arm's failed grasp might contain RGB-D images,
3D poses, joint and torque data, planned trajectories, control commands, model
outputs, device logs, and the final sorting result. Images describe the
environment, trajectories describe the action, sensor data records the
execution process, and the business outcome shows whether the task succeeded.
+
+Open formats allow teams to retain this data over the long term, evolve it,
and share it among different tools. Doris organizes these forms of evidence by
Episode into an interactive query. The query first filters by time, device, and
status, then retrieves similar situations through full-text and vector search,
and finally joins model versions with business outcomes to compare failure
rates, retry rates, and task duration.
+
+
+
+Figure 4: Closed-loop Physical AI Episode analysis. Doris organizes
multisensor evidence, model context, and business outcomes into a reusable data
feedback loop.
+
+## Agentic AI: Using real-time analytics to connect models with business
context
+
+Large language models can understand general knowledge, but they do not
inherently know the enterprise's current state. They cannot know whether an
order has been canceled, whether inventory is sufficient, whether a device is
online, whether a customer still has permission, or whether a particular
anomaly is currently surging without access to current enterprise data.
+
+The challenge for an agent is therefore broader than generating SQL. It needs
a continuous supply of fresh, trustworthy, explainable, and
permission-compliant data context.
+
+### Real-time writes: Bringing business changes directly into the open
lakehouse
+
+Doris integration with Iceberg has already progressed from querying a data
source to writing efficiently to an open data foundation. `INSERT`, `INSERT
OVERWRITE`, CTAS, and row-level updates are already available. Doris 5.0 will
bring the same write capabilities to Paimon and add Variant writes for both
Iceberg and Paimon.
+
+Users will be able to use unified SQL to write structured and Variant data
directly to open tables. They will not need to write the data to Doris internal
tables first or build another synchronization pipeline. Real-time business
events, tool-call parameters, and continuously changing model results can be
stored directly as open data assets.
+
+Within the execution pipeline, Doris BEs generate data files through
distributed parallel execution. The FE coordinates snapshots and atomic
commits, and it handles idempotent retries and failure cleanup. Doris manages
efficient writes, commit consistency, and schema evolution through a common
implementation, so applications do not need to assemble open-table transactions
themselves.
+
+### Real-time analytics: Building agent context from the latest open data
+
+Orders, device states, behavioral events, and model-derived results written to
Iceberg or Paimon can be shared among different compute engines. Doris queries
the latest data directly, combines it with real-time business facts in internal
tables, and applies cross-table joins and permission filtering to construct
fresh, verifiable business context for agents.
+
+For event streams with stricter freshness requirements, the new Fluss Catalog
uses Union Read to combine real-time increments in Fluss with historical data
tiered to Paimon and queries them as a single table. The resulting agent
context includes both the complete history and changes that have just occurred.
+
+
+
+Figure 5: A real-time Agentic AI feedback loop using open formats selected
according to workload requirements. Doris provides read, write, and Variant
capabilities for Iceberg or Paimon selected by the user, along with read and
multimodal search capabilities for Lance. It then constructs agent context
through unified SQL and MPP OLAP.
+
+## Multimodal lakehouse use cases with Doris
+
+### Scenario 1: Real-time decision-making for customer service agents
+
+Knowledge documents, product images, and samples of similar faults are stored
in Lance. Orders, support tickets, and historical service data are stored in
Iceberg and Paimon. The latest inventory, warranty, and customer status data is
loaded into Doris internal tables.
+
+After an agent receives a customer question, Doris performs semantic
retrieval, keyword matching, permission and status filtering, then joins the
results with repair success rates and inventory data. The result is more than
"the most similar document." It is a resolution that the current customer can
act on.
+
+### Scenario 2: Selecting high-value Episodes for Physical AI
+
+Autonomous driving or robotics teams store video, point clouds, and embeddings
in Lance. They write continuously changing model outputs and task results to
Paimon as Variant data, while retaining historical evaluation results in
Iceberg. These teams need to answer a question beyond "Which Episodes are
similar?" They need to know, "Which Episodes are worth investing in for
annotation, training, and review?"
+
+After the query pipeline described above filters the Episodes, high-confidence
anomalies enter training and regression datasets. Episodes with inconsistent
evidence go to manual review. Failure and intervention rates aggregated by
model version then show whether a new strategy has improved a specific scenario.
+
+### Scenario 3: From similar defects to root-cause analysis in industrial
quality inspection
+
+A quality engineer uploads an image of a defect. Doris retrieves visually
similar samples from Lance, matches defect descriptions in Variant and
full-text fields, and joins the results with continuously updated equipment
events in Paimon and with batch, process, and repair history in Iceberg.
+
+The system then determines whether defects are occurring in clusters, which
equipment and batches are affected, how accurately different model versions
identify the defect, and whether yields recover in subsequent batches. This
expands a similar-image search into a complete closed-loop quality analysis.
+
+## Conclusion: Keep data in open formats and use Doris for search and analytics
+
+Doris has one primary direction for the multimodal lakehouse: extend the
unified search and real-time analytics capabilities already proven on internal
tables to data in Iceberg, Paimon, Lance, and Fluss. Data can remain in open
formats for search and analytics. Structured data, Variant data, full-text
data, and vectors do not need to reside in separate systems. Doris will provide
the unified execution and serving layer above this data:
+
+- Use unified SQL to access Doris internal tables and multiple open formats.
+- Use Variant, full-text search, vector search, and structured filtering to
query different forms of data.
+- Use MPP OLAP to turn retrieved results into business metrics and decisions.
+- Use real-time writes to supply the latest business state that agents
otherwise lack.
+- Use unified permission filtering so applications and agents see only the
data they are authorized to access.
+
+A series of articles about Doris 5.0 will examine these capabilities for open
formats such as Iceberg, Paimon, Lance, and Fluss. The articles will cover
their technical principles, performance benchmarks, and representative
implementation scenarios.
+
+The multimodal lakehouse capabilities for Iceberg, Paimon, and Lance have been
merged into the Doris 4.2 release branch and are scheduled for official release
with version 4.2 at the end of September. Additional capabilities, including
Fluss support, have also entered the Doris main branch and are planned for
release with Doris 5.0 at the end of the year.
diff --git a/blog/apache-doris-lance-physical-ai-analytics.md
b/blog/apache-doris-lance-physical-ai-analytics.md
new file mode 100644
index 00000000000..c26ccecd8af
--- /dev/null
+++ b/blog/apache-doris-lance-physical-ai-analytics.md
@@ -0,0 +1,346 @@
+---
+title: 'Apache Doris x Lance for Autonomous Driving and Robotics Analytics'
+summary: 'Doris and Lance combine episode search, structured filters, and OLAP
analysis to help teams find hard cases and diagnose failures in Physical AI
data.'
+description: 'How Apache Doris and Lance support hard-case mining for
autonomous driving and failure analysis for robotics with open multimodal
datasets.'
+picked: "true"
+order: "2"
+keywords:
+ - 'Apache Doris'
+ - 'Lance Catalog'
+ - 'Physical AI'
+ - 'episode analytics'
+date: '2026-09-08'
+author: 'Apache Doris · Wenqiang Li'
+tags:
+ - 'Tech Sharing'
+image: '/images/blogs/apache-doris-lance-physical-ai-analytics/cover.jpg'
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+> **Abstract:**
+>
+> Apache Doris 5.0 Multimodal Lakehouse Preview (1) | Scenario solutions:
Autonomous vehicles and embodied intelligence systems generate massive volumes
of multimodal data every day. The practical problem is finding high-value
episodes quickly and identifying the scenarios, versions, and tasks associated
with failures. This article explains how Apache Doris and Lance combine
semantic search, structured filtering, and OLAP analytics for hard-case mining,
failure attribution, and closed-lo [...]
+
+Autonomous vehicles generate video, point clouds, vehicle states, and model
outputs every day. Each robot task also produces multiple camera streams, joint
states, action trajectories, language instructions, and execution results.
Storing this data is only part of the problem. Teams must also answer the
questions that determine how efficiently they can iterate on their models:
+
+- Which segments contain scenarios similar to the current accident?
+- Has the intervention rate of the new model increased at night, in
construction zones, or around irregularly shaped obstacles?
+- Which types of robot tasks are most likely to fail, and what similar visual
or action patterns appear before failure?
+- How can selected high-value episodes be passed to annotation, training, and
regression evaluation?
+
+Traditional data warehouses handle structured aggregation well. Vector
databases specialize in image similarity search, while object storage holds
large numbers of files at low cost. Physical AI needs these capabilities to
work together, so semantic search, structured filtering, business metrics, and
raw evidence can be used in the same analytical workflow.
+
+Apache Doris support for Lance Catalog provides that connection. Lance stores
open multimodal datasets and vector indexes designed for AI. Doris queries the
data directly and performs predicate pushdown, parallel scans, vector
retrieval, global Top-K selection, and OLAP aggregation within a single SQL
execution framework. Teams do not need to copy the data into a dedicated
analytical replica first, and their AI data assets remain available to more
than one compute system.
+
+## 1. Why physical AI needs semantic search and analytics, not vector search
alone
+
+Data for autonomous driving and embodied intelligence is naturally organized
around episodes. An episode is a reconstructable physical process rather than a
single image: sensors observe the environment, a model makes a decision, a
control system executes an action, and the physical world produces an outcome.
+
+In autonomous driving, finding images similar to the five seconds before an
accident is only the first step. Engineering teams must further constrain the
search by vehicle model, sensor version, road type, weather, model version, and
time range. They then need to compare intervention rates, false-positive rates,
or planning deviations among the candidate segments. The same applies to
embodied intelligence. A failed grasp involves visual semantics, but it must
also be examined in the cont [...]
+
+Finding high-value data therefore requires at least three layers of
computation:
+
+1. Use structured conditions to narrow the search space to the correct
business context.
+2. Use vector similarity to find related scenarios that the labeling system
does not yet cover.
+3. Use aggregation, joins, sorting, and window analysis to turn candidate
segments into model evaluation findings and actionable datasets.
+
+Lance and Doris handle different parts of this process. Lance manages AI
datasets, random access, and vector indexes. Doris brings the retrieval results
into an explainable, composable SQL analytics system.
+
+## 2. Overall architecture: Lance manages AI data assets, and Doris provides
unified queries and analytics
+
+
+
+**Figure 1. Apache Doris and Lance in the Physical AI data architecture**
+
+In this architecture, upstream systems continue to use tools such as the Lance
SDK, Ray, and Spark to generate and maintain Lance datasets. These datasets
include features, embeddings, labels, model outputs, and existing Lance vector
indexes. The data can reside in a local file system, S3-compatible object
storage, or OSS.
+
+Apache Doris uses Lance Catalog to discover datasets and schemas. During a
query, Doris fixes the snapshot, divides fragments or index segments, runs
scans and searches in parallel, and performs global Top-K selection and OLAP
computation.
+
+Apache Doris covers a layer that can otherwise be fragmented across separate
tools before and after training. Engineers, algorithm teams, business analysts,
and agents can use the same system to find data, validate models, locate
problems, and construct the next training dataset.
+
+## 3. Autonomous driving: Find similar video and build a closed hard-case
mining loop
+
+### 3.1 Finding high-value scenarios is the bottleneck
+
+In autonomous driving development, hard-case mining means finding high-value
scenarios within massive volumes of driving data where models are prone to
false positives, false negatives, abnormal braking, or human intervention.
These segments make up only a small proportion of all fleet data, but they
directly affect model safety and coverage of long-tail scenarios.
+
+Collecting the data is usually easier than finding the right records within
the full set of driving data. The main bottlenecks and corresponding solutions
include:
+
+- **Anomalous segments are rare, and their context is scattered.** Vehicles
continuously generate data from multiple cameras, point clouds, localization
systems, vehicle states, and model outputs. An anomaly may exist only in a
segment lasting a few seconds and may occur across different vehicle models,
roads, weather conditions, and sensor versions. Doris can first narrow the
search by business conditions such as time, vehicle model, road, weather, and
model version.
+- **Existing labels cannot cover all semantics.** Queries based only on time,
vehicle, or labels can easily miss semantically similar segments that have not
yet been labeled. Lance stores episode metadata, embeddings, and vector
indexes, and Doris reuses those indexes to retrieve similar scenarios.
+- **Similarity results are not yet engineering conclusions.** Vector
similarity search alone may return results from irrelevant business contexts
and cannot directly determine issue frequency, model-version regressions, or
sample value. Once the candidate results enter Doris, distributed statistics
and version comparisons can identify the episodes that should be included in
annotation, training, and evaluation.
+
+### 3.2 A practical hard-case mining workflow
+
+Suppose a team wants to find segments that are visually and semantically
similar to an emergency-braking event involving an irregularly shaped obstacle.
The event occurred on urban roads, and the team wants to determine whether the
problem is concentrated in a particular perception model version:
+
+1. The Lance dataset stores `episode_id`, time, road and weather labels,
vehicle model, model version, intervention or emergency-braking results, image
or segment embeddings, and raw media location information.
+2. Doris pushes down conditions such as time range, urban roads, vehicle
model, and model version to narrow the search space before generating vector
candidates.
+3. Doris reuses existing Lance vector indexes and searches them in parallel.
Fragments not covered by an index are automatically supplemented with Flat
Search.
+4. Each parallel task returns local candidates, which Doris merges into a
global Top-K result.
+5. SQL aggregates intervention rates, emergency-braking rates, and sample
counts by model version, weather, and road type to produce a hard-case
distribution.
+6. Candidate `episode_id` values are sent for human review, annotation,
inclusion in a training dataset, or regression evaluation. The Lance SDK reads
the raw media precisely.
+
+
+
+**Figure 2. Autonomous-driving hard-case mining: from fleet data to model
iteration**
+
+This workflow does not require every capability to live in one storage format.
Instead, it keeps similar scenarios connected to their business context.
Algorithm engineers receive more than a page of similar images. They can
determine which versions exhibit similar problems, whether a regression is
occurring, how many vehicles and road conditions are affected, and which
episodes are most useful for the next training cycle.
+
+### 3.3 Find similar scenarios with a single SQL query
+
+Suppose an upstream tool has already created a `scene_embedding` index in the
Lance dataset. Doris can start a search through `vector_search()`:
+
+```SQL
+SELECT
+ episode_id,
+ event_time,
+ road_type,
+ weather,
+ model_version,
+ intervention,
+ _distance
+FROM vector_search(
+ "table" = "lance_ai.autodrive.scene_episodes",
+ "column" = "scene_embedding",
+ "query_vector" = "[0.12, -0.08, 0.31, ...]",
+ "top_k" = "200",
+ "metric" = "cosine",
+ "nprobes" = "20",
+ "refine_factor" = "10",
+ "filter" = "road_type = 'urban' AND event_time >= '2026-08-01'",
+ "use_index" = "true"
+)
+ORDER BY _distance ASC, episode_id;
+
+
+```
+
+Here, `filter` performs pre-filtering. It first restricts the search to the
correct scenario range, then generates vector candidates. This behavior usually
fits the semantics of hard-case mining better than post-filtering with `WHERE`
outside the TVF. A smaller `_distance` value indicates greater similarity.
Adding a unique key explicitly as the second sort field produces a stable
return order when distances are equal.
+
+Vector candidates can also feed standard SQL aggregation. For example, the
following query calculates the distribution of interventions in similar
scenarios by model version:
+
+```SQL
+WITH similar_scenes AS (
+ SELECT model_version, weather, intervention, _distance
+ FROM vector_search(
+ "table" = "lance_ai.autodrive.scene_episodes",
+ "column" = "scene_embedding",
+ "query_vector" = "[0.12, -0.08, 0.31, ...]",
+ "top_k" = "1000",
+ "metric" = "cosine",
+ "filter" = "road_type = 'urban'",
+ "use_index" = "true"
+ )
+)
+SELECT
+ model_version,
+ weather,
+ COUNT(*) AS scene_count,
+ SUM(CASE WHEN intervention THEN 1 ELSE 0 END) AS intervention_count
+FROM similar_scenes
+GROUP BY model_version, weather
+ORDER BY intervention_count DESC;
+
+
+```
+
+## 4. Embodied intelligence: Make every episode searchable, comparable, and
reviewable
+
+### 4.1 Robot data requires cross-frequency alignment, random access, and
cross-episode analytics
+
+An embodied intelligence task typically produces data in several forms and at
different frequencies. Robot arm states and control signals may be collected at
hundreds of hertz, while multiple cameras record environmental changes at tens
of frames per second. Other data includes language instructions, frame-level
labels, intermediate model results, and information about task success,
failure, and retries.
+
+If this data is stored separately in video files, columnar files, and JSON,
teams must maintain timestamp alignment and version mappings across those
files. They must also repeatedly implement data-assembly logic for training,
playback, and problem analysis. A random read may require locating the video
segment first, then finding the corresponding state and action records. When
schemas or features change, separate copies of the data can become inconsistent.
+
+Training frameworks focus on reading batches of samples at high throughput.
Research and development analytics has a different job: locating similar
failures among millions of episodes, comparing policy and hardware versions,
and determining whether problems originate in perception, planning, control, or
execution. Random access alone cannot provide this type of cross-episode
aggregation and attribution. Traditional data warehouse full-table scans are
also unsuitable for all multimodal s [...]
+
+Lance organizes states, actions, features, embeddings, and media location
information by episode and timeline while preserving random-access paths for
training and review. Doris performs conditional filtering, vector retrieval,
and cross-episode aggregation over the supported fields. With this division of
responsibilities, the same robot data can support model training, unified
analysis, and review.
+
+### 4.2 From failed tasks to a closed training-data loop
+
+For failures involving robot arms attempting to grasp transparent or
reflective objects, teams can use the following workflow:
+
+1. Data from multiple cameras, joint states, action trajectories,
instructions, policy versions, and task results enters Lance by episode.
+2. An upstream model generates visual or trajectory embeddings and creates
vector indexes in Lance.
+3. Doris first performs pre-filtering by robot model, task type, policy
version, and `success = false`.
+4. Vector search finds images or action segments similar to the target failure
pattern.
+5. Doris aggregates failure reasons, retry counts, task duration, hardware
versions, and policy versions to determine whether the issue is related to
perception, planning, control, or hardware.
+6. High-value episodes are sent for review, relabeling, and training. The
corrected model is then subjected to regression validation against the same
dataset.
+
+
+
+**Figure 3. Embodied-intelligence episode analysis: from multi-frequency
sensors to a closed training loop**
+
+This workflow also works well for agentic data processing. An agent can use
SQL to define the task scope, invoke vector search to locate similar failures,
and generate distribution statistics and a list of candidate episodes. Each
step retains its filter conditions, SQL, and data versions. Reviewers can
inspect how the results were produced instead of relying on a model to search
file directories or a vector database without a traceable procedure.
+
+### 4.3 Example: Search for robot failure patterns
+
+The upstream system generates `query_vector` with the same embedding model
used for `frame_embedding`. For example, it can pass a reference image, a
representative video frame, or an action trajectory to the model and then
provide the resulting floating-point array to `vector_search()`. The query
vector's dimensions, numeric type, and normalization method must match the
vector column in the dataset and the settings used when the index was created.
The `[-0.04, 0.27, 0.18, ...]` value in [...]
+
+```SQL
+SELECT
+ episode_id,
+ robot_model,
+ policy_version,
+ task_type,
+ failure_reason,
+ retry_count,
+ _distance
+FROM vector_search(
+ "table" = "lance_ai.robotics.task_episodes",
+ "column" = "frame_embedding",
+ "query_vector" = "[-0.04, 0.27, 0.18, ...]",
+ "top_k" = "300",
+ "metric" = "cosine",
+ "filter" = "task_type = 'pick' AND success = false",
+ "use_index" = "true"
+)
+ORDER BY _distance ASC, episode_id;
+
+
+```
+
+The results can then be aggregated by `policy_version`, `robot_model`, or
`failure_reason`. The query can instead return `episode_id` and media location
fields, allowing an application to use the Lance SDK to open the corresponding
video frames, point clouds, or other Blob content for review.
+
+> Availability: Lance Catalog support is in the Doris 4.2 release branch and
is scheduled for public release at the end of September. The SQL below shows
the planned public syntax.
+
+## 5. How to use Lance Catalog in Apache Doris
+
+### 5.1 Connect to a Lance dataset in object storage
+
+The following example uses S3-compatible object storage. In production
environments, supply access keys through secure credential management rather
than writing them directly into shared scripts.
+
+```SQL
+CREATE CATALOG lance_ai PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "s3://<bucket>/physical-ai",
+ "s3.endpoint" = "https://<object-storage-endpoint>",
+ "s3.region" = "<region>",
+ "s3.access_key" = "<access-key>",
+ "s3.secret_key" = "<secret-key>"
+);
+
+
+```
+
+Doris also supports local file systems, OSS, and Lance REST Catalogs
authenticated with Bearer Tokens, API Keys, or custom headers. Filesystem
Catalog maps namespaces by directory: a first-level directory under the
warehouse root corresponds to a Doris database, and a `.lance` directory
corresponds to a table.
+
+### 5.2 Discover datasets, schemas, and existing indexes
+
+```SQL
+SHOW DATABASES FROM lance_ai;
+SHOW TABLES FROM lance_ai.autodrive;
+DESC lance_ai.autodrive.scene_episodes;
+SHOW INDEX FROM lance_ai.autodrive.scene_episodes;
+
+
+```
+
+`SHOW INDEX` displays indexes that tools in the Lance ecosystem have already
created.
+
+### 5.3 Run vector queries through the Catalog
+
+```SQL
+SELECT episode_id, event_time, model_version, _distance
+FROM vector_search(
+ "table" = "lance_ai.autodrive.scene_episodes",
+ "column" = "scene_embedding",
+ "query_vector" = "[0.12, -0.08, 0.31, ...]",
+ "top_k" = "200",
+ "metric" = "cosine",
+ "filter" = "road_type = 'urban'",
+ "use_index" = "true"
+)
+ORDER BY _distance ASC, episode_id;
+
+
+```
+
+Catalog creation, dataset discovery, and vector search all remain within the
same Catalog namespace. `filter` restricts candidates to urban-road scenarios
before candidate generation, while `_distance` returns the similarity distance.
The query vector's dimensions and distance metric must match those used by the
upstream system to generate embeddings and create the Lance index.
+
+Pre-filtering and post-filtering differ in when their conditions take part in
candidate generation.
+
+- A condition in the `filter` parameter of `vector_search()` is a pre-filter.
Lance first filters the data and then performs vector retrieval within that
subset. Therefore, `top_k` represents the Top-K results among the filtered
data. This is suitable for conditions such as road type, time range, vehicle
model, tenant, or data permissions that must constrain the search space.
+- A condition in `WHERE` outside the TVF is a post-filter. Vector search
produces candidates first, and Doris then filters those candidates. This is
suitable for expressions that Lance cannot currently push down or for scenarios
that require semantic retrieval before business filtering. However, the final
result may contain fewer than `top_k` entries, and the system will not
automatically expand the retrieval range to fill the requested count.
+
+A query can combine both approaches. Put strong constraints that define the
search scope in `filter`, place complex calculations or remaining conditions in
the outer `WHERE`, and use `EXPLAIN` to confirm which conditions have been
pushed down.
+
+### 5.4 Use EXPLAIN to confirm predicate pushdown and index use
+
+Before running a query, use `EXPLAIN` with the same `vector_search()` query.
In the execution plan, `lancePushdownPredicate` indicates that pre-search
conditions have been pushed down, while `lanceSearchIndexSegments` indicates
that Lance index segments have been planned. Doris supplements new fragments
not covered by the index with Flat Search so that the query includes all data
in the current snapshot.
+
+```SQL
+EXPLAIN
+SELECT episode_id, _distance
+FROM vector_search(
+ "table" = "lance_ai.autodrive.scene_episodes",
+ "column" = "scene_embedding",
+ "query_vector" = "[0.12, -0.08, 0.31, ...]",
+ "top_k" = "200",
+ "metric" = "cosine",
+ "filter" = "road_type = 'urban'",
+ "use_index" = "true"
+)
+ORDER BY _distance ASC, episode_id;
+
+
+```
+
+### 5.5 Use a TVF for temporary reads without creating a Catalog
+
+If you only need to analyze a specific Lance dataset temporarily and do not
want to manage it through a Catalog, you can use the S3 TVF directly. The TVF
is suitable for path-based reads and scalar analysis. If you need to reuse
datasets, schemas, and vector indexes from a Catalog, use the Lance Catalog and
`vector_search()` described above.
+
+```SQL
+SELECT episode_id, event_time, model_version
+FROM s3(
+ "uri" = "s3://<bucket>/physical-ai/autodrive/scene_episodes.lance",
+ "s3.endpoint" = "https://<object-storage-endpoint>",
+ "s3.region" = "<region>",
+ "s3.access_key" = "<access-key>",
+ "s3.secret_key" = "<secret-key>",
+ "format" = "lance"
+)
+WHERE event_time >= '2026-08-01';
+
+
+```
+
+## 6. Conclusion: Connect AI data to decisions
+
+Autonomous driving and embodied intelligence systems will continue to produce
large amounts of data. The scarce resource is the set of high-value episodes
that can explain model problems, cover long-tail scenarios, and inform the next
iteration.
+
+Apache Doris and Lance provide a direct path from multimodal assets to
business decisions while retaining open formats. Lance organizes and indexes AI
data and supports random access. Doris makes the data filterable, searchable,
aggregatable, and understandable by both people and agents through unified SQL.
+
+With this workflow, finding similar scenarios leads directly to questions
about which version is affected, how often the issue occurred, whether it is a
regression, and which training dataset should include it. Multimodal data can
then participate in the closed loop for model iteration and business
decision-making.
+
+### 6.1 Doris and Lance development roadmap
+
+The multimodal lakehouse capabilities for Iceberg, Paimon, Lance, and other
formats have currently been merged into the Doris 4.2 release branch and are
planned for official release with version 4.2 at the end of September.
+
+Future work will focus on the performance, stability, and usability of Doris
and Lance in the following areas:
+
+- Establishing reproducible benchmarks across different data volumes, vector
dimensions, Top-K values, filter selectivity, index coverage, concurrency
levels, and cold or warm caches.
+- Measuring P50, P95, and P99 latency, throughput, recall, bytes scanned,
object-storage requests, network traffic, peak CPU usage, and peak memory usage.
+- Continuing to optimize split balancing, large Top-K merges, high-dimensional
vectors, lookups of wide result columns, object-storage reads, and cache paths.
+- Providing scenario-specific recommendations for parameters such as
`nprobes`, `refine_factor`, `ef`, and `use_index`, and developing
capacity-planning and troubleshooting guides.
+- Improving Lance table-management capabilities.
+- Supporting data export to Lance files.
diff --git a/blog/apache-doris-paimon-agentic-ai-data-loop.md
b/blog/apache-doris-paimon-agentic-ai-data-loop.md
new file mode 100644
index 00000000000..0bc26aebc02
--- /dev/null
+++ b/blog/apache-doris-paimon-agentic-ai-data-loop.md
@@ -0,0 +1,231 @@
+---
+title: 'Apache Doris x Paimon 2.0: Closing the Agentic AI Data Loop'
+summary: 'See how Paimon stores changing business facts and vectors while
Doris connects retrieval, analysis, and SQL write-back for agents.'
+description: 'A technical look at Apache Doris and Paimon 2.0 for Agentic AI,
covering vector indexes, fresh business context, SQL queries, and feedback
write-back.'
+keywords:
+ - 'Apache Doris'
+ - 'Apache Paimon 2.0'
+ - 'Agentic AI'
+ - 'vector search'
+date: '2026-09-09'
+author: 'Apache Doris · Wenqiang Li'
+tags:
+ - 'Tech Sharing'
+image: '/images/blogs/apache-doris-paimon-agentic-ai-data-loop/cover.jpg'
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+> **Abstract**
+>
+> [Apache Doris 5.0 Multimodal Lakehouse
Preview](/blog/apache-doris-5-multimodal-lakehouse-preview) (2) | Scenario
solutions:
+>
+> Agentic AI is moving enterprise AI beyond one-off question answering and
into a continuous "perceive-retrieve-analyze-act-feedback" cycle. Paimon 2.0
manages changing business facts, multimodal data, and indexes in one system,
while Doris uses unified SQL to connect vector retrieval, real-time analytics,
and result write-back. Together, they give agents current, consistent, and
verifiable business context.
+
+---
+
+Large models determine what an agent can understand. The data platform
determines what the agent knows and can trust at a given moment, along with
what it retains after taking action. Agentic AI is moving enterprise AI beyond
one-off question answering and into a continuous
"perceive-retrieve-analyze-act-feedback" cycle.
+
+Agents need access to more than documents and knowledge bases. They also need
current orders, inventory, device status, images, videos, vectors, model
outputs, and historical results. A real request requires more than semantic
retrieval. It may combine vector and full-text search with structured
filtering, cross-table joins, metric computation, and permission checks. This
creates two requirements for the data platform. The underlying data must
support continuous updates and schema evolut [...]
+
+Apache Paimon 2.0 and Apache Doris address these requirements at different
layers. Paimon 2.0 manages continuously changing business facts, multimodal
content, model features, and indexes. Doris provides unified SQL for reads,
writes, vector retrieval, relational computation, and application serving.
Together, they supply agents with current, consistent, and verifiable business
context.
+
+
+
+## 1. Core challenges of Agentic AI
+
+Traditional BI primarily answers, "What happened in the past?" Traditional RAG
primarily answers, "Which documents are relevant to this question?" Agentic AI
must answer additional questions: Does the current state permit an action? Do
separate pieces of evidence support the same conclusion? What happened after
similar actions in the past? How will this action become part of the context
for the next request? Consider an industrial fault-diagnosis agent that
receives a new defect image. R [...]
+
+The agent must also determine whether those defects came from the same
equipment model and production stage. It needs to check whether sensor metrics
were abnormal before or after the event, whether the equipment is currently
under maintenance, whether similar problems are occurring in a cluster, and
which remediation method improved the yield of subsequent batches. Requests of
this kind expose four data challenges.
+
+### 1.1 Context: continuously growing multimodal data
+
+A business record is rarely completed in one operation. Orders, equipment
records, and user states may arrive first. Images, videos, and logs appear
later. Different jobs then add OCR results, captions, embeddings, model scores,
and human-generated labels asynchronously. Model upgrades add new versions of
features to the same data. Agents therefore need more than a static wide table.
They need a data asset that can keep acquiring fields, content, features, and
indexes.
+
+### 1.2 Retrieval produces candidates; business analysis produces answers
+
+Vector retrieval can find semantically or visually similar objects, but it
cannot determine inventory, permissions, equipment status, time ranges, or
business outcomes on its own. To produce context on which an agent can act, the
system must combine vector or full-text retrieval with structured filtering,
joins, aggregations, window computations, and business-specific ranking.
+
+### 1.3 Agents need current and consistent context
+
+When business facts and model features are spread across lake tables, object
storage, vector databases, and search systems, each system may have different
update times, versions, and deletion states. A stale index may still return a
delisted product, a device under maintenance, or a withdrawn document and pass
it to an agent.
+
+### 1.4 Agent actions must feed back into the data loop
+
+Agents produce classifications, summaries, risk scores, tool-call results, and
processing states. Human review adds final labels. If those results remain in
the application layer, the next analysis cannot use the latest feedback. The
data platform must support reads and write-back while preserving transactional
consistency. Agentic AI therefore needs more than a vector database. It needs a
complete data pipeline in which multimodal data can evolve, indexes can expand
their coverage, busi [...]
+
+## 2. Paimon 2.0: continuously evolving the data required by agents
+
+Paimon has traditionally used Flink to ingest database CDC and message streams
continuously. It maintains real-time lake tables on object storage and supports
updates, deletions, snapshots, and incremental reads. Paimon 2.0 expands that
scope from "continuously updated structured rows" to "continuously growing
multimodal data assets."
+
+- `BLOB` stores large objects such as images, videos, and audio in separate
files. Queries do not need to read the object contents when the column is not
selected.
+- `VECTOR<t, n>` provides fixed-dimension semantics and a specialized storage
layout for embeddings.
+- `VARIANT` stores events, model outputs, and tool-call parameters with
flexible structures, while allowing frequently used fields to be extracted into
typed subcolumns.
+- Data Evolution allows models to add or update only changed columns without
repeatedly rewriting unchanged large objects and business fields.
+- Global Row ID aligns data files, BLOBs, vectors, indexes, and deletion
states with the same logical record.
+- Global Index incorporates scalar, full-text, and vector indexes into the
lake table's metadata and snapshot system.
+
+These capabilities do not force every type of data into the same file format.
Each modality can use a suitable physical layout while remaining consistent
with the same table, snapshot, and logical record. The system can identify
newly written data and the current index coverage, then address freshness by
backfilling indexes or selecting an appropriate query mode.
+
+Paimon 2.0 handles how agent data is stored, updated, evolved, and indexed. It
does not, by itself, perform cross-table joins, global Top-K computation,
real-time metric calculation, permission checks, or high-concurrency serving.
Turning this open data into context that agents can use also requires Doris's
query and execution capabilities.
+
+## 3. Doris and Paimon integration: turning open data into agent context
+
+Doris support for Paimon 2.0 goes beyond scanning Paimon files as ordinary
Parquet files. The integration connects three paths: snapshot-consistent
lake-table reads, multimodal retrieval through Vector Index, and standard SQL
writes to Paimon.
+
+### 3.1 Reading the latest business facts while preserving Paimon table
semantics
+
+Doris discovers databases and tables through the Paimon Catalog and caches
their metadata. Within one SQL statement, it binds a consistent snapshot and
schema. The query planner understands manifests, Primary Key merging, Deletion
Vector, incremental ranges, Time Travel, Branch, and Tag, so it does not treat
Paimon Primary Key tables as ordinary file scans. Agents can retrieve a
consistent business state from a specified snapshot and use incremental queries
to detect changes in orders, i [...]
+
+### 3.2 Vector retrieval: bringing Vector Index into the Doris query plan
+
+Doris FE plans and distributes splits, while BE calls the Paimon Rust Reader
through the C ABI. A vector query first reads the Vector Index, searches index
shards in parallel, and merges the candidates. It then pushes the candidate
range down to files, Row Groups, and Parquet Pages. The reader loads vectors,
BLOB metadata, and other wide columns only after confirming matches. Results
from Rust enter C++ through the Arrow C Data Interface and are converted into
Doris Blocks.
+
+The candidate set then enters the Doris MPP engine for scalar filtering,
cross-table joins, aggregation, permission checks, reranking, and global Top-K
computation. Vector retrieval is a candidate-generation stage within a
distributed SQL plan, not a separate system beside Doris. Paimon Vector Index
finds relevant data, while Doris determines whether that data is valid,
important, and actionable in the current business context.
+
+Community work and Kwai's production deployments have validated this path.
Paimon external tables connect to Doris through the Rust Reader, with data
scales ranging from millions to billions of records and Top-K values ranging
from 100 to 1 million. Candidate pushdown and on-demand materialization limit
the cost of reading wide vectors.
+
+
+
+### 3.3 Real-time writes: writing to Paimon with standard SQL
+
+An Agentic AI data pipeline cannot stop after reading data. If aggregation
results, model labels, risk scores, and human-review states produced by Doris
still require separate Spark or Flink jobs for write-back, the architecture
again has fragmented transactional boundaries and additional operational
pipelines.
+
+For Doris 4.2, the Paimon Catalog is planned to expand from read-only access
to integrated read and write support, including:
+
+- `INSERT INTO ... SELECT` and `INSERT ... VALUES` for appending query results
or application data to Paimon.
+- `INSERT OVERWRITE` for overwriting non-partitioned tables or data in static
or dynamic partitions.
+- `UPDATE`, `DELETE`, and `MERGE` for updating business states, model labels,
and human feedback in Primary Key tables.
+- `CREATE TABLE` and schema evolution for adding, deleting, renaming, and
changing the types of fields.
+- Variant reads and writes, allowing model results and tool-call parameters
with continuously changing structures to enter open lake tables directly.
+
+Doris distributes data across BEs according to Paimon's partitioning and
bucket rules, and the BEs generate files in parallel. The FE then collects
Commit Messages from multiple writers and creates an atomically visible Paimon
snapshot. When a failure or retry occurs, the commit coordinator handles
termination, cleanup, deduplication, and idempotent recovery. This prevents
partially visible data and duplicate commits. Results produced by agents can
return to the same open dataset and bec [...]
+
+## 4. Execution framework: how an agent request completes retrieval, analysis,
and write-back
+
+
+
+For vector retrieval in an agent request, Paimon Vector Index first narrows
the candidates. Doris then performs the business analysis. After a vector query
enters Doris, execution proceeds in three stages:
+
+1. **Plan the context.** Doris FE parses the Vector SQL, binds the Paimon
snapshot and schema used by the query, and generates parallel splits.
+2. **Retrieve and read on demand.** BE invokes Paimon Rust through the C ABI.
Vector Index produces Top-K candidates, after which the reader prunes files,
Row Groups, and Pages according to the candidate range and materializes only
matching columns.
+3. **Generate the business answer.**
+
+ 1. Arrow data is converted into Doris Blocks and enters the MPP engine for
joins, filters, aggregation, reranking, and global Top-K computation. The
result is then returned to the agent through SQL, an API, or a tool call.
+ 2. When an agent or human confirmation produces new labels, states, or
processing results, Doris starts another write path. Nereids generates a DML
plan, data is routed to parallel writers on the BEs, and Commit Messages are
returned to the FE for aggregation and atomic commit as a new Paimon snapshot.
The new results then become available for the next round of incremental reads,
index backfilling, and agent analysis.
+ 3. With further native integration, Doris will directly consume Paimon
Global Index Results, use Global Row IDs to read matching rows, and understand
index coverage, the `fast` / `full` / `detail` freshness modes, and the
Deletion Vector of the current snapshot. Manifests, index shards, vector pages,
and matching data pages will also be managed separately through a tiered local
cache.
+
+## 5. Production use: Kwai brings vector retrieval back to Paimon lake tables
+
+
+
+In Kwai's production environment, vectors and business data remain in Paimon,
while Doris handles retrieval, row lookup, and analysis. Kwai needed to provide
large-scale vector retrieval on its existing Paimon real-time lakehouse while
keeping business data, vectors, permission fields, and update states consistent.
+
+Copying vectors, filter fields, and return fields into a separate vector
database would create a second copy of the business data and another
synchronization pipeline that would need long-term maintenance. As model
versions, business states, and deletion states change, the retrieval system
could return stale results that no longer match the current business state.
+
+Kwai chose to keep vectors and business data in Paimon Primary Key tables and
connect vector retrieval to Doris external-table queries:
+
+1. Paimon tables store business fields and 2,048-dimensional vectors inline.
An IVF-RQ Vector Index retrieves candidates, while remote Alluxio provides
caching for data and index reads.
+2. Users continue to use Doris SQL. Doris FE plans and distributes splits, and
BE invokes the Paimon Rust Reader through the C ABI.
+3. Paimon Rust executes vector retrieval in parallel and merges the
candidates. Candidate ranges are pushed down to files, Row Groups, and Parquet
Pages, avoiding the need to scan and decode the complete vector column in
advance.
+4. When only business identifiers are required, queries can skip materializing
wide vectors. When vectors need to be returned, candidate identifiers are used
to look up the corresponding rows, reading only matching pages and records.
+5. Read results are converted into Doris Blocks through the Arrow C Data
Interface and continue through MPP computations such as filtering, joins,
aggregation, and Top-K.
+
+Production validation covered datasets with tens of millions and hundreds of
millions of records. Public tests used 16 million and 128 million real
2,048-dimensional vectors, with Top-K values of 100, 10,000, and 1 million.
Recall through the Paimon external-table path ranged from 96.0% to 99.6%. When
comparing "Paimon external tables with remote Alluxio caching" against "Doris
internal tables with local caching," query latency was 1.07 to 4.18 times that
of the latter. At a Top-K of 1 m [...]
+
+These results show that users can keep open data in Paimon and use Doris SQL
for vector retrieval and business analysis, even with hundreds of millions of
high-dimensional vectors and Top-K values as large as 1 million. They do not
have to copy the entire dataset into Doris internal tables or another vector
database first.
+
+For Agentic AI, retrieval results can be joined directly with the latest
business state in Paimon. Doris can then apply permission filters, compute
metrics, and perform relational analysis. Agents receive similar objects
together with context that matches current business facts and can be verified
before the system acts on it.
+
+> Availability: The Paimon capabilities used in these examples are in the
Doris 4.2 release branch and are scheduled for public release at the end of
September. Use the examples as preview syntax until that release is available.
+
+## 6. Getting started
+
+Teams already using Flink and Paimon do not need to change their existing data
pipeline as a first step. They can create a Paimon Catalog in Doris:
+
+```SQL
+CREATE CATALOG paimon_catalog PROPERTIES (
+ 'type' = 'paimon',
+ 'warehouse' = 's3://example-bucket/warehouse',
+ 's3.endpoint' = 'https://object-storage.example.com',
+ 's3.region' = 'region-id',
+ 's3.access_key' = 'YOUR_ACCESS_KEY',
+ 's3.secret_key' = 'YOUR_SECRET_KEY'
+);
+
+
+```
+
+Perform structured analysis through Doris SQL:
+
+```SQL
+SELECT device_model, defect_type, COUNT(*) AS defect_count
+FROM paimon_catalog.quality.inspection_records
+WHERE event_time >= NOW() - INTERVAL 30 DAY
+GROUP BY device_model, defect_type
+ORDER BY defect_count DESC;
+```
+
+In the community-developed vector retrieval path, express Top-K queries
through Doris SQL:
+
+```SQL
+SELECT id
+FROM paimon_catalog.quality.vector_items
+ORDER BY l2_distance_approximate(embedding, [...])
+LIMIT 100;
+
+
+```
+
+After the analysis is complete, write the aggregated results directly back to
Paimon:
+
+```SQL
+INSERT INTO paimon_catalog.quality.daily_defect_summary
+SELECT DATE(event_time), device_model, defect_type, COUNT(*)
+FROM paimon_catalog.quality.inspection_records
+GROUP BY DATE(event_time), device_model, defect_type;
+```
+
+For Primary Key tables that require continuous state corrections, use `MERGE`
to incorporate human reviews or new model results:
+
+```SQL
+MERGE INTO paimon_catalog.quality.inspection_records AS target
+USING review_results AS source
+ON target.record_id = source.record_id
+WHEN MATCHED THEN
+ UPDATE SET review_status = source.review_status,
+ defect_type = source.defect_type
+WHEN NOT MATCHED THEN
+ INSERT (record_id, review_status, defect_type)
+ VALUES (source.record_id, source.review_status, source.defect_type);
+```
+
+## 7. Conclusion: using business facts as they emerge
+
+The data problem in Agentic AI is broader than adding a vector field or
deploying a vector database. The system must keep changing business facts,
multimodal content, model features, retrieval indexes, and action results
consistent. It must then turn that data into verifiable context when an agent
needs it.
+
+Paimon 2.0 provides an open foundation for storing, updating, evolving, and
indexing this data. Apache Doris connects Paimon snapshot reads, Vector Index,
MPP OLAP, and standard SQL writes. This allows agents to maintain a continuous
"perceive changes-retrieve evidence-analyze and decide-take action-write back
feedback" loop.
+
+Doris users do not need to maintain a separate closed copy of their data for
AI workloads. The data can remain in Paimon while Doris handles retrieval and
analysis. Doris can then write query results and agent feedback back to the
open lake tables. Paimon allows the data used by agents to evolve continuously,
while Doris turns that data into current, verifiable business context on which
agents can act.
+
+We will publish more articles about the Apache Doris 5.0 multimodal lakehouse
and open formats such as Iceberg and Paimon. The series will cover the
underlying technical principles, performance evaluations, and typical
deployment scenarios.
+
+---
+
+Multimodal lakehouse capabilities for Iceberg, Paimon, Lance, and other
formats have now been merged into the Doris 4.2 release branch and are
scheduled for official release with version 4.2 at the end of September.
diff --git
a/blog/apache-doris-python-udf-calling-the-python-ecosystem-from-sql.mdx
b/blog/apache-doris-python-udf-calling-the-python-ecosystem-from-sql.mdx
index a3ac17e9ce9..b3ef05ea31f 100644
--- a/blog/apache-doris-python-udf-calling-the-python-ecosystem-from-sql.mdx
+++ b/blog/apache-doris-python-udf-calling-the-python-ecosystem-from-sql.mdx
@@ -2,8 +2,6 @@
'title': 'Apache Doris Python UDF: Calling the Python Ecosystem from SQL
for Agent-Era'
'summary': 'Apache Doris adds Python UDF in 4.1.3. This enables teams to
create and call Python functions directly in SQL, bringing Pandas, PyArrow, and
the rest of the Python ecosystem into the Doris query path.'
'description': 'Apache Doris adds Python UDF in 4.1.3. This enables teams
to create and call Python functions directly in SQL, bringing Pandas, PyArrow,
and the rest of the Python ecosystem into the Doris query path.'
- 'picked': "true"
- 'order': "4"
'date': '2026-8-6'
'author': 'velodb.io · Haopeng Li & Zhenqi Lin'
'externalLink':
'https://www.velodb.io/blog/apache-doris-python-udf-calling-the-python-from-sql-for-agent-era'
diff --git a/blog/apache-doris-variant-open-lakehouse.md
b/blog/apache-doris-variant-open-lakehouse.md
new file mode 100644
index 00000000000..fcd3129aa48
--- /dev/null
+++ b/blog/apache-doris-variant-open-lakehouse.md
@@ -0,0 +1,216 @@
+---
+title: 'Apache Doris 5.0 Variant Across Internal and Open Lakehouse Tables'
+summary: 'A look at Variant in Doris internal tables and open lake formats,
with an agent-trace example and guidance for choosing where to store data.'
+description: 'How Apache Doris 5.0 plans to use Variant across internal,
Iceberg, and Paimon tables, with SQL examples and selection guidance.'
+keywords:
+ - 'Apache Doris 5.0'
+ - 'Variant'
+ - 'Iceberg'
+ - 'Paimon'
+ - 'semi-structured data'
+date: '2026-09-15'
+author: 'Apache Doris · Mingyu Chen'
+tags:
+ - 'Tech Sharing'
+image: '/images/blogs/apache-doris-variant-open-lakehouse/cover.jpg'
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+> **Preview: Apache Doris 5.0 multimodal lakehouse (4) | Scenario and
solution**
+>
+> Apache Doris introduced Variant as a column type in version 2.1 for storing
semi-structured data such as JSON. Users do not need to define fields before
creating a table. During ingestion, Doris automatically splits the data into
typed subcolumns. This retains the flexibility of JSON ingestion while
providing analytical performance close to columnar storage. Apache Doris 5.0
will extend the Variant type from internal tables to open lake formats such as
Iceberg and Paimon. The same VARI [...]
+
+## 1. Agent traces: The divide between Variant producers and consumers
+
+JSON is a natural representation for an AI Agent execution trace. Protocols
such as MCP and A2A use JSON-RPC, while tool-call parameters, model output, and
span attributes for each step all contain nested structures. A complete trace
can range from tens of KB to several MB and may be seven or eight levels deep.
Tool lists and parameters also change as models evolve. Eight fields yesterday
may become twelve today, so defining a stable schema in advance is often
impossible.
+
+Analytical systems have traditionally handled this type of data in three ways:
+
+| Approach | Advantage | Cost |
+|-|-|-|
+| Flatten data through ETL into a wide table | Fast queries | Every new
upstream field requires changes to jobs and tables, followed by historical
backfills |
+| Store data as JSON strings or JSONB | Flexible ingestion | Every SQL query
must parse the entire JSON document |
+| Hard-code the structure with STRUCT or MAP | Better read performance | Still
strongly typed and insufficiently flexible |
+
+The Variant type addresses the trade-off between flexibility and query
performance. Apache Doris has provided the Variant column type since version
2.1 for storing semi-structured data such as JSON:
+
+- When creating a table, users declare a single VARIANT column without
defining its fields in advance.
+- During ingestion, Doris automatically splits JSON into typed subcolumns,
performs type inference and promotion, stores infrequently used fields in
sparse columns, and allows indexes on subcolumns.
+- Queries access values by path and read only the relevant subcolumns instead
of parsing the entire JSON document.
+
+Variant combines the ingestion flexibility of JSON with query performance
close to columnar storage.
+
+For Agent observability platforms, the recommended data model is:
+
+- Store stable fields such as `trace_id`, `tool_name`, and latency as regular
columns.
+- Store inputs, outputs, and tool parameters in a VARIANT column, allowing
upstream systems to add fields without altering the table.
+- Use regular SQL with path-based value access for failure attribution, cost
aggregation by tool, and effectiveness comparisons across model versions.
+- Use inverted indexes for full-text searches over text in model outputs.
+
+Later Apache Doris releases added Variant features such as Schema Template,
secondary indexes at the subcolumn level, DOC mode for JSON documents hundreds
of MB in size, and NestedGroup for searching nested arrays.
+
+Changes in enterprise data architectures created another problem. Agent traces
are produced by ingestion pipelines running on Flink or Spark. The data usually
lands in Paimon first, then feeds training, evaluation, observability, and
other workloads. Before version 5.0, Doris could process Variant data only in
its own format. Consumers that wanted to analyze the data with Variant had to
write and synchronize it to both internal tables and the lake. This duplicated
storage and introduced [...]
+
+
+
+Other Variant use cases have the same problem:
+
+- **Event tracking:** Event properties can be stored in a VARIANT column,
allowing product teams to add properties without altering the table. The same
event data is also a shared enterprise asset that algorithm teams need for
training features. This requires another copy in Iceberg.
+- **IoT telemetry:** Device properties vary by model and firmware. A wide
table would be excessively sparse, while a VARIANT column allows one SQL query
to aggregate data across device models. Complete datasets must also be retained
for years, and object storage with an open format is the only way to reduce
storage costs significantly. Real-time alerts and historical analysis must
therefore access different systems, which fragments the user experience.
+
+Variant needs to follow the data. When data resides in the lake, Variant must
be available there too, and the compute engine must support unified analysis of
that Variant data.
+
+## 2. Variant standardization: Bringing semi-structured data into the "one
copy, multiple engines" model
+
+Semi-structured data in a lake traditionally had two options: storage as
strings or conversion to STRUCT through ETL. As this data became more widely
used, the performance and operational costs of those approaches became harder
to ignore. Variant began moving toward open standardization in 2025:
+
+- In March 2025, Apache Parquet 2.11 defined a binary encoding for Variant and
introduced the Shredding specification. Writers can split frequently accessed
fields into typed columns while retaining the remaining data in a binary value,
giving the file itself a columnar shape. The specification was finalized in
August of the same year.
+- In June 2025, the Apache Iceberg V3 specification was approved, making
Variant a first-class table type. Apache Paimon has supported Variant and
Shredding since version 1.4.
+- Spark 4.0/4.1, Flink 2.1, and DuckDB subsequently added support, while Trino
began offering experimental support. Snowflake and Databricks also announced
general availability of their respective support for Variant in open formats in
2026.
+
+The open lakehouse model of "one copy of data shared by multiple engines" is
extending to semi-structured data. With a standard format, any compliant engine
can read and write Variant data in a lake. We therefore decided to extend the
Variant processing capabilities available for internal tables to standard
Variant formats.
+
+
+
+An engine needs to solve three problems to use Variant effectively in a lake:
+
+1. **High-performance reads:** The engine must be able to use the Shredded
layout directly.
+2. **Variant production:** An engine that can read but not write Variant
cannot write its analytical results back to the open lakehouse for reuse.
+3. **Interoperability between internal and external formats:** Variant should
behave consistently across formats.
+
+The following sections describe the unified Variant solution planned for
Apache Doris 5.0. A later article about Variant internals will explain how
Doris reads the Shredded layout.
+
+## 3. Doris 5.0: One Variant type, one SQL syntax, one execution engine
+
+Doris 5.0 will bring the Variant analytical capabilities of internal tables to
open lake formats such as Iceberg and Paimon while keeping the user experience
consistent across formats.
+
+
+
+### 3.1 Three layers of unification
+
+- **Upper layer: Unified type and SQL syntax.** Doris has one VARIANT type,
with consistent SQL syntax and semantics for internal and external tables.
+
+ - Unified path access syntax: `payload['event']['type']`
+ - Unified conversion logic: use `CAST` for type conversion and
`VARIANT_TYPE` to inspect the actual type
+ - Unified constructor: use `PARSE_TO_VARIANT` to construct values from JSON
text
+ - Unified nesting: ARRAY, MAP, and STRUCT support nested VARIANT values
+ - Unified NULL semantics: SQL NULL is distinguished from JSON null
+
+- **Middle layer: Unified execution operators.** Data read from internal table
files or Parquet files in the lake is converted into the execution engine's
internal Variant column representation. The same operators then process the
data, providing consistent behavior.
+
+ - Variant values from internal and lake tables can appear in the same SQL
statement and participate in UNION, JOIN, and other operations.
+ - Internal and external tables can write to each other, for example, `INSERT
INTO LakeTable SELECT ... FROM DorisTable`.
+ - Variant can participate directly in GROUP BY and DISTINCT, with
deduplication based on logical values rather than byte representations.
+
+- **Lower layer: Unified format access interfaces.** Logical abstractions for
Scan and Sink operators allow each format to implement only its own reader and
writer. The Doris internal table format, for example, supports subcolumn
splitting, inverted indexes, BloomFilters, real-time ingestion, and primary-key
updates. Open lake formats support both Plain and Shredded layouts, with each
format implementing its own write layout as described in Section 3.2.
Format-specific differences remain [...]
+
+### 3.2 Capability matrix
+
+The following Variant capabilities for data lakes are planned for Doris 5.0:
+
+| Area | Capability | Iceberg | Paimon |
+| --- | --- | --- | --- |
+| Read | File layout | Plain, Shredded, and mixed within the same table |
Plain, Shredded, and mixed within the same table |
+| Read | Nested Variant (STRUCT / ARRAY / MAP) | Supported | Supported |
+| Read | Path extraction, CAST, predicate filtering, and page-level pruning |
Supported | Supported |
+| Read | Existing lake-format capabilities | Time Travel, Deletion Vector, and
schema evolution | Append tables and primary-key tables |
+| Read | Asynchronous materialized views built on lake Variant tables |
Supported | Supported |
+| Write | INSERT INTO / INSERT OVERWRITE / INSERT ... SELECT | Supported
(format-version 3 + Parquet) | Supported (Append tables, primary-key tables,
and partitioned and bucketed tables) |
+| Write | Row-level UPDATE / DELETE / MERGE INTO | Supported | Supported |
+| Write | Nested Variant writes | Supported | Supported |
+| Write | On-disk layout | Plain (see note below) | Written according to the
Shredding Schema, specified explicitly or inferred automatically |
+| Interoperability | Spark writes → Doris reads; Doris writes → Spark reads |
Verified | Verified |
+| Interoperability | Internal table ↔ lake table INSERT ... SELECT |
Bidirectional | Bidirectional |
+| Interoperability | Query internal and lake tables in the same SQL statement
| Supported | Supported |
+
+> Iceberg currently writes only the Plain layout. Reading the Shredded layout
is fully supported. Shredded writes are planned for a later iteration, as
described in the conclusion.
+
+> Availability: The lake Variant capabilities and SQL workflow below are
planned for Doris 5.0. They describe the intended behavior rather than a
currently released feature.
+
+### 3.3 Example: Unified lakehouse analysis of Agent traces
+
+The following example uses an Agent observability platform.
+
+First, Flink writes trace data to Paimon. Stable fields are stored as regular
columns, while variable data is stored in a VARIANT column.
+
+```sql
+-- Flink writes the paimon.obs.agent_traces trace table:
+-- (trace_id, tool_name, latency_ms, dt, payload VARIANT)
+
+-- 1. Analyze failures in lake tables using path extraction, type conversion,
and predicates
+SELECT tool_name, COUNT(*) AS failures
+FROM paimon.obs.agent_traces
+WHERE CAST(payload['result']['status'] AS STRING) = 'error'
+ AND CAST(payload['model']['version'] AS STRING) = 'v2'
+GROUP BY tool_name;
+
+-- 2. Load hot data into internal tables for real-time alerts and full-text
search
+INSERT INTO internal.obs.agent_traces_hot
+SELECT * FROM paimon.obs.agent_traces WHERE dt = CURRENT_DATE();
+
+-- 3. Write derived results and the VARIANT column back to the lake for other
engines
+INSERT INTO paimon.obs.failed_traces
+SELECT trace_id, tool_name, payload
+FROM internal.obs.agent_traces_hot
+WHERE CAST(payload['result']['status'] AS STRING) = 'error';
+```
+
+In this pipeline:
+
+- The lake table and internal table use the same VARIANT type, so they do not
require separate schemas.
+- Internal and lake tables use the same SQL query syntax.
+- Standard SQL `INSERT ... SELECT` statements handle data synchronization and
write-back.
+
+The same pipeline also works for event tracking and IoT:
+
+- **Event tracking:** Events are shared as Iceberg Variant data. Spark creates
features, while Doris produces reports. Both use one copy of the data and one
schema-evolution process.
+- **IoT:** The complete dataset remains in the lake for long-term retention.
Hot data enters internal tables through `INSERT ... SELECT` for real-time
alerting, while hot and cold data use the same data model.
+
+## 4. Recommendations for choosing a Variant format
+
+The following comparison of Variant in internal and external tables can help
users select a format.
+
+
+
+| Dimension | Variant in internal tables | Variant in the lake |
+|-|-|-|
+| Storage and format | Doris proprietary format | Parquet Variant in Iceberg
V3 or Paimon |
+| Engines that can read and write | Doris | ✅ **Any standards-compliant
engine** |
+| Query performance | ✅ **Best performance through subcolumns and indexes** |
Depends on the Shredding strategy; readers access subcolumns directly |
+| Indexing and search | ✅ **Inverted indexes, BloomFilter, DOC mode, and
NestedGroup** | Parquet statistics with page-level pruning |
+| Schema control | ✅ **Schema Template, with Doris splitting subcolumns** |
The writer's Shredding strategy |
+| Ingestion latency and updates | ✅ **Real-time ingestion and primary-key
updates** | Batch or micro-batch commits; Iceberg row-level updates |
+| Cost | Integrated or decoupled storage and compute | ✅ **Object storage with
low per-unit storage cost** |
+
+Variant in internal tables provides more features and better read and write
performance. Variant in the lake allows multiple engines to access the same
data and offers lower storage costs.
+
+The recommendations are:
+
+| Scenario | Recommendation |
+|-|-|
+| Data is primarily produced and consumed within Doris; workloads require
subsecond real-time ingestion and updates, full-text search, or
high-concurrency point lookups and filtering | Variant in internal tables |
+| Data is an enterprise asset shared by multiple engines; ingestion is
primarily batch or micro-batch; long-term retention and storage costs matter;
Spark or Flink production pipelines already exist | Variant in the lake |
+| Both scenarios apply | Put hot data in internal tables and retain the
complete dataset in the lake; query both with one SQL statement; accelerate
lake Variant tables with asynchronous materialized views; write processing
results back to the lake |
+
+## 5. Conclusion: Variant follows the data
+
+Doris introduced Variant for internal tables in version 2.1 and plans to
support open lake formats in version 5.0. Future work will strengthen Shredded
writes for Variant and improve the performance of reading Variant from data
lakes. These changes are intended to meet enterprise requirements for
semi-structured data analysis and cover more use cases.
+
+The next article, *Apache Doris Variant in the Lake: How Shredding Works and
Performance Benchmarks*, will examine the implementation details of Variant and
compare the performance characteristics of different formats. It will explain
in more detail how Variant works and how to use it.
diff --git
a/blog/dynamic-json-in-agent-workloads-apache-doris-clickhouse-elasticsearch-and-opensearch.mdx
b/blog/dynamic-json-in-agent-workloads-apache-doris-clickhouse-elasticsearch-and-opensearch.mdx
index c5405a52181..ff14f596c21 100644
---
a/blog/dynamic-json-in-agent-workloads-apache-doris-clickhouse-elasticsearch-and-opensearch.mdx
+++
b/blog/dynamic-json-in-agent-workloads-apache-doris-clickhouse-elasticsearch-and-opensearch.mdx
@@ -3,7 +3,7 @@
'summary': 'AI agent logs create wide, fast-changing JSON payloads that
are difficult to analyze with predictable latency. We compare how Apache Doris,
ClickHouse, Elasticsearch, and OpenSearch perform in such scenario.'
'description': 'AI agent logs create wide, fast-changing JSON payloads
that are difficult to analyze with predictable latency. We compare how Apache
Doris, ClickHouse, Elasticsearch, and OpenSearch perform in such scenario.'
'picked': "true"
- 'order': "1"
+ 'order': "3"
'date': '2026-8-27'
'author': 'velodb.io · Max Li'
'externalLink':
'https://www.velodb.io/blog/dynamic-json-in-agent-workloads-apache-doris-clickhouse-elasticsearch-and-opensearch'
diff --git
a/blog/from-spark-to-apache-doris-how-kwai-made-a-b-testing-metrics-145x-faster-at-scale.mdx
b/blog/from-spark-to-apache-doris-how-kwai-made-a-b-testing-metrics-145x-faster-at-scale.mdx
index 11be348410f..bab13569358 100644
---
a/blog/from-spark-to-apache-doris-how-kwai-made-a-b-testing-metrics-145x-faster-at-scale.mdx
+++
b/blog/from-spark-to-apache-doris-how-kwai-made-a-b-testing-metrics-145x-faster-at-scale.mdx
@@ -2,8 +2,6 @@
'title': 'From Spark to Apache Doris: How Kwai Made A/B Testing Metrics
145x Faster at Scale'
'summary': 'Kwai migrated its company-wide A/B testing metrics pipeline
from Spark to Apache Doris. The rebuild delivered 145x faster metrics
computation and cut resource consumption by 72%.'
'description': 'Kwai migrated its company-wide A/B testing metrics
pipeline from Spark to Apache Doris. The rebuild delivered 145x faster metrics
computation and cut resource consumption by 72%.'
- 'picked': "true"
- 'order': "3"
'date': '2026-8-12'
'author': 'Kwai · Siwei Zeng'
'externalLink':
'https://www.velodb.io/blog/from-spark-to-apache-doris-how-kwai-made-a-b-testing-metrics-145x-faster-at-scale'
diff --git a/blog/kwai-paimon-vector-search-apache-doris.md
b/blog/kwai-paimon-vector-search-apache-doris.md
new file mode 100644
index 00000000000..bb6e8febbc8
--- /dev/null
+++ b/blog/kwai-paimon-vector-search-apache-doris.md
@@ -0,0 +1,351 @@
+---
+title: 'Kwai Vector Search on Paimon with Apache Doris'
+summary: 'Kwai queries Paimon vector indexes through Doris SQL. This article
explains the Rust reader, index pushdown, and production benchmarks.'
+description: 'Kwai production practice with Apache Doris and Paimon vector
indexes, including IVF-RQ, Top-K pushdown, test conditions, recall, and query
latency.'
+picked: "true"
+order: "4"
+keywords:
+ - 'Apache Doris'
+ - 'Apache Paimon'
+ - 'Kwai'
+ - 'vector index'
+ - 'IVF-RQ'
+date: '2026-09-10'
+author: 'Apache Doris · Wenqiang Li; Kwai · Junrui Li, Shiyao Yu, Fangyuan
Deng'
+tags:
+ - 'Best Practice'
+image: '/images/blogs/kwai-paimon-vector-search-apache-doris/cover.jpg'
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+> **Introduction:**
+>
+> **Kwai** uses Apache Doris in production to query vector indexes directly on
Apache Paimon tables. Vectors and business data remain in Paimon, while Doris
SQL initiates retrieval and pushes execution down to the data lake. Existing
ingestion pipelines and OLAP access patterns do not change. This article
explains the indexing foundation in Paimon 2.0, the read path developed jointly
by the Doris and Paimon communities, and benchmark results for tens of millions
and hundreds of millions [...]
+
+As LLM and RAG applications have become more common, semantic search has
become a standard data platform requirement. One common design synchronizes
vectors, filter fields, and result fields into a standalone vector database.
The main cost is not the initial synchronization but the ongoing maintenance.
Synchronization pipelines must run continuously, which creates a data freshness
window. Vectors and business fields are split across two systems, as are access
control and governance. Team [...]
+
+Kwai took a different approach. Vectors and business columns remain in Paimon
tables, and the Vector Index in Paimon 2.0 handles retrieval. Doris reads
indexes and data directly through Paimon Rust. It retrieves candidates and
looks up rows in parallel by bucket. Users continue to write Doris SQL with the
same syntax they use for Doris internal tables.
+
+> Availability: Kwai has validated this integration in production, while the
Doris 4.2 capabilities described here are scheduled for public release at the
end of September.
+
+## 1. Data foundation: indexing in Paimon 2.0
+
+### 1.1 Physical separation of indexes and data
+
+The Vector Index in Paimon 2.0 works with ordinary Paimon tables and has two
forms: the Global Vector Index for append tables and the PK Vector Index for
primary-key tables. Both use the same index implementation and retrieval model,
but they build indexes and retrieve matching rows differently. Kwai's business
tables are primary-key tables, so the rest of this article focuses on the PK
Vector Index.
+
+Both forms store index files separately from data files. A Snapshot references
a Data Manifest and an Index Manifest. The Index Manifest records each index
file's type, field, location, and coverage. For append tables, it records the
covered Row ID range. For primary-key tables, it records the covered source
data files. Adding an index therefore does not require a different data layout,
and the same table can still support batch processing, stream processing, and
regular OLAP reads.
+
+Figure 1 shows how index files relate to data files.
+
+
+
+*Figure 1: Index files and data files are stored separately. The same Snapshot
links them through two types of Manifest. Index segments for primary-key tables
are generated during compaction and record their source files.*
+
+Vector columns are declared as `ARRAY<FLOAT>` or fixed-length `VECTOR<FLOAT,
N>`. For primary-key tables, table properties configure vector indexes, which
are built automatically during compaction without a separate trigger. The
following Spark SQL specifies the indexed column, index type, and distance
metric when creating a primary-key table:
+
+```sql
+CREATE TABLE vector_items (
+item_id BIGINT,
+embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;2048',
+payload STRING
+) TBLPROPERTIES (
+'primary-key' = 'item_id',
+'bucket' = '32',
+'deletion-vectors.enabled' = 'true',
+'pk-vector.index.columns' = 'embedding',
+'fields.embedding.pk-vector.index.type' = 'ivf-rq',
+'fields.embedding.pk-vector.distance.metric' = 'l2'
+);
+```
+
+- Spark SQL does not have a `VECTOR` type. The column comment
`__VECTOR_FIELD;2048` declares `ARRAY<FLOAT>` as a fixed-length,
2,048-dimensional vector.
+- Build parameters such as `nlist` are passed as JSON through
`fields.<col>.pk-vector.index.options`.
+- The reranking multiplier `refine_factor` can be set in either table
properties or query options.
+
+After data is written, compaction merges files within a bucket into a level
above Level 0 and generates an index segment for the vector column. A new
Snapshot references the segment after it is registered in the Index Manifest.
The segment records the source data files it covers, which allows matches to
map back to physical row positions within those files. Newly written data does
not belong to an index segment until the next compaction. Section 2.2 explains
how retrieval handles this in [...]
+
+### 1.2 Five types of vector indexes
+
+The Paimon 2.0 Vector Index currently supports five index implementations,
each with different characteristics and trade-offs:
+
+| Index | Main characteristic | Typical trade-off |
+| --- | --- | --- |
+| IVF-Flat | Stores the original vectors after clustering | Higher recall, but
a larger index |
+| IVF-PQ | Product quantization | Smaller index, but some quantization loss |
+| IVF-SQ | 8-bit scalar quantization of residuals | Balanced throughput,
recall, and storage |
+| IVF-RQ | Rotated residual quantization | Higher compression, but requires
tuning around the quality inflection point |
+| DiskANN | Graph search with persistently stored vectors for reranking |
Suitable for local SSDs or a fully local cache |
+
+The index types balance query performance, recall, index size, and build time
differently. Storing hundreds of millions of 2,048-dimensional vectors at their
original precision produces a correspondingly large index and increases the
amount of data read from remote storage. RQ uses rotated residual quantization
to achieve greater compression, though it must be tuned to meet quality
targets. Kwai chose IVF-RQ as a balance between compression and recall. **The
measured Recall in Section 3. [...]
+
+Figure 2 compares the five index types and shows Kwai's choice.
+
+
+
+*Figure 2: Each index type makes a different trade-off between index size and
recall. Kwai chose IVF-RQ and used its higher compression to reduce remote
reads.*
+
+## 2. Bringing indexes back to the data lake: how Doris directly queries
Paimon indexes
+
+A compute engine needs the following capabilities to use indexes on lake data
directly for retrieval:
+
+- A high-performance data read path.
+- Awareness of index distribution and index coverage.
+- Optimized index access for common vector-retrieval workloads such as Top-K.
+
+Together, these capabilities allow the engine to retrieve and analyze lake
data efficiently. Section 2.1 covers the read path. Section 2.2 explains index
awareness and retrieval pushdown. Sections 2.3 and 2.4 describe two
index-access optimizations for Top-K workloads.
+
+### 2.1 High-performance reads: from JNI to Paimon Rust
+
+In earlier Doris versions, reading Paimon data in MOR (Merge-On-Read)
scenarios required the Paimon Java SDK. Doris called the SDK through JNI to
read and merge the data. This approach had two problems:
+
+- Doris could not track and control JVM memory at query granularity. If a
large query exhausted memory, Doris lacked fine-grained visibility and control
over its usage.
+- MOR used substantial memory, and high concurrency could cause JVM OOM
errors. Increasing the JVM heap reduced the native memory available to the BE.
+
+After testing both options, Kwai chose Paimon Rust. The official Paimon team
maintains Paimon Rust and keeps its MOR and index reads in sync with the Java
implementation. Tests found that Paimon Rust provided two to three times the
read performance of Paimon Java and was approximately 10% slower than the
native Doris reader. Most of the difference came from converting Arrow's
in-memory format into Doris Blocks. Paimon Rust uses native memory, so Doris
can include its consumption in the e [...]
+
+At the code level, the Doris BE, which is the C++ execution process, calls the
Paimon Rust Reader through a C ABI. Rust and C++ exchange results through the
Arrow C Data Interface. Doris then converts those results into Doris Blocks and
passes them to the execution layer. Because Arrow is a columnar format, C++ can
consume the data produced by Rust directly by column without an intermediate
row-by-row conversion.
+
+Figure 3 shows this read path.
+
+
+
+*Figure 3: Doris BE calls Paimon Rust through a C ABI. Paimon Rust returns
results in Arrow's columnar format, which Doris converts into Blocks. Regular
reads and vector retrieval use the same path.*
+
+### 2.2 Pushing retrieval down to the data lake
+
+Unlike a regular read, a vector Top-K query does not know which rows it needs
before reading starts. It must identify those rows through the index first.
Without the index, a vector Top-K query written as `ORDER BY ... LIMIT` becomes
a full computation. Doris loads the entire vector column into BE memory,
calculates distances row by row, and sorts the results. For 128 million rows
with 2,048 dimensions, one query would need to transfer approximately 1 TB of
vector data.
+
+The query engine must push retrieval down to the index on the data lake. To do
this correctly, it must account for four properties of vector indexes on Paimon
primary-key tables:
+
+- Indexes are organized by `(Partition, Bucket)`. A bucket is both the
data-sharding unit and the indexing unit.
+- Paimon's background compaction process generates indexes, so indexes cover
only compacted data. Newly written L0 files are not included in any index.
Correct retrieval must merge three operations: index retrieval, exact fallback
over uncovered files, and Deletion Vector mask computation.
+- An index segment is valid only when its set of covered source files still
matches the bucket's current set of active files. Checking this condition
requires the complete file set for the entire bucket.
+- Kwai's business tables are Paimon primary-key tables. Rows in primary-key
tables can be updated and merged, so they do not have stable global Row IDs.
Instead, the index stores physical coordinates: the data file and the row
number within that file. Retrieval uses these physical row numbers to fetch
rows from the data files.
+
+Doris uses the following index-query plan to account for these properties:
+
+- The Doris FE, the Frontend responsible for query planning, enumerates scan
units by bucket for the Paimon table and obtains the complete file list for
each unit.
+- Doris dispatches scan units to different BE nodes. Multiple BEs process the
file lists for their assigned buckets in parallel. Within each bucket, they
perform retrieval, exact fallback, reranking, row lookup, and Deletion Vector
application.
+- Doris's sort-and-merge operator computes the global Top-K across buckets.
+
+In this design, Paimon Rust performs vector retrieval, while Doris schedules
the distributed work and merges the results.
+
+Figure 4 shows whole-bucket dispatch and parallel processing by bucket.
+
+
+
+*Figure 4: Each scan unit contains one complete bucket. The BE performs index
retrieval, exact fallback, and deletion masking within that bucket. Doris then
merges the Top-K across buckets.*
+
+### 2.3 Top-K pushdown and index-only scan
+
+A standard vector-retrieval SQL query sorts rows by distance and returns the
first K. The Doris FE uses an optimizer rule to recognize this pattern and
rewrite the execution plan:
+
+- Add a retrieval-score column to the scan node.
+- Rewrite the distance expression in the original SQL as a conversion of this
score, which avoids calculating the distance again.
+- Change the sort order from ascending distance to descending score.
+- When the vector column does not appear in the `SELECT` list, do not read the
original vector column. This Index-Only Scan removes vector-column reads and
transfers.
+
+The optimizer applies the rule when all of the following conditions hold:
+
+- The distance function matches the metric configured for the index.
+- A vector index exists on the column.
+- The SQL contains no aggregation or `JOIN`.
+
+If any condition is not met, Doris uses a regular scan to preserve correct
results.
+
+The same rule handles queries with filters and pagination. The FE uses
partition predicates for partition pruning and retains only buckets from
relevant partitions. Predicates that can be converted into Paimon predicates
are pushed down to Paimon Rust, which uses them to narrow the candidate set
before retrieval. This provides pre-filtering. If the filter column is not
indexed, the reader must scan that column. Doris functions such as `length(a) >
1` cannot be pushed down, so Doris appli [...]
+
+Users can write the same SQL they use for vector retrieval on Doris internal
tables. They do not need to modify their queries to use vector indexes on
Paimon tables.
+
+Figure 5 shows how the optimizer identifies and rewrites the query.
+
+
+
+*Figure 5: A vector Top-K query that meets all three conditions is rewritten
to sort by score. If the query does not select the vector column, Doris does
not read it. If any condition is unmet, Doris uses a regular scan.*
+
+### 2.4 Late materialization for large Top-K queries: separating retrieval
from materialization
+
+A typical Top-K SQL query looks like this:
+
+```Plain Text
+select id, embedding from vec_table
+order by l2_distance_approximate(embedding, [...])
+limit <K>;
+```
+
+This SQL retrieves the `id` and `embedding` columns in a single scan and then
performs retrieval. When K is small, this execution method adds little overhead.
+
+When K is large, the cost of a single scan grows with it. Each bucket must
read all user columns for K candidate rows, although the global merge will
discard most of those rows. With K = 1 million and a 2,048-dimensional vector
column, each candidate row carries 8 KB of vector data (`2,048 × 4` bytes). As
more candidates are discarded, more of those reads are wasted.
+
+Kwai currently rewrites these queries as self-joins. A subquery performs
retrieval and returns only IDs, then joins the original table to fetch vectors
for the matching rows, as shown in Section 3.2. During the join, Doris pushes
the candidate ID set down to the Paimon read layer. The reader uses this set to
exclude unrelated files and row groups. It then uses Parquet page-level indexes
to skip unrelated pages and materializes the vector column only for matching
rows. In the Section 3.3 [...]
+
+The next execution plan is still under development. It separates retrieval and
materialization into three stages:
+
+- Candidate stage: Each BE performs retrieval on its assigned buckets without
reading user columns and produces only a candidate set.
+- Merge stage: All candidates are collected for global Top-K selection and
reranking. The result is a list of rows to materialize, represented as data
files and row ranges within those files. The list is grouped by data file, and
matching row ranges are merged.
+- Materialization stage: Read user columns only for rows in the row list.
+
+The merge stage must run on a single node for algorithmic reasons. Reranking
needs the global candidate set to calculate the correct Top-K, and grouping and
merging row ranges also require the complete set. The aggregated candidate set
is only on the order of megabytes, and this stage performs no hash or range
repartitioning.
+
+Figure 6 compares the one-pass scan with the three-stage plan.
+
+
+
+*Figure 6: A one-pass scan makes every bucket read wide columns for K
candidates. The three-stage plan first produces candidates alone, then
materializes matching rows after the global Top-K step.*
+
+The expected benefits are:
+
+- As Top-K grows, more candidate rows are discarded, so the system can avoid
more wide-column reads.
+- The materialization stage replaces the user-written self-join and removes
that join from the execution plan.
+- The FE uses a cost model to choose between the two forms automatically.
Users do not need to know which form it selects or change their SQL.
+
+Section 3 reports measured performance for the current one-pass form. Section
4 gives the current status of the three-stage implementation.
+
+## 3. Performance: scale and results in Kwai's production environment
+
+### 3.1 Deployment and benchmark methodology
+
+Kwai already uses Doris external tables in production to query vector indexes
on Paimon tables. The deployment is configured as follows:
+
+| Dimension | Choice |
+| --- | --- |
+| Table type | Paimon primary-key table (PK Table) |
+| Vector storage | Inline, in the same table as the business columns |
+| Vector index | IVF-RQ |
+| Vector dimensions | 2,048 (real production data) |
+| Read path | Paimon Rust + Doris BE |
+| Storage acceleration | Remote Alluxio cache |
+| Query interface | Doris SQL |
+| Cluster size | 64 CU |
+
+The tests use the following dataset sizes:
+
+| Scale | Rows | Data volume | Buckets |
+| --- | ---: | ---: | ---: |
+| Tens of millions | 16 million | Approximately 150 GB | 32 |
+| Hundreds of millions | 128 million | Approximately 1.1 TB | 256 |
+
+### 3.2 Two query types
+
+The first query type returns only business identifiers. The vector index
produces the Top-K directly:
+
+```sql
+select id
+from vec_table
+order by l2_distance_approximate(embedding, [...]) /* 2,048-dimensional query
vector */
+limit <K>; -- K is 100, 10,000, or 1,000,000
+```
+
+This is a typical Index-Only Scan test case. It primarily measures the cost of
retrieval itself.
+
+The second query type returns both business identifiers and vectors. With the
current implementation, users rewrite these queries as self-joins. A subquery
retrieves the Top-K IDs, then joins the original table to fetch vectors for the
matching rows:
+
+```sql
+select t1.id, embedding
+from (select id, embedding from vec_table) t1
+join (
+select id
+from vec_table
+order by l2_distance_approximate(embedding, [...]) /* 2,048-dimensional query
vector */
+limit <K>
+) t2 on t1.id = t2.id;
+```
+
+The "Business identifier + vector" column in the Section 3.3 table uses this
self-join SQL. It primarily measures the cost of reading and materializing the
vector column. After the three-stage plan described in Section 2.4 is
implemented, users will be able to write the same query directly as:
+
+```sql
+select id, embedding
+from vec_table
+order by l2_distance_approximate(embedding, [...]) /* 2,048-dimensional query
vector */
+limit <K>; -- K is 100, 10,000, or 1,000,000
+```
+
+The materialization stage will then replace the self-join, and the SQL will be
identical to a query against a Doris internal table.
+
+### 3.3 Measured results from tens of millions to hundreds of millions of
vectors
+
+The following table reports query performance and recall for different
parameter combinations. Both latency columns measure end-to-end latency.
Queries that return only business identifiers use the first SQL statement in
Section 3.2. Queries that also return vectors use the self-join SQL. The values
are representative results from individual queries:
+
+| Vector count | Top-K | Business identifier | Business identifier + vector |
Recall |
+| ---: | ---: | ---: | ---: | ---: |
+| 16 million | 100 | 1.4 seconds | 2.4 seconds | 96.0% |
+| 128 million | 100 | 1.8 seconds | 2.5 seconds | 97.0% |
+| 16 million | 10,000 | 1.8 seconds | 2.3 seconds | 99.2% |
+| 128 million | 10,000 | 2.0 seconds | 4.6 seconds | 98.5% |
+| 16 million | 1 million | 7.9 seconds | 65 seconds | 97.8% |
+| 128 million | 1 million | 8.0 seconds | 76 seconds | 99.6% |
+
+The table supports two conclusions:
+
+- Index-retrieval latency changes little with data scale. When the row count
grows from 16 million to 128 million, latency for returning only business
identifiers increases by 0.4 seconds, 0.2 seconds, and 0.1 seconds across the
three Top-K settings, respectively.
+- When Top-K = 1 million, vector-column materialization is the bottleneck, not
retrieval.
+
+These are representative results from individual queries. The table does not
cover QPS.
+
+### 3.4 Comparing vector retrieval on Paimon tables and Doris internal tables
+
+Kwai also compared vector retrieval on Paimon tables with retrieval on Doris
internal tables. The team loaded the same Paimon data into Doris internal
tables and queried it through Doris's own vector indexes. The following table
gives the results.
+
+| Vector count | Top-K | Returned data | Doris internal table (local cache) |
Paimon external table (Alluxio) | External / internal |
+| ---: | ---: | --- | ---: | ---: | ---: |
+| 16 million | 100 | Business identifier | 0.49 seconds | 1.4 seconds | 2.86× |
+| 16 million | 100 | Business identifier + vector | 0.7 seconds | 2.4 seconds
| 3.43× |
+| 128 million | 100 | Business identifier | 0.7 seconds | 1.8 seconds | 2.57× |
+| 128 million | 100 | Business identifier + vector | 0.8 seconds | 2.5 seconds
| 3.13× |
+| 16 million | 10,000 | Business identifier | 0.58 seconds | 1.8 seconds |
3.10× |
+| 16 million | 10,000 | Business identifier + vector | 1.4 seconds | 2.3
seconds | 1.64× |
+| 128 million | 10,000 | Business identifier | 0.7 seconds | 2.0 seconds |
2.86× |
+| 128 million | 10,000 | Business identifier + vector | 1.1 seconds | 4.6
seconds | 4.18× |
+| 16 million | 1 million | Business identifier | 7.2 seconds | 7.9 seconds |
1.10× |
+| 16 million | 1 million | Business identifier + vector | 61 seconds | 65
seconds | 1.07× |
+| 128 million | 1 million | Business identifier | 7.2 seconds | 8.0 seconds |
1.11× |
+| 128 million | 1 million | Business identifier + vector | 56 seconds | 76
seconds | 1.36× |
+
+- External-table latency is 1.07 to 4.18 times internal-table latency. When
the query returns only business identifiers, the absolute difference is 0.7 to
1.3 seconds.
+- At Top-K = 1 million, the difference narrows to 1.07 to 1.36 times. Most of
the latency at this setting comes from row lookup and materialization. Both
sides read the same amount of data, which reduces the effect of cache location.
+
+Figure 7 summarizes the results measured by Kwai.
+
+
+
+*Figure 7: Recall is 96.0% to 99.6% at both scales. Queries that return only
business identifiers finish within 2.0 seconds for Top-K values up to 10,000.
At Top-K = 1 million, materialization accounts for most of the latency.*
+
+Internal tables perform better, but returning only business identifiers adds
just 0.7 to 1.3 seconds of query latency in absolute terms. In return, Kwai
gets the following operational benefits:
+
+- Better data freshness between writing data to Paimon and making it
searchable.
+- Unified storage without redundant data copies.
+- No compute-resource overhead for transferring and converting data between
different systems.
+- No separate access-control and governance systems to maintain.
+
+## 4. Results and roadmap of joint development by the Doris and Paimon
communities
+
+Kwai, the Apache Doris community, and the Apache Paimon community developed
this read path together. Kwai has contributed more than 120 PRs to the Apache
Paimon and Apache Paimon Rust communities. The Doris-side changes are planned
for merger into the Doris community within the next two months.
+
+Kwai is also working on the following changes to improve vector retrieval
performance when Doris queries Paimon:
+
+- Three-stage late materialization: The plan described in Section 2.4
separates candidate retrieval from vector-column materialization. At operating
points such as Top-K = 1 million, the system will stop reading wide columns for
candidate rows that are later discarded. Users will also no longer need to
write self-joins manually.
+- Local index caching: Add local disk and memory caches within the calling
process, with Doris providing consistent-hash distribution.
+- Full-text retrieval and Hybrid Search: The full-text retrieval path is
already complete on the Paimon side. After the Doris-side integration is
complete, users will be able to combine structured filtering, keyword matching,
and semantic retrieval on the same lake table.
+- Completing retrieval semantics: Add similarity-range retrieval. Kwai has
begun an internal grayscale rollout and plans to contribute the feature to the
community after large-scale production validation. Retrieval parameters will
also become overridable per query instead of remaining limited to table-level
configuration.
+
+## Learn more
+
+The multimodal lakehouse capabilities for Paimon, Iceberg, Lance, and other
formats have been merged into the **Doris 4.2** release branch. They are
scheduled for official release with version 4.2 at the **end of September**.
diff --git
a/blog/postgresql-cdc-to-apache-doris-real-time-sync-with-one-sql-statement.mdx
b/blog/postgresql-cdc-to-apache-doris-real-time-sync-with-one-sql-statement.mdx
index a88b5aed534..d16214fdf1c 100644
---
a/blog/postgresql-cdc-to-apache-doris-real-time-sync-with-one-sql-statement.mdx
+++
b/blog/postgresql-cdc-to-apache-doris-real-time-sync-with-one-sql-statement.mdx
@@ -2,8 +2,6 @@
'title': 'PostgreSQL CDC to Apache Doris: Real-Time Sync with One SQL
Statement'
'summary': 'Apache Doris offers native CDC for PostgreSQL, replacing
separate CDC, Kafka, Flink, or Spark components with one built-in streaming
job.'
'description': 'Apache Doris offers native CDC for PostgreSQL, replacing
separate CDC, Kafka, Flink, or Spark components with one built-in streaming
job.'
- 'picked': "true"
- 'order': "2"
'date': '2026-8-21'
'author': 'velodb.io · Dylan Wu'
'externalLink':
'https://www.velodb.io/blog/postgresql-cdc-to-apache-doris-real-time-sync-with-one-sql-statement'
diff --git a/doc-tools/HOW-TO-ADD-BLOG.md b/doc-tools/HOW-TO-ADD-BLOG.md
index 4b0fe4a5241..8a4db8cfece 100644
--- a/doc-tools/HOW-TO-ADD-BLOG.md
+++ b/doc-tools/HOW-TO-ADD-BLOG.md
@@ -222,7 +222,67 @@ Set the cover in front matter:
image: '/images/blogs/<slug>/cover.png'
```
-A horizontal cover works best with the current cards. Recent Blog covers
commonly use an aspect ratio close to `1800 × 766`.
+### Create the cover image
+
+Use a horizontal cover at `1800 × 766` pixels (approximately `2.35:1`)
whenever possible. A slightly larger image with the same aspect ratio is
acceptable, but do not stretch an image to fit. Use `cover.jpg` for an opaque
cover and `cover.png` only when transparency or lossless rendering is necessary.
+
+Before creating a cover, inspect several recent covers under
`static/images/blogs/` and choose examples with a similar article type. The new
cover should look like part of the same Blog collection while still
communicating the subject of the article.
+
+Follow these visual guidelines:
+
+- use a clean light, off-white, or pale-blue background with restrained
gradients, geometric shapes, or fine wave patterns;
+- use dark, high-contrast typography and a limited set of accent colors
derived from the Apache Doris brand and the article subject;
+- keep one clear visual focus and enough empty space for the design to remain
readable when reduced to a Blog card;
+- keep the title or subtitle short, and verify every visible word at full size
and thumbnail size;
+- avoid dense paragraphs, small labels, screenshots used as a background,
decorative clutter, watermarks, QR codes, calls to action, trial offers,
community-group promotions, and commercial branding unrelated to the article;
+- keep important text and logos away from the outer edges so common card crops
do not cut them off.
+
+For an integration article or a company case study, the official logos can be
the main visual. Use the official Apache Doris logo together with the official
project or company logo, preserve their aspect ratios and clear space, and
separate them with a simple `×` or `+` when appropriate. Do not ask an image
model to recreate a logo or use a guessed wordmark. Obtain the logo from an
authoritative source and composite the original asset into the cover.
+
+Use this workflow:
+
+1. identify the article's single main idea and select two or three existing
Blog covers as style references;
+2. collect the official logo assets when the topic centers on another project
or company;
+3. create the cover at the target aspect ratio and inspect it at both full
size and card-thumbnail size;
+4. verify logo accuracy, spelling, contrast, edge clearance, and visual
balance;
+5. compress the approved cover using the process below and keep only the
compressed file in the article image directory;
+6. set the final path in the front matter and verify that the path, file name,
and extension match exactly.
+
+### Compress and replace source images
+
+Compress every cover and content image before the Blog is complete. Follow the
detailed settings in the [image compression
guide](./prompts/img-compression-guide.md).
+
+Choose the output format according to the image:
+
+- convert opaque photographs, generated illustrations, and diagrams without
transparency to JPEG with Pillow using `quality=85` and `optimize=True`;
+- keep PNG for transparent images, flat-color graphics, or screenshots whose
sharp text and edges degrade noticeably in JPEG;
+- preserve the original pixel dimensions during compression unless the image
is being resized deliberately to meet the cover specification;
+- never change the aspect ratio by stretching the image.
+
+Compress to a temporary output first. Do not delete or overwrite the source
until the compressed image has been inspected. For JPEG output, the standard
conversion is:
+
+```python
+from pathlib import Path
+from PIL import Image
+
+files = [Path('source-image.png')]
+
+for source in files:
+ output = source.with_suffix('.jpg')
+ image = Image.open(source).convert('RGB')
+ image.save(output, 'JPEG', quality=85, optimize=True)
+ print(f'{source}: {source.stat().st_size} -> {output}:
{output.stat().st_size}')
+```
+
+After compression:
+
+1. open the compressed image and check text, diagrams, fine lines, colors, and
logos for visible artifacts;
+2. confirm that its dimensions and aspect ratio are correct and that its file
size is smaller;
+3. if the extension or file name changed, update the front matter and every
Markdown reference;
+4. verify that every `/images/...` reference resolves to an existing file with
the same case-sensitive name;
+5. delete the uncompressed source and any superseded variants from the
repository.
+
+The final article directory should contain only the compressed images used by
the article. Do not keep both the original PNG and the converted JPEG after all
references point to the JPEG.
When the supplied Markdown references images:
diff --git
a/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/agent-data-loop.jpg
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/agent-data-loop.jpg
new file mode 100644
index 00000000000..4d3adf457c9
Binary files /dev/null and
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/agent-data-loop.jpg
differ
diff --git
a/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/cover.jpg
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/cover.jpg
new file mode 100644
index 00000000000..4bbe29f0bc9
Binary files /dev/null and
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/cover.jpg
differ
diff --git
a/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/physical-ai-episode.jpg
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/physical-ai-episode.jpg
new file mode 100644
index 00000000000..947f44cf700
Binary files /dev/null and
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/physical-ai-episode.jpg
differ
diff --git
a/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/search-evolution.jpg
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/search-evolution.jpg
new file mode 100644
index 00000000000..1534411b116
Binary files /dev/null and
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/search-evolution.jpg
differ
diff --git
a/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/unified-execution.jpg
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/unified-execution.jpg
new file mode 100644
index 00000000000..617a8b59ee7
Binary files /dev/null and
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/unified-execution.jpg
differ
diff --git
a/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/unified-query-architecture.jpg
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/unified-query-architecture.jpg
new file mode 100644
index 00000000000..7f0a6fb9408
Binary files /dev/null and
b/static/images/blogs/apache-doris-5-multimodal-lakehouse-preview/unified-query-architecture.jpg
differ
diff --git
a/static/images/blogs/apache-doris-lance-physical-ai-analytics/autonomous-driving-hard-cases.jpg
b/static/images/blogs/apache-doris-lance-physical-ai-analytics/autonomous-driving-hard-cases.jpg
new file mode 100644
index 00000000000..df2caf8d422
Binary files /dev/null and
b/static/images/blogs/apache-doris-lance-physical-ai-analytics/autonomous-driving-hard-cases.jpg
differ
diff --git
a/static/images/blogs/apache-doris-lance-physical-ai-analytics/cover.jpg
b/static/images/blogs/apache-doris-lance-physical-ai-analytics/cover.jpg
new file mode 100644
index 00000000000..f38855fc646
Binary files /dev/null and
b/static/images/blogs/apache-doris-lance-physical-ai-analytics/cover.jpg differ
diff --git
a/static/images/blogs/apache-doris-lance-physical-ai-analytics/physical-ai-data-architecture.jpg
b/static/images/blogs/apache-doris-lance-physical-ai-analytics/physical-ai-data-architecture.jpg
new file mode 100644
index 00000000000..7200387b620
Binary files /dev/null and
b/static/images/blogs/apache-doris-lance-physical-ai-analytics/physical-ai-data-architecture.jpg
differ
diff --git
a/static/images/blogs/apache-doris-lance-physical-ai-analytics/robot-episode-analysis.jpg
b/static/images/blogs/apache-doris-lance-physical-ai-analytics/robot-episode-analysis.jpg
new file mode 100644
index 00000000000..9f131593bcf
Binary files /dev/null and
b/static/images/blogs/apache-doris-lance-physical-ai-analytics/robot-episode-analysis.jpg
differ
diff --git
a/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/agent-vector-search.jpg
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/agent-vector-search.jpg
new file mode 100644
index 00000000000..efce4814c9b
Binary files /dev/null and
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/agent-vector-search.jpg
differ
diff --git
a/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/cover.jpg
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/cover.jpg
new file mode 100644
index 00000000000..c0d47f555d7
Binary files /dev/null and
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/cover.jpg differ
diff --git
a/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/kwai-vector-search.jpg
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/kwai-vector-search.jpg
new file mode 100644
index 00000000000..4bd6e8263dc
Binary files /dev/null and
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/kwai-vector-search.jpg
differ
diff --git
a/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/open-ai-lakehouse-architecture.jpg
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/open-ai-lakehouse-architecture.jpg
new file mode 100644
index 00000000000..281802609d9
Binary files /dev/null and
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/open-ai-lakehouse-architecture.jpg
differ
diff --git
a/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/vector-index-query-plan.jpg
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/vector-index-query-plan.jpg
new file mode 100644
index 00000000000..eb44c3f9732
Binary files /dev/null and
b/static/images/blogs/apache-doris-paimon-agentic-ai-data-loop/vector-index-query-plan.jpg
differ
diff --git
a/static/images/blogs/apache-doris-variant-open-lakehouse/agent-trace-dual-write.jpg
b/static/images/blogs/apache-doris-variant-open-lakehouse/agent-trace-dual-write.jpg
new file mode 100644
index 00000000000..b421ac8e276
Binary files /dev/null and
b/static/images/blogs/apache-doris-variant-open-lakehouse/agent-trace-dual-write.jpg
differ
diff --git a/static/images/blogs/apache-doris-variant-open-lakehouse/cover.jpg
b/static/images/blogs/apache-doris-variant-open-lakehouse/cover.jpg
new file mode 100644
index 00000000000..6b1431f217f
Binary files /dev/null and
b/static/images/blogs/apache-doris-variant-open-lakehouse/cover.jpg differ
diff --git
a/static/images/blogs/apache-doris-variant-open-lakehouse/hot-cold-data.jpg
b/static/images/blogs/apache-doris-variant-open-lakehouse/hot-cold-data.jpg
new file mode 100644
index 00000000000..28295f17b4b
Binary files /dev/null and
b/static/images/blogs/apache-doris-variant-open-lakehouse/hot-cold-data.jpg
differ
diff --git
a/static/images/blogs/apache-doris-variant-open-lakehouse/open-variant-standard.jpg
b/static/images/blogs/apache-doris-variant-open-lakehouse/open-variant-standard.jpg
new file mode 100644
index 00000000000..490d36af196
Binary files /dev/null and
b/static/images/blogs/apache-doris-variant-open-lakehouse/open-variant-standard.jpg
differ
diff --git
a/static/images/blogs/apache-doris-variant-open-lakehouse/unified-variant-execution.jpg
b/static/images/blogs/apache-doris-variant-open-lakehouse/unified-variant-execution.jpg
new file mode 100644
index 00000000000..23af5139b51
Binary files /dev/null and
b/static/images/blogs/apache-doris-variant-open-lakehouse/unified-variant-execution.jpg
differ
diff --git
a/static/images/blogs/kwai-paimon-vector-search-apache-doris/benchmark-overview.jpg
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/benchmark-overview.jpg
new file mode 100644
index 00000000000..f2b2779f670
Binary files /dev/null and
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/benchmark-overview.jpg
differ
diff --git
a/static/images/blogs/kwai-paimon-vector-search-apache-doris/bucket-parallelism.jpg
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/bucket-parallelism.jpg
new file mode 100644
index 00000000000..c6e3511129f
Binary files /dev/null and
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/bucket-parallelism.jpg
differ
diff --git
a/static/images/blogs/kwai-paimon-vector-search-apache-doris/cover.jpg
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/cover.jpg
new file mode 100644
index 00000000000..9677fa7e56f
Binary files /dev/null and
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/cover.jpg differ
diff --git
a/static/images/blogs/kwai-paimon-vector-search-apache-doris/ivf-rq-tradeoffs.jpg
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/ivf-rq-tradeoffs.jpg
new file mode 100644
index 00000000000..0870b87f781
Binary files /dev/null and
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/ivf-rq-tradeoffs.jpg
differ
diff --git
a/static/images/blogs/kwai-paimon-vector-search-apache-doris/late-materialization.jpg
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/late-materialization.jpg
new file mode 100644
index 00000000000..83cd79c494c
Binary files /dev/null and
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/late-materialization.jpg
differ
diff --git
a/static/images/blogs/kwai-paimon-vector-search-apache-doris/paimon-data-index-manifests.jpg
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/paimon-data-index-manifests.jpg
new file mode 100644
index 00000000000..dedb251488e
Binary files /dev/null and
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/paimon-data-index-manifests.jpg
differ
diff --git
a/static/images/blogs/kwai-paimon-vector-search-apache-doris/rust-reader.jpg
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/rust-reader.jpg
new file mode 100644
index 00000000000..913a7ada733
Binary files /dev/null and
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/rust-reader.jpg
differ
diff --git
a/static/images/blogs/kwai-paimon-vector-search-apache-doris/top-k-pushdown.jpg
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/top-k-pushdown.jpg
new file mode 100644
index 00000000000..b46048d89f5
Binary files /dev/null and
b/static/images/blogs/kwai-paimon-vector-search-apache-doris/top-k-pushdown.jpg
differ
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
