GitHub user MisterRaindrop created a discussion: [DISCUSSION] Making Apache
Cloudberry an Agent-Native Analytical Database
I'd like to discuss a possible direction for Apache Cloudberry: making
Cloudberry an **open analytical backend for AI agents**, rather than building
another agent framework inside the database.
AI agents such as Codex, Claude, Maka, and other MCP-compatible agents are
increasingly becoming a new interface for interacting with data systems.
The basic idea is:
```text
Agent
Codex / Claude / Maka
/ | \
/ | \
Web Cloudberry MCP Apps
│
┌──────┼──────────┐
▼ ▼ ▼
SQL RAG Semantic
│ │
Iceberg Lance
```
Cloudberry already has an MCP server and a mature MPP analytical engine.
Therefore, I don't think Cloudberry needs to build another agent runtime.
The external agent can be responsible for:
- reasoning and planning
- conversation and context
- web search
- external MCP tools
- multi-step tool orchestration
Cloudberry can focus on what an analytical database is good at:
- SQL analytics
- MPP execution
- structured data analysis
- vector / semantic retrieval
- structured + unstructured analysis
The responsibility boundary could be:
```text
Agent:
decide what to do
Cloudberry:
decide how to analyze the data efficiently
```
## Structured + AI-Native Data
Cloudberry already provides a strong foundation for structured analytical
workloads through SQL, MPP execution, and lakehouse integration.
One possible missing piece is an AI-native data layer.
A possible architecture is:
```text
Cloudberry
│
Unified Analytics
│
┌──────────┴──────────┐
▼ ▼
Iceberg Lance
│ │
Structured AI-native Data
Data Vector / Text /
Multimodal Data
```
Iceberg can continue to serve structured analytical/lakehouse workloads.
Lance could complement it as an external data source optimized for AI-oriented
datasets.
This could eventually allow queries combining structured business data with
semantic or multimodal data.
For example:
> Find customers whose revenue dropped by more than 20% in the last three
> months, then analyze their support tickets to determine the most common
> complaints.
Conceptually:
```text
Iceberg / Cloudberry
↓
Structured SQL analysis
↓
Declining customers
↓
Lance semantic/vector search
↓
Support documents
↓
Cloudberry aggregation
↓
Agent reasoning
```
This is closer to **analytical RAG** than traditional Top-K RAG.
## Why Lance?
The goal is not simply to add another vector index.
Lance is interesting because it is designed around AI-oriented datasets
containing combinations of:
```text
metadata
text
vectors
images
audio/video references
multimodal features
```
This makes it potentially useful as an AI-native data layer alongside Iceberg.
Cloudberry could remain responsible for SQL, JOIN, aggregation, MPP execution,
and distributed query planning, while Lance provides storage and retrieval
capabilities for AI-oriented datasets.
## Why Not Just pgvector?
Cloudberry already supports pgvector, and pgvector is a good solution for
storing and searching vectors inside PostgreSQL-compatible relational tables.
I see pgvector and Lance as solving different problems.
```text
pgvector
Cloudberry native table
│
├── relational columns
└── vector column
```
This is a natural solution when embeddings are part of relational data.
The proposed Lance integration targets external AI-native datasets:
```text
Lance Dataset
│
├── metadata
├── text
├── vectors
├── multimodal data
└── vector indexes
```
Therefore, Lance would not replace pgvector.
The three layers could coexist:
```text
pgvector
→ vectors in native Cloudberry tables
Iceberg
→ structured lakehouse datasets
Lance
→ AI-native / multimodal datasets
Cloudberry
→ unified MPP analytical engine
```
A simple way to describe the distinction is:
> **pgvector provides vectors inside PostgreSQL. Lance provides an AI-oriented
> dataset layer. Cloudberry provides the analytical engine across them.**
## RAG and Hybrid Retrieval
Once Cloudberry can access Lance datasets, we could expose retrieval
capabilities through the existing MCP server.
For example:
```text
execute_query()
vector_search()
hybrid_search()
search_documents()
retrieve_context()
```
An external agent could then combine these tools.
For example:
```text
Codex / Claude
│
MCP
│
Cloudberry
/ \
/ \
SQL Analytics RAG
│ │
Iceberg Lance
```
The important point is that Cloudberry does not need to know whether the caller
is Codex, Claude, Maka, or another agent.
MCP provides the common interface.
## Analytical RAG
A longer-term opportunity is to go beyond traditional RAG.
Traditional RAG usually works as:
```text
Documents
↓
Vector Search
↓
Top-K
↓
LLM
```
This works well for retrieval, but not for questions such as:
> What percentage of all customer complaints last year were related to query
> latency, grouped by month?
This requires:
```text
Large document dataset
↓
Semantic retrieval/filtering
↓
AI extraction/classification
↓
Cloudberry MPP
↓
COUNT / GROUP BY / JOIN
```
This could be an interesting area where Cloudberry's existing analytical engine
provides capabilities beyond a standalone vector database.
## Semantic Layer
Another possible future direction is a lightweight semantic layer.
Instead of requiring an agent to infer business meaning directly from physical
schemas, Cloudberry could expose concepts such as:
```text
Metrics:
revenue
churn_rate
Dimensions:
region
customer
product
Relationships:
customer → orders
customer → support_tickets
```
The external agent could use this semantic information before generating SQL or
retrieval requests.
This could improve the reliability of natural-language analytics without
requiring Cloudberry itself to implement an agent runtime.
## Proposed First Version
I think the first implementation should remain intentionally small.
### Phase 1: Read-only Lance FDW
The initial goal could simply be:
> **Allow Cloudberry to query an existing Lance Dataset as an external table.**
For example:
```sql
CREATE FOREIGN TABLE lance_documents (
id bigint,
customer_id bigint,
content text,
embedding float4[]
)
SERVER lance_server
OPTIONS (
uri 's3://bucket/documents.lance'
);
```
Initially support:
- schema mapping
- sequential scan
- projection pushdown
- filter pushdown
- read-only access
No INSERT / UPDATE / DELETE would be required initially.
Data could be generated by existing Lance tools or a simple export utility.
### Phase 2: Vector Top-K Pushdown
Then support queries such as:
```sql
SELECT id, content
FROM lance_documents
WHERE customer_id IN (...)
ORDER BY embedding <-> query_embedding
LIMIT 20;
```
Instead of:
```text
Read all vectors
↓
Cloudberry distance calculation
↓
Sort
↓
LIMIT
```
Cloudberry could recognize:
```text
Vector distance
+
LIMIT
```
and push the operation into Lance:
```text
Cloudberry Planner
↓
Vector Top-K Pushdown
↓
Lance Vector Index
↓
Top-K candidates
```
This would be the first step toward a vector-aware analytical engine.
### Phase 3: Distributed Vector Top-K
Cloudberry's MPP architecture could later provide:
```text
QD
│
┌──────────┼──────────┐
▼ ▼ ▼
QE0 QE1 QE2
│ │ │
Lance Lance Lance
│ │ │
Top-K Top-K Top-K
└──────────┼──────────┘
▼
Global Top-K
```
This could become a Cloudberry-specific capability rather than simply exposing
Lance APIs.
## Possible Roadmap
The whole direction could be explored incrementally:
```text
Read-only Lance FDW
↓
Vector Top-K Pushdown
↓
Distributed Vector Top-K
↓
Hybrid Search / RAG
↓
MCP Analytical Tools
↓
Analytical RAG
↓
Semantic Layer
```
Each stage is independently useful.
The first experiment is deliberately narrow:
> **Can Cloudberry efficiently query Lance datasets and push down Vector Top-K
> operations?**
If that proves useful, the higher-level AI analytical capabilities can be
explored incrementally.
## Long-Term Goal
The goal is not to turn Cloudberry into another vector database or another
agent framework.
Instead, the idea is to explore whether Cloudberry can evolve from:
```text
MPP Analytical Database
```
toward:
```text
Agent-Native Analytical Database
Structured Analytics
+
AI-Native / Multimodal Data
+
RAG / Semantic Search
+
MCP
```
External agents such as Codex, Claude, Maka, or any other MCP-compatible system
could then use Cloudberry as an open-source analytical backend.
I'd especially like feedback from the community on:
1. Does Lance make sense as an external AI-native data source for Cloudberry?
2. Should Lance integration start as an independent FDW/extension?
3. Does Iceberg + Lance + Cloudberry MPP provide useful capabilities beyond
existing pgvector support?
4. Would vector/RAG/analytical-search tools be useful additions to the existing
MCP server?
5. Does the broader idea of an **agent-native analytical database** fit
Cloudberry's long-term direction?
GitHub link: https://github.com/apache/cloudberry/discussions/1967
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
