SEZ9 commented on issue #11304:
URL: https://github.com/apache/seatunnel/issues/11304#issuecomment-5179061270
@DanielLeens Thanks for the structural feedback — narrowing to an MVP is the
right call. I've since done a deeper survey of the agent-memory ecosystem
(Mem0/Zep/Letta/LangMem architectures, cloud memory services, and what existing
ETL tools actually cover), and this comment proposes a phased plan that follows
your four-point reduction, answering the five design questions along the way.
## Ecosystem facts that shape the choice
- **No open-source ETL tool has a dedicated connector for any agent-memory
API today** (Mem0/Zep/Letta). Mem0's 25+ integrations all target agent
frameworks (LangChain, CrewAI, ...), not data-integration platforms. Airbyte's
"Agent Connectors" are explicitly positioned as lightweight API clients, "not
designed for data replication". The lifecycle gap in the issue description is
real and unclaimed.
- Mem0 is the most active open-source memory engine (~59k stars),
self-hostable as three containers (FastAPI + pgvector + Neo4j) with a stable
REST API — which matters for us because it makes deterministic e2e testing with
testcontainers practical.
- Memory frameworks handle their own real-time conversational ingestion
(`add()` during chat). The unserved need is **bulk/continuous integration with
enterprise systems**: hydration from CRM/OLTP/warehouse, CDC-driven freshness,
deletion propagation, archival export. That's a data-movement problem, not an
agent-runtime problem — squarely our side of the boundary you drew.
## Phase 1 MVP (following the four-point reduction)
| Your question | MVP answer |
|---|---|
| Which backend first | **Mem0**, self-hosted REST API (Platform API is a
compatible superset) |
| Which direction | **source → memory only** (sink). Cold-start hydration is
the most universal first need, and a pure sink introduces no new abstractions |
| Which operations | **upsert + delete**, mapped to Mem0's `add`/`delete`
with explicit `user_id` / `agent_id` / `run_id` scoping options. Upsert covers
hydration and CDC freshness; delete lays the foundation for compliance
propagation later |
| Validation plan | e2e tests spinning the Mem0 stack via testcontainers;
docs (en/zh); two runnable examples — batch hydration (JDBC → Mem0) and
freshness (MySQL-CDC → Mem0 upsert/delete) |
**On the remaining design questions:**
- **Reusable contract vs connector-specific**: Phase 1 deliberately adds
*no* shared "memory abstraction" to SeaTunnel core. `connector-mem0` is a plain
sink built on `connector-http-base` (the same foundation as the 12+ existing
HTTP-family connectors). Whether a `MemorySink` contract is worth extracting is
a question we should answer *after* a second backend exists — one instance
doesn't justify an abstraction.
- **Compatibility impact**: none. No engine changes, no connector-API
changes, standard OptionRule config only.
- **Dependencies/licensing**: pure REST over the existing HTTP client — no
memory-backend SDK dependency at all in phase 1.
One thing worth highlighting: the transform side needs **zero new work**.
SeaTunnel's existing LLM transform (summarization/fact extraction) and
embedding transform already cover the encode-and-summarize stages of a memory
pipeline, so the MVP demo can show a complete `CRM conversations → LLM fact
extraction → embedding → Mem0` flow using only shipped components plus the new
sink.
## Where this goes after the MVP (direction, not commitment)
- **Phase 2 — bidirectional + compliance**: Mem0 *source* for memory →
lakehouse archival/analytics (pairs directly with existing Iceberg/Paimon
sinks); a delete-propagation pattern (CDC deletes → memory scope deletion with
an audit trail) targeting GDPR/RTBF — the hardest requirement for
finance/healthcare adopters and the most differentiated capability in this
space. Evaluation of a second backend (Zep/Graphiti or a cloud memory API)
happens here, and only then the shared-contract discussion.
- **Phase 3 — convergence with the AI CLI roadmap (#11616)**: memory
pipelines as a natural-language scenario ("hydrate Mem0 from our CRM, scoped by
user_id"), and memory-pipeline tasks added to the CLI accuracy benchmark so
generation quality for this connector family is measured, not assumed.
These later phases stay out of scope for the phase-1 design; they're listed
so the MVP's operation choice (upsert **+ delete**) is understood as deliberate
groundwork.
If this reduction looks right, I'll write up the phase-1 design as a proper
STIP-style proposal (config surface, error semantics, retry/idempotency
behavior, e2e matrix) as a sub-issue for review before any code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
