CryoThrust opened a new issue, #12102: URL: https://github.com/apache/seatunnel/issues/12102
## Summary This STIP proposes a Phase 1 connector-local sink that moves SeaTunnel records into a self-hosted Mem0-compatible memory service. It is the narrowed implementation path from #11304: **source to memory only**, one backend family, no new engine/core/SPI abstraction, and no agent-runtime behavior. The goal is durable batch hydration and CDC-driven freshness for enterprise agent-memory data, not conversational memory management. ## Goals 1. Add a Connector-V2 sink built on the existing HTTP-base extension points. 2. Make the record-to-request mapping explicit and validate invalid combinations at startup. 3. Define an honest, testable delivery contract for the backend asynchronous add API. 4. Provide deterministic mock-server coverage, EN/ZH docs, and batch/CDC examples. ## Non-goals - No `MemorySink` abstraction in engine/core/SPI. - No Mem0 SDK or provider-specific runtime dependency. - No agent runtime, embedding, summarization, retrieval, or conversational ingestion protocol. These remain upstream transforms/application responsibilities. - No source/export/archival path or bi-temporal history model. - No exactly-once claim without a documented, backend-enforced idempotency contract. ## Proposed Connector Boundary The initial module is a plain HTTP-family sink. It owns provider request/response translation, authentication forwarding, sanitization, and status classification; SeaTunnel retains normal source checkpoint and retry behavior. Existing HTTP-base request building, header configuration, retry hooks, and response-body sanitization are reused rather than introducing shared memory APIs. No engine, connector API, checkpoint-state, or existing connector configuration changes are required. ## Configuration and Row Contract The implementation must use unambiguous option names following existing Connector-V2 conventions. The intended logical contract is: | Concern | Required contract | | --- | --- | | Endpoint | `endpoint` identifies one Mem0-compatible service base URL. | | Authentication | Existing HTTP-base authentication/header mechanism; credentials must never appear in diagnostics. | | Operation | `operation` is an explicit default. A CDC override may use `operation_field`; supported values are `add` and `delete`. | | Scope | At least one configured scope mapping is required for `add`: `user_id`, `agent_id`, `app_id`, or `run_id`; each mapping is a literal or row path. | | Content | `messages_field` is required for `add`; it resolves to the provider request payload. | | Metadata | `metadata_field` is optional and must be a JSON object when present. | | Record identity | `event_id_field` is optional and is only used as a deterministic correlation value; it must not create an exactly-once claim by itself. | | Delete | `memory_id_field` is mandatory for a row-level `delete`. Scope deletion is a separately named, explicitly configured operation; null content never implies deletion. | Examples of combinations that must fail at startup: - `add` without `messages_field` or without any scope mapping; - `delete` without `memory_id_field`; - an `operation_field` that can emit unknown values without a configured rejection policy; - a metadata mapping that is not an object; - simultaneous literal and row-path mappings for the same scope field. ## Delivery and Failure Semantics Mem0 documents `POST /v3/memories/add/` as an asynchronous, additive operation that returns an `event_id`; terminal processing can be inspected separately. The Phase 1 delivery boundary is therefore: > An `add` record is acknowledged when the configured endpoint returns a successful acceptance response. This is **accepted, at-least-once** delivery, not completed delivery and not exactly-once delivery. The sink does not poll event completion or keep provider event state in this phase. An event-polling/completed-delivery mode would require explicit timeout, checkpoint, recovery, and ordering design and is deferred. Retry classification: - Retry transport failures and HTTP 408, 429, and 5xx using the existing connector retry policy. - Do not retry authentication/authorization, malformed request, or other terminal 4xx responses. - Error diagnostics retain status and a sanitized, bounded provider body; secrets, access tokens, and full payloads are never logged. - For a batch/async submission with individual outcomes, only accepted records are acknowledged. Rejected records must propagate through the normal connector failure path with a safe operation/scope correlation value. ## Delete Capability Gate The additive endpoint is not an update/delete API. The implementation PR must name the exact documented Mem0-compatible row-delete endpoint, request shape, and acceptance semantics before enabling `delete`. If the selected target version does not expose a stable documented delete endpoint, the first implementation ships `add` only and rejects `delete` during option validation. It must never silently map a delete record to the additive endpoint. This preserves a clean CDC evolution path while preventing unverified destructive behavior. ## Test Matrix The implementation PR must include deterministic mock-server tests for: 1. all valid and invalid configuration combinations, including literal and row-path scopes; 2. add request serialization, scope mapping, metadata validation, and correlation propagation; 3. the explicit 2xx acceptance boundary and returned `event_id`; 4. retryable transport/408/429/5xx and non-retryable 4xx outcomes; 5. bounded sanitization of response bodies and headers; 6. partial accepted/rejected batch behavior where supported by the endpoint; 7. delete request serialization and failure behavior only after the endpoint capability gate is resolved. A real self-hosted Mem0 Testcontainers profile is optional and may be added only after image licensing, startup time, and CI reliability are demonstrated. A deterministic mock server is the required acceptance test. ## Documentation and Examples The connector documentation must state the accepted at-least-once guarantee, operation/capability limits, scope requirements, retry outcomes, and sensitive-data handling. Two examples are required: 1. JDBC batch hydration into a scoped memory collection. 2. MySQL-CDC freshness pipeline using an explicit row operation mapping; delete is included only after the capability gate is accepted. ## Compatibility This is additive: a new connector module, examples, tests, and documentation only. No existing config, SPI, engine lifecycle, state format, or checkpoint compatibility changes. A second backend must exist before proposing any reusable cross-provider memory abstraction. ## Review Questions 1. Is accepted at-least-once delivery the appropriate Phase 1 contract for the documented asynchronous add operation? 2. Which exact Mem0-compatible delete endpoint/version should be treated as the Phase 1 capability gate? 3. Does the proposed logical mapping fit existing HTTP-base configuration conventions, or should it be adjusted before implementation? 4. Is a mock-server-only first PR preferred before evaluating Testcontainers? Related: #11304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
