CryoThrust commented on issue #11304: URL: https://github.com/apache/seatunnel/issues/11304#issuecomment-5550817551
Following the maintainer feedback, here is a reviewable Phase 1 design boundary before any connector implementation. ## Scope Add one connector-local Mem0-compatible **sink** on top of the existing HTTP connector base. The first phase is source-to-memory only and supports two explicit operations: `upsert` and `delete`. There is no new engine/core/SPI abstraction, no Mem0 SDK dependency, no conversational ingestion, and no memory source or archival flow. ## Configuration and row contract The connector should make the mapping explicit rather than infer semantics from null values: - `endpoint` and authentication settings identify the Mem0-compatible REST service; - `operation` is a required mode (`upsert` or `delete`), with an optional row-field override for CDC jobs; - `user_id` is required for `upsert` and may be a literal or a row path; - `agent_id` and `run_id` are optional literal/row-path scope fields; - `memory` and optional `metadata` are row paths for `upsert`; - `memory_id` is required for row-level `delete`; - scope deletion is a separate explicit mode and is never inferred from a null memory field. Startup validation should reject ambiguous combinations (for example, `delete` without `memory_id` or an explicit scope-delete mode, and an `upsert` without a content mapping). The exact option names can follow existing HTTP connector conventions after the API review. ## Delivery and failure semantics The connector must state at-least-once delivery unless the backend provides a stable idempotency contract. When a source event identifier is available, derive a deterministic request key from that identifier; otherwise derive it from operation, scope, memory id, and content. Attach it only if the Mem0-compatible endpoint documents an idempotency header. Do not claim exactly-once based on a client-side hash alone. Retry only transient transport failures, 408, 429, and 5xx responses, using the existing HTTP retry policy. Authentication/authorization and invalid-request responses are non-retryable and should surface a sanitized status/body. If a batch is partially accepted, acknowledge only successful records and report rejected records with operation and scope identifiers; never turn the whole batch into a false success. ## Backend boundary The connector should target the documented Mem0-compatible REST operations through the HTTP base and keep provider-specific request/response translation inside the connector. The implementation must not assume that a particular Mem0 deployment supports idempotency headers or scope deletion; unsupported capabilities should fail during validation or be documented as at-least-once behavior. A second backend will be evaluated separately before considering a shared memory abstraction. ## Test and acceptance matrix The first implementation PR should include deterministic mock-server tests for: 1. valid/invalid option combinations and literal/row-path scope mapping; 2. `upsert` and `delete` request serialization; 3. stable request-key generation and the documented delivery guarantee; 4. 2xx success, 408/429/5xx retry, non-retryable 4xx, malformed responses, and sanitized bodies; 5. partial batch failure and acknowledgement boundaries. An optional Testcontainers profile can exercise a real self-hosted Mem0 image only after startup time, licensing, and CI reliability are confirmed. Two examples should demonstrate JDBC batch hydration and MySQL-CDC upsert/delete. The PR should explicitly state that embeddings and summarization happen upstream through existing SeaTunnel transforms. If this boundary is accepted, the next step is a small implementation PR containing only the connector, tests, examples, and documentation, followed by a separate evaluation of a second backend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
