tisonkun opened a new pull request, #4708: URL: https://github.com/apache/iggy/pull/4708
## Summary Add a ScopeDB sink connector (`iggy_connector_scopedb_sink`) that appends consumed Iggy batches to an existing ScopeDB table via the published `scopedb-client 0.5.0` crate. - **Write modes & schema validation**: Supports `mapped` mode (filtering top-level JSON object fields to writable table columns) and `raw` mode (writing binary, UTF-8 text, JSON object, or arbitrary JSON payloads into a configured `payload_column`), with optional `iggy_*` metadata and `iggy_headers` columns validated against the table catalog on startup. - **Batching, flow control & retries**: Delegates request splitting (`batch_size`, `max_request_bytes`), concurrency (`max_concurrent_requests`), zstd-compressed NDJSON framing, and bounded retry backoff to `scopedb_client::AppendWriter` with `AppendFailurePolicy::Continue` so sibling sub-batches still settle when one sub-batch fails. - **Error classification & replay semantics**: Retries validated HTTP 429 admission refusals (up to three total attempts), keeps ambiguous outcomes (`5xx`, connection drops, invalid row counts, and timeouts) unretried by default unless `retry_ambiguous = true` is configured, and filters/replays remaining valid rows at most once when a definite `400`/`422` response provides complete row-error indexes without replaying already-committed sibling batches. - **Zero-copy JSON bridging & single-pass row encoding**: Moves owned `Payload::Json` values directly via `owned_value_into_serde_json` and retains pre-serialized `RawValue` rows after per-row byte-limit validation so `AppendWriter` emits each row without a second serialization pass. ## Validation - `prek run` and `cargo clippy -p iggy_connector_scopedb_sink --all-targets -- -D warnings` pass. - 5 unit tests and 15 HTTP integration tests in `core/connectors/sinks/scopedb_sink/tests/scopedb_sink.rs` pass against an in-process fake ScopeDB server covering partial batch recovery, whole-batch errors, invalid success counts, bounded 429 retries, ambiguous 503/timeout handling, and cancellation during in-flight HTTP requests. - End-to-end live ScopeDB tests in `core/connectors/sinks/scopedb_sink/tests/live_scopedb.rs` verified against a live ScopeDB instance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
