ryankert01 opened a new pull request, #3575: URL: https://github.com/apache/iggy/pull/3575
## Which issue does this PR address? Relates to #3215 > Stacked on #3574 (in-request retry). Until that merges, this PR's diff includes the retry commit; it reduces to the CSV-only diff once #3574 lands. ## Rationale Follow-up to the Doris sink (#3215). JSON Stream Load parses more expensively on the Doris BE than CSV; a 10k-row benchmark shows CSV loads ~21% faster server-side at roughly half the body size. This adds CSV as an opt-in output format. ## What changed? The sink emitted JSON Stream Load only. This adds an opt-in `output_format = "csv"` (default stays JSON). CSV is positional where JSON is name-mapped, so it requires `columns` to pin the column order — `open()` fails on `output_format = "csv"` without it. The hand-rolled encoder uses control-char framing (`column_separator` `\x01`, `line_delimiter` `\x02`) with `enclose`/`escape` quoting so embedded separators/quotes/newlines round-trip. Doris escapes with a prefix char (`\"`), not RFC-4180 quote-doubling, so the `csv` crate does not fit and no new dependency is added. JSON `null` and missing keys map to `\N`, empty string to `""`, numbers and bools emit bare, nested values stringify to JSON. The field is `output_format` (not `format`) to avoid an env-override collision with the runtime's top-level `plugin_config_format`. ## Local Execution - Passed - Pre-commit hooks: checks run manually (the `license-headers` hook cannot run on this machine's bash 3.2 — its script needs `mapfile`; `hawkeye check` passes directly and CI enforces it). `cargo fmt`, `cargo clippy -p iggy_connector_doris_sink --all-targets -- -D warnings`, `cargo test -p iggy_connector_doris_sink` (50 unit tests), `markdownlint`, and `taplo` all pass. Verified end-to-end against a real `apache/doris:4.0.3-all-slim` container: the connector-driven CSV test round-trips a value with a comma/quote/backslash byte for byte; the `#[ignore]`d JSON-vs-CSV benchmark reports CSV ~21% faster (server LoadTimeMs 37 vs 47, body 520KB vs 1060KB). The pre-existing 1000-row bulk test is flaky on the local constrained Doris BE (reproduces without these changes); it passes in CI. ## AI Usage 1. Claude Code (Anthropic). 2. Implemented the `Format` enum + `output_format` config, the hand-rolled CSV encoder, the connector-driven correctness test, the `#[ignore]`d benchmark, and docs — after researching Doris CSV Stream Load escaping/NULL semantics and confirming them against the running container. 3. 50 unit tests cover the encoder (escaping, null vs empty, column ordering, nested values); an integration test round-trips hazardous bytes through real Doris; the benchmark measures the load-time delta. Full crate suite, clippy, and lint pass locally. 4. Yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
