javier opened a new pull request, #4505: URL: https://github.com/apache/iggy/pull/4505
## Which issue does this PR address? Closes #4086 ## Rationale Iggy has no QuestDB sink. QWP is QuestDB's native and fastest ingestion protocol, and it gives typed columns, per-flush acks, store-and-forward and multi-host failover, which line protocol does not. The issue has the details. ## What changed? Iggy could reach QuestDB only through the InfluxDB sink over line protocol, which loses typed columns, acks and failover, and is not QuestDB's fastest ingestion path. The new sink in `core/connectors/sinks/questdb_sink` writes over QWP with `questdb-rs`, and the same plugin works with QuestDB open source and QuestDB Enterprise through params in the connect string. Delivery is at-least-once, as with QuestDB's official Kafka connector, so target tables should use `DEDUP UPSERT KEYS`. A window that fails is logged with its exact offsets, because the runtime commits offsets before `consume` and cannot replay it (#2928, #2927). Notes for review: JSON numbers default to DOUBLE (`integer_columns` keeps chosen columns exact), flushes are capped by bytes as well as rows, and an ack that has not arrived yet is reported as lag rather than loss. The README covers configuration and these trade-offs. ## Local Execution - Passed: clippy, 115 unit tests, and the testcontainers integration suite (25 tests against a real QuestDB). - Pre-commit hooks ran: `prek run` passed on the branch. - Also manually tested against a three-node QuestDB Enterprise cluster. With rows streaming continuously, I set the primary read-only, promoted a replica, then switched back. All 60,500 rows landed on every node, with no gaps and no duplicates. ## AI Usage 1. Claude (Claude Code). 2. Claude wrote most of the implementation, following a design I drafted and modeled on QuestDB's Kafka connector, which we wrote by hand. We went through multiple design and review iterations. 3. Every iteration went through Iggy's own `connector-review` skill, the adversarial multi-agent review in `.claude/skills`, and we repeated it until it came back clean. Before opening the PR, I also ran a final review with the code-review skill I use for QuestDB PRs, adapted to Iggy. Beyond the unit and testcontainers suites, I tested it against QuestDB OSS locally and against the Enterprise cluster described above, which I maintain by hand for demos. 4. Yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
