atharvalade commented on PR #3103: URL: https://github.com/apache/iggy/pull/3103#issuecomment-4524426822
The biggest thing I caught was a credential leak where the derived Debug on S3Sink would dump the full AWS access key and secret key into logs whenever the struct got printed. I replaced that with a manual Debug impl that just shows the bucket name and added a regression test so it never sneaks back in. On the metrics side, the received count was being incremented after processing the batch, which meant if something failed halfway through you could lose track of how many messages actually got dropped. I moved that counter up front and added logic to correctly mark the unprocessed remainder as lost. I also cleaned up the dependency situation since `once_cell` was listed but never used, `simd-json `was only needed in tests but lived in regular deps, and the `cargo-machete` ignore list was papering over all of it. The `max_retries` field got renamed to `max_attempts` with a serde alias so existing configs still work, and the README had a contradiction where it said timestamp was wall clock time when the code actually derives it from the first message in the buffer, so I fixed the docs to match reality. For the lost marker file that records data gaps, I routed it through the retry logic so a transient S3 hiccup does not silently eat your gap record. Then I wrote about twenty unit tests covering the pure functions like retry status classification, flush payload extraction, config validation, and the Debug redaction. Finally I stood up a full integration test suite using MinIO in a container that validates the sink actually writes `jsonl` to S3 with the correct path layout and rotates files properly when the message count threshold is hit. /ready -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
