numinnex opened a new issue, #4728: URL: https://github.com/apache/iggy/issues/4728
### Bug description Three smaller problems in the Iceberg sink. 1. **Credential logging.** `open()` (`core/connectors/sinks/iceberg_sink/src/sink.rs`) logs the first three characters of both `store_access_key_id` and `store_secret_access_key`. Both fields are plain `String`. They should be `SecretString`, and the log line should not print any part of them. 2. **Uncounted drops in the dynamic router.** `DynamicRouter::route_data` skips a message when the route field is missing or unsupported, when the table name is invalid, or when the table does not load. At most it logs a warning. The batch still reports success and nothing increments the error count, so the rows disappear. 3. **Partial commits across tables.** With more than one target table, both routers commit table by table. If table k fails, the tables before it keep their rows and the batch is reported as failed. Once failed batches are redelivered (#4232), those earlier tables get the rows twice. ### Affected area / component Connectors: `iggy_connector_iceberg_sink` ### Versions master @ 3c265dd89 ### Proposed fix 1. Switch both credential fields to `SecretString` and remove the partial logging. 2. Treat an unroutable message as a failure, or at least count it, and log its offset. 3. Make multi-table writes safe to redeliver, for example with the per-commit snapshot marker proposed for catalog retries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
