johnclara edited a comment on issue #1637: URL: https://github.com/apache/iceberg/issues/1637#issuecomment-718265869
@rdblue our pipeline is: kafka topic -> akka streams service -> ingestion iceberg table -> spark compaction job -> compacted iceberg table the akka streams service only provides at least once delivery guarantees so we have to be okay with duplicate data anyways in the ingestion table the way we implemented our table's snapshot commit actually fails a lot due to DynamoDB rate limiting so we have to have retries around commits, which can lead to duplicate data files (not just duplicate rows) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
