ethanlin01x opened a new issue, #3731: URL: https://github.com/apache/iggy/issues/3731
### Description Add a native Rust sink connector for Amazon DynamoDB, allowing Iggy stream messages to be written directly into DynamoDB tables. This is tracked in the connector ecosystem roadmap: #2753 ### Affected area / component Connectors ### Proposed solution **Crate:** [`aws-sdk-dynamodb`](https://crates.io/crates/aws-sdk-dynamodb) (official AWS SDK, async/Tokio compatible) **Key design decisions:** 1. **Batching** — `batch_write_item` has a hard limit of 25 items per request. The `consume()` handler will split incoming message batches into chunks of ≤25 before sending. 2. **Idempotency** — The connector will support a configurable `partition_key_field` (and optional `sort_key_field`) mapped from message payload. `batch_write_item` uses `PutRequest` which overwrites existing items on key conflict, making repeated delivery idempotent as long as the primary key is deterministic. 3. **UnprocessedItems handling** — `batch_write_item` can return partial failures via `UnprocessedItems` without throwing an error. The connector will retry unprocessed items with exponential backoff rather than silently dropping them. 4. **Throughput errors** — `ProvisionedThroughputExceededException` will be handled with retry logic compatible with both Provisioned and On-Demand capacity modes. 5. **Item size limit** — DynamoDB enforces a 400KB per-item limit. Oversized messages will be logged and skipped rather than failing the entire batch. ### Alternatives considered _No response_ ### Contribution - [x] I'm willing to submit a pull request to implement this feature ### Good first issue - [ ] I think this could be a good first issue for a new contributor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
