ethanlin01x commented on code in PR #3996: URL: https://github.com/apache/iggy/pull/3996#discussion_r3944274588
########## core/connectors/sinks/dynamodb_sink/README.md: ########## @@ -0,0 +1,95 @@ +# DynamoDB Sink Connector + +A sink connector that consumes messages from Iggy streams and writes them to an +Amazon DynamoDB table with `BatchWriteItem` through the official AWS SDK. + +## Configuration + +```toml +[plugin_config] +table = "iggy_messages" +region = "us-east-1" +# endpoint = "http://localhost:8000"; +# access_key_id = "..." +# secret_access_key = "..." +# session_token = "..." +partition_key_field = "iggy_id" +# sort_key_field = "iggy_offset" +batch_size = 25 +include_metadata = true +include_checksum = true +include_origin_timestamp = true +max_item_size = 409600 +max_retries = 3 +retry_delay = "500ms" +max_retry_delay = "5s" +verbose_logging = false +``` + +- `table`: Target DynamoDB table. The table must already exist. +- `region`: AWS region. Falls back to the default AWS region chain when unset. +- `endpoint`: Custom endpoint URL, for DynamoDB Local or a VPC endpoint. +- `access_key_id` / `secret_access_key` / `session_token`: Static credentials. + Provide both `access_key_id` and `secret_access_key`, or neither. When both + are omitted the connector uses the default AWS credential chain. +- `partition_key_field`: Item attribute used as the table partition key. + Defaults to `iggy_id`. +- `sort_key_field`: Item attribute used as the table sort key. Only set this + when the table has a sort key. +- `batch_size`: Items per `BatchWriteItem` request. Defaults to `25`, which is + also the DynamoDB limit, so larger values are clamped. +- `include_metadata`: Add `iggy_stream`, `iggy_topic`, `iggy_partition_id`, + `iggy_offset`, and `iggy_timestamp` to each item. Defaults to `true`. +- `include_checksum`: Add `iggy_checksum`. Defaults to `true`. +- `include_origin_timestamp`: Add `iggy_origin_timestamp`. Defaults to `true`. +- `max_item_size`: Maximum item size in bytes. Defaults to `409600` (400 KB), + which is also the DynamoDB limit, so larger values are clamped. +- `max_retries`: Retries after the first attempt. Defaults to `3`. +- `retry_delay`: First retry delay as a humantime string. Defaults to `500ms`. +- `max_retry_delay`: Upper bound of the exponential backoff. Defaults to `5s`. +- `verbose_logging`: Log per-batch results at info level. Defaults to `false`. + +## Behavior + +JSON objects are written attribute by attribute, so a message field becomes a +DynamoDB attribute of the matching type. JSON arrays and scalars are nested +under a `payload` attribute, because a DynamoDB item must be a map. Text +payloads go into `payload` as a string. Raw payloads are parsed as JSON when +possible, otherwise they are stored as binary. Protobuf, FlatBuffer, and Avro +payloads are not supported and are skipped with a warning. + +Metadata attributes are written after the payload, so they overwrite payload +fields of the same name. Review Comment: Fixed in bef6bb677. I took the first option: the README now says message headers are not written, and an item holds only the payload and the enabled iggy_* attributes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
