The GitHub Actions job "flink-connector-aws: build on pull request" on flink-connector-aws.git/main has succeeded. Run started by GitHub user gguptp (triggered by ferenc-csaky).
Head commit for run: 1ed75ae27de0b525ba9414f35743451db9875393 / Abhi Gupta <[email protected]> [FLINK-41365][Connectors/DynamoDB] Make non-blocking GetRecords calls in the DynamoDB Streams split reader The split reader made one blocking GetRecords call at a time, so a slow or throttled call on one shard delayed every other shard on the reader. With many shards per reader and high call latency (for example, cross-region reads), each shard was polled far less often than the configured poll delay. By default, the split reader now makes non-blocking calls: 1. Each fetch() starts a GetRecords call (or a child shard discovery call) for every shard that is ready, not paused, and has no call in progress. 2. The fetch then returns the records of one completed call. If no call is complete, it waits until a call completes, a shard is ready, or wakeUp() is called, for at most 100 ms. Each shard has at most one call in progress, and new option flink.dynamodbstreams.getrecords.max-in-flight (default 100) limits the calls in progress per source reader, so that a reader buffers a bounded number of responses. Shards that the limit keeps waiting get the next free calls. Held responses of paused splits do not count, so that paused splits cannot block the splits that watermark alignment waits for. New option flink.dynamodbstreams.getrecords.blocking (default false) keeps the previous blocking behavior. Both modes store the same split state, so a job can switch in either direction and restore from its last checkpoint. Details: - StreamProxy gets getRecordsAsync and listShardsWithFilterAsync, implemented with DynamoDbStreamsAsyncClient. They follow the blocking methods: the shard iterator store, the retry after ExpiredIteratorException, the read from TRIM_HORIZON after TrimmedDataAccessException, and the empty response after ResourceNotFoundException. - A call that fails after all SDK retries fails fetch(), as the blocking call does. - If a split is paused while its call is in progress, the reader holds the response until the split resumes and starts no second call for it. - The async client has the same region, credentials, endpoint override and user agent prefix as the sync client. Its Netty HTTP client uses HTTP/1.1, because DynamoDB Streams does not support HTTP/2. AWSClientUtil gets a createAwsAsyncClient overload that takes a ClientOverrideConfiguration builder, as createAwsSyncClient does, so that the caller can set the retry strategy. - In non-blocking mode, the split readers use the configured retry mode (AWS_RETRY_MODE or the profile's retry_mode) without a circuit breaker. All calls of a reader share the retry token bucket of the strategy, and with many calls in progress, one burst of errors would spend the whole bucket. The adaptive modes cannot turn off the circuit breaker, so they use the standard mode. Blocking mode keeps the default retry strategy. - In non-blocking mode the stream proxy has only the async client. - All split readers of a source reader now share one stream proxy, and the source reader closes it. The fetcher manager creates a new split reader each time it starts a fetcher, for example after all splits of the previous fetcher finished. A proxy per split reader created a new SDK client, HTTP client and Netty event loop each time. The proxy is created with the first split reader, so a source reader without splits opens no clients. - The proxy removes the stored shard iterator at the end of a shard. - The enumerator still uses only the sync client. Testing: unit tests run on JDK 11, 17 and 21, and the existing reader tests run in both modes. A test sends one call through the real Netty client to a local HTTP/1.1 server. A load test on Amazon Managed Service for Apache Flink (Flink 2.2, parallelism 512) read a stream with tens of thousands of shards from TRIM_HORIZON. It read about 30 hours of backlog at 200k-250k records/s and caught up in about 20 minutes. Once caught up, it made about the same number of GetRecords calls as the blocking reader on the same stream. The load test ran before the in-flight limit was added. Generated-by: Claude Code (Claude Opus 5.5) Report URL: https://github.com/apache/flink-connector-aws/actions/runs/42663805063 With regards, GitHub Actions via GitBox
