The GitHub Actions job "flink-connector-aws: build on pull request" on 
flink-connector-aws.git/main has succeeded.
Run started by GitHub user gguptp (triggered by ferenc-csaky).

Head commit for run:
1ed75ae27de0b525ba9414f35743451db9875393 / Abhi Gupta <[email protected]>
[FLINK-41365][Connectors/DynamoDB] Make non-blocking GetRecords calls in the 
DynamoDB Streams split reader

The split reader made one blocking GetRecords call at a time, so a slow or
throttled call on one shard delayed every other shard on the reader. With many
shards per reader and high call latency (for example, cross-region reads), each
shard was polled far less often than the configured poll delay.

By default, the split reader now makes non-blocking calls:

1. Each fetch() starts a GetRecords call (or a child shard discovery call) for
   every shard that is ready, not paused, and has no call in progress.
2. The fetch then returns the records of one completed call. If no call is
   complete, it waits until a call completes, a shard is ready, or wakeUp() is
   called, for at most 100 ms.

Each shard has at most one call in progress, and new option
flink.dynamodbstreams.getrecords.max-in-flight (default 100) limits the calls in
progress per source reader, so that a reader buffers a bounded number of
responses. Shards that the limit keeps waiting get the next free calls. Held
responses of paused splits do not count, so that paused splits cannot block the
splits that watermark alignment waits for.

New option flink.dynamodbstreams.getrecords.blocking (default false) keeps the
previous blocking behavior. Both modes store the same split state, so a job can
switch in either direction and restore from its last checkpoint.

Details:
- StreamProxy gets getRecordsAsync and listShardsWithFilterAsync, implemented
  with DynamoDbStreamsAsyncClient. They follow the blocking methods: the shard
  iterator store, the retry after ExpiredIteratorException, the read from
  TRIM_HORIZON after TrimmedDataAccessException, and the empty response after
  ResourceNotFoundException.
- A call that fails after all SDK retries fails fetch(), as the blocking call
  does.
- If a split is paused while its call is in progress, the reader holds the
  response until the split resumes and starts no second call for it.
- The async client has the same region, credentials, endpoint override and
  user agent prefix as the sync client. Its Netty HTTP client uses HTTP/1.1,
  because DynamoDB Streams does not support HTTP/2. AWSClientUtil gets a
  createAwsAsyncClient overload that takes a ClientOverrideConfiguration
  builder, as createAwsSyncClient does, so that the caller can set the retry
  strategy.
- In non-blocking mode, the split readers use the configured retry mode
  (AWS_RETRY_MODE or the profile's retry_mode) without a circuit breaker. All
  calls of a reader share the retry token bucket of the strategy, and with many
  calls in progress, one burst of errors would spend the whole bucket. The
  adaptive modes cannot turn off the circuit breaker, so they use the standard
  mode. Blocking mode keeps the default retry strategy.
- In non-blocking mode the stream proxy has only the async client.
- All split readers of a source reader now share one stream proxy, and the
  source reader closes it. The fetcher manager creates a new split reader
  each time it starts a fetcher, for example after all splits of the previous
  fetcher finished. A proxy per split reader created a new SDK client, HTTP
  client and Netty event loop each time. The proxy is created with the first
  split reader, so a source reader without splits opens no clients.
- The proxy removes the stored shard iterator at the end of a shard.
- The enumerator still uses only the sync client.

Testing: unit tests run on JDK 11, 17 and 21, and the existing reader tests run
in both modes. A test sends one call through the real Netty client to a local
HTTP/1.1 server. A load test on Amazon Managed Service for Apache Flink (Flink
2.2, parallelism 512) read a stream with tens of thousands of shards from
TRIM_HORIZON. It read about 30 hours of backlog at 200k-250k records/s and
caught up in about 20 minutes. Once caught up, it made about the same number
of GetRecords calls as the blocking reader on the same stream. The load test ran
before the in-flight limit was added.

Generated-by: Claude Code (Claude Opus 5.5)

Report URL: 
https://github.com/apache/flink-connector-aws/actions/runs/42663805063

With regards,
GitHub Actions via GitBox

Reply via email to