mas-chen opened a new pull request, #20215:
URL: https://github.com/apache/flink/pull/20215
…id source
## What is the purpose of the change
Hybrid Source gets into tight busy loop since availability future is marked
completed and never refreshed.
## Brief change log
- Introduce MultipleFuturesAvailabilityHelper to help manage availability
future
## Verifying this change
This change added tests and can be verified as follows:
- Added basic unit test to verify that availability future is refreshed
- Existing tests exercise that source continues to emit records properly
I verified this fix on the latest 1.14 branch.
Test setup: Hybrid Source setup (bounded read on short csv file and then
switchover to unbounded read from Kafka)
Test Jobs:
1. Hybrid Source without fix
2. Hybrid Source with fix
3. Kafka Source
I have attached the flamegraphs to the jira ticket. Additionally, I
confirmed that test job 2 and test job 3 CPU usage similar running in EKS (2%
CPU usage) vs test job 1 (~80%).
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): yes
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes
- The serializers: no
- The runtime per-record code paths (performance sensitive): no. This is
not per-record but this is performance impacting
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
