curcur commented on pull request #17229:
URL: https://github.com/apache/flink/pull/17229#issuecomment-955988634
Thanks @rkhachatryan for the PR, I have a few high-level + implementation
questions. Let's discuss offline:
- Item 1
seizeCapacity: multiple task thread?
1. hasCapacity(): inFlightBytesCounter < maxBytesInFlight;
isAvaialble(): what is that, how it is different from hasCapacity()
2. Even waken up, it still not guranteed to have enough capacity
(inFlightBytesCounter + bytes) still possible to exceed capacity
what's the purpose to split seizeCapacity into these two steps?
- Item 2
releaseCapacity: IO tread, single IO thread, multiple IO thread?
How it is related to upload task scheduler?
UploadThrottle - AvailabilityProvider
BatchingStateChangeUploader contains AvailabilityProvider
- Item 3
BatchingStateChangeUploader.upload
even if it grabs the lock and seize capacity, it may still not able to
upload all bytes
Why it is designed this way
(Let's discuss this offline)
- Item 4
BatchingStateChangeUploader per FsStateChangelogStorage inlcuded in
TaskStateManager
per TM
- Item 5
Why it is allowed to be null?
environment.getTaskStateManager().getStateChangelogStorage() == null
- Item 6
Now Backpressure seems has one more state:
idleTimeMsPerSecond
busyTimeMsPerSecond
backPressuredTimeMsPerSecond -- backpressure because of downstream /
backpressure because of changelog IO
- Item 7
I found there are several locks in a few different places, are they all
necessary (asking this because of the DFS PR is not reviewed by me, so needs a
bit more input)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
