curcur edited a comment on pull request #17229:
URL: https://github.com/apache/flink/pull/17229#issuecomment-955988634
Thanks @rkhachatryan for the PR, I have a few high-level + implementation
questions. Let's discuss offline:
- Item 1
`seizeCapacity`: multiple task thread?
1. `hasCapacity()`: inFlightBytesCounter < maxBytesInFlight;
`isAvaialble()`: what is that, how it is different from `hasCapacity()`
2. Even waken up, it still not guranteed to have enough capacity
(`inFlightBytesCounter + bytes`) still possible to exceed capacity
what's the purpose to split `seizeCapacity` into these two steps?
- Item 2
`releaseCapacity`: IO tread, single IO thread, multiple IO thread?
How it is related to `upload` task scheduler?
`UploadThrottle` - AvailabilityProvider
`BatchingStateChangeUploader` contains AvailabilityProvider
- Item 3
`BatchingStateChangeUploader.upload`
even if it grabs the lock and seize capacity, it may still not able to
upload all bytes
Why it is designed this way
- Item 4
`BatchingStateChangeUploader` per `FsStateChangelogStorage` inlcuded in
`TaskStateManager`
per TM
- Item 5
Why it is allowed to be null?
`environment.getTaskStateManager().getStateChangelogStorage() == null`
- Item 6
Now Backpressure seems has one more state:
`idleTimeMsPerSecond`
`busyTimeMsPerSecond`
`backPressuredTimeMsPerSecond`
-- backpressure because of downstream / backpressure because of changelog IO
- Item 7
I found there are several locks in a few different places, are they all
necessary (asking this because of the DFS PR is not reviewed by me, so needs a
bit more input)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
