errose28 commented on PR #3286:
URL: https://github.com/apache/ozone/pull/3286#issuecomment-1095357010
Hey thanks for the quick PR and discussions, sorry I am a bit late joining
here. @sodonnel definitely raised some valid points. Here is how I was thinking
this could work, let me know your thoughts.
- Each allocate block request checks that `existing open key size` + `the
block to be allocated` + `current quota usage` <= `max quota`. If this
condition fails, allocate block fails.
- Each key commit checks that `total key size` + `current quota usage` <=
`max quota`. If so, it updates the quota usage and commits the key.
I think this would address most of the already mentioned problems:
- Open keys, which are owned by the system, are not counted against users'
quotas.
- If a large key write exceeds the quota, it fails as soon as possible (when
the block pushing it over the quota is allocated), which might be the best we
can do in a streaming scenario.
- This also prevents large uncommitted key writes from vastly exceeding
the quota.
- Quota usage only reflects committed data.
- Open keys that cumulatively exceed the quota can coexist independently,
and only need to acknowledge each other once committed.
- For example:
1. 1GB quota remaining
2. Two 1GB open keys being written: k1 and k2.
- k1 and k2 both successfully allocate all blocks they need and
write them. Neither is committed so they have no knowledge of each other.
3. k1 is committed.
4. k2 either calls commit, or attempts to allocate another block.
This would cause it to exceed the quota that k1 just filled up, so k2's write
fails at this point.
The case @captainzmc mentioned where remaining quota is less than 3X block
size would still be a problem with this approach. I am not sure of a good
solution for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
