s0nskar commented on code in PR #2727:
URL: https://github.com/apache/celeborn/pull/2727#discussion_r1795446725
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/quota/QuotaManager.scala:
##########
@@ -39,10 +39,23 @@ class QuotaManager(celebornConf: CelebornConf,
configService: ConfigService) ext
}
}
- def checkQuotaSpaceAvailable(
+ private def getQuotaWithWatermark(
userIdentifier: UserIdentifier,
- resourceResumption: ResourceConsumption): (Boolean, String) = {
+ quotaWatermark: Double): Quota = {
val quota = getQuota(userIdentifier)
+ Quota(
+ (quota.diskBytesWritten * quotaWatermark).toLong,
Review Comment:
@waitinfuture Currently Celeborn consider quota as soft limits and just stop
allowing new jobs on use Celeborn if the quota is breached while the existing
jobs keeps on running while breaching the quota. IMO there are two shortcoming
in current quota implementation –
- It allows new jobs to onboard even if quota is just about to be breached.
Ideally it should stop onboarding jobs after a certain threshold to not go
overboard with the quota limits (This PR addresses that)
- Killing jobs if quota for an user is breached (I was planning to implement
this but noticed that @leixm raised a PR to handle this today -
https://github.com/apache/celeborn/pull/2801)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
