xianjingfeng commented on issue #1247: URL: https://github.com/apache/incubator-uniffle/issues/1247#issuecomment-1783027164
> > shuffleMetaMap is useless, right? > > I don't think ShuffleMetaMap is useless. I believe its intention is to track every shuffleId's shuffle size and last read time. It should be updated in a thread safe manner. Without this metadata, it would be hard to track how many data each shuffle has been wrote, data management operations such as: quota limit per app/shuffle, back pressure or traffic throttle would be impossible. > > > And size is just for judging whether the local disk can be written, right? > > Currently, there's a capacity configuration for shuffle server. It is possible that the shuffle server's disk is shared by other services, although not recommended. So we cannot simply check if the disk has free space to determine whether it's writable. > > If we are going to remove LocalStorageMeta, I think we may have to add it back in the future to support advanced features. It's okay to keep it. Let's focus on how do we solve this problem and how can we avoid this problem from happening again in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
