oOKevinOo commented on PR #2081: URL: https://github.com/apache/incubator-celeborn/pull/2081#issuecomment-1803112148
> > > Hi, I've created a Jira ticket for you, https://issues.apache.org/jira/browse/CELEBORN-1118. > > > > > > Is this PR a subtask of [Support memory file storage](https://www.mail-archive.com/[email protected]/msg00613.html)? > > Not really. We discussed with @oOKevinOo offline, he implemented this PR without aware of https://www.mail-archive.com/[email protected]/msg00613.html . I think we can merge this first, since this PR has a good result for small shuffle files, @oOKevinOo could you share some of your test? The memory storage will take longer time before GA. We are currently facing serious shuffle read performance issues, but the disk performance is far from expectations. Through service investigation, we suspect that small files have a significant impact on disk performance, so we raised this PR. This is our indicator data based on the pressure measurement of this PR: > > > Hi, I've created a Jira ticket for you, https://issues.apache.org/jira/browse/CELEBORN-1118. > > > > > > Is this PR a subtask of [Support memory file storage](https://www.mail-archive.com/[email protected]/msg00613.html)? > > Not really. We discussed with @oOKevinOo offline, he implemented this PR without aware of https://www.mail-archive.com/[email protected]/msg00613.html . I think we can merge this first, since this PR has a good result for small shuffle files, @oOKevinOo could you share some of your test? The memory storage will take longer time before GA. > > > Hi, I've created a Jira ticket for you, https://issues.apache.org/jira/browse/CELEBORN-1118. > > > > > > Is this PR a subtask of [Support memory file storage](https://www.mail-archive.com/[email protected]/msg00613.html)? > > Not really. We discussed with @oOKevinOo offline, he implemented this PR without aware of https://www.mail-archive.com/[email protected]/msg00613.html . I think we can merge this first, since this PR has a good result for small shuffle files, @oOKevinOo could you share some of your test? The memory storage will take longer time before GA. i tick into test discription -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
