FMX commented on PR #3523: URL: https://github.com/apache/celeborn/pull/3523#issuecomment-3485769664
@AmandeepSingh285 Thanks for your enthusiasm about this PR, but this pr's functionality can be replaced by tuning the `diskGroupGradient`. I added some calculations down here to clarify that you just need to tweak your configuration. The algorithm is to allocator more workload to faster disk groups, not the worker groups, while your modification changed the original purpose. For example, we have 9 workers; each worker has only one storage directory. <!--StartFragment--><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta><byte-sheet-html-origin data-id="1762259347864" data-version="4" data-is-embed="false" data-grid-line-hidden="false" data-importRangeRawData-spreadSource="https://bytedance.larkoffice.com/wiki/W5E1wBAj5i9wobkmS9jcLWwLnIh"; data-importRangeRawData-range="'Sheet1'!A1:H9"> | 1 | 1.1 | 1.21 | 1.331 | 1.4641 | 6.1051 | -- | -- | -- | -- | -- | -- | -- | -- gradient | 0.1 | | | | | | total task count group allocation ratio | 0.163797481 | 0.180177229 | 0.198194952 | 0.218014447 | 0.239815892 | | 5000 group allocation result | 818.987404 | 900.8861444 | 990.9747588 | 1090.072235 | 1199.079458 | | | | | | | | | | 1 | 1.3 | 1.69 | 2.197 | 2.8561 | 9.0431 | gradient | 0.3 | | | | | | total task count group allocation ratio | 0.110581548 | 0.143756013 | 0.186882817 | 0.242947662 | 0.31583196 | | 5000 group allocation result | 552.9077418 | 718.7800644 | 934.4140837 | 1214.738309 | 1579.159801 | | </byte-sheet-html-origin><!--EndFragment--> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
