zuston commented on PR #1652: URL: https://github.com/apache/incubator-uniffle/pull/1652#issuecomment-2081949303
> 1. Is current design compatible for balance feature? I think yes. But in current stage, the load balance will not be inclued in this PR, because let make someone misunderstand this feature. And I want to say that the name of faulty server is also not proper for the writing failure due to the NO_BUFFER. So I rename it to `receivingFailureServer` , if you have better name, please tell me. > 2. The possible risks are: > 1) The memory cost, will it cause we use too much memory to store the data stucture I think it will not bring too much burden, and the data structure is not the biggest cost for the memory. This is a overthinking point. > 2) If a task fails many times, will it produce a bad influence. Firstly, I want to know the bad influence you mentioned? Could you help list these. And I hope we could discuss the specific problems or guess. Anyway, I will give my thought or guess for your consideration. The process of writing failure will do the following things: 1. Ask the shuffleManager to request more replacement server from coordinator, this will bring extra requests for coordinator. But this is no problems, the assignment is a lightweight operation. 2. And then register to the servers for those replacement servers. This will introduce extra server burder, but this is also not a problems. Because in the original logic, the assignment servers number may reach 100+. 3. The multiple writing failure will reassign multiple times. This will make the task run slower than the normal task, due to multiple retry. But I think this is acceptable that compared with the task retry failure or stage retry. And to avoid too much multiple reassignment for one partition, I have introduced the extra limit to avoid too much reassignment servers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
