jerqi commented on issue #825: URL: https://github.com/apache/incubator-uniffle/issues/825#issuecomment-1519077664
> > > > A critical problem is that how to inform the servers that we will assign a new server. > > > > > > > > > `inform the servers` what servers do you mean? > > > > > > I mean that we should how to inform the executors that we have updated shuffle servers. Do we only know the shuffle server allocation at the beginning of the stage or the beginning of the stage re-attempt? Could we add shuffle server for some partitions when the stage is running? If we can, how to inform the executors that we have assign new shuffle servers? > > Each task will retrieve new shuffle assignment from driver with local cache on the executor side. So once a new server is added, new tasks will get new shuffle server assignments. Old tasks wouldn't get a chance to update shuffle server assignments. It would either failed/abort due to bad shuffle server or write shuffle data successfully. Will it cause too much pressure if we don't executor's level shuffle server info cache. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
