advancedxy commented on PR #749: URL: https://github.com/apache/incubator-uniffle/pull/749#issuecomment-1486540366
> > Would you mind to post a design doc or improvement proposal here to illustrate your idea more clear, especially about the compatibility issues here. > > I think the reason of this changed has been exploded in #731 description. WDYT? The detailed design is to be compatible with different apps. From #731, I get the need to expand task attempt id. But rather than to increase the length of task_attempt_id or make it configurable, there might be other ways to solve this issue. For example, you can roll over the the task_attempt_id to zero and start again once it exceeds the capacity as long as the number of one shuffle map tasks doesn't exceeds the maximum number of the TASK_ATTEMPT_ID_MAX_LENGTH. The reason this is possible is that the block ids are bound to shuffle ids, it shouldn't interfere with each other. Also, I'm more concerned about the compatibility issues. To make the length of fields in block ids adaptive itself is appealing, it's just that we really have to think it through and make sure it wouldn't break things. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
