zhengchenyu commented on issue #1249: URL: https://github.com/apache/incubator-uniffle/issues/1249#issuecomment-1772566879
<img width="601" alt="image" src="https://github.com/apache/incubator-uniffle/assets/10381583/caa17b0c-a2b4-4720-8171-9542e30b1fba";> Upper picture show the data format change. In brief, we have three data format: Format B: the raw format. It is a sequence of keylen, valuelen, key, value, ending with two -1. Format A: Format b is compressed according to uniffle encoding. Format C: Add checksum to format b (IFile format) and compress it according to hadoop encoding. > The yellow is component of Map. Blue is shuffle server. Green is component of Reduce. What's the key of the promblem which is described by 8683254720974c7254f9c5c2fd902743ae0ab273? We write the format B to OnDiskMapOutput, so MergeManagerImpl can not parse it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
