squito commented on issue #23510: [SPARK-26590][CORE] make fetch-block-to-disk backward compatible URL: https://github.com/apache/spark/pull/23510#issuecomment-455237408 I kinda agree with Tom after thinking about this a bit more, that maybe its not worth adding . The worry I have about trying to make it backwards compatible is that we might not be testing it regularly and it gets inadvertently broken later on. I guess I'm fine either way. > AFAIK the streaming response is very different from chunk fetch response. not really that different -- there is a small header (which is different in each case), followed by the bulk of the response which is the actual data of the shuffle block (the same in both cases). Now, the *client* does very different things with that response based on the first header -- the data is always a stream as some level, but the client may decide to buffer it all into memory or not.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
