cloud-fan commented on issue #23510: [SPARK-26590][CORE] make fetch-block-to-disk backward compatible URL: https://github.com/apache/spark/pull/23510#issuecomment-454274318 > so the client accepts it, but will the client stream to disk still, or will it fallback to still fetching to memory? When the old server returns a normal chunk fetch response, the new client will process it just like the client has sent a normal chunk fetch request, and put the data in memory. > It seems it should be possible to stream to disk, as the server is really sending virtually the same bytes either way (just a different header, more or less) AFAIK the streaming response is very different from chunk fetch response. The chunk fetch response will send the data in one message, so the client already puts the data in memory when it receives the message. The stream response is a notice of the following small messages, and the real data is sent via many small messages, so that client has a chance to put it in disk incrementally.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
