squito commented on issue #23510: [SPARK-26590][CORE] make fetch-block-to-disk backward compatible URL: https://github.com/apache/spark/pull/23510#issuecomment-454233253 Thanks for doing this @cloud-fan -- I have one high level question before looking more carefully at the code. When you say: > new client <-> old server: The ChunkFetchRequest message contains the fetchAsStream flag, but the old server doesn't know about it and stops reading the message right before the fetchAsStream part. Then the old server returns normal chunk fetch response, and new client accept it. so the client accepts it, but will the client stream to disk still, or will it fallback to still fetching to memory? It seems it should be possible to stream to disk, as the server is really sending virtually the same bytes either way (just a different header, more or less). That said, I think the actually plumbing is rather complicated to update (this is something @attilapiros looked at earlier). I can see about testing this with old versions of the shuffle server -- and especially if you think this makes stream-to-disk work even with the old shuffle server, than we can also test out shuffling large blocks too.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
