Github user squito commented on the issue:
https://github.com/apache/spark/pull/23058
can we also make the same change to `TaskResultGetter`? We could avoid the
same inefficiency for large task results that also get fetched to disk. (That
would actually get us closer to removing the 2gb block limit on task results,
but to fully remove it, we'd need to fix the task serialization in the
executor, which is a little trickier.)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
