Github user massie commented on the pull request:
https://github.com/apache/spark/pull/7268#issuecomment-120094947
@kayousterhout I like that this PR deletes more lines than it adds; the
Spark shuffle code needs that. I just looked over the code and it shouldn't
derail my work in any way. I'll just rebase my work on top of this once it's
merged.
I'm not sure if this would be in scope for this PR, but it would be ideal
if the `ShuffleBlockFetcherIterator` was able to allow consumers to: a)
differentiate between local and remote blocks and b) provide `File`-level
access to local blocks. For the Parquet shuffle manager, I have to do a
needless `Files.copy()` using the `InputStream` in order to get `File` access
to local blocks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
