Github user massie commented on the pull request:
https://github.com/apache/spark/pull/7265#issuecomment-119349112
Map-side aggregation is supported in this pull request. If you look at the
`ParquetShuffleManager.parquetShuffleCanBeUsed()` method, you'll see that it
generates an Avro schema which take into consideration which class of objects
will be spilled. The `ParquetShuffleWriter.write()` method, aggregates and then
writes using this schema (the `ParquetShuffleReader.read()` just uses the
schema stored in Parquet by the writer).
The `ExternalSorter` currently uses the Spark `Serializer` interface to
read/write objects during the spill. If you like, I can easily have the Parquet
shuffle reader follow that same pattern for now since it will work, even though
it's not ideal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
