[
https://issues.apache.org/jira/browse/SPARK-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated SPARK-6986:
----------------------------
Component/s: (was: Spark Core)
Description: SPARK-4550 has exposed the interfaces. We can safely
enable Serializer2 to support sort merge. (was: Our existing Java and Kryo
serializer are both general-purpose serialize. They treat every object
individually and encode the type of an object to underlying stream. For Spark,
it is common that we serialize a collection with records having the same types
(for example, records of a DataFrame). For these cases, we do not need to write
out types of records and we can take advantage the type information to build
specialized serializer. To do so, seems we need to extend the interface of
SerializationStream/DeserializationStream, so a
SerializationStream/DeserializationStream can have more information about
objects passed in (for example, if an object is key/value pair, a key, or a
value). )
Target Version/s: 1.4.0
Summary: Makes SparkSqlSerializer2 support sort-based shuffle with
sort merge (was: Make SerializationStream/DeserializationStream understand
key/value semantic)
> Makes SparkSqlSerializer2 support sort-based shuffle with sort merge
> --------------------------------------------------------------------
>
> Key: SPARK-6986
> URL: https://issues.apache.org/jira/browse/SPARK-6986
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Reporter: Yin Huai
> Assignee: Yin Huai
>
> SPARK-4550 has exposed the interfaces. We can safely enable Serializer2 to
> support sort merge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
