GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/5924
[SPARK-7311] Introduce internal Serializer API method for determining if
serializers support object relocation
This patch extends the `Serializer` interface with a new `@Private` API
which allows serializers to indicate whether they support relocation of
serialized objects in serializer stream output.
This relocatibilty property is described in more detail in
`Serializer.scala`, but in a nutshell a serializer supports relocation if
reordering the bytes of serialized objects in serialization stream output is
equivalent to having re-ordered those elements prior to serializing them. The
optimized shuffle path introduced in #4450 and #5868 both rely on serializers
having this property; this patch just centralizes the logic for determining
whether a serializer has this property. I also added tests and comments
clarifying when this works for KryoSerializer.
This change allows the optimizations in #4450 to be applied for shuffles
that use `SqlSerializer2`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-7311
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5924.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5924
----
commit b9624eefa111c012b013417a4c754358066e29ed
Author: Josh Rosen <[email protected]>
Date: 2015-05-01T21:43:56Z
Expand serializer API and use new function to help control when new
UnsafeShuffle path is used.
Conflicts:
core/src/main/scala/org/apache/spark/shuffle/unsafe/UnsafeShuffleManager.scala
commit 86d4dcdf0cb09b75d14ef201262ea771995ac7cb
Author: Josh Rosen <[email protected]>
Date: 2015-05-01T21:47:33Z
Flag that SparkSqlSerializer2 supports relocation
commit 450fa2182c5345b08f6fb5eea74f9f5d96822a48
Author: Josh Rosen <[email protected]>
Date: 2015-05-03T19:57:07Z
Back out accidental log4j.properties change
commit 0ba75e61ed32140f17e559cd5378ed3d3ba6ed38
Author: Josh Rosen <[email protected]>
Date: 2015-05-03T19:52:15Z
Add tests for serializer relocation property.
I verified that the Kryo tests will fail if we remove the auto-reset
check in KryoSerializer. I also checked that this test fails if we
mistakenly enable this flag for JavaSerializer. This demonstrates that
the test case is actually capable of detecting the types of bugs that it's
trying to prevent.
Of course, it's possible that certain bugs will only surface when
serializing
specific data types, so we'll still have to be cautious when overriding
`supportsRelocationOfSerializedObjects` for new serializers.
commit 2c1233a4b2754fe94e076bb0aac806474555730a
Author: Josh Rosen <[email protected]>
Date: 2015-05-03T20:08:35Z
Small refactoring of SerializerPropertiesSuite to enable test re-use:
This lays some groundwork for re-using this test logic for serializers
defined
in other subprojects (those projects can just declare a test-jar dependency
on Spark core).
commit 4aa61b2ff8a494881937104299103bb5fdce6f8b
Author: Josh Rosen <[email protected]>
Date: 2015-05-03T20:18:39Z
Add missing newline
commit 123b99286e3a5e6586818d302ec79abc4f0e958e
Author: Josh Rosen <[email protected]>
Date: 2015-05-05T21:30:28Z
Cleanup for submitting as standalone patch.
commit 0a7ebd7311c42707d2576cda9a7b28ce96d2973f
Author: Josh Rosen <[email protected]>
Date: 2015-05-05T21:35:59Z
Clarify reason why SqlSerializer2 supports this serializer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
