Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5924#discussion_r29718856
--- Diff: core/src/main/scala/org/apache/spark/serializer/Serializer.scala
---
@@ -63,6 +63,39 @@ abstract class Serializer {
/** Creates a new [[SerializerInstance]]. */
def newInstance(): SerializerInstance
+
+ /**
+ * :: Private ::
+ * Returns true if this serializer supports relocation of its serialized
objects and false
+ * otherwise. This should return true if and only if reordering the
bytes of serialized objects
+ * in serialization stream output is equivalent to having re-ordered
those elements prior to
+ * serializing them. More specifically, the following should hold if a
serializer supports
+ * relocation:
+ *
+ * {{{
+ * serOut.open()
+ * position = 0
+ * serOut.write(obj1)
+ * serOut.flush()
+ * position = # of bytes writen to stream so far
+ * obj1Bytes = output[0:position-1]
+ * serOut.write(obj2)
+ * serOut.flush()
+ * position2 = # of bytes written to stream so far
+ * obj2Bytes = output[position:position2-1]
+ * serIn.open([obj2bytes] concatenate [obj1bytes]) should return (obj2,
obj1)
+ * }}}
+ *
+ * In general, this property should hold for serializers that are
stateless and that do not
+ * write special metadata at the beginning or end of the serialization
stream.
+ *
+ * This API is private to Spark; this method should not be overridden in
third-party subclasses
+ * or called in user code and is subject to removal in future Spark
releases.
+ *
+ * See SPARK-7311 for more details.
+ */
+ @Private
+ private[spark] def supportsRelocationOfSerializedObjects: Boolean = false
--- End diff --
I'm not set on this name, by the way; happy to change if someone thinks of
a less verbose name that's not misleading.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
