Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5924#discussion_r29718002
--- Diff: core/src/main/scala/org/apache/spark/serializer/Serializer.scala
---
@@ -63,6 +63,39 @@ abstract class Serializer {
/** Creates a new [[SerializerInstance]]. */
def newInstance(): SerializerInstance
+
+ /**
+ * :: Private ::
+ * Returns true if this serializer supports relocation of its serialized
objects and false
+ * otherwise. This should return true if and only if reordering the
bytes of serialized objects
+ * in serialization stream output is equivalent to having re-ordered
those elements prior to
+ * serializing them. More specifically, the following should hold if a
serializer supports
+ * relocation:
+ *
+ * {{{
+ * serOut.open()
+ * position = 0
+ * serOut.write(obj1)
+ * serOut.flush()
+ * position = # of bytes writen to stream so far
+ * obj1Bytes = output[0:position-1]
+ * serOut.write(obj2)
+ * serOut.flush()
+ * position2 = # of bytes written to stream so far
+ * obj2Bytes = output[position:position2-1]
+ * serIn.open([obj2bytes] concatenate [obj1bytes]) should return (obj2,
obj1)
+ * }}}
+ *
+ * In general, this property should hold for serializers that are
stateless and that do not
+ * write special metadata at the beginning or end of the serialization
stream.
+ *
+ * This API is private to Spark; this method should not be overridden in
third-party subclasses
+ * or called in user code and is subject to removal in future Spark
releases.
+ *
+ * See SPARK-7311 for more details.
+ */
+ @Private
--- End diff --
I'm not sure that we want to commit to this as a stable public API, which
is why I've chosen to mark this as private and leave comments warning users
that this API is private and subject to change. If someone can think of a
better way to restrict use / implementation of this method, I'd be happy to
incorporate that change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
