[
https://issues.apache.org/jira/browse/SPARK-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14985011#comment-14985011
]
Sean Owen commented on SPARK-11449:
-----------------------------------
FWIW if the purpose of this class was to be a serializable factory for a
stream, then I'd actually not expect it to manage the stream at all: it makes
the stream for you and then you have to deal with it. But it does, mostly to
provide {{toArray}}.
One fix is to remove close() and tell callers to manage the stream directly.
And then also toArray(). It's only used in tests and PythonRDD. A smaller
change would be to keep toArray() but have it open the stream, read it, and
close it. How about that? It's {{Experimental}} so we can change it. If anyone
really worries about keeping close() it can stay a no-op.
> Improve documentation on close behavior of PortableDataStream
> -------------------------------------------------------------
>
> Key: SPARK-11449
> URL: https://issues.apache.org/jira/browse/SPARK-11449
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Herman van Hovell
> Priority: Minor
>
> {{PortableDataStream}}'s close behavior caught me by surprise the other day.
> I assumed incorrectly that closing the inputstream it provides would also
> close the {{PortableDataStream}}. This leads to quite a confusing situation
> in when you try to reuse the {{PortableDataStream}}: the state of the
> {{PortableDataStream}} indicates that it is open, whereas the underlying
> inputstream is actually closed.
> I'd like either to improve the documentation, or add an {{InputStream}}
> wrapper that closes the {{PortableDataStream}} when you close the
> {{InputStream}}. Any thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
