[
https://issues.apache.org/jira/browse/SPARK-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987743#comment-14987743
]
Apache Spark commented on SPARK-10949:
--------------------------------------
User 'JoshRosen' has created a pull request for this issue:
https://github.com/apache/spark/pull/9439
> Upgrade Snappy Java to 1.1.2
> ----------------------------
>
> Key: SPARK-10949
> URL: https://issues.apache.org/jira/browse/SPARK-10949
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.5.0, 1.5.1
> Reporter: Adam Roberts
> Priority: Minor
>
> Snappy now supports concatenation of serialized streams, this patch contains
> a version number change and the "does not support" test is now a "supports"
> test.
> Note: I do have the pull request for this already created, tested to be OK on
> Intel 64 bit Linux, IBM Power 8 LE and Linux on IBM Z Systems (all with IBM
> Java 8). Also note that I was required to delete my m2 cache in order to
> resolve incompatible class exceptions when building.
> Snappy 1.1.2 changelog mentions:
> snappy-java-1.1.2 (22 September 2015)
> This is a backward compatible release for 1.1.x.
> Add AIX (32-bit) support.
> There is no upgrade for the native libraries of the other platforms.
> A major change since 1.1.1 is a support for reading concatenated results of
> SnappyOutputStream(s)
> snappy-java-1.1.2-RC2 (18 May 2015)
> Fix #107: SnappyOutputStream.close() is not idempotent
> snappy-java-1.1.2-RC1 (13 May 2015)
> SnappyInputStream now supports reading concatenated compressed results of
> SnappyOutputStream
> There has been no compressed format change since 1.0.5.x. So You can read the
> compressed results interchangeablly between these versions.
> Fixes a problem when java.io.tmpdir does not exist.
> From https://github.com/xerial/snappy-java/blob/develop/Milestone.md and up
> to date at the time of this pull request
> Also note https://github.com/xerial/snappy-java/issues/103
> "@xerial not sure how feasible or likely it is for this to happen, but it'd
> help tremendously Spark's performance because we are experimenting with a new
> shuffle path that uses channel.transferTo to avoid user space copying.
> However, for that to work, we'd need the underlying format to support
> concatenation. As far we know, LZF has this property, and Snappy might also
> have it (but snappy-java implementation doesn't support it)."
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
