GitHub user a-roberts opened a pull request:
https://github.com/apache/spark/pull/8995
[SPARK-10949] Update Snappy version to 1.1.2
Snappy now supports concatenation of serialized streams, this patch
contains a version number change and the "does not support" test is now a
"supports" test.
Snappy 1.1.2 changelog mentions:
snappy-java-1.1.2 (22 September 2015)
This is a backward compatible release for 1.1.x.
Add AIX (32-bit) support.
There is no upgrade for the native libraries of the other platforms.
A major change since 1.1.1 is a support for reading concatenated results of
SnappyOutputStream(s)
snappy-java-1.1.2-RC2 (18 May 2015)
Fix #107: SnappyOutputStream.close() is not idempotent
snappy-java-1.1.2-RC1 (13 May 2015)
SnappyInputStream now supports reading concatenated compressed results of
SnappyOutputStream
There has been no compressed format change since 1.0.5.x. So You can read
the compressed results interchangeablly between these versions.
Fixes a problem when java.io.tmpdir does not exist.
From https://github.com/xerial/snappy-java/blob/develop/Milestone.md and up
to date at the time of this pull request
Also note https://github.com/xerial/snappy-java/issues/103
"@xerial not sure how feasible or likely it is for this to happen, but it'd
help tremendously Spark's performance because we are experimenting with a new
shuffle path that uses channel.transferTo to avoid user space copying. However,
for that to work, we'd need the underlying format to support concatenation. As
far we know, LZF has this property, and Snappy might also have it (but
snappy-java implementation doesn't support it)."
Would be useful to have this in both the 1.5 and 1.6 branches
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/a-roberts/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8995.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8995
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
