Github user ksakellis commented on the pull request:
https://github.com/apache/spark/pull/3119#issuecomment-62447986
@vanzin regarding testing... yes, I did try what you recommended - I added
a note in the commit message. Snappy does some static initialization so if it
was
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3119#issuecomment-62464530
BTW with sort based shuffle, it might be ok to roll back the default
compression to lzf, since we no longer open a large number of concurrent
streams. It is now only one
Github user ksakellis commented on the pull request:
https://github.com/apache/spark/pull/3119#issuecomment-62485529
@pwendell good point. I'll just make this throw an exception when the you
try to create a SnappyCompressionCodec and snappy is not available. I think
that is better
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3119#issuecomment-62313527
Could this instead just throw an exception when Snappy is configured but
not supported? We typically try not to silently mutate configs in the
background in favor of
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3119#issuecomment-62348827
I agree with @pwendell - it'd be better to just fail fast here rather than
picking LZF based on the the availability of the library.
---
If your project is set up for it,
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3119#issuecomment-62226594
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user ksakellis opened a pull request:
https://github.com/apache/spark/pull/3119
[SPARK-4079] [CORE] Default to LZF if Snappy not available
By default, snappy is the compression codec used. If Snappy is not
available, Spark currently throws a stack trace. Now Spark falls back
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3119#issuecomment-61885458
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3119#discussion_r19907994
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -42,28 +43,52 @@ trait CompressionCodec {
def compressedOutputStream(s:
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3119#discussion_r19908142
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -42,28 +43,52 @@ trait CompressionCodec {
def compressedOutputStream(s:
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3119#discussion_r19908213
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -42,28 +43,52 @@ trait CompressionCodec {
def compressedOutputStream(s:
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3119#discussion_r19908270
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -42,28 +43,52 @@ trait CompressionCodec {
def compressedOutputStream(s:
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/3119#issuecomment-61886283
Just small nits. re: testing, can't you write a test where you set the
system property to make snappy fail as part of that test?
---
If your project is set up for it,
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3119#discussion_r19909185
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -42,28 +43,52 @@ trait CompressionCodec {
def compressedOutputStream(s:
14 matches
Mail list logo
