Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/18805
Merging to master. Thanks for seeing this through!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83282/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #83282 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83282/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #83282 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83282/testReport)**
for PR 18805 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83204/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #83204 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83204/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #83204 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83204/testReport)**
for PR 18805 at commit
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
ping.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82911/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #82911 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82911/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #82911 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82911/testReport)**
for PR 18805 at commit
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
Created https://github.com/luben/zstd-jni/issues/47.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
```
$ ldd linux/amd64/libzstd-jni.so
ldd: warning: you do not have execution permission for
`linux/amd64/libzstd-jni.so'
linux/amd64/libzstd-jni.so: /lib64/libc.so.6: version
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
Good news is that I can reproduce it on the amplab machine, so I'll try to
play around with the zstd-jni code a bit.
---
-
To
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
Yeah but that would also cause it to fail locally if it were the cause, and
it passes for me. I can't really figure out from the rest of the logs if
something obvious is wrong, so I guess the best
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/18805
This seems to be caused by a issue in the `zstd-jni` library. It probably
uses the wrong `ClassLoader` to load the native library, and as a result it
cannot find the library & load it.
---
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
Turns out that's caused by SparkContext failing to clean up after itself
when the `UnsatisfiedLinkError` happens, so those errors are red herrings...
---
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
I haven't been able to reproduce the issue locally, but looking at the
jenkins logs I see a bunch of exceptions like these:
```
17/10/13 06:53:26.609 dispatcher-event-loop-15 ERROR
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
(I'll file a bug and send a PR for it separately, btw.)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82729/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #82729 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82729/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #82729 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82729/testReport)**
for PR 18805 at commit
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/18805
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
Same test failed, so looks like there's a real non-infra-related issue...
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82644/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #82644 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82644/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #82644 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82644/testReport)**
for PR 18805 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
The
[code](https://github.com/luben/zstd-jni/blob/master/src/main/java/com/github/luben/zstd/util/Native.java)
overwrites the original exception message that might shed some light on what's
going
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
Yes, the binary distribution is included in the zstd-jni jar file.
---
-
To unsubscribe, e-mail:
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18805
Does the package include a binary distribution for Linux?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
@vanzin - May be the test time outs are related to one test failure
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82608/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #82608 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82608/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #82608 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82608/testReport)**
for PR 18805 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #3938 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3938/testReport)**
for PR 18805 at commit
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18805
I didn't look into it. I can restart the test here though
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #3938 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3938/testReport)**
for PR 18805 at commit
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
@srowen - Did you get a chance to look into this ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
Thanks for looking into this @srowen. Its weird, I dont understand that
either. Also, I am not able to reproduce this issue on my laptop.
---
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18805
Hm, this might be a real error. It seems to be hanging at:
```
[info] ExternalAppendOnlyMapSuite:
...
[info] - simple cogroup (56 milliseconds)
[info] - spilling (3 seconds,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #3930 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3930/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #3930 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3930/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #3927 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3927/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #3927 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3927/testReport)**
for PR 18805 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81967/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #81967 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81967/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #81967 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81967/testReport)**
for PR 18805 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81911/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #81911 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81911/testReport)**
for PR 18805 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #81911 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81911/testReport)**
for PR 18805 at commit
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
Updated with zstd-jni versin 1.3.1-1 and also updated the license to
include zstd-jni license. @srowen - How does that look from licensing
prospective?
---
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18805
It looks like zstd-jni has now been updated to pull 1.3.1
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18805
Yes, licenses have to be updated, that's the one type of thing that's not
optional.
But Marcelo is right that the library actually doesn't yet include the
newer dependency with the right
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
>> I think this will be OK but we do need to add these two licenses to
licenses/ (see the convention there) and also add a line for each in LICENSE
here.
@srowen - Does that need to be
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
Just to make sure, this is ok if the JNI wrapper does not bring the zstd
native library with it; otherwise it also needs to be updated since it still
has the old BSD+PATENTS licence
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18805
It's dual-licensed (see bottom of
https://github.com/facebook/zstd/blob/v1.3.1/README.md ) and one is 3-clause
BSD. This is fine as it's a "Category A" license:
Github user markhamstra commented on the issue:
https://github.com/apache/spark/pull/18805
In addition to LICENSE, there is also COPYING in the v1.3.1 release:
https://github.com/facebook/zstd/blob/v1.3.1/LICENSE
https://github.com/facebook/zstd/blob/v1.3.1/COPYING
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/18805
Thank you for confirming, @vanzin ! I see.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
The patents grant (`PATENTS` file) seems to have been dropped from the repo:
https://github.com/facebook/zstd
That makes it plain BSD, which is fine for Spark.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/18805
Hi, @sitalkedia .
facebook/zstd#775 was `Consider re-licensing to Apache License v2`.
But, I found that the new one is not Apache License v2. It's about adding
GPLv2. I'm
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
Relevant PR - https://github.com/facebook/zstd/pull/801/files
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
We have released new zstd version
(https://github.com/facebook/zstd/releases) with modified BSD + GPLv2 license.
@rxin, @srowen, @markhamstra - Can you confirm this looks fine from
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
https://github.com/facebook/zstd/issues/775
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
Just an update on this - I am in talk with our internal team to relicense
zstd library. This might take some time though. I will keep you updated.
@discipleforteen - Unfortunately, we do
Github user discipleforteen commented on the issue:
https://github.com/apache/spark/pull/18805
how about TPCDS ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
@rxin - Sure, let me talk to folks internally to see if it is possible to
relicense. Otherwise, we might have to upgrade to hadoop 2.9.0, which will come
with its own zstd implementation.
---
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18805
@sitalkedia anyway you can talk to the FB team that does that one and
relicense, similar to RocksDB?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
Yes, the native library has the BSD+patents license:
https://github.com/luben/zstd-jni/tree/master/src/main/native
---
If your project is set up for it, you can reply to this email and have your
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18805
oh it might actually fall into that as it has the patents:
https://github.com/facebook/zstd/blob/dev/PATENTS
---
If your project is set up for it, you can reply to this email and have your
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18805
https://www.apache.org/legal/resolved.html
Seems to list bsd-2 and bsd-3 as ok.
https://github.com/facebook/zstd/blob/dev/LICENSE
not sure how the Facebook BSD+Patents license was
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18805
> Let me know if we are fine using https://github.com/luben/zstd-jni
library which uses BSD license.
Isn't the license of the native library (not the JNI wrapper) the same or
very similar
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
>> How big is the dependency that's getting pulled in?
zstd-jni library actually is a very thin library and is not pulling any
dependency of its own, so I would not be worried about
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18805
Our compression codec is actually completely decoupled from Hadoops, but
dependency management (and licensing) can be annoying to deal with.
---
If your project is set up for it, you can reply to
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18805
Got it, thanks for the reminder. I think the question is mostly about
license and dependency weight then. I think we'd want to use whatever Hadoop
provides.
---
If your project is set up for it,
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18805
> Why does this need to be in Spark?
@srowen you already asked that question and it has been answered on the
jira as well as the old pr. A user cannot add zstd compression to the internal
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18805
Why does this need to be in Spark? and what are the licensing terms of the
native code underneath (just suspicious because it's often GPL)? can a user not
just add this with their app?
I
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80148/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #80148 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80148/testReport)**
for PR 18805 at commit
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18805
How big is the dependency that's getting pulled in? If we are adding more
compression codecs maybe we should retire some old ones, or move them into a
separate package so downstream apps can
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #80148 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80148/testReport)**
for PR 18805 at commit
Github user tejasapatil commented on the issue:
https://github.com/apache/spark/pull/18805
re build failure: you can repro that locally by running
"./dev/test-dependencies.sh". Its failing due to introducing a new dep... you
need to add it to `dev/deps/spark-deps-hadoop-XXX`
---
If
Github user tejasapatil commented on the issue:
https://github.com/apache/spark/pull/18805
In `Benchmark` section the values for `Lz4` are all zeros which feels
confusing while reading.. first thing I thought is they were absolute values
but they are supposed to be relative
---
If
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
Any idea what is the build failure about?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #80144 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80144/testReport)**
for PR 18805 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80144/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #80144 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80144/testReport)**
for PR 18805 at commit
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
jenkins retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/18805
Please note that few minor improvements I have made when comapring to old
PR - #17303
1. Use zstd compression level 1 instead of 3, which is significantly faster.
2. Wrap the zstd
1 - 100 of 117 matches
Mail list logo
