Github user mallman commented on the issue:
https://github.com/apache/spark/pull/17295
> LGTM, cc @mallman to check the unmap part
LGTM, too. Sorry for the late reply... I've been away the past two weeks.
---
If your project is set up for it, you can reply to this email and
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17295
thanks, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75323/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #75323 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75323/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #75323 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75323/testReport)**
for PR 17295 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75267/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #75267 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75267/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #75267 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75267/testReport)**
for PR 17295 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17295
LGTM, cc @mallman to check the `unmap` part
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75226/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #75226 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75226/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #75226 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75226/testReport)**
for PR 17295 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17295
I removed `StorageUtils.unmap()` in my last commit (see commit message for
details). That makes the confusion go away.
The replication tests fail from time to time but they seem to be flaky
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75120/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #75120 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75120/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #75120 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75120/testReport)**
for PR 17295 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74997/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74997 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74997/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74997 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74997/testReport)**
for PR 17295 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74991/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74991 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74991/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74991 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74991/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74989 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74989/testReport)**
for PR 17295 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74989/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74989 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74989/testReport)**
for PR 17295 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74911/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74911 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74911/testReport)**
for PR 17295 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74905/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74905 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74905/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74911 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74911/testReport)**
for PR 17295 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17295
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74906/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74906 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74906/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74906 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74906/testReport)**
for PR 17295 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74905 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74905/testReport)**
for PR 17295 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17295
> shall we also transfer shuffle blocks after decryption?
No. That's explained in the PR description.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17295
makes sense. one more question, ideally, shall we also transfer shuffle
blocks after decryption?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17295
> What's the actual difference? previously we transfer encrypted data?
Yes. The previous version of the code would transfer the encrypted file
over to the receiver, and the encrypted data
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17295
> The penalty comes when transferring that encrypted data from disk. If the
data ends up in memory again, it is as efficient as before; but if the
evicted block needs to be transferred
Github user sameeragarwal commented on the issue:
https://github.com/apache/spark/pull/17295
cc @cloud-fan @ueshin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17295
> Is there any case where it is transfered in encrypted form in supported
cases ?
No, with these changes, only shuffle data is transferred in encrypted form.
---
If your project is set up
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17295
> Not really sure what you mean here. But transferring encrypted data
without RPC encryption is not really secure, since the encryption key is
transferred to executors using an RPC. There's even a
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17295
> Just to be clear, I would prefer if we consistently did things - either
encrypt all blocks while transferring (irrespective of sasl being enabled or
not); or depend only on sasl for channel
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17295
> This can be solved by tagging the block data with a prefix byte
Sure, it could be solved in different ways. I just happened to prefer the
one in this patch, since I think it's less
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74555/
Test PASSed.
---
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17295
Just to be clear, I would prefer if we consistently did things - either
encrypt all blocks while transferring (irrespective of sasl being enabled or
not); or depend only on sasl for channel
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17295
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74555 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74555/testReport)**
for PR 17295 at commit
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17295
> First, keep in mind that there's no metadata that tells the receiver
whether a block is encrypted or not. This means that methods like
BlockManager.get, which can read block data from either
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17295
> Isn't it not simpler to transmit block contents in encrypted format
without decryption?
First, keep in mind that there's no metadata that tells the receiver
whether a block is encrypted
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17295
I have not looked at the implementation in detail, but can you comment on
why the change w.r.t plain text block data to remote executor ? Isn't it not
simpler to transmit block contents in
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17295
**[Test build #74555 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74555/testReport)**
for PR 17295 at commit
62 matches
Mail list logo
