Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs
[SPARK-17714](https://issues.apache.org/jira/browse/SPARK-17714) has been
created for further investigation.
---
If your project is set up for it, you can reply to this email and
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Thanks @tgravescs yes. I have created a PR
[15286](https://github.com/apache/spark/pull/15286).
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14659
its there to make sure everyone is using the same classloader and to handle
if it they are chained. I'm not really familiar with all the scenarios of the
repl. I see the Suite itself is getting
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
@tgravescs @srowen Thanks. Using `Class.forName` which uses
`this.getClass().getClassLoader()` by default makes all the tests passed (both
sbt and maven). However there must be some reason we
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14659
Yeah its definitely caused by the call in Task to the
Utils.CallerContext.setCurrent(). It seems to be from it using the
ExecutorClassLoader to go try to check the class for the executor and I
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs @srowen
Give an intermediate update, If using `Class.forName `instead of
Utils.classForName`, Maven build and all of the tests will be passed.
```
def
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
@tgravescs @srowen Sorry for the failure. I am looking into it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14659
this is a new feature so generally don't put them into point releases which
are bug fixes.
@srowen thanks for pointing out, I will take alook.
---
If your project is set up for it, you
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14659
@Sherry302 @tgravescs oops it looks like this causes master Maven builds to
fail:
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs Should we also commit this PR to Branch-2? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Thanks a lot for the review. @tgravescs @cnauroth @steveloughran @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14659
thanks @Sherry302
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14659
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65934/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65934 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65934/consoleFull)**
for PR 14659 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65934 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65934/consoleFull)**
for PR 14659 at commit
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs Thanks a lot for the comments. I have updated the PR to
rename local vals and remove the `@since` in `Utils.scala`.
---
If your project is set up for it, you can reply to this
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14659
Couple minor things otherwise looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65799/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65799 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65799/consoleFull)**
for PR 14659 at commit
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs Thank you very much. Yes. I have updated the PR to make the
string of the caller context shorter.
---
If your project is set up for it, you can reply to this email and have your
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65799 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65799/consoleFull)**
for PR 14659 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65677/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65677 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65677/consoleFull)**
for PR 14659 at commit
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs Thank you so much for the review. I have updated the PR
based on your every comment.
The only one question left is this one (in `Task`) "are these params all
optional just
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65677 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65677/consoleFull)**
for PR 14659 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14659
also just to verify , did you test this with multiple tasks running in the
same executor? Just want to make sure the audit log showed each task properly
---
If your project is set up for it,
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14659
So correct me if I'm missing something but this is actually going to set
the caller context for everything (HDFS and YARN) correct? The caller context
is a static thing you set and its just
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs Could you please review this again? I have updated the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65445/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65445 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65445/consoleFull)**
for PR 14659 at commit
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65445 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65445/consoleFull)**
for PR 14659 at commit
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
There is "0 failures (±0)" in the Test Result page. All tests passed. I'll
re-trigger again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65427/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65427 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65427/consoleFull)**
for PR 14659 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65427 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65427/consoleFull)**
for PR 14659 at commit
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs Thank you very much for the review. I have updated the PR
based on your every comment, including adding a CallerContext class, updating
java doc, and made the caller context string
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65422 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65422/consoleFull)**
for PR 14659 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65422/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #65422 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65422/consoleFull)**
for PR 14659 at commit
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
@tgravescs Sure. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14659
I'll try to look at this later today or tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs Could you please review this PR? Thank you very much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64757/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #64757 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64757/consoleFull)**
for PR 14659 at commit
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
@steveloughran Thank you very much. I have updated the PR based on your
comments. Also, I have added an unit test.
---
If your project is set up for it, you can reply to this email and have
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
@srowen Thanks all the same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #64757 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64757/consoleFull)**
for PR 14659 at commit
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/14659
This context is just something passed over IPC to provide a general string
for the audit logs, the main actual access of it is in the HDFS audit log
```
HdfsAuditLogger
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14659
I'm not sure I know enough about the APIs and need for this to say I would
review and merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @srowen Could you please review this PR again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @srowen Could you please review this PR? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64380/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #64380 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64380/consoleFull)**
for PR 14659 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #64380 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64380/consoleFull)**
for PR 14659 at commit
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
The only failure is 'basic functionality', but it passed locally. I'll
re-trigger again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64372/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #64372 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64372/consoleFull)**
for PR 14659 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #64372 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64372/consoleFull)**
for PR 14659 at commit
Github user cnauroth commented on the issue:
https://github.com/apache/spark/pull/14659
+1 (non-binding) for the latest revision. I think this is an improvement,
speaking from the perspective of the HDFS audit log. Thank you, @Sherry302 .
:-)
---
If your project is set up for
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Thanks a lot for adding me as âcontributorâ in Hadoop :) @steveloughran
@cnauroth
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @steveloughran Thanks a lot for the comments.
In the audit log, if users set some configuration in spark-defaults.conf
like `spark.eventLog.dir hdfs://localhost:9000/spark-history`,
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @cnauroth Thank you very much for the review and suggestion. I have
removed the spaces in the value of the caller context, and prepended "Spark"
instead (refer to the commit
Github user cnauroth commented on the issue:
https://github.com/apache/spark/pull/14659
Actually a HADOOP JIRA... I filed
[HADOOP-13528](https://issues.apache.org/jira/browse/HADOOP-13528) for
follow-up.
---
If your project is set up for it, you can reply to this email and have
Github user cnauroth commented on the issue:
https://github.com/apache/spark/pull/14659
> Chris: maybe the CallerContext class could check for bad characters,
including spaces, newlines, "," and quotation marks .. the usual things to
break parsers.
Yes, good conversation to
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/14659
Chris: maybe the CallerContext class could check for bad characters,
including spaces, newlines, "," and quotation marks .. the usual things to
break parsers.
There's also
Github user cnauroth commented on the issue:
https://github.com/apache/spark/pull/14659
Hello @Sherry302 . Thank you for the patch.
From the HDFS perspective, we recommend against using spaces in the value
of the caller context. The HDFS audit log frequently gets parsed by
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/14659
having some problems adding you as a contributor; JIRA scale issues,
browser problems , I've asked others to try and do it. Start with the coding;
I'll sort out the contributor entry
---
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @steveloughran Thank you very much for the comments. I have created an
Hadoop jira [HADOOP-13527 ](https://issues.apache.org/jira/browse/HADOOP-13527)
and attached the patch, could you please
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/14659
That Caller context doesn't list Spark as one of the users in its
LimitedPrivate scope. Add a Hadoop patch there and I'll get it in. This avoids
arguments later when someone breaks the API,
Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @srowen . Thank you so much for the review. Sorry for the test
failure and late update. The failure reasons are that âjobIDâ were
none or there was no âspark.app.nameâ in
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63910/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #63910 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63910/consoleFull)**
for PR 14659 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14659
**[Test build #63910 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63910/consoleFull)**
for PR 14659 at commit
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14659
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14659
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
88 matches
Mail list logo
