[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-28 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Hi, @tgravescs [SPARK-17714](https://issues.apache.org/jira/browse/SPARK-17714) has been created for further investigation. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-28 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Thanks @tgravescs yes. I have created a PR [15286](https://github.com/apache/spark/pull/15286). --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-28 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14659 its there to make sure everyone is using the same classloader and to handle if it they are chained. I'm not really familiar with all the scenarios of the repl. I see the Suite itself is getting

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-28 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 @tgravescs @srowen Thanks. Using `Class.forName` which uses `this.getClass().getClassLoader()` by default makes all the tests passed (both sbt and maven). However there must be some reason we

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-28 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14659 Yeah its definitely caused by the call in Task to the Utils.CallerContext.setCurrent(). It seems to be from it using the ExecutorClassLoader to go try to check the class for the executor and I

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-28 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Hi, @tgravescs @srowen Give an intermediate update, If using `Class.forName `instead of Utils.classForName`, Maven build and all of the tests will be passed. ``` def

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-28 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 @tgravescs @srowen Sorry for the failure. I am looking into it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-28 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14659 this is a new feature so generally don't put them into point releases which are bug fixes. @srowen thanks for pointing out, I will take alook. --- If your project is set up for it, you

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-28 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14659 @Sherry302 @tgravescs oops it looks like this causes master Maven builds to fail: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-27 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Hi, @tgravescs Should we also commit this PR to Branch-2? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-27 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Thanks a lot for the review. @tgravescs @cnauroth @steveloughran @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-27 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14659 thanks @Sherry302 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-27 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14659 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65934/ Test PASSed. ---

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14659 **[Test build #65934 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65934/consoleFull)** for PR 14659 at commit

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14659 **[Test build #65934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65934/consoleFull)** for PR 14659 at commit

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-26 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Hi, @tgravescs Thanks a lot for the comments. I have updated the PR to rename local vals and remove the `@since` in `Utils.scala`. --- If your project is set up for it, you can reply to this

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14659 Couple minor things otherwise looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65799/ Test PASSed. ---

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14659 **[Test build #65799 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65799/consoleFull)** for PR 14659 at commit

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-22 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Hi, @tgravescs Thank you very much. Yes. I have updated the PR to make the string of the caller context shorter. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14659 **[Test build #65799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65799/consoleFull)** for PR 14659 at commit

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65677/ Test PASSed. ---

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14659 **[Test build #65677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65677/consoleFull)** for PR 14659 at commit

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-20 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Hi, @tgravescs Thank you so much for the review. I have updated the PR based on your every comment. The only one question left is this one (in `Task`) "are these params all optional just

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14659 **[Test build #65677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65677/consoleFull)** for PR 14659 at commit