[GitHub] spark pull request: SPARK-3874: Provide stable TaskContext API

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2803#issuecomment-59161674 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/382/consoleFull) for PR 2803 at commit

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18875997 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18876000 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18875991 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18875998 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18875996 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18875995 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18875993 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18875990 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18876002 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18876004 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics

2014-10-15 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2667#discussion_r18876036 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala --- @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-3874: Provide stable TaskContext API

2014-10-15 Thread ScrapCodes
Github user ScrapCodes commented on a diff in the pull request: https://github.com/apache/spark/pull/2803#discussion_r18876515 --- Diff: project/MimaExcludes.scala --- @@ -50,7 +50,11 @@ object MimaExcludes {

[GitHub] spark pull request: [SPARK-3923] Decrease Akka heartbeat interval ...

2014-10-15 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2784#issuecomment-59163637 Thanks, LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3923] Decrease Akka heartbeat interval ...

2014-10-15 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2784#issuecomment-59163981 Minor: Your PR title looks misleading ! :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-3944][Core] Code re-factored as suggest...

2014-10-15 Thread Shiti
GitHub user Shiti opened a pull request: https://github.com/apache/spark/pull/2810 [SPARK-3944][Core] Code re-factored as suggested You can merge this pull request into a Git repository by running: $ git pull https://github.com/Shiti/spark master Alternatively you can review

[GitHub] spark pull request: [SPARK-3944][Core] Code re-factored as suggest...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2810#issuecomment-59164110 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-10-15 Thread viper-kun
Github user viper-kun commented on the pull request: https://github.com/apache/spark/pull/2471#issuecomment-59164199 @vanzin , is it ok to go? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-59165099 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-59165094 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21762/consoleFull) for PR 2701 at commit

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

2014-10-15 Thread gvramana
Github user gvramana commented on a diff in the pull request: https://github.com/apache/spark/pull/2762#discussion_r18877380 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala --- @@ -159,7 +162,10 @@ private[hive] case class

[GitHub] spark pull request: SPARK-3874: Provide stable TaskContext API

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2803#issuecomment-59166771 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/382/consoleFull) for PR 2803 at commit

[GitHub] spark pull request: promote the speed of convert files to RDDS

2014-10-15 Thread surq
GitHub user surq opened a pull request: https://github.com/apache/spark/pull/2811 promote the speed of convert files to RDDS about convert files to RDDS there are 3 loops with files sequence in spark source. loops files sequence: 1、files.map(...)

[GitHub] spark pull request: promote the speed of convert files to RDDS

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2811#issuecomment-59167172 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

2014-10-15 Thread gvramana
Github user gvramana commented on a diff in the pull request: https://github.com/apache/spark/pull/2762#discussion_r18877950 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala --- @@ -186,6 +230,51 @@ private[hive] trait HiveInspectors {

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

2014-10-15 Thread gvramana
Github user gvramana commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-59168016 unfortunately :( I also have worked and implemented the same as part of https://github.com/apache/spark/pull/2802 Anyways I will rework on my pull request for

[GitHub] spark pull request: SPARK-3926 [CORE] Result of JavaRDD.collectAsM...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2805#issuecomment-59174952 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21763/consoleFull) for PR 2805 at commit

[GitHub] spark pull request: SPARK-3926 [CORE] Result of JavaRDD.collectAsM...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2805#issuecomment-59175385 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-3926 [CORE] Result of JavaRDD.collectAsM...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2805#issuecomment-59175382 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21763/consoleFull) for PR 2805 at commit

[GitHub] spark pull request: promote the speed of convert files to RDDS

2014-10-15 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/2811#discussion_r18881912 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -27,6 +27,7 @@ import org.apache.spark.rdd.RDD

[GitHub] spark pull request: promote the speed of convert files to RDDS

2014-10-15 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/2811#issuecomment-59176947 Looks good to me about the improvement. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: promote the speed of convert files to RDDS

2014-10-15 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/2811#issuecomment-59177082 Besides would you mind creating a related JIRA and change the title like other PR. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-3945]Properties of hive-site.xml is inv...

2014-10-15 Thread luogankun
Github user luogankun commented on the pull request: https://github.com/apache/spark/pull/2800#issuecomment-59183382 @liancheng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...

2014-10-15 Thread adrian-wang
GitHub user adrian-wang opened a pull request: https://github.com/apache/spark/pull/2812 [SPARK-3939][SQL] NPE caused by SessionState.out not set in thriftserver2 There would be an NPE caused by SessionState.out not set in thriftserver2. You can merge this pull request into a Git

[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59186353 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21764/consoleFull) for PR 2812 at commit

[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59189745 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59189741 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21764/consoleFull) for PR 2812 at commit

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-10-15 Thread mattf
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/2471#issuecomment-59191797 Well, good luck with adding something like that to HDFS... that is not the responsibility of filesystems. just so we're on the same page, i'm not advocating

[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...

2014-10-15 Thread timothysc
Github user timothysc commented on the pull request: https://github.com/apache/spark/pull/2126#issuecomment-59201681 I believe @tgravescs is the only committer on this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...

2014-10-15 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2126#issuecomment-59214378 @pwendell @mateiz Any committers that are more familiar with the mesos stuff that could look at this? --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: improve spark on yarn doc(issus 3629 on jira)

2014-10-15 Thread ssjssh
GitHub user ssjssh opened a pull request: https://github.com/apache/spark/pull/2813 improve spark on yarn doc(issus 3629 on jira) Right now running-on-yarn.md starts off with a big list of config options, and only then tells you how to submit an app. It would be better to put that

[GitHub] spark pull request: improve spark on yarn doc(issus 3629 on jira)

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2813#issuecomment-59215162 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-3944][Core] Code re-factored as suggest...

2014-10-15 Thread markhamstra
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/2810#issuecomment-59223588 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoopUtil should not u...

2014-10-15 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/2814 SPARK-1209 [CORE] SparkHadoopUtil should not use package org.apache.hadoop (This is just a look at what completely moving the classes would look like. I know Patrick flagged that as maybe not OK,

[GitHub] spark pull request: SPARK-3926 [CORE] Result of JavaRDD.collectAsM...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2805#issuecomment-59226192 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21766/consoleFull) for PR 2805 at commit

[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoopUtil should not u...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59226216 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21765/consoleFull) for PR 2814 at commit

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-10-15 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2471#issuecomment-59231360 @mattf Always in for a beer but unfortunately I'm not in NY... Also, you mention rotating a lot. This is not rotating. This is cleaning up, as in deleting

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-10-15 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2471#issuecomment-59231417 @viper-kun haven't had a chance to look at the diff again, but it seems there are merge conflicts now. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-3944][Core] Code re-factored as suggest...

2014-10-15 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2810#issuecomment-59235424 Hi @Shiti , could you write a less cryptic PR title? This becomes the git commit summary, so it's recommended that it describe the actual change. Thanks! --- If your

[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59237336 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-3926 [CORE] Result of JavaRDD.collectAsM...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2805#issuecomment-59237316 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21766/consoleFull) for PR 2805 at commit

[GitHub] spark pull request: SPARK-3926 [CORE] Result of JavaRDD.collectAsM...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2805#issuecomment-59237321 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3923] Increase Akka heartbeat pause abo...

2014-10-15 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2784#issuecomment-59237464 Updated, but I think we should always give PRs a name opposite to what they actually do. Keeps things interesting. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-3923] Increase Akka heartbeat pause abo...

2014-10-15 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2784#issuecomment-59237808 above below? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...

2014-10-15 Thread shaneknapp
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59239541 jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...

2014-10-15 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59239623 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-3926 [CORE] Result of JavaRDD.collectAsM...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2805#issuecomment-59241056 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21769/consoleFull) for PR 2805 at commit

[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59241716 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59241697 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21765/consoleFull) for PR 2814 at commit

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-15 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r18908858 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,496 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18908910 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -83,216 +79,163 @@ object SparkSubmit { * (4) the main class

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18909075 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -33,30 +34,25 @@ import org.apache.spark.util.Utils * a layer over the

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18909183 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -83,216 +79,163 @@ object SparkSubmit { * (4) the main class

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18909253 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -83,216 +79,163 @@ object SparkSubmit { * (4) the main class

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18909359 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -83,216 +79,163 @@ object SparkSubmit { * (4) the main class

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18909554 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -83,216 +79,163 @@ object SparkSubmit { * (4) the main class

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18909952 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -83,216 +79,163 @@ object SparkSubmit { * (4) the main class

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18910093 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -397,9 +347,9 @@ object SparkSubmit { * Provides an indirection layer for

[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59246755 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18910401 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -17,201 +17,286 @@ package org.apache.spark.deploy

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18910527 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -17,201 +17,286 @@ package org.apache.spark.deploy

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18910722 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -17,201 +17,286 @@ package org.apache.spark.deploy

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911158 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -227,91 +312,92 @@ private[spark] class SparkSubmitArguments(args:

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911121 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -17,201 +17,286 @@ package org.apache.spark.deploy

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911187 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -227,91 +312,92 @@ private[spark] class SparkSubmitArguments(args:

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911284 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -227,91 +312,92 @@ private[spark] class SparkSubmitArguments(args:

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911428 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -398,22 +478,117 @@ private[spark] class SparkSubmitArguments(args:

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911516 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -398,22 +478,117 @@ private[spark] class SparkSubmitArguments(args:

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911832 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -398,22 +478,117 @@ private[spark] class SparkSubmitArguments(args:

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911938 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitDriverBootstrapper.scala --- @@ -50,71 +51,69 @@ private[spark] object

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911947 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitDriverBootstrapper.scala --- @@ -50,71 +51,69 @@ private[spark] object

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18911964 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitDriverBootstrapper.scala --- @@ -50,71 +51,69 @@ private[spark] object

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2516#discussion_r18912066 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1479,6 +1479,14 @@ private[spark] object Utils extends Logging {

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-10-15 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2516#issuecomment-59251504 @tigerquoll you'll need to merge this with current master, since there are conflicts. You may be able to clean up some code since the PR I mentioned before is now checked

[GitHub] spark pull request: SPARK-3926 [CORE] Result of JavaRDD.collectAsM...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2805#issuecomment-59251747 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21769/consoleFull) for PR 2805 at commit

[GitHub] spark pull request: SPARK-3926 [CORE] Result of JavaRDD.collectAsM...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2805#issuecomment-59251760 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-15 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r18912692 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,496 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: Remove Bytecode Inspection for Join Eliminatio...

2014-10-15 Thread jegonzal
GitHub user jegonzal opened a pull request: https://github.com/apache/spark/pull/2815 Remove Bytecode Inspection for Join Elimination Removing bytecode inspection from triplet operations and introducing explicit join elimination flags. The explicit flags make the join elimination

[GitHub] spark pull request: Remove Bytecode Inspection for Join Eliminatio...

2014-10-15 Thread jegonzal
Github user jegonzal commented on the pull request: https://github.com/apache/spark/pull/2815#issuecomment-59263992 @ankurdave and @rxin I have not updated the applications to use the new explicit flags. I will do that in this PR pending approval for the API changes. --- If your

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-15 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-59265338 Friendly ping. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: Remove Bytecode Inspection for Join Eliminatio...

2014-10-15 Thread Ishiihara
Github user Ishiihara commented on a diff in the pull request: https://github.com/apache/spark/pull/2815#discussion_r18917918 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Graph.scala --- @@ -195,6 +195,12 @@ abstract class Graph[VD: ClassTag, ED: ClassTag] protected ()

[GitHub] spark pull request: Remove Bytecode Inspection for Join Eliminatio...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2815#issuecomment-59265527 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21771/consoleFull) for PR 2815 at commit

[GitHub] spark pull request: Remove Bytecode Inspection for Join Eliminatio...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2815#issuecomment-59265670 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21771/consoleFull) for PR 2815 at commit

[GitHub] spark pull request: Remove Bytecode Inspection for Join Eliminatio...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2815#issuecomment-59265673 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-59266259 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21772/consoleFull) for PR 2701 at commit

[GitHub] spark pull request: Remove Bytecode Inspection for Join Eliminatio...

2014-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2815#issuecomment-59266720 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...

2014-10-15 Thread adrian-wang
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59270336 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...

2014-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59271430 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21773/consoleFull) for PR 2812 at commit

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-15 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r18920574 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,496 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-15 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r18920718 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,496 @@ +/* + * Licensed to the Apache

  1   2   3   >