[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13739286 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +201,78 @@ case class If(predicate:

[GitHub] spark pull request: [SPARK-2135][SQL] Use planner for in-memory sc...

2014-06-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1072 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-45980206 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SQL] Support transforming TreeNodes with Opti...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1074#issuecomment-45980616 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SQL] Support transforming TreeNodes with Opti...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1074#issuecomment-45980618 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15752/ --- If your project is set up for it, you can

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread vrilleup
Github user vrilleup commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-45981092 @rezazadeh Thank you for the comments! I added another method with the original method as you suggested. @mengxr I have tested it with real matrices. Here are

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-45981311 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-45981303 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-13 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/916#issuecomment-45981871 @dorx Do you think this works for extreme large data set with really small sample size? e.g. n = 1.0x10^11 while sample = 1 ? in that case, the final adjusted fraction

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once semantics for...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1071#issuecomment-45981900 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once semantics for...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1071#issuecomment-45981906 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-13 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-45982040 Hey @andrewor14, I sort of avoided EOF blocks for nested ifs, since that can look a bit less nicer because of no indentation. --- If your project is set up for it,

[GitHub] spark pull request: SPARK-1729. Make Flume pull data from source, ...

2014-06-13 Thread joyyoj
Github user joyyoj commented on a diff in the pull request: https://github.com/apache/spark/pull/807#discussion_r13740432 --- Diff: external/flume/src/main/scala/org/apache/spark/streaming/flume/EventTransformer.scala --- @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-45982605 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15753/ --- If your project

[GitHub] spark pull request: SPARK-1729. Make Flume pull data from source, ...

2014-06-13 Thread joyyoj
Github user joyyoj commented on a diff in the pull request: https://github.com/apache/spark/pull/807#discussion_r13740434 --- Diff: external/flume/src/main/scala/org/apache/spark/streaming/flume/EventTransformer.scala --- @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1729. Make Flume pull data from source, ...

2014-06-13 Thread joyyoj
Github user joyyoj commented on a diff in the pull request: https://github.com/apache/spark/pull/807#discussion_r13740438 --- Diff: external/flume/src/main/scala/org/apache/spark/streaming/flume/EventTransformer.scala --- @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-45982604 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-13 Thread li-zhihui
Github user li-zhihui commented on a diff in the pull request: https://github.com/apache/spark/pull/900#discussion_r13740596 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -48,6 +48,10 @@ class

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once semantics for...

2014-06-13 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/1071#discussion_r13740605 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDDLike.scala --- @@ -48,7 +49,17 @@ private[sql] trait SchemaRDDLike { */

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-45983110 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-45983120 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-13 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/900#discussion_r13740722 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -48,6 +48,10 @@ class

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once semantics for...

2014-06-13 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/1071#discussion_r13740823 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala --- @@ -22,45 +22,69 @@ import org.apache.spark.rdd.RDD import

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once semantics for...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1071#issuecomment-45983713 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once semantics for...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1071#issuecomment-45983726 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Workaround in Spark for ConcurrentModification...

2014-06-13 Thread nishkamravi2
Github user nishkamravi2 commented on the pull request: https://github.com/apache/spark/pull/1000#issuecomment-45983893 @rxin does this look ok? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Workaround in Spark for ConcurrentModification...

2014-06-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1000#issuecomment-45983980 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: Workaround in Spark for ConcurrentModification...

2014-06-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1000#issuecomment-45983988 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Workaround in Spark for ConcurrentModification...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1000#issuecomment-45984049 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Workaround in Spark for ConcurrentModification...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1000#issuecomment-45984058 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-13 Thread rahulsinghaliitd
Github user rahulsinghaliitd commented on a diff in the pull request: https://github.com/apache/spark/pull/1067#discussion_r13741193 --- Diff: core/src/main/scala/org/apache/spark/metrics/sink/CsvSink.scala --- @@ -53,11 +53,14 @@ private[spark] class CsvSink(val property:

[GitHub] spark pull request: [SPARK-2014] Make PySpark store RDDs in MEMORY...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1051#issuecomment-45985416 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2014] Make PySpark store RDDs in MEMORY...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1051#issuecomment-45985401 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-45986611 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-45986612 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15754/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-45987365 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15756/ --- If your project

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-45987363 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once semantics for...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1071#issuecomment-45987364 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: Workaround in Spark for ConcurrentModification...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1000#issuecomment-45989954 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once semantics for...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1071#issuecomment-45989960 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15757/ --- If your

[GitHub] spark pull request: [SPARK-2014] Make PySpark store RDDs in MEMORY...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1051#issuecomment-45989955 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2014] Make PySpark store RDDs in MEMORY...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1051#issuecomment-45989958 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15759/ --- If your project is set up for it, you can

[GitHub] spark pull request: Workaround in Spark for ConcurrentModification...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1000#issuecomment-45989959 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15758/ --- If your

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once semantics for...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1071#issuecomment-45989953 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742986 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742980 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742995 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala --- @@ -201,6 +202,31 @@ class RowMatrix( } /**

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13743003 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala --- @@ -201,6 +202,31 @@ class RowMatrix( } /**

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742972 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742973 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13743001 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala --- @@ -201,6 +202,31 @@ class RowMatrix( } /**

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742975 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742981 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742970 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742971 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742989 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala --- @@ -201,6 +202,31 @@ class RowMatrix( } /**

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742968 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/964#discussion_r13742976 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala --- @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-13 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-45990973 @vrilleup This implementation looks good to me and thanks for the experiments! Besides the inline comments, we should think when to switch from ARPACK to dense SVD. ARPACK

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-13 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/916#issuecomment-45992111 @colorant Tried the following with the new implementation: ~~~ val rdd = sc.parallelize(0 until 10, 1).flatMap(i = Iterator.fill(10)(0)) // 10^10

[GitHub] spark pull request: [HOTFIX] add math3 version to pom

2014-06-13 Thread mengxr
GitHub user mengxr opened a pull request: https://github.com/apache/spark/pull/1075 [HOTFIX] add math3 version to pom Passed `mvn package`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mengxr/spark takeSample-fix

[GitHub] spark pull request: [HOTFIX] add math3 version to pom

2014-06-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1075 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: SPARK-2058: Overriding config from SPARK_HOME ...

2014-06-13 Thread EugenCepoi
Github user EugenCepoi commented on the pull request: https://github.com/apache/spark/pull/997#issuecomment-45997041 Thx for the comments. Updated the PR based on them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: SPARK-2134: Report metrics before application ...

2014-06-13 Thread rahulsinghaliitd
GitHub user rahulsinghaliitd opened a pull request: https://github.com/apache/spark/pull/1076 SPARK-2134: Report metrics before application finishes You can merge this pull request into a Git repository by running: $ git pull https://github.com/Guavus/spark SPARK-2134

[GitHub] spark pull request: SPARK-2134: Report metrics before application ...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1076#issuecomment-46001763 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Resolve sbt warnings during build

2014-06-13 Thread huitseeker
Github user huitseeker commented on the pull request: https://github.com/apache/spark/pull/1069#issuecomment-46002024 @witgo @pwendell @srowen Correct : adding `-language:postfixOps` to your `scalac` line (`scalacOptions` for sbt) will remove the warning. --- If your project is set

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46002435 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46002447 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Resolve sbt warnings during build

2014-06-13 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1069#issuecomment-46002724 Yes, but then what warnings is this resolving? I understand one compiler flag is shorter than imports. I also understand that these language features are

[GitHub] spark pull request: Resolve sbt warnings during build

2014-06-13 Thread huitseeker
Github user huitseeker commented on the pull request: https://github.com/apache/spark/pull/1069#issuecomment-46004365 It resolves the warning the following warning, which occurs in different (but easy-to-reach) contexts depending on whether you are running with the `-language` option

[GitHub] spark pull request: Resolve sbt warnings during build

2014-06-13 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1069#issuecomment-46004821 Yes, I understand that, but these warnings are already suppressed with imports, and the advertised change here is to resolve warnings. It just exchanges mechanisms for

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46004992 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46005001 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46005097 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46005098 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15761/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46005347 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46005348 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15760/ --- If your project is set up for it, you can

[GitHub] spark pull request: Resolve sbt warnings during build

2014-06-13 Thread huitseeker
Github user huitseeker commented on the pull request: https://github.com/apache/spark/pull/1069#issuecomment-46006160 Fair enough. I just meant to provide context. Note that [this commit](https://github.com/apache/spark/commit/41c4a33105c74417192925db355019ba1badeab2) has

[GitHub] spark pull request: Resolve sbt warnings during build

2014-06-13 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1069#issuecomment-46006402 I'm more concerned with consistency than anything. We shouldn't do it both ways, and I don't think that compiler arg should have been added. A compiler flag is not a bad

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46006821 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46006828 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-13 Thread darabos
Github user darabos commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46007190 Thanks for the feedback! I've added JSON (de)serialization code for the new field. Patched in your change (thanks!). And added one more line to the top of the stack

[GitHub] spark pull request: [SPARK-2135][SQL] Use planner for in-memory sc...

2014-06-13 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1072#issuecomment-46008503 @marmbrus @liancheng I think this broke branch-1.0 from building: Error I am seeing: [INFO] Compiling 5 Scala sources

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46009163 Can you please be more specific on your description of this problem? Does it only occur in yarn cluster mode (or is it both cluster and client mode)?

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/969#discussion_r13749661 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -131,6 +132,13 @@ private[spark] class SparkSubmitArguments(args:

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-13 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/900#discussion_r13749934 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -48,6 +48,10 @@ class

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46011803 @tgravescs This problem only found in the yarn cluster mode. In the yarn client mode. [YarnClientSchedulerBackend.scala#L64](

[GitHub] spark pull request: SPARK-2134: Report metrics before application ...

2014-06-13 Thread rahulsinghaliitd
Github user rahulsinghaliitd commented on a diff in the pull request: https://github.com/apache/spark/pull/1076#discussion_r13750359 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -154,6 +154,8 @@ private[spark] class Master( }

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/969#discussion_r13750513 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -131,6 +132,13 @@ private[spark] class SparkSubmitArguments(args:

[GitHub] spark pull request: [SPARK-1930] The Container is running beyond p...

2014-06-13 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/894#discussion_r13751088 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala --- @@ -88,6 +88,18 @@ private[yarn] class

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46015458 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15762/ --- If your project is set up for it, you can

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46015455 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46018045 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46018031 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1930] The Container is running beyond p...

2014-06-13 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/894#discussion_r13752934 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala --- @@ -88,6 +88,18 @@ private[yarn] class YarnAllocationHandler(

[GitHub] spark pull request: [SPARK-1930] The Container is running beyond p...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/894#issuecomment-46022353 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1930] The Container is running beyond p...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/894#issuecomment-46022380 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46023406 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46023409 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15763/ --- If your project

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-13 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/969#discussion_r13755838 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -45,6 +44,14 @@ class ClientArguments(val args:

  1   2   3   >