[GitHub] spark pull request: SPARK-2158 Clean up core/stdout file from File...

2014-06-17 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1100#discussion_r13844029 --- Diff: core/src/test/scala/org/apache/spark/util/FileAppenderSuite.scala --- @@ -130,7 +130,8 @@ class FileAppenderSuite extends FunSuite with

[GitHub] spark pull request: SPARK-2158 Clean up core/stdout file from File...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1100#issuecomment-46269736 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Double check in doGetLocal to avoid read on re...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1103#issuecomment-46269743 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15841/ --- If your

[GitHub] spark pull request: added compatibility for python 2.6 for ssh_rea...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/941#issuecomment-46269746 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15842/ --- If your project

[GitHub] spark pull request: [SPARK-1768] History server enhancements.

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/718#issuecomment-46269745 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15844/ --- If your project is set up for it, you can

[GitHub] spark pull request: Double check in doGetLocal to avoid read on re...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1103#issuecomment-46269738 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: MLlib documentation fix

2014-06-17 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1098#issuecomment-46270697 LGTM. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: MLlib documentation fix

2014-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1098 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1102#discussion_r13844478 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/ui/WorkerPage.scala --- @@ -122,6 +110,7 @@ private[spark] class WorkerPage(parent:

[GitHub] spark pull request: Double check in doGetLocal to avoid read on re...

2014-06-17 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1103#issuecomment-46271308 jira ticket: https://github.com/apache/spark/pull/1103 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1102#discussion_r13844551 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/ApplicationPage.scala --- @@ -57,43 +57,54 @@ private[spark] class ApplicationPage(parent:

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1102#discussion_r13844576 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/ApplicationPage.scala --- @@ -57,43 +57,54 @@ private[spark] class ApplicationPage(parent:

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1102#issuecomment-46271457 LGTM, just a few minor comments. Thanks a bunch! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2130] End-user friendly String repr for...

2014-06-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1096#issuecomment-46271813 Looks good, thanks for this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Double check in doGetLocal to avoid read on re...

2014-06-17 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1103#issuecomment-46271800 Though with this patch, it is still possible that in the even rare case, the info is removed, and then add back again before this synchronized block is entered. then

[GitHub] spark pull request: [SPARK-2130] End-user friendly String repr for...

2014-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1096 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1099#issuecomment-46271992 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1099#issuecomment-46272002 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: added compatibility for python 2.6 for ssh_rea...

2014-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/941 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: added compatibility for python 2.6 for ssh_rea...

2014-06-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/941#issuecomment-46272633 Thanks - I've merged this and back ported it into 0.9 and 1.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1099#issuecomment-46273682 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1099#issuecomment-46273683 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15845/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-46273865 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46274071 LGTM - thanks for the work on this. I'm going to go ahead and merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/981 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-17 Thread codeboyyong
Github user codeboyyong commented on the pull request: https://github.com/apache/spark/pull/1099#issuecomment-46276348 @mengxr . I fix the format . And I did the test on hadoop-2.0.5-alpha and hadoop 2.2.0. The command line now works as usual. And my local yarn cluster is not working

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-46279341 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-46279342 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15846/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2144] ExecutorsPage reports incorrect #...

2014-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1080 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-17 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-46282079 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-46282254 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-46282247 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Refactor DriverRunner and DriverRunnerTest

2014-06-17 Thread CrazyJvm
Github user CrazyJvm closed the pull request at: https://github.com/apache/spark/pull/1066 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46283995 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46283977 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2163] class LBFGS optimize with Double ...

2014-06-17 Thread BaiGang
GitHub user BaiGang opened a pull request: https://github.com/apache/spark/pull/1104 [SPARK-2163] class LBFGS optimize with Double tolerance instead of Int https://issues.apache.org/jira/browse/SPARK-2163 This pull request includes the change for **[SPARK-2163]**:

[GitHub] spark pull request: [SPARK-2163] class LBFGS optimize with Double ...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1104#issuecomment-46289980 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-46290063 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15847/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46290061 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured ra...

2014-06-17 Thread li-zhihui
Github user li-zhihui commented on a diff in the pull request: https://github.com/apache/spark/pull/900#discussion_r13851868 --- Diff: yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala --- @@ -77,6 +77,12 @@ private[spark] class

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured ra...

2014-06-17 Thread li-zhihui
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/900#issuecomment-46291702 @tgravescs I add a commit support yarn-cluster. A little issue, the YarnClusterSchedulerBackend can't get --num-executors as totalExecutors

[GitHub] spark pull request: [SparkSQL] allow UDF on struct

2014-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/796 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: Follow up of PR #1071 for Java API

2014-06-17 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1085#issuecomment-46294986 Oh, sorry. Good catch! I must not have done the last push to apache part... --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SparkSQL] allow UDF on struct

2014-06-17 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/796#issuecomment-46294743 Thanks! I've merged this into master and 1.0 as https://issues.apache.org/jira/browse/SPARK-2164 --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1055 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-17 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1055#issuecomment-46295906 Thanks! merged into master and 1.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-17 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13853977 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala --- @@ -282,4 +290,33 @@ trait HiveTypeCoercion {

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-17 Thread darabos
Github user darabos commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-46303967 Thanks Patrick! @ankurdave: Do you want to add this to the storage UI? I can probably do it too if you're busy. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-1395] Fix local: URI support in Yarn ...

2014-06-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/560#discussion_r13858909 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -342,24 +352,16 @@ trait ClientBase extends Logging {

[GitHub] spark pull request: [SPARK-1395] Fix local: URI support in Yarn ...

2014-06-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/560#discussion_r13858930 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -378,15 +378,34 @@ trait ClientBase extends Logging { }

[GitHub] spark pull request: [SPARK-1395] Fix local: URI support in Yarn ...

2014-06-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/560#discussion_r13860204 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -386,12 +388,31 @@ trait ClientBase extends Logging { object

[GitHub] spark pull request: [SPARK-1395] Fix local: URI support in Yarn ...

2014-06-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/560#discussion_r13861962 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -464,71 +485,74 @@ object ClientBase extends Logging {

[GitHub] spark pull request: [SPARK-1395] Fix local: URI support in Yarn ...

2014-06-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/560#discussion_r13862215 --- Diff: yarn/common/src/test/scala/org/apache/spark/deploy/yarn/ClientBaseSuite.scala --- @@ -68,6 +77,65 @@ class ClientBaseSuite extends FunSuite {

[GitHub] spark pull request: Minor fix

2014-06-17 Thread WangTaoTheTonic
GitHub user WangTaoTheTonic opened a pull request: https://github.com/apache/spark/pull/1105 Minor fix The value env is never used in SparkContext.scala. Add detailed comment for method setDelaySeconds in MetadataCleaner.scala instead of the unsure one. You can merge

[GitHub] spark pull request: Minor fix

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1105#issuecomment-46317911 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1395] Fix local: URI support in Yarn ...

2014-06-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/560#discussion_r13864023 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -342,24 +352,16 @@ trait ClientBase extends Logging {

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/969#discussion_r13865835 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -45,6 +44,25 @@ class ClientArguments(val args:

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/969#discussion_r13866068 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -45,6 +44,25 @@ class ClientArguments(val args:

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46324012 Sorry if I wan't clear. I didn't mean to update ClientArguments with that logic, I just meant that is the logic we want overall. We don't want to duplicate code if

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46324473 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46324474 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15849/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46324445 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46324631 note I think it woudl be easiest just to have YarnClientSchedulerBackend ignore the configs spark.yarn.dist.archives/spark.yarn.dist.files and only do the backwards

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46326177 spark-defaults.conf|command | path | | - spark.yarn.dist.archives /other/path |`./bin/spark-submit --archives /some/path

[GitHub] spark pull request: [SPARK-1395] Fix local: URI support in Yarn ...

2014-06-17 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/560#discussion_r13867773 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -342,24 +352,16 @@ trait ClientBase extends Logging {

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-17 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-46327168 `spark.yarn.dist.*` related is concentrated in `ClientArguments` is a good idea --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-1768] History server enhancements.

2014-06-17 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/718#issuecomment-46327284 Guess I need to figure out how to exclude HistoryServer from compatibility checks. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: Optimize the schedule procedure in Master

2014-06-17 Thread WangTaoTheTonic
GitHub user WangTaoTheTonic opened a pull request: https://github.com/apache/spark/pull/1106 Optimize the schedule procedure in Master If the waiting driver array is too big, the drivers in it will be dispatched to the first worker we get(if it has enough resources), with or

[GitHub] spark pull request: Optimize the schedule procedure in Master

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1106#issuecomment-46328690 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1395] Fix local: URI support in Yarn ...

2014-06-17 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/560#discussion_r13870892 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -342,24 +352,16 @@ trait ClientBase extends Logging {

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1102#discussion_r13872230 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/ui/WorkerPage.scala --- @@ -122,6 +110,7 @@ private[spark] class WorkerPage(parent:

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1102#discussion_r13873224 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/ui/WorkerPage.scala --- @@ -122,6 +110,7 @@ private[spark] class WorkerPage(parent:

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1102#issuecomment-46339000 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1102#issuecomment-46339026 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-46339194 You are getting test failures because the patch doesn't merge cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-1768] History server enhancements.

2014-06-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/718#issuecomment-46340525 We should exclude the entire org.apache.spark.deploy package. @vanzin checkout `project/MimaExcludes.scala` and add `excludePackage` for 1.1 --- If your project is set

[GitHub] spark pull request: [SPARK-1768] History server enhancements.

2014-06-17 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/718#discussion_r13873985 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationHistoryProvider.scala --- @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-1768] History server enhancements.

2014-06-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/718#issuecomment-46340917 LGTM pending getting the MIMA stuff working. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Listing of instances to be terminated before t...

2014-06-17 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/270#discussion_r13874310 --- Diff: ec2/spark_ec2.py --- @@ -691,12 +691,18 @@ def real_main(): setup_cluster(conn, master_nodes, slave_nodes, opts, True) elif

[GitHub] spark pull request: Optimize the schedule procedure in Master

2014-06-17 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/1106#issuecomment-46341281 did you see any performance impact on the current strategy, randomization at the start of every schedule point is used not only at Master but also TaskSchedulerImpl...

[GitHub] spark pull request: Listing of instances to be terminated before t...

2014-06-17 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/270#discussion_r13874441 --- Diff: ec2/spark_ec2.py --- @@ -691,12 +691,18 @@ def real_main(): setup_cluster(conn, master_nodes, slave_nodes, opts, True) elif

[GitHub] spark pull request: Listing of instances to be terminated before t...

2014-06-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/270#issuecomment-46341757 Made some minor comments. Overall, this seems like a great to have. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: Listing of instances to be terminated before t...

2014-06-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/270#issuecomment-46341975 @j-martin one other thing. Would you mind creating a brief JIRA to describe this feature? You can just create an account on the site. Many people follow our issue

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-17 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13874731 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala --- @@ -282,4 +290,33 @@ trait HiveTypeCoercion

[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...

2014-06-17 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/1107 [WIP] SPARK-2157 Ability to write tight firewall rules for Spark https://issues.apache.org/jira/browse/SPARK-2157 This pull request adds the ability to specify every port opened by Spark

[GitHub] spark pull request: SPARK-1868: Users should be allowed to cogroup...

2014-06-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/813#issuecomment-46343152 @douglaz if you up-merge this with master I think the tests should pass fine (currently it's not merging cleanly). I'd like to get this merged soon if possible, so let

[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-46343356 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-46343337 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-1063 Add .sortBy(f) method on RDD

2014-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/369#issuecomment-46343296 I will test this today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-46343552 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15851/ --- If your project is set up for it, you can

[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-46343551 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1102#issuecomment-46343881 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1102#issuecomment-46343883 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15850/ --- If your

[GitHub] spark pull request: [SPARK-2130] End-user friendly String repr for...

2014-06-17 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/1096#issuecomment-46344583 @pwendell thanks for taking care of this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-1063 Add .sortBy(f) method on RDD

2014-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/369#issuecomment-46345169 This looks good to me. I will merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-1293 [SQL] [WIP] Parquet support for nes...

2014-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/360#discussion_r13876519 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetConverter.scala --- @@ -0,0 +1,667 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1102#issuecomment-46345817 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2147 / 2161] Show removed executors on ...

2014-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1102#issuecomment-46345846 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1768] History server enhancements.

2014-06-17 Thread vanzin
Github user vanzin closed the pull request at: https://github.com/apache/spark/pull/718 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-1768] History server enhancements.

2014-06-17 Thread vanzin
GitHub user vanzin reopened a pull request: https://github.com/apache/spark/pull/718 [SPARK-1768] History server enhancements. Two improvements to the history server: - Separate the HTTP handling from history fetching, so that it's easy to add new backends later

  1   2   3   >