[GitHub] spark issue #19746: [SPARK-22346][ML] VectorSizeHint Transformer for using V...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19746 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19746: [SPARK-22346][ML] VectorSizeHint Transformer for using V...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19746 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84449/ Test PASSed. ---

[GitHub] spark issue #19746: [SPARK-22346][ML] VectorSizeHint Transformer for using V...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19746 **[Test build #84449 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84449/testReport)** for PR 19746 at commit

[GitHub] spark issue #19887: [SPARK-21168] KafkaRDD should always set kafka clientId.

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19887 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19886: [SPARK-22162][BRANCH-2.2] Executors and the driver shoul...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19886 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84448/ Test PASSed. ---

[GitHub] spark issue #19886: [SPARK-22162][BRANCH-2.2] Executors and the driver shoul...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19886 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19886: [SPARK-22162][BRANCH-2.2] Executors and the driver shoul...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19886 **[Test build #84448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84448/testReport)** for PR 19886 at commit

[GitHub] spark pull request #19887: [SPARK-21168] KafkaRDD should always set kafka cl...

2017-12-04 Thread liu-zhaokun
GitHub user liu-zhaokun opened a pull request: https://github.com/apache/spark/pull/19887 [SPARK-21168] KafkaRDD should always set kafka clientId. There are no a number of other places that a client ID should be set,and I think we should use consumer.clientId in the clientId

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154836110 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigurationStepsOrchestrator.scala --- @@ -0,0

[GitHub] spark issue #19871: [SPARK-20728][SQL] Make OrcFileFormat configurable betwe...

2017-12-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19871 Could you review this again, @cloud-fan and @gatorsmile ? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19773: [SPARK-22546][SQL] Supporting for changing column dataTy...

2017-12-04 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/19773 gental ping @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154834535 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/Client.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154834099 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -119,5 +139,60 @@ private[spark] object Config

[GitHub] spark issue #19877: [SPARK-22681]Accumulator should only be updated once for...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19877 **[Test build #84452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84452/testReport)** for PR 19877 at commit

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154833977 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigurationStepsOrchestrator.scala --- @@ -0,0

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154833969 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigurationStepsOrchestrator.scala --- @@ -0,0

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154833947 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/Client.scala --- @@ -0,0 +1,234 @@ +/* + *

[GitHub] spark pull request #19877: [SPARK-22681]Accumulator should only be updated o...

2017-12-04 Thread carsonwang
Github user carsonwang commented on a diff in the pull request: https://github.com/apache/spark/pull/19877#discussion_r154833757 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -1832,6 +1832,27 @@ class DAGSchedulerSuite extends

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154833745 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/BaseDriverConfigurationStep.scala --- @@ -0,0 +1,162

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154833630 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigurationStepsOrchestrator.scala --- @@ -0,0

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154831875 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/Client.scala --- @@ -0,0 +1,234 @@ +/* + *

[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...

2017-12-04 Thread merlintang
Github user merlintang commented on a diff in the pull request: https://github.com/apache/spark/pull/19885#discussion_r154827513 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1428,6 +1428,12 @@ private object Client extends

[GitHub] spark pull request #19882: [SPARK-22672][SQL][TEST] Refactor ORC Tests

2017-12-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19882#discussion_r154824305 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala --- @@ -614,11 +531,63 @@ class OrcQuerySuite extends

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

2017-12-04 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 @vanzin please help to review, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19870: [SPARK-22665][SQL] Avoid repartitioning with empt...

2017-12-04 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19870 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19878: [SPARK-22682][SQL] HashExpression does not need to creat...

2017-12-04 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19878 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...

2017-12-04 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19885#discussion_r154822603 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1428,6 +1428,12 @@ private object Client extends

[GitHub] spark issue #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC disassoci...

2017-12-04 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19741 @devaraj-kavali looks like you'll need to update the kubernetes backend now... --- - To unsubscribe, e-mail:

[GitHub] spark issue #19870: [SPARK-22665][SQL] Avoid repartitioning with empty list ...

2017-12-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19870 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19746: [SPARK-22346][ML] VectorSizeHint Transformer for using V...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19746 **[Test build #84451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84451/testReport)** for PR 19746 at commit

[GitHub] spark issue #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC disassoci...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19741 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84445/ Test FAILed. ---

[GitHub] spark issue #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC disassoci...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19741 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/19884 @zsxwing, fyi after applying your Netty upgrade patch to Arrow, and then your other patch for Spark, all of the Spark Scala/Java tests pass ---

[GitHub] spark issue #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC disassoci...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19741 **[Test build #84445 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84445/testReport)** for PR 19741 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

2017-12-04 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 @merlintang would you please add the problem to your PR description, currently it is a WASB problem in which userInfo is honored to differentiate filesystems. Please add the scenario to the

[GitHub] spark pull request #19746: [SPARK-22346][ML] VectorSizeHint Transformer for ...

2017-12-04 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/19746#discussion_r154820556 --- Diff: mllib/src/test/scala/org/apache/spark/ml/feature/VectorSizeHintSuite.scala --- @@ -0,0 +1,173 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r154820118 --- Diff: python/pyspark/sql/session.py --- @@ -324,11 +324,12 @@ def range(self, start, end=None, step=1, numPartitions=None):

[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 Can we set the `names` for `_createFromRDD` -> `_inferSchema` too?: ```python >>> spark.createDataFrame(spark.sparkContext.parallelize([[None, 1], ["a", None], [1, 1]]),

[GitHub] spark issue #19882: [SPARK-22672][SQL][TEST] Refactor ORC Tests

2017-12-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19882 Thank you so much, @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19878: [SPARK-22682][SQL] HashExpression does not need t...

2017-12-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19878#discussion_r154819410 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -730,23 +776,29 @@ case class HiveHash(children:

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19717 **[Test build #84450 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84450/testReport)** for PR 19717 at commit

[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...

2017-12-04 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/19885#discussion_r154816425 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1428,6 +1428,12 @@ private object Client extends Logging

[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...

2017-12-04 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/19885#discussion_r154819072 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1428,6 +1428,12 @@ private object Client extends Logging

[GitHub] spark pull request #19878: [SPARK-22682][SQL] HashExpression does not need t...

2017-12-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19878#discussion_r154819030 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -610,25 +637,44 @@ case class HiveHash(children:

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154815505 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/DriverKubernetesCredentialsStep.scala --- @@ -0,0

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154814332 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/Client.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154816844 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigurationStepsOrchestrator.scala --- @@ -0,0

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154816823 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigurationStepsOrchestrator.scala --- @@ -0,0

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154814488 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/Client.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154814516 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/Client.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154814933 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/DriverConfigurationStepsOrchestrator.scala --- @@ -0,0

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154818466 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -119,5 +139,60 @@ private[spark] object

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154818296 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -590,6 +600,11 @@ private[deploy] class

[GitHub] spark pull request #19878: [SPARK-22682][SQL] HashExpression does not need t...

2017-12-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19878#discussion_r154818350 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -610,25 +637,44 @@ case class HiveHash(children:

[GitHub] spark pull request #19770: [SPARK-21571][WEB UI] Spark history server leaves...

2017-12-04 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19770#discussion_r154817100 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -643,6 +633,44 @@ private[history] class

[GitHub] spark pull request #19770: [SPARK-21571][WEB UI] Spark history server leaves...

2017-12-04 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19770#discussion_r154817305 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala --- @@ -658,6 +659,103 @@ class FsHistoryProviderSuite extends

[GitHub] spark pull request #19770: [SPARK-21571][WEB UI] Spark history server leaves...

2017-12-04 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19770#discussion_r154816346 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala --- @@ -658,6 +659,103 @@ class FsHistoryProviderSuite extends

[GitHub] spark issue #19883: [SPARK-22684][SQL] datetime functions should not generat...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19883 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/8/ Test PASSed. ---

[GitHub] spark issue #19883: [SPARK-22684][SQL] datetime functions should not generat...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19883 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19884 cc @zsxwing as well, I saw you opened a JIRA about this - SPARK-22656 --- - To unsubscribe, e-mail:

[GitHub] spark issue #19883: [SPARK-22684][SQL] datetime functions should not generat...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19883 **[Test build #8 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/8/testReport)** for PR 19883 at commit

[GitHub] spark issue #19882: [SPARK-22672][SQL][TEST] Refactor ORC Tests

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19882 Whoa big class list. Will take a look soon within tomorrow as well. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19886: [SPARK-22162][BRANCH-2.2] Executors and the driver shoul...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19886 **[Test build #84448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84448/testReport)** for PR 19886 at commit

[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

2017-12-04 Thread sadikovi
Github user sadikovi commented on a diff in the pull request: https://github.com/apache/spark/pull/19810#discussion_r154815012 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -193,38 +195,68 @@ case class

[GitHub] spark pull request #19746: [SPARK-22346][ML] VectorSizeHint Transformer for ...

2017-12-04 Thread MrBago
Github user MrBago commented on a diff in the pull request: https://github.com/apache/spark/pull/19746#discussion_r154815581 --- Diff: mllib/src/test/scala/org/apache/spark/ml/feature/VectorSizeHintSuite.scala --- @@ -0,0 +1,173 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19810: [SPARK-22599][SQL] In-Memory Table Pruning withou...

2017-12-04 Thread sadikovi
Github user sadikovi commented on a diff in the pull request: https://github.com/apache/spark/pull/19810#discussion_r154814573 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala --- @@ -52,6 +52,68 @@ object InMemoryRelation {

[GitHub] spark issue #19746: [SPARK-22346][ML] VectorSizeHint Transformer for using V...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19746 **[Test build #84449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84449/testReport)** for PR 19746 at commit

[GitHub] spark issue #19882: [SPARK-22672][SQL][TEST] Refactor ORC Tests

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19882 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84443/ Test PASSed. ---

[GitHub] spark issue #19882: [SPARK-22672][SQL][TEST] Refactor ORC Tests

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19882 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19770: [SPARK-21571][WEB UI] Spark history server leaves...

2017-12-04 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19770#discussion_r154814962 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -643,6 +633,44 @@ private[history] class

[GitHub] spark issue #19882: [SPARK-22672][SQL][TEST] Refactor ORC Tests

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19882 **[Test build #84443 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84443/testReport)** for PR 19882 at commit

[GitHub] spark issue #19886: [SPARK-22162][BRANCH-2.2] Executors and the driver shoul...

2017-12-04 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19886 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19886: [SPARK-22162][BRANCH-2.2] Executors and the driver shoul...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19886 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-04 Thread rezasafi
Github user rezasafi commented on the issue: https://github.com/apache/spark/pull/19848 The branch 2.2 PR for this fix is here: https://github.com/apache/spark/pull/19886 --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19886: [SPARK-22162][BRANCH-2.2] Executors and the drive...

2017-12-04 Thread rezasafi
GitHub user rezasafi opened a pull request: https://github.com/apache/spark/pull/19886 [SPARK-22162][BRANCH-2.2] Executors and the driver should use consistent JobIDs in the RDD commit protocol I have modified SparkHadoopMapReduceWriter so that executors and the driver always use

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154813008 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -590,6 +600,11 @@ private[deploy] class SparkSubmitArguments(args:

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

2017-12-04 Thread merlintang
Github user merlintang commented on the issue: https://github.com/apache/spark/pull/19885 @jerryshao can you review this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19885: [SPARK-22587] Spark job fails if fs.defaultFS and...

2017-12-04 Thread merlintang
GitHub user merlintang opened a pull request: https://github.com/apache/spark/pull/19885 [SPARK-22587] Spark job fails if fs.defaultFS and application jar are d… …ifferent url ## What changes were proposed in this pull request? Two filesystems comparing does

[GitHub] spark pull request #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to ...

2017-12-04 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19884#discussion_r154811610 --- Diff: pom.xml --- @@ -185,7 +185,7 @@ 2.8 1.8 1.0.0 -0.4.0 +0.8.0-SNAPSHOT --- End diff --

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/19884 > Great, @BryanCutler . Could you put the highlight in the PR description, too? Sure, thanks @dongjoon-hyun ! Will do, just want to go back and check the release notes first ---

[GitHub] spark pull request #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to ...

2017-12-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19884#discussion_r154810506 --- Diff: pom.xml --- @@ -185,7 +185,7 @@ 2.8 1.8 1.0.0 -0.4.0 +0.8.0-SNAPSHOT --- End diff --

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/19884 This is a WIP to start updating Spark to use Arrow 0.8.0 which will be released soon. TODO: - [ ] Update to reflect Java API changes - [ ] Update to reflect Python API

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19884 Great, @BryanCutler . Could you put the highlight in the PR description, too? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19872: WIP: [SPARK-22274][PySpark] User-defined aggregat...

2017-12-04 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r154809806 --- Diff: python/pyspark/sql/group.py --- @@ -89,8 +89,15 @@ def agg(self, *exprs): else: # Columns assert

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/19884 The highlights that pertain to Spark for the update from Arrow versoin 0.4.1 to 0.8.0 include: * Java refactoring for more simple API * Type support for DecimalType, ArrayType *

[GitHub] spark issue #19591: [SPARK-11035][core] Add in-process Spark app launcher.

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19591 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19591: [SPARK-11035][core] Add in-process Spark app launcher.

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19591 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84442/ Test FAILed. ---

[GitHub] spark issue #19591: [SPARK-11035][core] Add in-process Spark app launcher.

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19591 **[Test build #84442 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84442/testReport)** for PR 19591 at commit

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19717 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19717 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84440/ Test PASSed. ---

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19717 **[Test build #84440 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84440/testReport)** for PR 19717 at commit

[GitHub] spark issue #5503: [MLLIB][WIP] SPARK-4638: Kernels feature for MLLIB

2017-12-04 Thread mandar2812
Github user mandar2812 commented on the issue: https://github.com/apache/spark/pull/5503 It didn't make it into spark. Instead I started the [DynaML](https://github.com/transcendent-ai-labs/DynaML) project and made kernels a key module of it, along with spark support. ---

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19884 **[Test build #84447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84447/testReport)** for PR 19884 at commit

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19884 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84447/ Test FAILed. ---

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19884 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19884 **[Test build #84447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84447/testReport)** for PR 19884 at commit

[GitHub] spark pull request #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to ...

2017-12-04 Thread BryanCutler
GitHub user BryanCutler opened a pull request: https://github.com/apache/spark/pull/19884 [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0 ## What changes were proposed in this pull request? Upgrade Spark to Arrow 0.8.0 for Java and Python ## How was this

[GitHub] spark issue #19870: [SPARK-22665][SQL] Avoid repartitioning with empty list ...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19870 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19870: [SPARK-22665][SQL] Avoid repartitioning with empty list ...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19870 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84441/ Test PASSed. ---

<    1   2   3   4   5   >