[GitHub] spark issue #19511: [SPARK-22293][SQL] Avoid unnecessary traversal in Resolv...

2017-10-16 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/19511 Hi, @cloud-fan @gatorsmile. Would you mind take a look? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19511: [SPARK-22293][SQL] Avoid unnecessary traversal in Resolv...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19511 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19511: [SPARK-22293][SQL] Avoid unnecessary traversal in...

2017-10-16 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/19511 [SPARK-22293][SQL] Avoid unnecessary traversal in ResolveReferences ## What changes were proposed in this pull request? We don't need traverse the children expression to determine whether

[GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Tim...

2017-10-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/18664#discussion_r145037361 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala --- @@ -55,6 +55,12 @@ object ArrowWriter { case (Double

[GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Tim...

2017-10-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/18664#discussion_r145036404 --- Diff: python/pyspark/sql/types.py --- @@ -1619,11 +1619,38 @@ def to_arrow_type(dt): arrow_type = pa.decimal(dt.precision, dt.scale)

[GitHub] spark pull request #19370: [SPARK-18136] Fix setup of SPARK_HOME variable on...

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19370#discussion_r145039851 --- Diff: bin/find-spark-home.cmd --- @@ -0,0 +1,44 @@ +@echo off + +rem +rem Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145039356 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,43 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark issue #19476: [SPARK-22062][CORE] Spill large block to disk in BlockMa...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19476 **[Test build #82833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82833/testReport)** for PR 19476 at commit [`bba8ea1`](https://github.com/apache/spark/commit/bb

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-16 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145035994 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,43 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark issue #19508: [SPARK-20783][SQL][Follow-up] Create ColumnVector to abs...

2017-10-16 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19508 I will check this on Wed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark pull request #19496: [SPARK-22271][SQL]mean overflows and returns null...

2017-10-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19496#discussion_r145034923 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2103,4 +2103,13 @@ class DataFrameSuite extends QueryTest with Share

[GitHub] spark pull request #19496: [SPARK-22271][SQL]mean overflows and returns null...

2017-10-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19496#discussion_r145034645 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2103,4 +2103,13 @@ class DataFrameSuite extends QueryTest with Share

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145034007 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,43 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) for row

[GitHub] spark issue #19510: [SPARK-22292][Mesos] Added spark.mem.max support for Mes...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19510 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19510: [SPARK-22292][Mesos] Added spark.mem.max support ...

2017-10-16 Thread windkit
GitHub user windkit opened a pull request: https://github.com/apache/spark/pull/19510 [SPARK-22292][Mesos] Added spark.mem.max support for Mesos ## What changes were proposed in this pull request? To limit the amount of resources a spark job accept from Mesos, currently we

[GitHub] spark issue #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19269 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19269 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82829/ Test PASSed. ---

[GitHub] spark issue #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19269 **[Test build #82829 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82829/testReport)** for PR 19269 at commit [`416da98`](https://github.com/apache/spark/commit/4

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145032365 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,43 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) for row

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145032174 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,43 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hin...

2017-10-16 Thread hhbyyh
Github user hhbyyh commented on the issue: https://github.com/apache/spark/pull/17862 Please let me know if there's any unresolved comments. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-16 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145029289 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,43 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18664 **[Test build #82832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82832/testReport)** for PR 18664 at commit [`a3ba4ac`](https://github.com/apache/spark/commit/a3

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145029049 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,43 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark pull request #19505: [SPARK-20396][SQL][PySpark][FOLLOW-UP] groupby()....

2017-10-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19505#discussion_r145027904 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala --- @@ -137,11 +137,15 @@ object ExtractPythonUDFs extends R

[GitHub] spark issue #18747: [SPARK-20822][SQL] Generate code to directly get value f...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18747 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145028238 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,43 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark issue #19505: [SPARK-20396][SQL][PySpark][FOLLOW-UP] groupby().apply()...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19505 **[Test build #82831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82831/testReport)** for PR 19505 at commit [`85f250d`](https://github.com/apache/spark/commit/85

[GitHub] spark issue #18747: [SPARK-20822][SQL] Generate code to directly get value f...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18747 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82830/ Test FAILed. ---

[GitHub] spark issue #18747: [SPARK-20822][SQL] Generate code to directly get value f...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18747 **[Test build #82830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82830/testReport)** for PR 18747 at commit [`34982c6`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-16 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19488 After I rethinking about it, non-determinstic expressions should not qualify this rule. Please remove it. Thanks! --- - To un

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-16 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r145026444 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -23,21 +23,56 @@ import org.apache.spark.sql.cat

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-16 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r145026389 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ColumnarBatchScan.scala --- @@ -19,10 +19,10 @@ package org.apache.spark.sql.execution

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-16 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r145026383 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ColumnarBatchScan.scala --- @@ -84,6 +82,7 @@ private[sql] trait ColumnarBatchScan extends

[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19363 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19363 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82828/ Test PASSed. ---

[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19363 **[Test build #82828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82828/testReport)** for PR 19363 at commit [`955e3c9`](https://github.com/apache/spark/commit/9

[GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Tim...

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18664#discussion_r145024991 --- Diff: python/pyspark/sql/tests.py --- @@ -3383,6 +3403,42 @@ def test_vectorized_udf_varargs(self): res = df.select(f(col('id')))

[GitHub] spark pull request #19488: [SPARK-22266][SQL] The same aggregate function wa...

2017-10-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19488#discussion_r145025165 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -205,14 +205,17 @@ object PhysicalAggregation {

[GitHub] spark pull request #19505: [SPARK-20396][SQL][PySpark][FOLLOW-UP] groupby()....

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19505#discussion_r145012436 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala --- @@ -137,11 +137,15 @@ object ExtractPythonUDFs exte

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19509 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82825/ Test PASSed. ---

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19509 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19509 **[Test build #82825 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82825/testReport)** for PR 19509 at commit [`95a9658`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #18747: [SPARK-20822][SQL] Generate code to directly get value f...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18747 **[Test build #82830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82830/testReport)** for PR 18747 at commit [`34982c6`](https://github.com/apache/spark/commit/34

[GitHub] spark issue #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19269 **[Test build #82829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82829/testReport)** for PR 19269 at commit [`416da98`](https://github.com/apache/spark/commit/41

[GitHub] spark pull request #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r145018796 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/SimpleWritableDataSource.scala --- @@ -0,0 +1,252 @@ +/* + * Licensed to the Ap

[GitHub] spark pull request #19269: [SPARK-22026][SQL][WIP] data source v2 write path

2017-10-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r145017823 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/SimpleWritableDataSource.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the Ap

[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...

2017-10-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19363 LGTM, pending jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark pull request #19488: [SPARK-22266][SQL] The same aggregate function wa...

2017-10-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19488#discussion_r145017429 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -205,14 +205,17 @@ object PhysicalAggregation {

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19439 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19439 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82827/ Test PASSed. ---

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19439 **[Test build #82827 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82827/testReport)** for PR 19439 at commit [`697585f`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #19482: [SPARK-22264][DEPLOY] Add timeout for eventlog replaying...

2017-10-16 Thread caneGuy
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19482 Ok,thanks @vanzin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark pull request #19482: [SPARK-22264][DEPLOY] Add timeout for eventlog re...

2017-10-16 Thread caneGuy
Github user caneGuy closed the pull request at: https://github.com/apache/spark/pull/19482 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145013312 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -653,15 +663,34 @@ private[spark] class BlockManager( require(bl

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145013046 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -653,15 +663,34 @@ private[spark] class BlockManager( require(

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18664 **[Test build #82826 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82826/testReport)** for PR 18664 at commit [`9894519`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18664 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82826/ Test FAILed. ---

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18664 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145011923 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -509,11 +508,10 @@ class BlockManagerSuite extends SparkFunSuite wit

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145011775 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -684,7 +713,7 @@ private[spark] class BlockManager( // tak

[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19363 **[Test build #82828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82828/testReport)** for PR 19363 at commit [`955e3c9`](https://github.com/apache/spark/commit/95

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145010440 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -653,15 +663,34 @@ private[spark] class BlockManager( require(bl

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-16 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r145010250 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -469,6 +474,12 @@ trait MesosSch

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82823/ Test PASSed. ---

[GitHub] spark issue #19499: [SPARK-22279][SQL][WIP] Turn on spark.sql.hive.convertMe...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19499 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19495 **[Test build #82823 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82823/testReport)** for PR 19495 at commit [`2f35889`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #19499: [SPARK-22279][SQL][WIP] Turn on spark.sql.hive.convertMe...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19499 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82824/ Test PASSed. ---

[GitHub] spark issue #19499: [SPARK-22279][SQL][WIP] Turn on spark.sql.hive.convertMe...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19499 **[Test build #82824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82824/testReport)** for PR 19499 at commit [`cf7edbb`](https://github.com/apache/spark/commit/c

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-16 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r145009696 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -380,7 +389,8 @@

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82821/ Test PASSed. ---

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145009567 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -662,7 +662,9 @@ private[spark] object SparkConf extends Logging { "spark.ya

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19488 **[Test build #82821 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82821/testReport)** for PR 19488 at commit [`6159dcc`](https://github.com/apache/spark/commit/6

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-16 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r145009167 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -653,15 +663,34 @@ private[spark] class BlockManager( require(bl

[GitHub] spark issue #19381: [SPARK-10884][ML] Support prediction on single instance ...

2017-10-16 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/19381 cc @jkbradley @yanboliang thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #19497: [SPARK-21549][CORE] Respect OutputFormats with no/invali...

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19497 Thank you @mridulm. I regret that I raised this here, causing confusion. Let's talk more in another place. I will cc you (and @jiangxb1987) when I happened to file up a JIRA or see similar issue

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19439 **[Test build #82827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82827/testReport)** for PR 19439 at commit [`697585f`](https://github.com/apache/spark/commit/69

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19439 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r145007975 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,129 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contri

[GitHub] spark issue #19497: [SPARK-21549][CORE] Respect OutputFormats with no/invali...

2017-10-16 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/19497 @HyukjinKwon Thanks for clarifying. The way I look at it is: `saveAsHadoopFile` is explicitly referring to `Output the RDD to any Hadoop-supported file system` in its description (and na

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82820/ Test PASSed. ---

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19495 **[Test build #82820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82820/testReport)** for PR 19495 at commit [`52deee7`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #14321: [SPARK-8971][ML] Add stratified sampling to ML CrossVali...

2017-10-16 Thread idlecool
Github user idlecool commented on the issue: https://github.com/apache/spark/pull/14321 Hi @sethah, any plans to work on it again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional com

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19509 **[Test build #82825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82825/testReport)** for PR 19509 at commit [`95a9658`](https://github.com/apache/spark/commit/95

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18664 **[Test build #82826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82826/testReport)** for PR 18664 at commit [`9894519`](https://github.com/apache/spark/commit/98

[GitHub] spark pull request #19509: [SPARK-22290][core] Avoid creating Hive delegatio...

2017-10-16 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/19509 [SPARK-22290][core] Avoid creating Hive delegation tokens when not necessary. Hive delegation tokens are only needed when the Spark driver has no access to the kerberos TGT. That happens only in

[GitHub] spark pull request #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19451#discussion_r144997755 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceExceptWithFilter.scala --- @@ -0,0 +1,73 @@ +/* + * License

[GitHub] spark issue #19497: [SPARK-21549][CORE] Respect OutputFormats with no/invali...

2017-10-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19497 I agree this was focussed more on the regression introduced and it should be good enough already, and I am talking about a different thing for behaviour change. Let me organise my idea

[GitHub] spark issue #19452: [SPARK-22136][SS] Evaluate one-sided conditions early in...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19452 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82819/ Test PASSed. ---

[GitHub] spark issue #19452: [SPARK-22136][SS] Evaluate one-sided conditions early in...

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19452 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19452: [SPARK-22136][SS] Evaluate one-sided conditions early in...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19452 **[Test build #82819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82819/testReport)** for PR 19452 at commit [`23db42c`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #19499: [SPARK-22279][SQL][WIP] Turn on spark.sql.hive.convertMe...

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19499 **[Test build #82824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82824/testReport)** for PR 19499 at commit [`cf7edbb`](https://github.com/apache/spark/commit/cf

[GitHub] spark issue #19500: [SPARK-22280][SQL][TEST] Improve StatisticsSuite to test...

2017-10-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19500 Thank you so much, @gatorsmile and @wzhfy . :D --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For addit

[GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Tim...

2017-10-16 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/18664#discussion_r144994750 --- Diff: python/pyspark/sql/tests.py --- @@ -3086,18 +3086,35 @@ class ArrowTests(ReusedPySparkTestCase): @classmethod def setUp

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19439 **[Test build #82822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82822/testReport)** for PR 19439 at commit [`697585f`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19439 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82822/ Test FAILed. ---

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19439 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

  1   2   3   4   5   >