[GitHub] spark pull request #19508: [SPARK-20783][SQL][Follow-up] Create ColumnVector...

2017-10-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19508#discussion_r145329814 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala --- @@ -495,6 +474,8 @@ private[columnar] cas

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestamp suppo...

2017-10-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18664 Will take another look tomorrow. Thanks for your work! @BryanCutler --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.a

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestamp suppo...

2017-10-17 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/18664 I'd cc @gatorsmile, @cloud-fan for final check. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestamp suppo...

2017-10-17 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18664 LGTM too --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spa

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-17 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/19524 Looks reasonable, my only thing is putting in a forced exited inside of shellutil.py is sort of a random place to the version check. --- ---

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-17 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19509 I see, thanks for the explanation. I didn't think about such scenario. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.

[GitHub] spark issue #19507: [WEB-UI] Add count in fair scheduler pool page

2017-10-17 Thread guoxiaolongzte
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/19507 @ajbozarth Sorry, upload the code before I accidentally withdrew the parenthesis. I rejoined the parenthesis. I have fixed it. ---

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19524 seems fine. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.

[GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestam...

2017-10-17 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/18664#discussion_r145327316 --- Diff: python/pyspark/serializers.py --- @@ -259,11 +261,13 @@ def load_stream(self, stream): """ Deserialize ArrowRecordBatches

[GitHub] spark pull request #19508: [SPARK-20783][SQL][Follow-up] Create ColumnVector...

2017-10-17 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19508#discussion_r145323960 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala --- @@ -495,6 +474,8 @@ private[columnar] case

[GitHub] spark pull request #19508: [SPARK-20783][SQL][Follow-up] Create ColumnVector...

2017-10-17 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19508#discussion_r145323573 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnAccessor.scala --- @@ -63,9 +63,6 @@ private[columnar] abstract class BasicC

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19488 **[Test build #82874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82874/testReport)** for PR 19488 at commit [`506c410`](https://github.com/apache/spark/commit/50

[GitHub] spark issue #19511: [SPARK-22293][SQL] Avoid unnecessary traversal in Resolv...

2017-10-17 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/19511 OK, close it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark issue #19511: [SPARK-22293][SQL] Avoid unnecessary traversal in Resolv...

2017-10-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19511 @ConeyLiu I just saw extra codes/logics are added. Maybe close it first? --- - To unsubscribe, e-mail: reviews-unsubscr...@spa

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19480 @kiszk Does this look good to you? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19269 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82872/ Test PASSed. ---

[GitHub] spark issue #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19269 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19269 **[Test build #82872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82872/testReport)** for PR 19269 at commit [`9e12d9f`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19480 cc @rednaxelafx Do you have a bandwidth to review this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #18747: [SPARK-20822][SQL] Generate code to directly get value f...

2017-10-17 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18747 ping @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark issue #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19521 `SPARK-15474 ` is zero row. The above case is zero column. Are they the same issues? --- - To unsubscribe, e-mail: reviews-un

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19524 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19524 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82871/ Test PASSed. ---

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19524 **[Test build #82871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82871/testReport)** for PR 19524 at commit [`26eeae1`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #19522: [SPARK-22249][FOLLOWUP][SQL] Check if list of value for ...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19522 **[Test build #82873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82873/testReport)** for PR 19522 at commit [`e95bc7b`](https://github.com/apache/spark/commit/e9

[GitHub] spark issue #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-17 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19521 Thank you for review, @gatorsmile . 1. The test case was added at #15898 (SPARK-18457). I guess Parquet returns `null`, but we had better have explicit test cases. I will try to extend

[GitHub] spark pull request #19523: [SPARK-22301][SQL] Add rule to Optimizer for In w...

2017-10-17 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19523#discussion_r145318964 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -102,7 +102,8 @@ case class InMemoryTableSc

[GitHub] spark issue #19522: [SPARK-22249][FOLLOWUP][SQL] Check if list of value for ...

2017-10-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19522 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@sp

[GitHub] spark issue #19272: [Spark-21842][Mesos] Support Kerberos ticket renewal and...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19272 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82870/ Test PASSed. ---

[GitHub] spark issue #19272: [Spark-21842][Mesos] Support Kerberos ticket renewal and...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19272 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark issue #19272: [Spark-21842][Mesos] Support Kerberos ticket renewal and...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19272 **[Test build #82870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82870/testReport)** for PR 19272 at commit [`e522150`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19521 We can save an empty DataFrame as an ORC table, but we are unable to fetch it from the table. ```Scala val rddNoCols = sparkContext.parallelize(1 to 10).map(_ => Row.empty)

[GitHub] spark issue #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-17 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19521 Also LGTM Regarding the test case you posted, does Parquet return `null` or `empty string`? --- - To unsubscribe, e-

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-17 Thread shaneknapp
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/19524 yeah, this makes sense. i don't think we've officially supported 2.6 in a while, esp for tests and i'd be ok w/removing the backports. this makes for a much clearer exit case. @rxin @

[GitHub] spark issue #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-17 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19521 Thank you for review, @cloud-fan ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comman

[GitHub] spark issue #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-17 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19521 looks good, no new dependencies introduced, just upgrading. cc @srowen to double check. Thanks! --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19523: [SPARK-22301][SQL] Add rule to Optimizer for In w...

2017-10-17 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19523#discussion_r145306880 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -204,6 +204,7 @@ case class In(value: Expression, lis

[GitHub] spark pull request #19523: [SPARK-22301][SQL] Add rule to Optimizer for In w...

2017-10-17 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19523#discussion_r145307826 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -102,7 +102,8 @@ case class InMemoryTableScanExe

[GitHub] spark pull request #19495: [SPARK-22278][SS] Expose current event time water...

2017-10-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19495 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82869/ Test PASSed. ---

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19495 **[Test build #82869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82869/testReport)** for PR 19495 at commit [`ed9d3a2`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19269 **[Test build #82872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82872/testReport)** for PR 19269 at commit [`9e12d9f`](https://github.com/apache/spark/commit/9e

[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19459 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19459 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82866/ Test PASSed. ---

[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...

2017-10-17 Thread pmackles
Github user pmackles commented on the issue: https://github.com/apache/spark/pull/19515 Hi @ArtRand - Based on my testing and interpretation of the code, ```SPARK_DRIVER_MEMORY``` has no affect on MesosClusterDispatcher. Heap size always winds up being set to ```-Xmx1G``` (the default

[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19459 **[Test build #82866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82866/testReport)** for PR 19459 at commit [`81ddfa9`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82865/ Test PASSed. ---

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19488 **[Test build #82865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82865/testReport)** for PR 19488 at commit [`ece2062`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19524 **[Test build #82871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82871/testReport)** for PR 19524 at commit [`26eeae1`](https://github.com/apache/spark/commit/26

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-17 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19524 cc @JoshRosen, @holdenk and @shaneknapp, could you take a look and see if makes sense please? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19524: [SPARK-22302][INFRA] Remove manual backports for ...

2017-10-17 Thread HyukjinKwon
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/19524 [SPARK-22302][INFRA] Remove manual backports for subprocess and print explicit message for < Python 2.7 ## What changes were proposed in this pull request? Seems there is a mistake fro

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-10-17 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r145301221 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -194,6 +198,27 @@

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-10-17 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r145300847 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCredentialRenewer.scala --- @@ -0,0 +1,154 @@ +/* +

[GitHub] spark issue #19272: [Spark-21842][Mesos] Support Kerberos ticket renewal and...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19272 **[Test build #82870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82870/testReport)** for PR 19272 at commit [`e522150`](https://github.com/apache/spark/commit/e5

[GitHub] spark issue #19519: [SPARK-21840][core] Add trait that allows conf to be dir...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19519 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82863/ Test PASSed. ---

[GitHub] spark issue #19519: [SPARK-21840][core] Add trait that allows conf to be dir...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19519 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19519: [SPARK-21840][core] Add trait that allows conf to be dir...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19519 **[Test build #82863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82863/testReport)** for PR 19519 at commit [`d4466f2`](https://github.com/apache/spark/commit/d

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-10-17 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r145298313 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCredentialRenewer.scala --- @@ -0,0 +1,154 @@ +/* +

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-17 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r145298003 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -380,7 +389,8 @@

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82864/ Test PASSed. ---

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19495 **[Test build #82864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82864/testReport)** for PR 19495 at commit [`0d788fe`](https://github.com/apache/spark/commit/0

[GitHub] spark pull request #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-17 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r145296142 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Command.scala --- @@ -0,0 +1,114 @@ +/* + * Licen

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r145294156 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -173,6 +178,90 @@ private[

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r145294352 --- Diff: docs/running-on-mesos.md --- @@ -501,23 +503,74 @@ See the [configuration page](configuration.html) for information on Spark config spark

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r145294610 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -122,7 +126,8 @@ private[m

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r145294074 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -17,10 +17,14 @@

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestamp suppo...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18664 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestamp suppo...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18664 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82862/ Test PASSed. ---

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestamp suppo...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18664 **[Test build #82862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82862/testReport)** for PR 18664 at commit [`e428cbe`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19480 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82861/ Test PASSed. ---

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19480 **[Test build #82861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82861/testReport)** for PR 19480 at commit [`bce3616`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82860/ Test PASSed. ---

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-17 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145294433 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,43 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19488 **[Test build #82860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82860/testReport)** for PR 19488 at commit [`9c33a0c`](https://github.com/apache/spark/commit/9

[GitHub] spark pull request #19317: [SPARK-22098][CORE] Add new method aggregateByKey...

2017-10-17 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/19317#discussion_r145294297 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -180,6 +180,56 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])

[GitHub] spark pull request #19317: [SPARK-22098][CORE] Add new method aggregateByKey...

2017-10-17 Thread ConeyLiu
Github user ConeyLiu closed the pull request at: https://github.com/apache/spark/pull/19317 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19511: [SPARK-22293][SQL] Avoid unnecessary traversal in Resolv...

2017-10-17 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/19511 Hi @gatorsmile, if we can combine the two traverse, this should be simplify the code not complicate. However, this can't get big performance improvement. And I can close it if this change unnecessa

[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...

2017-10-17 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19459 BTW, https://github.com/apache/spark/pull/19459#discussion_r145034007 looks missed :). --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-17 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145293860 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,39 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19480 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82858/ Test PASSed. ---

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19480 **[Test build #82858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82858/testReport)** for PR 19480 at commit [`95b0ad8`](https://github.com/apache/spark/commit/9

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-17 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145293209 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,39 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark issue #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19437 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82868/ Test PASSed. ---

[GitHub] spark issue #19515: [SPARK-22287][MESOS] SPARK_DAEMON_MEMORY not honored by ...

2017-10-17 Thread ArtRand
Github user ArtRand commented on the issue: https://github.com/apache/spark/pull/19515 Hello @pmackles, thanks for this. It would be set to the value of `SPARK_DRIVER_MEMORY` by default correct? What to you think about introducing a new envvar (`SPARK_DISPATCHER_MEMORY`) so that it ca

[GitHub] spark issue #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19437 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19437 **[Test build #82868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82868/testReport)** for PR 19437 at commit [`b2a3675`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19495 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82859/ Test PASSed. ---

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-17 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145292822 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,39 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) fo

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-17 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19488 LGTM except one comment, thanks for working on it! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For addit

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19495 **[Test build #82859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82859/testReport)** for PR 19495 at commit [`037e036`](https://github.com/apache/spark/commit/0

[GitHub] spark pull request #19488: [SPARK-22266][SQL] The same aggregate function wa...

2017-10-17 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19488#discussion_r145292665 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -2677,4 +2678,29 @@ class SQLQuerySuite extends QueryTest with SharedSQ

[GitHub] spark pull request #19488: [SPARK-22266][SQL] The same aggregate function wa...

2017-10-17 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19488#discussion_r145292679 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -2677,4 +2678,29 @@ class SQLQuerySuite extends QueryTest with SharedSQ

[GitHub] spark issue #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19437 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

  1   2   3   4   5   >