[GitHub] spark issue #22713: [SPARK-25691][SQL] Use semantic equality in AliasViewChi...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22713 **[Test build #97380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97380/testReport)** for PR 22713 at commit [`c4aaa8d`](https://github.com/apache/spark/commit/c4

[GitHub] spark issue #22639: [SPARK-25647][k8s] Add spark streaming compatibility sui...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22639 **[Test build #97381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97381/testReport)** for PR 22639 at commit [`c63c54e`](https://github.com/apache/spark/commit/c6

[GitHub] spark issue #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite failures w...

2018-10-15 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22512 I made a new pr to address the Literal issue #22724 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additio

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22379 **[Test build #97385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97385/testReport)** for PR 22379 at commit [`93d094f`](https://github.com/apache/spark/commit/93

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22721 **[Test build #97387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97387/testReport)** for PR 22721 at commit [`37fed41`](https://github.com/apache/spark/commit/37

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22263 **[Test build #97379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97379/testReport)** for PR 22263 at commit [`8809311`](https://github.com/apache/spark/commit/88

[GitHub] spark issue #22219: [SPARK-25224][SQL] Improvement of Spark SQL ThriftServer...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22219 **[Test build #97388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97388/testReport)** for PR 22219 at commit [`2a41b70`](https://github.com/apache/spark/commit/2a

[GitHub] spark issue #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Support par...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20999 **[Test build #97382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97382/testReport)** for PR 20999 at commit [`2085088`](https://github.com/apache/spark/commit/20

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22379 **[Test build #97383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97383/testReport)** for PR 22379 at commit [`b26e49e`](https://github.com/apache/spark/commit/b2

[GitHub] spark issue #22219: [SPARK-25224][SQL] Improvement of Spark SQL ThriftServer...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22219 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97376/ Test PASSed. ---

[GitHub] spark issue #22219: [SPARK-25224][SQL] Improvement of Spark SQL ThriftServer...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22219 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22219: [SPARK-25224][SQL] Improvement of Spark SQL ThriftServer...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22219 **[Test build #97376 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97376/testReport)** for PR 22219 at commit [`f05570c`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #22723: [SPARK-25729][CORE]It is better to replace `minPartition...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97375/ Test PASSed. ---

[GitHub] spark issue #22723: [SPARK-25729][CORE]It is better to replace `minPartition...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22723 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22723: [SPARK-25729][CORE]It is better to replace `minPartition...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22723 **[Test build #97375 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97375/testReport)** for PR 22723 at commit [`b80bf66`](https://github.com/apache/spark/commit/b

[GitHub] spark pull request #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small ...

2018-10-15 Thread 10110346
GitHub user 10110346 opened a pull request: https://github.com/apache/spark/pull/22725 [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files via BinaryFileRDD ## What changes were proposed in this pull request? This is a follow up of #21601, `StreamFileInputFormat` and `Wh

[GitHub] spark issue #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Support par...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20999 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark issue #22715: [SPARK-25727][SQL] Add outputOrdering to otherCopyArgs i...

2018-10-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22715 @cloud-fan , sure, I'll submit a follow-up PR for this. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.o

[GitHub] spark pull request #22713: [SPARK-25691][SQL] Use semantic equality in Alias...

2018-10-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/22713#discussion_r225123064 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala --- @@ -112,8 +112,8 @@ object EliminateView extends Rule[LogicalP

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

2018-10-15 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22198 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h..

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

2018-10-15 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22198 ping @maryannxue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...

[GitHub] spark pull request #22708: [SPARK-21402] Fix java array/map of structs deser...

2018-10-15 Thread vofque
Github user vofque commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225121081 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala --- @@ -271,32 +272,41 @@ object JavaTypeInference {

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-10-15 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22379 While addressing the review comments, I also reviewed at the same time. The change looks pretty good to go. For https://github.com/apache/spark/pull/22379#issuecomment-429738031, I gues

[GitHub] spark issue #22715: [SPARK-25727][SQL] Add outputOrdering to otherCopyArgs i...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22715 Hi @mgaido91 , since you are the major author of this part, do you have time to open a PR and move `outputOrdering` to the main constructor? Thanks in advance! --- -

[GitHub] spark pull request #22713: [SPARK-25691][SQL] Use semantic equality in Alias...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22713#discussion_r225119359 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala --- @@ -112,8 +112,8 @@ object EliminateView extends Rule[Logical

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-10-15 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22379 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark pull request #22724: [SPARK-25734][SQL}

2018-10-15 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/22724 [SPARK-25734][SQL} ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this

[GitHub] spark pull request #22708: [SPARK-21402] Fix java array/map of structs deser...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225118613 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala --- @@ -271,32 +272,41 @@ object JavaTypeInference {

[GitHub] spark pull request #22713: [SPARK-25691][SQL] Use semantic equality in Alias...

2018-10-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22713#discussion_r225115756 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala --- @@ -112,8 +112,8 @@ object EliminateView extends Rule[LogicalPla

[GitHub] spark issue #22715: [SPARK-25727][SQL] Add outputOrdering to otherCopyArgs i...

2018-10-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22715 sorry @gatorsmile that is fine, my bad. A late LGTM to this, despite probably @cloud-fan 's comment make, we probably should have just put it as a part of the main constructor... --- ---

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97377/ Test FAILed. ---

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22721 **[Test build #97377 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97377/testReport)** for PR 22721 at commit [`8a7f4af`](https://github.com/apache/spark/commit/8

[GitHub] spark pull request #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Supp...

2018-10-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/20999#discussion_r225113719 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -523,35 +523,114 @@ case class AlterTableRenamePartitionCommand(

[GitHub] spark pull request #22598: [SPARK-25501][SS] Add kafka delegation token supp...

2018-10-15 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22598#discussion_r225111947 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/TokenUtil.scala --- @@ -0,0 +1,111 @@ +/* + * Licensed to the

[GitHub] spark pull request #22598: [SPARK-25501][SS] Add kafka delegation token supp...

2018-10-15 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22598#discussion_r225111758 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/TokenUtil.scala --- @@ -0,0 +1,111 @@ +/* + * Licensed to the

[GitHub] spark issue #22029: [SPARK-24395][SQL] IN operator should return NULL when c...

2018-10-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22029 Yes, I summarized earlier the behavior for `(a, b) in (struct_col1, struct_col2, ...)`. You can also check the PR description in "Summarizing:". Instead, `input_struct_col in (struct_col1,

[GitHub] spark issue #22561: [SPARK-25548][SQL]In the PruneFileSourcePartitions optim...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22561 Your rewrite is applied to the entire predicate expression tree, how about the case `NOT(p AND x)`? If `p` is true, `x` is false, then `NOT(p AND x)` is true. But `NOT(true AND true)` is false.

[GitHub] spark pull request #22598: [SPARK-25501][SS] Add kafka delegation token supp...

2018-10-15 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22598#discussion_r225109193 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -647,4 +647,42 @@ package object config { .stringConf

[GitHub] spark issue #22677: [SPARK-25683][Core] Make AsyncEventQueue.lastReportTimes...

2018-10-15 Thread shivusondur
Github user shivusondur commented on the issue: https://github.com/apache/spark/pull/22677 @jiangxb1987 @dongjoon-hyun I have modified accordingly. Tested locally with existing UT (to make drop the event, locally "LISTENER_BUS_EVENT_QUEUE_CAPACITY" is set to 1), using pri

[GitHub] spark pull request #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Supp...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20999#discussion_r225108214 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -523,35 +523,114 @@ case class AlterTableRenamePartitionCommand(

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-15 Thread ScrapCodes
Github user ScrapCodes commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r225105051 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark issue #22639: [SPARK-25647][k8s] Add spark streaming compatibility sui...

2018-10-15 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/22639 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22029: [SPARK-24395][SQL] IN operator should return NULL when c...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22029 Do you know how other databases behave for `(a, b) in (struct_col1, struct_col2, ...)` and `input_struct_col in (struct_col1, struct_col2, ...)`? Since `(a, b)` may be treated specially, we need t

[GitHub] spark pull request #22713: [SPARK-25691][SQL] Use semantic equality in Alias...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22713#discussion_r225103000 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RemoveRedundantAliasAndProjectSuite.scala --- @@ -124,4 +124,11 @@ class Re

[GitHub] spark pull request #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Supp...

2018-10-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/20999#discussion_r225101976 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -523,35 +523,114 @@ case class AlterTableRenamePartitionCommand(

[GitHub] spark pull request #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Supp...

2018-10-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/20999#discussion_r225101681 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala --- @@ -382,6 +382,30 @@ case class OuterReference(

[GitHub] spark issue #22708: [SPARK-21402] Fix java array/map of structs deserializat...

2018-10-15 Thread vofque
Github user vofque commented on the issue: https://github.com/apache/spark/pull/22708 In a nutshell: First we calculate field ordinals of array elements using java bean field information (which doesn't guarantee any particular field order). Then we apply these ordinals to the a

[GitHub] spark issue #22708: [SPARK-21402] Fix java array/map of structs deserializat...

2018-10-15 Thread vofque
Github user vofque commented on the issue: https://github.com/apache/spark/pull/22708 The original problem is described here: https://issues.apache.org/jira/browse/SPARK-21402 I'll try to explain what happens in detail. Let's consider this data structure: root

[GitHub] spark pull request #22721: [SPARK-25403][SQL] Refreshes the table after inse...

2018-10-15 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22721#discussion_r225092331 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -2370,4 +2370,26 @@ class HiveDDLSuite ))

[GitHub] spark pull request #22713: [SPARK-25691][SQL] Use semantic equality in Optim...

2018-10-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/22713#discussion_r225091990 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RemoveRedundantAliasAndProjectSuite.scala --- @@ -124,4 +124,11 @@ class Rem

[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...

2018-10-15 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @vanzin could I get a merge pls? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mai

[GitHub] spark issue #22029: [SPARK-24395][SQL] IN operator should return NULL when c...

2018-10-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22029 > It looks to me that this is another instance of special handling (a, b, ..), like #21403 Yes, we might say so. The problem is specific to how `=` is implemented for struct when one or mo

[GitHub] spark pull request #22279: [SPARK-25277][YARN] YARN applicationMaster metric...

2018-10-15 Thread LucaCanali
Github user LucaCanali commented on a diff in the pull request: https://github.com/apache/spark/pull/22279#discussion_r225088430 --- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala --- @@ -103,6 +103,14 @@ private[spark] class MetricsSystem private (

[GitHub] spark pull request #22279: [SPARK-25277][YARN] YARN applicationMaster metric...

2018-10-15 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/22279#discussion_r225086020 --- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala --- @@ -103,6 +103,14 @@ private[spark] class MetricsSystem private (

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r225082659 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ddl.scala --- @@ -34,11 +34,16 @@ import org.apache.spark.sql.types._ * @p

[GitHub] spark issue #22561: [SPARK-25548][SQL]In the PruneFileSourcePartitions optim...

2018-10-15 Thread eatoncys
Github user eatoncys commented on the issue: https://github.com/apache/spark/pull/22561 @cloud-fan which should be proved is that: the partitions returned of p' shound contain the partitions returned by p. Here, let p' = p && x, if x is true then p' == p; else if x is false, the pari

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r225079174 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ddl.scala --- @@ -34,11 +34,16 @@ import org.apache.spark.sql.types._ *

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-10-15 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22263 LGTM except for @dongjoon-hyun 's minor comment (we don't need the line 305) --- - To unsubscribe, e-mail: reviews-unsubscr...@spa

[GitHub] spark pull request #21990: [SPARK-25003][PYSPARK] Use SessionExtensions in P...

2018-10-15 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21990#discussion_r225077270 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -1136,4 +1121,27 @@ object SparkSession extends Logging { SparkSe

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3971/

[GitHub] spark pull request #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Supp...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20999#discussion_r225073862 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala --- @@ -382,6 +382,30 @@ case class OuterReference

[GitHub] spark pull request #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Supp...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20999#discussion_r225074108 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala --- @@ -382,6 +382,30 @@ case class OuterReference

[GitHub] spark pull request #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Supp...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20999#discussion_r225076055 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -523,35 +523,114 @@ case class AlterTableRenamePartitionCommand(

[GitHub] spark pull request #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Supp...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20999#discussion_r225074416 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -293,6 +293,31 @@ class AstBuilder(conf: SQLConf) exte

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22721 **[Test build #97377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97377/testReport)** for PR 22721 at commit [`8a7f4af`](https://github.com/apache/spark/commit/8a

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22721 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22263 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apa

[GitHub] spark issue #21322: [SPARK-24225][CORE] Support closing AutoClosable objects...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21322 Introducing a new concept which is similar to broadcast seems like an overkill. We can just update broadcast, to allow it to be memory-only. However, there might be simpler solutions to fi

[GitHub] spark issue #22219: [SPARK-25224][SQL] Improvement of Spark SQL ThriftServer...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22219 **[Test build #97376 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97376/testReport)** for PR 22219 at commit [`f05570c`](https://github.com/apache/spark/commit/f0

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97374/ Test FAILed. ---

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22721 **[Test build #97374 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97374/testReport)** for PR 22721 at commit [`8a7f4af`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #22561: [SPARK-25548][SQL]In the PruneFileSourcePartitions optim...

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22561 Please describe the idea clearly in the PR description. After reading the code, I think the idea is, for any predicate `p`, we can rewrite it to `p'` that only contains partition columns.

[GitHub] spark pull request #22714: [SPARK-25720][WEBUI] Support auto refresh page fo...

2018-10-15 Thread shahidki31
Github user shahidki31 closed the pull request at: https://github.com/apache/spark/pull/22714 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22714: [SPARK-25720][WEBUI] Support auto refresh page for the W...

2018-10-15 Thread shahidki31
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/22714 I am closing the PR, since already one PR is there for the webui auto refresh. Thanks. --- - To unsubscribe, e-mail: reviews-

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-10-15 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22379 I wouldn't too, but there's no way for using `schema_of_csv` otherwise .. --- - To unsubscribe, e-mail: reviews-unsubsc

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-10-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22379 I wouldn't make schema a `Columm`. `Column` means it's dynamic, while I think it should be a static value, to simplify the implementation. --- ---

[GitHub] spark issue #22720: [SPARK-25730][K8S] Delete executor pods from kubernetes ...

2018-10-15 Thread mikekap
Github user mikekap commented on the issue: https://github.com/apache/spark/pull/22720 @felixcheung opened a jira issue at https://issues.apache.org/jira/browse/SPARK-25730 --- - To unsubscribe, e-mail: reviews-unsu

[GitHub] spark issue #22723: [SPARK-25729][CORE]It is better to replace `minPartition...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22723 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22723: [SPARK-25729][CORE]It is better to replace `minPartition...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22723 **[Test build #97375 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97375/testReport)** for PR 22723 at commit [`b80bf66`](https://github.com/apache/spark/commit/b8

[GitHub] spark issue #22723: [SPARK-25729][CORE]It is better to replace `minPartition...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3970/

[GitHub] spark pull request #22723: [SPARK-25729][CORE]It is better to replace `minPa...

2018-10-15 Thread 10110346
GitHub user 10110346 opened a pull request: https://github.com/apache/spark/pull/22723 [SPARK-25729][CORE]It is better to replace `minPartitions` with `defaultParallelism` , when `minPartitions` is less than `defaultParallelism` ## What changes were proposed in this pull request?

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22721 **[Test build #97374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97374/testReport)** for PR 22721 at commit [`8a7f4af`](https://github.com/apache/spark/commit/8a

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3969/

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-10-15 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22379 Like .. ``` val csvData = "a,1,2" from_csv(column, schema = schema_of_csv(lit(csvData))) ``` --- - To

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22721 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97371/ Test FAILed. ---

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22721 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #22561: [SPARK-25548][SQL]In the PruneFileSourcePartitions optim...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22561 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97373/ Test FAILed. ---

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22721 **[Test build #97371 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97371/testReport)** for PR 22721 at commit [`8a7f4af`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #22561: [SPARK-25548][SQL]In the PruneFileSourcePartitions optim...

2018-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22561 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22561: [SPARK-25548][SQL]In the PruneFileSourcePartitions optim...

2018-10-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22561 **[Test build #97373 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97373/testReport)** for PR 22561 at commit [`a13fe23`](https://github.com/apache/spark/commit/a

<    1   2   3   4   5   6