[GitHub] [spark] SparkQA commented on pull request #30097: [SPARK-33140][SQL] remove SQLConf and SparkSession in all sub-class of Rule[QueryPlan]

2020-10-26 Thread GitBox
SparkQA commented on pull request #30097: URL: https://github.com/apache/spark/pull/30097#issuecomment-716575575 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34885/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716574140 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716574140 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
SparkQA removed a comment on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716490470 **[Test build #130282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130282/testReport)** for PR 30149 at commit

[GitHub] [spark] SparkQA commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
SparkQA commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716572355 **[Test build #130282 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130282/testReport)** for PR 30149 at commit

[GitHub] [spark] gaborgsomogyi commented on a change in pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore

2020-10-26 Thread GitBox
gaborgsomogyi commented on a change in pull request #26935: URL: https://github.com/apache/spark/pull/26935#discussion_r511974992 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ## @@ -89,16 +116,16 @@ trait StateStore {

[GitHub] [spark] gaborgsomogyi commented on a change in pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore

2020-10-26 Thread GitBox
gaborgsomogyi commented on a change in pull request #26935: URL: https://github.com/apache/spark/pull/26935#discussion_r511989317 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ## @@ -197,15 +212,26 @@

[GitHub] [spark] gaborgsomogyi commented on a change in pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore

2020-10-26 Thread GitBox
gaborgsomogyi commented on a change in pull request #26935: URL: https://github.com/apache/spark/pull/26935#discussion_r511816408 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ## @@ -81,6 +74,40 @@ trait StateStore {

[GitHub] [spark] SparkQA commented on pull request #30147: [SPARK-33240][SQL] Fail fast when fails to instantiate configured v2 session catalog

2020-10-26 Thread GitBox
SparkQA commented on pull request #30147: URL: https://github.com/apache/spark/pull/30147#issuecomment-716570681 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34889/

[GitHub] [spark] SparkQA commented on pull request #28363: [SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files

2020-10-26 Thread GitBox
SparkQA commented on pull request #28363: URL: https://github.com/apache/spark/pull/28363#issuecomment-716569398 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34888/

[GitHub] [spark] SparkQA commented on pull request #30151: [WIP][SPARK-33223][SS][UI]Structured Streaming Web UI state information

2020-10-26 Thread GitBox
SparkQA commented on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-716565253 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34886/

[GitHub] [spark] SparkQA commented on pull request #29970: [SPARK-33087][SQL] DataFrameWriterV2 should delegate table resolution to the analyzer

2020-10-26 Thread GitBox
SparkQA commented on pull request #29970: URL: https://github.com/apache/spark/pull/29970#issuecomment-716563096 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34887/

[GitHub] [spark] SparkQA commented on pull request #30097: [SPARK-33140][SQL] remove SQLConf and SparkSession in all sub-class of Rule[QueryPlan]

2020-10-26 Thread GitBox
SparkQA commented on pull request #30097: URL: https://github.com/apache/spark/pull/30097#issuecomment-716562140 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34885/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716538795 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716538787 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716538666 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] tanelk commented on pull request #30134: [SPARK-33225][SQL] Extract AliasHelper trait

2020-10-26 Thread GitBox
tanelk commented on pull request #30134: URL: https://github.com/apache/spark/pull/30134#issuecomment-716554760 > Have you checked all the place where `Alias`s are used in analyzer/optimizer? There were some places, but I did not feel that they deserved to get a method extracted.

[GitHub] [spark] SparkQA commented on pull request #30134: [SPARK-33225][SQL] Extract AliasHelper trait

2020-10-26 Thread GitBox
SparkQA commented on pull request #30134: URL: https://github.com/apache/spark/pull/30134#issuecomment-716551877 **[Test build #130292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130292/testReport)** for PR 30134 at commit

[GitHub] [spark] tanelk commented on a change in pull request #30134: [SPARK-33225][SQL] Extract AliasHelper trait

2020-10-26 Thread GitBox
tanelk commented on a change in pull request #30134: URL: https://github.com/apache/spark/pull/30134#discussion_r511962300 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AliasHelper.scala ## @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache

[GitHub] [spark] SparkQA commented on pull request #29414: [SPARK-32106][SQL] Implement script transform in sql/core

2020-10-26 Thread GitBox
SparkQA commented on pull request #29414: URL: https://github.com/apache/spark/pull/29414#issuecomment-716546987 **[Test build #130291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130291/testReport)** for PR 29414 at commit

[GitHub] [spark] SparkQA commented on pull request #30145: [SPARK-33233][SQL]CUBE/ROLLUP/GROUPING SETS support GROUP BY ordinal

2020-10-26 Thread GitBox
SparkQA commented on pull request #30145: URL: https://github.com/apache/spark/pull/30145#issuecomment-716546948 **[Test build #130290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130290/testReport)** for PR 30145 at commit

[GitHub] [spark] steveloughran commented on pull request #30019: [SPARK-33135][CORE] Use listLocatedStatus from FileSystem implementations

2020-10-26 Thread GitBox
steveloughran commented on pull request #30019: URL: https://github.com/apache/spark/pull/30019#issuecomment-716547112 @sunchao if you do want backporting, probably easiest if you lift it so I can review, though I could probably abuse the "I'm cherry picking a subset of a larger patch"

[GitHub] [spark] AmplabJenkins commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716545909 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
SparkQA commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716545894 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34884/

[GitHub] [spark] maropu commented on a change in pull request #30093: [SPARK-33183][SQL] Fix EliminateSorts bug when removing global sorts

2020-10-26 Thread GitBox
maropu commented on a change in pull request #30093: URL: https://github.com/apache/spark/pull/30093#discussion_r511958568 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -1242,6 +1242,13 @@ object SQLConf { .booleanConf

[GitHub] [spark] steveloughran commented on pull request #30141: [SPARK-33230][SQL] Hadoop committers to get unique job ID in "spark.sql.sources.writeJobUUID"

2020-10-26 Thread GitBox
steveloughran commented on pull request #30141: URL: https://github.com/apache/spark/pull/30141#issuecomment-716545472 @dongjoon-hyun > Do you think you can add a test case, @steveloughran ? not easily. Would need a new Hadoop committer (subclass of FileOutputCommitter easiest)

[GitHub] [spark] AngersZhuuuu commented on pull request #29414: [SPARK-32106][SQL] Implement script transform in sql/core

2020-10-26 Thread GitBox
AngersZh commented on pull request #29414: URL: https://github.com/apache/spark/pull/29414#issuecomment-716544979 also ping @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AngersZhuuuu commented on pull request #29414: [SPARK-32106][SQL] Implement script transform in sql/core

2020-10-26 Thread GitBox
AngersZh commented on pull request #29414: URL: https://github.com/apache/spark/pull/29414#issuecomment-716544630 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30145: [SPARK-33233][SQL]CUBE/ROLLUP/GROUPING SETS support GROUP BY ordinal

2020-10-26 Thread GitBox
AngersZh commented on a change in pull request #30145: URL: https://github.com/apache/spark/pull/30145#discussion_r511957287 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -3691,6 +3691,22 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] gaborgsomogyi commented on pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore

2020-10-26 Thread GitBox
gaborgsomogyi commented on pull request #26935: URL: https://github.com/apache/spark/pull/26935#issuecomment-716544437 Thanks for the explanation! Your point makes sense (`read-only state only copies the specific version of state from loaded map, which doesn't involve any file

[GitHub] [spark] AngersZhuuuu commented on pull request #29421: [SPARK-32388][SQL] TRANSFORM with schema-less mode should keep the same with hive

2020-10-26 Thread GitBox
AngersZh commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-716544398 ping @HyukjinKwon can merge now. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] steveloughran commented on pull request #30141: [SPARK-33230][SQL] Hadoop committers to get unique job ID in "spark.sql.sources.writeJobUUID"

2020-10-26 Thread GitBox
steveloughran commented on pull request #30141: URL: https://github.com/apache/spark/pull/30141#issuecomment-716544258 @rdblue I am going to add two things to the committers 1. Autogenerate in job setup. More specifically -generate in constructor, and in task setup, fail if that

[GitHub] [spark] gaborgsomogyi commented on pull request #30151: [WIP][SPARK-33223][SS][UI]Structured Streaming Web UI state information

2020-10-26 Thread GitBox
gaborgsomogyi commented on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-716543090 Sure, just wanted to check the jenkins tests. When the PR is polished I'll add the UI snapshot. This is

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30145: [SPARK-33233][SQL]CUBE/ROLLUP/GROUPING SETS support GROUP BY ordinal

2020-10-26 Thread GitBox
AngersZh commented on a change in pull request #30145: URL: https://github.com/apache/spark/pull/30145#discussion_r511955190 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1775,17 +1776,29 @@ class Analyzer(

[GitHub] [spark] steveloughran commented on a change in pull request #30141: [SPARK-33230][SQL] Hadoop committers to get unique job ID in "spark.sql.sources.writeJobUUID"

2020-10-26 Thread GitBox
steveloughran commented on a change in pull request #30141: URL: https://github.com/apache/spark/pull/30141#discussion_r511955017 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala ## @@ -164,6 +164,10 @@ object

[GitHub] [spark] attilapiros commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

2020-10-26 Thread GitBox
attilapiros commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r511954637 ## File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java ## @@ -0,0 +1,462 @@ +/* + *

[GitHub] [spark] HyukjinKwon commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
HyukjinKwon commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716542326 It does require to rewrite a lot of docs. I am thinking that in most cases there would not be too much diff except when rewriting parameters. By doing that, it can

[GitHub] [spark] SparkQA commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
SparkQA commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716540471 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34880/

[GitHub] [spark] AmplabJenkins commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716540487 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
SparkQA commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716538777 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34881/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716538486 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
SparkQA commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716538641 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34882/

[GitHub] [spark] AmplabJenkins commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716538787 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
SparkQA commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716538464 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34883/

[GitHub] [spark] AmplabJenkins commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716538486 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716538666 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30097: [SPARK-33140][SQL] remove SQLConf and SparkSession in all sub-class of Rule[QueryPlan]

2020-10-26 Thread GitBox
SparkQA commented on pull request #30097: URL: https://github.com/apache/spark/pull/30097#issuecomment-716538184 **[Test build #130289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130289/testReport)** for PR 30097 at commit

[GitHub] [spark] SparkQA commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
SparkQA commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716530027 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34884/

[GitHub] [spark] cloud-fan commented on a change in pull request #28026: [SPARK-31257][SQL] Unify create table syntax

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #28026: URL: https://github.com/apache/spark/pull/28026#discussion_r511939078 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala ## @@ -295,18 +295,61 @@ private[sql] object

[GitHub] [spark] SparkQA commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
SparkQA commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716529288 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34882/

[GitHub] [spark] cloud-fan commented on a change in pull request #28026: [SPARK-31257][SQL] Unify create table syntax

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #28026: URL: https://github.com/apache/spark/pull/28026#discussion_r511939078 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala ## @@ -295,18 +295,61 @@ private[sql] object

[GitHub] [spark] SparkQA commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP

2020-10-26 Thread GitBox
SparkQA commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-716527908 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34880/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-716526821 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-716526821 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
SparkQA commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716526674 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34883/

[GitHub] [spark] HeartSaVioR commented on pull request #30151: [WIP][SPARK-33223][SS][UI]Structured Streaming Web UI state information

2020-10-26 Thread GitBox
HeartSaVioR commented on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-716526327 Could you please paste the screenshots here as this PR addresses the UI change? That would help to see the proposal easier. Thanks in advance!

[GitHub] [spark] SparkQA removed a comment on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

2020-10-26 Thread GitBox
SparkQA removed a comment on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-716377176 **[Test build #130273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130273/testReport)** for PR 24173 at commit

[GitHub] [spark] SparkQA commented on pull request #30147: [SPARK-33240][SQL] Fail fast when fails to instantiate configured v2 session catalog

2020-10-26 Thread GitBox
SparkQA commented on pull request #30147: URL: https://github.com/apache/spark/pull/30147#issuecomment-716525618 **[Test build #130288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130288/testReport)** for PR 30147 at commit

[GitHub] [spark] SparkQA commented on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

2020-10-26 Thread GitBox
SparkQA commented on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-716525683 **[Test build #130273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130273/testReport)** for PR 24173 at commit

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30145: [SPARK-33233][SQL]CUBE/ROLLUP/GROUPING SETS support GROUP BY ordinal

2020-10-26 Thread GitBox
AngersZh commented on a change in pull request #30145: URL: https://github.com/apache/spark/pull/30145#discussion_r511932497 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -244,6 +244,7 @@ class Analyzer(

[GitHub] [spark] SparkQA commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
SparkQA commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716522333 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34881/

[GitHub] [spark] SparkQA commented on pull request #28363: [SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files

2020-10-26 Thread GitBox
SparkQA commented on pull request #28363: URL: https://github.com/apache/spark/pull/28363#issuecomment-716522115 **[Test build #130287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130287/testReport)** for PR 28363 at commit

[GitHub] [spark] gengliangwang closed pull request #30119: [SPARK-33204][UI]The 'Event Timeline' area cannot be opened when a spark application has some failed jobs.

2020-10-26 Thread GitBox
gengliangwang closed pull request #30119: URL: https://github.com/apache/spark/pull/30119 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] cloud-fan commented on a change in pull request #30093: [SPARK-33183][SQL] Fix EliminateSorts bug when removing global sorts

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #30093: URL: https://github.com/apache/spark/pull/30093#discussion_r511928699 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -1242,6 +1242,13 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan commented on a change in pull request #30134: [SPARK-33225][SQL] Extract AliasHelper trait

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #30134: URL: https://github.com/apache/spark/pull/30134#discussion_r511928131 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AliasHelper.scala ## @@ -0,0 +1,100 @@ +/* + * Licensed to the

[GitHub] [spark] cloud-fan commented on pull request #30134: [SPARK-33225][SQL] Extract AliasHelper trait

2020-10-26 Thread GitBox
cloud-fan commented on pull request #30134: URL: https://github.com/apache/spark/pull/30134#issuecomment-716519788 looks fine This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HeartSaVioR edited a comment on pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore

2020-10-26 Thread GitBox
HeartSaVioR edited a comment on pull request #26935: URL: https://github.com/apache/spark/pull/26935#issuecomment-716517527 I don't only intend to restrict the API usage on read-only path. The purpose of separation is to allow read-only implementation to be differentiated. e.g. In HDFS

[GitHub] [spark] gengliangwang commented on pull request #30119: [SPARK-33204 ][UI]The 'Event Timeline' area cannot be opened when a spark application has some failed jobs.

2020-10-26 Thread GitBox
gengliangwang commented on pull request #30119: URL: https://github.com/apache/spark/pull/30119#issuecomment-716519394 Thanks, merging to master This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29970: [SPARK-33087][SQL] DataFrameWriterV2 should delegate table resolution to the analyzer

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #29970: URL: https://github.com/apache/spark/pull/29970#issuecomment-715668894 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] cloud-fan commented on a change in pull request #30134: [SPARK-33225][SQL] Extract AliasHelper trait

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #30134: URL: https://github.com/apache/spark/pull/30134#discussion_r511926874 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AliasHelper.scala ## @@ -0,0 +1,100 @@ +/* + * Licensed to the

[GitHub] [spark] SparkQA commented on pull request #29970: [SPARK-33087][SQL] DataFrameWriterV2 should delegate table resolution to the analyzer

2020-10-26 Thread GitBox
SparkQA commented on pull request #29970: URL: https://github.com/apache/spark/pull/29970#issuecomment-716518254 **[Test build #130286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130286/testReport)** for PR 29970 at commit

[GitHub] [spark] HeartSaVioR commented on pull request #28363: [SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files

2020-10-26 Thread GitBox
HeartSaVioR commented on pull request #28363: URL: https://github.com/apache/spark/pull/28363#issuecomment-716518444 retest this, please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] HeartSaVioR commented on pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore

2020-10-26 Thread GitBox
HeartSaVioR commented on pull request #26935: URL: https://github.com/apache/spark/pull/26935#issuecomment-716517527 I don't only intend to restrict the API usage on read-only path. The purpose of separation is to allow read-only implementation to be differentiated. e.g. In HDFS state

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29421: [SPARK-32388][SQL] TRANSFORM with schema-less mode should keep the same with hive

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-716515589 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] attilapiros commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

2020-10-26 Thread GitBox
attilapiros commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r511923750 ## File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ## @@ -363,4 +363,26 @@ public boolean

[GitHub] [spark] maropu commented on a change in pull request #30145: [SPARK-33233][SQL]CUBE/ROLLUP/GROUPING SETS support GROUP BY ordinal

2020-10-26 Thread GitBox
maropu commented on a change in pull request #30145: URL: https://github.com/apache/spark/pull/30145#discussion_r511923270 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1775,17 +1776,29 @@ class Analyzer( //

[GitHub] [spark] AmplabJenkins commented on pull request #29421: [SPARK-32388][SQL] TRANSFORM with schema-less mode should keep the same with hive

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-716515589 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #29800: URL: https://github.com/apache/spark/pull/29800#discussion_r511922381 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala ## @@ -151,10 +169,69 @@ final class

[GitHub] [spark] SparkQA removed a comment on pull request #29421: [SPARK-32388][SQL] TRANSFORM with schema-less mode should keep the same with hive

2020-10-26 Thread GitBox
SparkQA removed a comment on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-716356599 **[Test build #130268 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130268/testReport)** for PR 29421 at commit

[GitHub] [spark] maropu commented on a change in pull request #30145: [SPARK-33233][SQL]CUBE/ROLLUP/GROUPING SETS support GROUP BY ordinal

2020-10-26 Thread GitBox
maropu commented on a change in pull request #30145: URL: https://github.com/apache/spark/pull/30145#discussion_r511921824 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -244,6 +244,7 @@ class Analyzer(

[GitHub] [spark] SparkQA commented on pull request #29421: [SPARK-32388][SQL] TRANSFORM with schema-less mode should keep the same with hive

2020-10-26 Thread GitBox
SparkQA commented on pull request #29421: URL: https://github.com/apache/spark/pull/29421#issuecomment-716514610 **[Test build #130268 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130268/testReport)** for PR 29421 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #29800: URL: https://github.com/apache/spark/pull/29800#discussion_r511921186 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala ## @@ -151,10 +169,69 @@ final class

[GitHub] [spark] cloud-fan commented on a change in pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #29800: URL: https://github.com/apache/spark/pull/29800#discussion_r511919183 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala ## @@ -151,10 +169,69 @@ final class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30150: [SPARK-32188][PYTHON][DOCS][FOLLOW-UP] Document Column APIs in API reference

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #30150: URL: https://github.com/apache/spark/pull/30150#issuecomment-716511023 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] maropu commented on a change in pull request #30145: [SPARK-33233][SQL]CUBE/ROLLUP/GROUPING SETS support GROUP BY ordinal

2020-10-26 Thread GitBox
maropu commented on a change in pull request #30145: URL: https://github.com/apache/spark/pull/30145#discussion_r511918123 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -3691,6 +3691,22 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] AmplabJenkins commented on pull request #30150: [SPARK-32188][PYTHON][DOCS][FOLLOW-UP] Document Column APIs in API reference

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #30150: URL: https://github.com/apache/spark/pull/30150#issuecomment-716511023 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30150: [SPARK-32188][PYTHON][DOCS][FOLLOW-UP] Document Column APIs in API reference

2020-10-26 Thread GitBox
SparkQA commented on pull request #30150: URL: https://github.com/apache/spark/pull/30150#issuecomment-716511005 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34878/

[GitHub] [spark] maropu commented on a change in pull request #30145: [SPARK-33233][SQL]CUBE/ROLLUP/GROUPING SETS support GROUP BY ordinal

2020-10-26 Thread GitBox
maropu commented on a change in pull request #30145: URL: https://github.com/apache/spark/pull/30145#discussion_r511917844 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -244,6 +244,7 @@ class Analyzer(

[GitHub] [spark] maropu commented on pull request #30145: [SPARK-33233][SQL]CUBE/ROLLUP/GROUPING SETS support GROUP BY ordinal

2020-10-26 Thread GitBox
maropu commented on pull request #30145: URL: https://github.com/apache/spark/pull/30145#issuecomment-716510003 Please fill the description?

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716508213 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716508213 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30138: [SPARK-33075][SQL] Enable auto bucketed scan by default (disable only for cached query)

2020-10-26 Thread GitBox
AmplabJenkins removed a comment on pull request #30138: URL: https://github.com/apache/spark/pull/30138#issuecomment-716507682 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30149: [SPARK-33243][PYTHON][BUILD] Add numpydoc into documentation dependency

2020-10-26 Thread GitBox
SparkQA commented on pull request #30149: URL: https://github.com/apache/spark/pull/30149#issuecomment-716508197 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34879/

[GitHub] [spark] cloud-fan commented on a change in pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #29800: URL: https://github.com/apache/spark/pull/29800#discussion_r511914182 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala ## @@ -141,6 +141,24 @@ final class

[GitHub] [spark] cloud-fan commented on a change in pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #29800: URL: https://github.com/apache/spark/pull/29800#discussion_r511914182 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala ## @@ -141,6 +141,24 @@ final class

[GitHub] [spark] AmplabJenkins commented on pull request #30138: [SPARK-33075][SQL] Enable auto bucketed scan by default (disable only for cached query)

2020-10-26 Thread GitBox
AmplabJenkins commented on pull request #30138: URL: https://github.com/apache/spark/pull/30138#issuecomment-716507682 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #29800: URL: https://github.com/apache/spark/pull/29800#discussion_r511913426 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExecBase.scala ## @@ -172,17 +179,41 @@ trait WindowExecBase extends

[GitHub] [spark] SparkQA removed a comment on pull request #30138: [SPARK-33075][SQL] Enable auto bucketed scan by default (disable only for cached query)

2020-10-26 Thread GitBox
SparkQA removed a comment on pull request #30138: URL: https://github.com/apache/spark/pull/30138#issuecomment-716360810 **[Test build #130270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130270/testReport)** for PR 30138 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

2020-10-26 Thread GitBox
cloud-fan commented on a change in pull request #29800: URL: https://github.com/apache/spark/pull/29800#discussion_r511912579 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExecBase.scala ## @@ -136,8 +136,15 @@ trait WindowExecBase extends

[GitHub] [spark] maropu commented on pull request #30134: [SPARK-33225][SQL] Extract AliasHelper trait

2020-10-26 Thread GitBox
maropu commented on pull request #30134: URL: https://github.com/apache/spark/pull/30134#issuecomment-716506505 NOTE: This PR comes from the comment in https://github.com/apache/spark/pull/30018#discussion_r510101973 cc: @cloud-fan @viirya @tanelk Thanks for the work. Have you

<    1   2   3   4   5   6   7   8   >