[GitHub] [spark] HyukjinKwon commented on pull request #28372: [SPARK-31578][R] Vectorize schema validation for arrow in types.R

2020-04-27 Thread GitBox
HyukjinKwon commented on pull request #28372: URL: https://github.com/apache/spark/pull/28372#issuecomment-620332859 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] SparkQA commented on pull request #28375: [SPARK-30282][SQL][FOLLOWUP] SHOW TBLPROPERTIES should support views

2020-04-27 Thread GitBox
SparkQA commented on pull request #28375: URL: https://github.com/apache/spark/pull/28375#issuecomment-620332881 **[Test build #121925 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121925/testReport)** for PR 28375 at commit

[GitHub] [spark] baohe-zhang opened a new pull request #28378: [SPARK-31584][WebUI] Fix NullPointerException when parsing event log with InMemoryStore

2020-04-27 Thread GitBox
baohe-zhang opened a new pull request #28378: URL: https://github.com/apache/spark/pull/28378 ### What changes were proposed in this pull request? https://github.com/apache/spark/pull/27716 introduced parent index for InMemoryStore. When the method "deleteParentIndex(Object key)" in

[GitHub] [spark] MichaelChirico commented on pull request #28367: [SPARK-31573][R] Apply fixed=TRUE as appropriate to regex usage in R

2020-04-27 Thread GitBox
MichaelChirico commented on pull request #28367: URL: https://github.com/apache/spark/pull/28367#issuecomment-620332664 same transient failure for linter here... This is an automated message from the Apache Git Service. To

[GitHub] [spark] HyukjinKwon commented on pull request #28374: [SPARK-31581][R] Replaces paste(sep="") to paste0

2020-04-27 Thread GitBox
HyukjinKwon commented on pull request #28374: URL: https://github.com/apache/spark/pull/28374#issuecomment-620332057 Oops, I missed that the title linked a wrong JIRA. This is an automated message from the Apache Git

[GitHub] [spark] MichaelChirico commented on pull request #28372: [SPARK-31578][R] Vectorize schema validation for arrow in types.R

2020-04-27 Thread GitBox
MichaelChirico commented on pull request #28372: URL: https://github.com/apache/spark/pull/28372#issuecomment-620331958 linter failure looks transient to me: > E: The repository 'https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Release' does not have a Release file.

[GitHub] [spark] HyukjinKwon commented on pull request #28372: [SPARK-31578][R] Vectorize schema validation for arrow in types.R

2020-04-27 Thread GitBox
HyukjinKwon commented on pull request #28372: URL: https://github.com/apache/spark/pull/28372#issuecomment-620331533 I am going to merge. The tests are already passed, and I don't believe removing the return causes any test or build to break.

[GitHub] [spark] HyukjinKwon commented on pull request #28374: [SPARK-31568][R] Replaces paste(sep="") to paste0

2020-04-27 Thread GitBox
HyukjinKwon commented on pull request #28374: URL: https://github.com/apache/spark/pull/28374#issuecomment-620331318 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] dongjoon-hyun commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
dongjoon-hyun commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620330778 Merged to master/3.0. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28194: [SPARK-31372][SQL][TEST] Display expression schema for double check.

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28194: URL: https://github.com/apache/spark/pull/28194#issuecomment-620329525 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28194: [SPARK-31372][SQL][TEST] Display expression schema for double check.

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #28194: URL: https://github.com/apache/spark/pull/28194#issuecomment-620329525 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28194: [SPARK-31372][SQL][TEST] Display expression schema for double check.

2020-04-27 Thread GitBox
SparkQA commented on pull request #28194: URL: https://github.com/apache/spark/pull/28194#issuecomment-620329099 **[Test build #121940 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121940/testReport)** for PR 28194 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620328344 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620328344 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
SparkQA removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620260494 **[Test build #121928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121928/testReport)** for PR 28373 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative t

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-620327352 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
SparkQA commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620327637 **[Test build #121928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121928/testReport)** for PR 28373 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative task

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-620327352 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative task

2020-04-27 Thread GitBox
SparkQA commented on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-620327086 **[Test build #121939 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121939/testReport)** for PR 26339 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620326566 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620326561 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620326561 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] dongjoon-hyun commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
dongjoon-hyun commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620326265 The above one is passed with `-Phive-1.2`. This is an automated message from the Apache Git Service. To

[GitHub] [spark] turboFei commented on a change in pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative ta

2020-04-27 Thread GitBox
turboFei commented on a change in pull request #26339: URL: https://github.com/apache/spark/pull/26339#discussion_r416265024 ## File path: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala ## @@ -236,13 +272,33 @@ class

[GitHub] [spark] SparkQA removed a comment on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
SparkQA removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620257651 **[Test build #121927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121927/testReport)** for PR 28373 at commit

[GitHub] [spark] SparkQA commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
SparkQA commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620325906 **[Test build #121927 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121927/testReport)** for PR 28373 at commit

[GitHub] [spark] venkata91 commented on a change in pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative t

2020-04-27 Thread GitBox
venkata91 commented on a change in pull request #26339: URL: https://github.com/apache/spark/pull/26339#discussion_r416259866 ## File path: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala ## @@ -236,13 +272,33 @@ class

[GitHub] [spark] venkata91 commented on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative task

2020-04-27 Thread GitBox
venkata91 commented on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-620320985 LGTM @turboFei . Except for the build checks issues. @cloud-fan @Ngone51 @dongjoon-hyun Can some one please review this change? This is critical for us in production.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #27627: URL: https://github.com/apache/spark/pull/27627#issuecomment-620316848 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28372: [SPARK-31578][R] Vectorize schema validation for arrow in types.R

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28372: URL: https://github.com/apache/spark/pull/28372#issuecomment-620316932 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #27627: URL: https://github.com/apache/spark/pull/27627#issuecomment-620316848 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #28372: [SPARK-31578][R] Vectorize schema validation for arrow in types.R

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #28372: URL: https://github.com/apache/spark/pull/28372#issuecomment-620316932 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28372: [SPARK-31578][R] Vectorize schema validation for arrow in types.R

2020-04-27 Thread GitBox
SparkQA commented on pull request #28372: URL: https://github.com/apache/spark/pull/28372#issuecomment-620316474 **[Test build #121937 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121937/testReport)** for PR 28372 at commit

[GitHub] [spark] SparkQA commented on pull request #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow

2020-04-27 Thread GitBox
SparkQA commented on pull request #27627: URL: https://github.com/apache/spark/pull/27627#issuecomment-620316504 **[Test build #121938 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121938/testReport)** for PR 27627 at commit

[GitHub] [spark] HeartSaVioR commented on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
HeartSaVioR commented on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620314357 That said, I'd support porting back this to 2.4 as it fixes the wrong code. This change might impact broader audiences (so technically saying the PR/commit title doesn't

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28372: [SPARK-31578][R] Vectorize schema validation for arrow in types.R

2020-04-27 Thread GitBox
HyukjinKwon commented on a change in pull request #28372: URL: https://github.com/apache/spark/pull/28372#discussion_r416251165 ## File path: R/pkg/R/types.R ## @@ -94,27 +94,23 @@ checkSchemaInArrow <- function(schema) { } # Both cases below produce a corrupt value

[GitHub] [spark] dongjoon-hyun commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
dongjoon-hyun commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620313129 Thank you for review and approval, @holdenk ! This is an automated message from the Apache Git Service.

[GitHub] [spark] dongjoon-hyun commented on pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))

2020-04-27 Thread GitBox
dongjoon-hyun commented on pull request #27803: URL: https://github.com/apache/spark/pull/27803#issuecomment-620312644 Oh, sure. I'll take a look tonight. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] MichaelChirico commented on a change in pull request #28372: [SPARK-31578][R] Vectorize schema validation for arrow in types.R

2020-04-27 Thread GitBox
MichaelChirico commented on a change in pull request #28372: URL: https://github.com/apache/spark/pull/28372#discussion_r416249467 ## File path: R/pkg/R/types.R ## @@ -94,27 +94,23 @@ checkSchemaInArrow <- function(schema) { } # Both cases below produce a corrupt value

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28374: [SPARK-31568][R] Replaces paste(sep="") to paste0

2020-04-27 Thread GitBox
HyukjinKwon commented on a change in pull request #28374: URL: https://github.com/apache/spark/pull/28374#discussion_r416248403 ## File path: R/pkg/R/mllib_classification.R ## @@ -338,7 +338,7 @@ setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula")

[GitHub] [spark] MichaelChirico commented on a change in pull request #28374: [SPARK-31568][R] paste0 replaces paste(sep="")

2020-04-27 Thread GitBox
MichaelChirico commented on a change in pull request #28374: URL: https://github.com/apache/spark/pull/28374#discussion_r416247438 ## File path: R/pkg/R/mllib_classification.R ## @@ -338,7 +338,7 @@ setMethod("spark.logit", signature(data = "SparkDataFrame", formula =

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28372: [SPARK-31578][R] Vectorize schema validation for arrow in types.R

2020-04-27 Thread GitBox
HyukjinKwon commented on a change in pull request #28372: URL: https://github.com/apache/spark/pull/28372#discussion_r416247772 ## File path: R/pkg/R/types.R ## @@ -94,27 +94,23 @@ checkSchemaInArrow <- function(schema) { } # Both cases below produce a corrupt value

[GitHub] [spark] dongjoon-hyun commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
dongjoon-hyun commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620311386 Thank you for review and approval, @HyukjinKwon . This is an automated message from the Apache Git

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28374: [SPARK-31568][R] paste0 replaces paste(sep="")

2020-04-27 Thread GitBox
HyukjinKwon commented on a change in pull request #28374: URL: https://github.com/apache/spark/pull/28374#discussion_r416247038 ## File path: R/pkg/R/mllib_classification.R ## @@ -338,7 +338,7 @@ setMethod("spark.logit", signature(data = "SparkDataFrame", formula = "formula")

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations

2020-04-27 Thread GitBox
dongjoon-hyun edited a comment on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620293684 All Scala/Java/Python test passed, but it's timeouted at R testing. I ran the R UT manually. ``` ══ testthat results

[GitHub] [spark] HeartSaVioR commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost

2020-04-27 Thread GitBox
HeartSaVioR commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620310834 Sorry for maybe out-of-topic, but I'm seeing that column metadata may have more issues due to not properly defined its usage and how it should be handled. I'm

[GitHub] [spark] HeartSaVioR edited a comment on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost

2020-04-27 Thread GitBox
HeartSaVioR edited a comment on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620310834 Sorry for maybe out-of-topic, but I'm seeing that column metadata may have more issues due to not properly defined its usage and how it should be handled. I'm

[GitHub] [spark] HeartSaVioR edited a comment on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
HeartSaVioR edited a comment on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620309415 > It sounds like the metadata is not available for TimeWindow expression when the function name(alias: String) copies the metadata to explicitMetadata. Thus, we do

[GitHub] [spark] HyukjinKwon commented on pull request #28367: [SPARK-31573][R] Apply fixed=TRUE as appropriate to regex usage in R

2020-04-27 Thread GitBox
HyukjinKwon commented on pull request #28367: URL: https://github.com/apache/spark/pull/28367#issuecomment-620309744 Let me rerun the AppVeyor build to make sure. This is an automated message from the Apache Git Service. To

[GitHub] [spark] HeartSaVioR commented on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
HeartSaVioR commented on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620309415 > It sounds like the metadata is not available for TimeWindow expression when the function name(alias: String) copies the metadata to explicitMetadata. Thus, we do not set

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620306513 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620306513 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
SparkQA commented on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620306258 **[Test build #121936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121936/testReport)** for PR 28377 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
dongjoon-hyun commented on a change in pull request #28377: URL: https://github.com/apache/spark/pull/28377#discussion_r416240326 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -1016,17 +1010,11 @@ class Column(val expr: Expression) extends

[GitHub] [spark] dongjoon-hyun commented on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
dongjoon-hyun commented on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620304478 Got it. You are right. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] gengliangwang commented on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation

2020-04-27 Thread GitBox
gengliangwang commented on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-620301870 I think either a separate PR or folding into #28208 is fine. Since #28208 is not merged yet, let's mark this one as WIP.

[GitHub] [spark] gatorsmile commented on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
gatorsmile commented on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620301600 Any other Expressions except `NamedExpression` has the field Metadata? This is an automated message from the

[GitHub] [spark] gengliangwang commented on a change in pull request #28208: [SPARK-31440][SQL] Improve SQL Rest API

2020-04-27 Thread GitBox
gengliangwang commented on a change in pull request #28208: URL: https://github.com/apache/spark/pull/28208#discussion_r416234324 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusStore.scala ## @@ -146,4 +146,6 @@ class

[GitHub] [spark] dongjoon-hyun commented on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
dongjoon-hyun commented on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620300747 @gatorsmile . Are you assuming `NamedExpression` only? This is an automated message from the Apache Git

[GitHub] [spark] gengliangwang commented on pull request #28208: [SPARK-31440][SQL] Improve SQL Rest API

2020-04-27 Thread GitBox
gengliangwang commented on pull request #28208: URL: https://github.com/apache/spark/pull/28208#issuecomment-620300958 Hi @erenavsarogullari , sorry for the late reply. I think we can output the whole SparkPlanGraph in json format: ``` nodes: [ {..}, {..}] edges: [{fromId:

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-620299880 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620299965 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620299965 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-620299880 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] gatorsmile commented on a change in pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
gatorsmile commented on a change in pull request #28377: URL: https://github.com/apache/spark/pull/28377#discussion_r416232467 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -1016,17 +1010,11 @@ class Column(val expr: Expression) extends Logging

[GitHub] [spark] SparkQA commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped

2020-04-27 Thread GitBox
SparkQA commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620299461 **[Test build #121935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121935/testReport)** for PR 28359 at commit

[GitHub] [spark] SparkQA commented on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned

2020-04-27 Thread GitBox
SparkQA commented on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-620299487 **[Test build #121934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121934/testReport)** for PR 28370 at commit

[GitHub] [spark] maropu commented on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned

2020-04-27 Thread GitBox
maropu commented on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-620298905 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] HeartSaVioR commented on a change in pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
HeartSaVioR commented on a change in pull request #28377: URL: https://github.com/apache/spark/pull/28377#discussion_r416230700 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -1016,17 +1010,11 @@ class Column(val expr: Expression) extends Logging

[GitHub] [spark] gatorsmile commented on a change in pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
gatorsmile commented on a change in pull request #28377: URL: https://github.com/apache/spark/pull/28377#discussion_r416230516 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -1016,17 +1010,11 @@ class Column(val expr: Expression) extends Logging

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620297211 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
dongjoon-hyun commented on a change in pull request #28377: URL: https://github.com/apache/spark/pull/28377#discussion_r416229855 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -1016,17 +1010,11 @@ class Column(val expr: Expression) extends

[GitHub] [spark] gengliangwang commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped

2020-04-27 Thread GitBox
gengliangwang commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620297439 Retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] gatorsmile edited a comment on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
gatorsmile edited a comment on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620296526 @dongjoon-hyun @holdenk My understanding is different from what you said above. It sounds like the metadata is not available for TimeWindow expression when

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620297203 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped

2020-04-27 Thread GitBox
SparkQA removed a comment on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620223140 **[Test build #121926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121926/testReport)** for PR 28359 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
dongjoon-hyun commented on a change in pull request #28377: URL: https://github.com/apache/spark/pull/28377#discussion_r416229855 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -1016,17 +1010,11 @@ class Column(val expr: Expression) extends

[GitHub] [spark] AmplabJenkins commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620297203 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped

2020-04-27 Thread GitBox
SparkQA commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620296891 **[Test build #121926 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121926/testReport)** for PR 28359 at commit

[GitHub] [spark] gatorsmile commented on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
gatorsmile commented on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620296526 @dongjoon-hyun @holdenk My understanding is different from what you said above. It sounds like the metadata is not available for TimeWindow expression when we

[GitHub] [spark] dbtsai commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath

2020-04-27 Thread GitBox
dbtsai commented on pull request #28376: URL: https://github.com/apache/spark/pull/28376#issuecomment-620296694 The other way we can do is changing `yarn.application.classpath` and `mapreduce.application.classpath` to an empty folder, but this is more hackey unfortunately.

[GitHub] [spark] holdenk commented on a change in pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
holdenk commented on a change in pull request #28373: URL: https://github.com/apache/spark/pull/28373#discussion_r416228152 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala ## @@ -320,4 +320,11 @@ class HiveOrcSourceSuite extends

[GitHub] [spark] maropu commented on a change in pull request #28194: [SPARK-31372][SQL][TEST] Display expression schema for double check.

2020-04-27 Thread GitBox
maropu commented on a change in pull request #28194: URL: https://github.com/apache/spark/pull/28194#discussion_r416227468 ## File path: sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala ## @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] gatorsmile commented on a change in pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
gatorsmile commented on a change in pull request #28377: URL: https://github.com/apache/spark/pull/28377#discussion_r416226948 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -1016,17 +1010,11 @@ class Column(val expr: Expression) extends Logging

[GitHub] [spark] maropu commented on a change in pull request #28368: [SPARK-31575][SQL] Synchronise global JVM security configuration modification

2020-04-27 Thread GitBox
maropu commented on a change in pull request #28368: URL: https://github.com/apache/spark/pull/28368#discussion_r416225641 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/DB2ConnectionProvider.scala ## @@ -48,7 +48,7 @@

[GitHub] [spark] dongjoon-hyun commented on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations

2020-04-27 Thread GitBox
dongjoon-hyun commented on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620293684 All Scala/Java/Python test passed, but it's timeouted R testing. I ran the R UT manually. ``` ══ testthat results

[GitHub] [spark] dongjoon-hyun commented on pull request #28377: [SPARK-27340][SS][2.4] Alias on TimeWindow expression may cause watermark metadata lost

2020-04-27 Thread GitBox
dongjoon-hyun commented on pull request #28377: URL: https://github.com/apache/spark/pull/28377#issuecomment-620293161 Yes. Correct. In `name` function, the metadata loss occurs. Without the metadata, some structured query fails at Analysis Stage, and another structure queries becomes

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27026: [SPARK-29596][Web UI] Task duration not updating for running tasks

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #27026: URL: https://github.com/apache/spark/pull/27026#issuecomment-620293018 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #27026: [SPARK-29596][Web UI] Task duration not updating for running tasks

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #27026: URL: https://github.com/apache/spark/pull/27026#issuecomment-620293018 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #27026: [SPARK-29596][Web UI] Task duration not updating for running tasks

2020-04-27 Thread GitBox
SparkQA commented on pull request #27026: URL: https://github.com/apache/spark/pull/27026#issuecomment-620292701 **[Test build #121933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121933/testReport)** for PR 27026 at commit

[GitHub] [spark] srowen commented on pull request #28376: [SPARK-31582] [Yarn] Being able to not populate Hadoop classpath

2020-04-27 Thread GitBox
srowen commented on pull request #28376: URL: https://github.com/apache/spark/pull/28376#issuecomment-620291609 I generally don't like yet another config option, but I don't object to this. Is there any simpler way to solve it?

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27026: [SPARK-29596][Web UI] Task duration not updating for running tasks

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #27026: URL: https://github.com/apache/spark/pull/27026#issuecomment-569298446 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] holdenk commented on pull request #27026: [SPARK-29596][Web UI] Task duration not updating for running tasks

2020-04-27 Thread GitBox
holdenk commented on pull request #27026: URL: https://github.com/apache/spark/pull/27026#issuecomment-620291254 Jenkins ok to test This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] AmplabJenkins removed a comment on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative t

2020-04-27 Thread GitBox
AmplabJenkins removed a comment on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-620290976 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] gatorsmile commented on a change in pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost

2020-04-27 Thread GitBox
gatorsmile commented on a change in pull request #28326: URL: https://github.com/apache/spark/pull/28326#discussion_r416221822 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ## @@ -1040,17 +1034,11 @@ class Column(val expr: Expression) extends Logging

[GitHub] [spark] AmplabJenkins commented on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative task

2020-04-27 Thread GitBox
AmplabJenkins commented on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-620290976 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative task

2020-04-27 Thread GitBox
SparkQA commented on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-620290577 **[Test build #121932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121932/testReport)** for PR 26339 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
dongjoon-hyun commented on a change in pull request #28373: URL: https://github.com/apache/spark/pull/28373#discussion_r416219047 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala ## @@ -320,4 +320,11 @@ class HiveOrcSourceSuite

[GitHub] [spark] holdenk commented on pull request #28343: [SPARK-31563][SQL] Fix failure of InSet.sql for collections of Catalyst's internal types

2020-04-27 Thread GitBox
holdenk commented on pull request #28343: URL: https://github.com/apache/spark/pull/28343#issuecomment-620289770 Thanks everyone :) This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10

2020-04-27 Thread GitBox
dongjoon-hyun commented on a change in pull request #28373: URL: https://github.com/apache/spark/pull/28373#discussion_r416219047 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala ## @@ -320,4 +320,11 @@ class HiveOrcSourceSuite

<    1   2   3   4   5   6   7   8   9   10   >