[GitHub] [spark] AmplabJenkins removed a comment on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
AmplabJenkins removed a comment on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-620274163 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121917/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
dongjoon-hyun edited a comment on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620274907 @zsxwing . For `branch-2.4`, shall we talk on the backporting PR? I didn't backport it directly because we need a further discussion about critical bug fix and the behavior change. BTW, without this, we cannot run long running Structure Streaming with State queries. @HeartSaVioR . Ya. Unfortunately, it looks like that. When this PR is made, I checked that the three commits has two authors and it still does, but merge-script seems to have a corner case and to miss it.. :( This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
dongjoon-hyun edited a comment on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620274907 @zsxwing . For `branch-2.4`, shall we talk on the backporting PR? I didn't backport it directly because we need a further discussion about critical bug fix and the behavior change. BTW, without this, we cannot run a long running Structure Streaming with State. @HeartSaVioR . Ya. Unfortunately, it looks like that. When this PR is made, I checked that the three commits has two authors and it still does, but merge-script seems to have a corner case and to miss it.. :( This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
AmplabJenkins removed a comment on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-620274146 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
dongjoon-hyun commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620274907 @zsxwing . For `branch-2.4`, shall we talk on the PR? I didn't backport it directly because we need a further discussion about critical bug fix and the behavior change. BTW, without this, we cannot run a long running Structure Streaming with State. @HeartSaVioR . Ya. Unfortunately, it looks like that. When this PR is made, I checked that the three commits has two authors and it still does, but merge-script seems to have a corner case and to miss it.. :( This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
AmplabJenkins commented on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-620274146 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
SparkQA commented on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-620273375 **[Test build #121917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121917/testReport)** for PR 28370 at commit [`bb324f9`](https://github.com/apache/spark/commit/bb324f946019b8d700c517cc5eb2f7c11dc70cfc). * This patch **fails from timeout after a configured wait of `400m`**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
SparkQA removed a comment on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-620078395 **[Test build #121917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121917/testReport)** for PR 28370 at commit [`bb324f9`](https://github.com/apache/spark/commit/bb324f946019b8d700c517cc5eb2f7c11dc70cfc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
HeartSaVioR commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620272637 Just FYI, looks like the merged commit doesn't reflect the credit properly (not showing as 2 authors) - maybe because of the authorship of the first commit. author and committer were swapped. Maybe ideal to ask about @LiangchangZ whether it's OK. Btw, I'd like to be sure about how to address the follow up PR across all branches, as it leaves up two tasks, backport PR for branch-2.4, follow up PR for master. Ideally we'd be better to sync up the commit, hence making backport PR don't contain the follow-up, and follow-up PR to be ported back as well. (Ideally it'd be nice to wait for couple of days more to address all of valid comments to avoid such situation.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
zsxwing commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620270939 Is it safe for a maintenance branch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620266159 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620266159 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
SparkQA removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620162279 **[Test build #121923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121923/testReport)** for PR 28373 at commit [`f4fe22d`](https://github.com/apache/spark/commit/f4fe22d1a4533ad1651501432b49e0fc47996e59). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
SparkQA commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620265114 **[Test build #121923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121923/testReport)** for PR 28373 at commit [`f4fe22d`](https://github.com/apache/spark/commit/f4fe22d1a4533ad1651501432b49e0fc47996e59). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
dongjoon-hyun commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620262595 Also, could you make a backporting PR to branch-2.4 please, @xuanyuanking ? cc @holdenk since she is the release manager for 2.4.6. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620260919 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
dongjoon-hyun commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620261253 Thank you, @xuanyuanking , @cloud-fan , @HeartSaVioR . The test passed almost and was time-outed `6 hr 42 min` during the end of Python Tests. In these days, this frequently happens. I'll merge this first. @xuanyuanking . Please try to address @HeartSaVioR 's comment as a follow-up `[TESTS]` PR. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620260919 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
SparkQA commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620260494 **[Test build #121928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121928/testReport)** for PR 28373 at commit [`4e346c7`](https://github.com/apache/spark/commit/4e346c7958332e3526332b583386250a4caf4498). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
HeartSaVioR commented on pull request #28258: URL: https://github.com/apache/spark/pull/28258#issuecomment-620259362 I'm not that familiar with standalone mode, so assume we would like to make it behave similar with yarn-cluster. How it behaves if supervise option is specified? In yarn-cluster mode it waits till the application has been killed - submit process would wait even the case of relaunching of AM. Does it work like so? And given the flag only affects the standalone mode and yarn has the same flag having prefix, it would be better to add prefix (`spark.submit.standalone.`?) to differentiate, but that's only me and others may have better idea. (Or even unifying the config if majority of resource managers are supporting this?) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
dongjoon-hyun commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620258789 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620258096 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620258096 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
SparkQA commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620257651 **[Test build #121927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121927/testReport)** for PR 28373 at commit [`4e346c7`](https://github.com/apache/spark/commit/4e346c7958332e3526332b583386250a4caf4498). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620255042 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620255042 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620248626 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121924/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620248616 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620248616 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
SparkQA removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620170277 **[Test build #121924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121924/testReport)** for PR 28373 at commit [`f4fe22d`](https://github.com/apache/spark/commit/f4fe22d1a4533ad1651501432b49e0fc47996e59). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
SparkQA commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620248096 **[Test build #121924 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121924/testReport)** for PR 28373 at commit [`f4fe22d`](https://github.com/apache/spark/commit/f4fe22d1a4533ad1651501432b49e0fc47996e59). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #28026: [SPARK-31257][SQL] Unify create table syntax
HeartSaVioR edited a comment on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-620244166 Simply thinking as user perspective, if we still support `EXTERNAL` keyword on creating table syntax then end users will try to execute the same query they do with Hive, and be questioned if it behaves differently. That's the main reason I proposed adding marker to "differentiate" the twos (opposite to the direction of this maybe), clearly indicating which space (Spark, or Hive compatible) they're in to execute the query. This is a debt on starting with Hive compatible DDL - Spark has been putting the great efforts on compatibility with Hive and attracts Hive users to migrate to Spark, but this also leads to misunderstand of end users Spark SQL should be compatible with Hive in any way. I don't think the unified create table syntax should cover all possible clauses on both Spark native and Hive, especially Hive side. This is a new start and we're not forced to guarantee compatibility with Hive. That might bring backward incompatibility, but this can be tolerated if we no longer want to treat Hive compatibility as the first class. (End users could still deal with beeline or so.) If then I think we should also make clear to the points we drop support - if we want to drop support of something then it should be clearly represented in syntax perspective, in this case, get rid of `EXTERNAL` keyword. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28026: [SPARK-31257][SQL] Unify create table syntax
HeartSaVioR commented on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-620244166 Simply thinking as user perspective, if we still support `EXTERNAL` keyword on creating table syntax then end users will try to execute the same query they do with Hive, and be questioned if it behaves differently. That's the main reason I proposed adding marker to "differentiate" the twos (opposite to the direction of this maybe), clearly indicating which space (Spark, or Hive compatible) they're in to execute the query. This is a debt on starting with Hive compatible DDL - Spark has been putting the great efforts on compatibility with Hive and attracts Hive users to migrate to Spark, but this also leads to misunderstand of end users Spark SQL should be compatible with Hive in any way. I don't think the unified create table syntax should cover all possible clauses on both Spark native and Hive, especially Hive side. This is a new start and we're not forced to guarantee compatibility with Hive. That might bring backward incompatibility, but this can be tolerated if we no longer want to treat Hive compatibility as the first class. (End users could still deal with beeline or so.) But then I think we should also make clear to the points we drop support - if we want to drop support of something then it should be clearly represented in syntax perspective, in this case, get rid of `EXTERNAL` keyword. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
HeartSaVioR commented on a change in pull request #28326:
URL: https://github.com/apache/spark/pull/28326#discussion_r416137750
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala
##
@@ -593,6 +593,17 @@ class EventTimeWatermarkSuite extends StreamTest with
BeforeAndAfter with Matche
}
}
+ test("SPARK-27340 Alias on TimeWindow expression cause watermark metadata
lost") {
+val inputData = MemoryStream[Int]
+val aliasWindow = inputData.toDF()
+ .withColumn("eventTime", $"value".cast("timestamp"))
+ .withWatermark("eventTime", "10 seconds")
+ .select(window($"eventTime", "5 seconds") as 'aliasWindow)
+// Check the eventTime metadata is kept in the top level alias.
+assert(aliasWindow.logicalPlan.output.exists(
+ _.metadata.contains(EventTimeWatermark.delayKey)))
Review comment:
```
val windowedAggregation = aliasWindow
.groupBy('aliasWindow)
.agg(count("*") as 'count)
.select($"aliasWindow".getField("start").cast("long").as[Long],
$"count".as[Long])
testStream(windowedAggregation)(
AddData(inputData, 10, 11, 12, 13, 14, 15),
CheckNewAnswer(),
AddData(inputData, 25), // Advance watermark to 15 seconds
CheckNewAnswer((10, 5)),
assertNumStateRows(2),
AddData(inputData, 10), // Should not emit anything as data less
than watermark
CheckNewAnswer(),
assertNumStateRows(2)
)
```
Let's append this to make the UT verifying E2E (yes this is same as other
UTs in this suite, and the revised UT fails on master branch even without
assertion to check metadata directly) - and then we no longer need to have
complicated stream-stream join UT.
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
##
@@ -991,4 +991,30 @@ class StreamingOuterJoinSuite extends StreamTest with
StateStoreMetricsTest with
)
}
}
+
+ test("SPARK-27340 Windowed left out join with Alias on TimeWindow") {
Review comment:
I guess this is to retain the efforts of origin PR, but based on root
cause, it should be pretty much easier to reproduce (and you actually did it in
EventTimeWatermarkSuite).
Let's remove this test in new commit (so that we can still retain the
credit) and append more code on new UT to do E2E test. I'll comment there for
code we need to add.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
AmplabJenkins removed a comment on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620223464 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121905/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
AmplabJenkins removed a comment on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620223460 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
AmplabJenkins removed a comment on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620223720 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
AmplabJenkins commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620223720 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
AmplabJenkins removed a comment on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620220204 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121922/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
AmplabJenkins commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620223460 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
SparkQA commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620223140 **[Test build #121926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121926/testReport)** for PR 28359 at commit [`90d3dbf`](https://github.com/apache/spark/commit/90d3dbf3aa671280b494a82ee25c0a13a5f532f1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
SparkQA commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620222318 **[Test build #121905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121905/testReport)** for PR 28326 at commit [`05ed338`](https://github.com/apache/spark/commit/05ed338a23829875683fca1efafa32340bad271f). * This patch **fails from timeout after a configured wait of `400m`**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
SparkQA removed a comment on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620005507 **[Test build #121905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121905/testReport)** for PR 28326 at commit [`05ed338`](https://github.com/apache/spark/commit/05ed338a23829875683fca1efafa32340bad271f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
AmplabJenkins removed a comment on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620220080 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
AmplabJenkins commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620220192 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
dongjoon-hyun commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620219816 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
AmplabJenkins commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620220080 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
SparkQA removed a comment on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620131504 **[Test build #121922 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121922/testReport)** for PR 28359 at commit [`90d3dbf`](https://github.com/apache/spark/commit/90d3dbf3aa671280b494a82ee25c0a13a5f532f1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
SparkQA commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-620219430 **[Test build #121922 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121922/testReport)** for PR 28359 at commit [`90d3dbf`](https://github.com/apache/spark/commit/90d3dbf3aa671280b494a82ee25c0a13a5f532f1). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28330: [SPARK-31377][SQL][TEST] Added unit tests to 'number of output rows metric' for some joins in SQLMetricSuite
AmplabJenkins commented on pull request #28330: URL: https://github.com/apache/spark/pull/28330#issuecomment-620216723 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28330: [SPARK-31377][SQL][TEST] Added unit tests to 'number of output rows metric' for some joins in SQLMetricSuite
AmplabJenkins removed a comment on pull request #28330: URL: https://github.com/apache/spark/pull/28330#issuecomment-620216723 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28330: [SPARK-31377][SQL][TEST] Added unit tests to 'number of output rows metric' for some joins in SQLMetricSuite
SparkQA removed a comment on pull request #28330: URL: https://github.com/apache/spark/pull/28330#issuecomment-620037823 **[Test build #121910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121910/testReport)** for PR 28330 at commit [`42d7ec4`](https://github.com/apache/spark/commit/42d7ec49ee534940a37f4acbc8c1758e466446f1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28330: [SPARK-31377][SQL][TEST] Added unit tests to 'number of output rows metric' for some joins in SQLMetricSuite
SparkQA commented on pull request #28330: URL: https://github.com/apache/spark/pull/28330#issuecomment-620215580 **[Test build #121910 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121910/testReport)** for PR 28330 at commit [`42d7ec4`](https://github.com/apache/spark/commit/42d7ec49ee534940a37f4acbc8c1758e466446f1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28375: [SPARK-30282][SQL][FOLLOWUP] SHOW TBLPROPERTIES should support views
AmplabJenkins removed a comment on pull request #28375: URL: https://github.com/apache/spark/pull/28375#issuecomment-620212480 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28375: [SPARK-30282][SQL][FOLLOWUP] SHOW TBLPROPERTIES should support views
AmplabJenkins commented on pull request #28375: URL: https://github.com/apache/spark/pull/28375#issuecomment-620212480 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28375: [SPARK-30282][SQL][FOLLOWUP] SHOW TBLPROPERTIES should support views
SparkQA commented on pull request #28375: URL: https://github.com/apache/spark/pull/28375#issuecomment-620211719 **[Test build #121925 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121925/testReport)** for PR 28375 at commit [`a9a7b39`](https://github.com/apache/spark/commit/a9a7b392ed7a285add8f6700af47e705a42c40a0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28355: [SPARK-31565][WEBUI][FOLLOWUP] Add font color setting of DAG-viz for query plan.
AmplabJenkins removed a comment on pull request #28355: URL: https://github.com/apache/spark/pull/28355#issuecomment-620209505 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #28375: [SPARK-30282][SQL][FOLLOWUP] SHOW TBLPROPERTIES should support views
imback82 commented on pull request #28375: URL: https://github.com/apache/spark/pull/28375#issuecomment-620209520 cc: @cloud-fan @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28355: [SPARK-31565][WEBUI][FOLLOWUP] Add font color setting of DAG-viz for query plan.
AmplabJenkins commented on pull request #28355: URL: https://github.com/apache/spark/pull/28355#issuecomment-620209505 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28355: [SPARK-31565][WEBUI][FOLLOWUP] Add font color setting of DAG-viz for query plan.
SparkQA removed a comment on pull request #28355: URL: https://github.com/apache/spark/pull/28355#issuecomment-620020043 **[Test build #121908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121908/testReport)** for PR 28355 at commit [`6982a17`](https://github.com/apache/spark/commit/6982a175113055139e7cb94cb0d3d5af135af773). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations
AmplabJenkins removed a comment on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620208929 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #26921: [SPARK-30282][SQL] Migrate SHOW TBLPROPERTIES to new framework
imback82 commented on pull request #26921: URL: https://github.com/apache/spark/pull/26921#issuecomment-620208827 I created a PR: https://github.com/apache/spark/pull/28375 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations
AmplabJenkins commented on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620208929 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 opened a new pull request #28375: [SPARK-30282][SQL][FOLLOWUP] SHOW TBLPROPERTIES should support views
imback82 opened a new pull request #28375:
URL: https://github.com/apache/spark/pull/28375
### What changes were proposed in this pull request?
This PR addresses two things:
- `SHOW TBLPROPERTIES` should supports view (a regression introduced by
#24963)
- `SHOW TBLPROPERTIES` on a temporary view should return empty result (2.4
behavior instead of throwing `AnalysisException`.
### Why are the changes needed?
It's a bug.
### Does this PR introduce any user-facing change?
Yes, now `SHOW TBLPROPERTIES` works on views:
```
scala> sql("CREATE VIEW view TBLPROPERTIES('p1'='v1', 'p2'='v2') AS SELECT 1
AS c1")
scala> sql("SHOW TBLPROPERTIES view").show(truncate=false)
+-+-+
|key |value|
+-+-+
|view.catalogAndNamespace.numParts|2|
|view.query.out.col.0 |c1 |
|view.query.out.numCols |1|
|p2 |v2 |
|view.catalogAndNamespace.part.0 |spark_catalog|
|p1 |v1 |
|view.catalogAndNamespace.part.1 |default |
+-+-+
```
And for a temporary view:
```
scala> sql("CREATE TEMPORARY VIEW tview TBLPROPERTIES('p1'='v1', 'p2'='v2')
AS SELECT 1 AS c1")
scala> sql("SHOW TBLPROPERTIES tview").show(truncate=false)
+---+-+
|key|value|
+---+-+
+---+-+
```
### How was this patch tested?
Added tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28355: [SPARK-31565][WEBUI][FOLLOWUP] Add font color setting of DAG-viz for query plan.
SparkQA commented on pull request #28355: URL: https://github.com/apache/spark/pull/28355#issuecomment-620208393 **[Test build #121908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121908/testReport)** for PR 28355 at commit [`6982a17`](https://github.com/apache/spark/commit/6982a175113055139e7cb94cb0d3d5af135af773). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations
SparkQA removed a comment on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620010568 **[Test build #121906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121906/testReport)** for PR 28371 at commit [`c4d0a69`](https://github.com/apache/spark/commit/c4d0a69c7ac20cfe5ad68c97f66f6a312a2e2a19). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations
SparkQA commented on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620207761 **[Test build #121906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121906/testReport)** for PR 28371 at commit [`c4d0a69`](https://github.com/apache/spark/commit/c4d0a69c7ac20cfe5ad68c97f66f6a312a2e2a19). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28330: [SPARK-31377][SQL][TEST] Added unit tests to 'number of output rows metric' for some joins in SQLMetricSuite
AmplabJenkins removed a comment on pull request #28330: URL: https://github.com/apache/spark/pull/28330#issuecomment-620205188 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28330: [SPARK-31377][SQL][TEST] Added unit tests to 'number of output rows metric' for some joins in SQLMetricSuite
AmplabJenkins commented on pull request #28330: URL: https://github.com/apache/spark/pull/28330#issuecomment-620205188 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28330: [SPARK-31377][SQL][TEST] Added unit tests to 'number of output rows metric' for some joins in SQLMetricSuite
SparkQA removed a comment on pull request #28330: URL: https://github.com/apache/spark/pull/28330#issuecomment-620024217 **[Test build #121909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121909/testReport)** for PR 28330 at commit [`882ba72`](https://github.com/apache/spark/commit/882ba724505201f940497cb48d8c2ce66573e135). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28330: [SPARK-31377][SQL][TEST] Added unit tests to 'number of output rows metric' for some joins in SQLMetricSuite
SparkQA commented on pull request #28330: URL: https://github.com/apache/spark/pull/28330#issuecomment-620203923 **[Test build #121909 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121909/testReport)** for PR 28330 at commit [`882ba72`](https://github.com/apache/spark/commit/882ba724505201f940497cb48d8c2ce66573e135). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620197128 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121919/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620197119 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620197119 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
SparkQA removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620087096 **[Test build #121919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121919/testReport)** for PR 28373 at commit [`4e2c009`](https://github.com/apache/spark/commit/4e2c0090c682bc6f67a550d83837986d08e01f19). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
SparkQA commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620196436 **[Test build #121919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121919/testReport)** for PR 28373 at commit [`4e2c009`](https://github.com/apache/spark/commit/4e2c0090c682bc6f67a550d83837986d08e01f19). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations
AmplabJenkins removed a comment on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620191367 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121904/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations
AmplabJenkins removed a comment on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620191359 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations
AmplabJenkins commented on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620191359 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations
SparkQA commented on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-620189990 **[Test build #121904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121904/testReport)** for PR 28371 at commit [`dd6cec5`](https://github.com/apache/spark/commit/dd6cec59085c4cb8bd93733b0d724cd07fb73cce). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28371: [SPARK-31577][SQL] Fix case-sensitivity and forward name conflict problems when check name conflicts of CTE relations
SparkQA removed a comment on pull request #28371: URL: https://github.com/apache/spark/pull/28371#issuecomment-62996 **[Test build #121904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121904/testReport)** for PR 28371 at commit [`dd6cec5`](https://github.com/apache/spark/commit/dd6cec59085c4cb8bd93733b0d724cd07fb73cce). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
dongjoon-hyun commented on a change in pull request #28326:
URL: https://github.com/apache/spark/pull/28326#discussion_r416084154
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
##
@@ -991,4 +991,30 @@ class StreamingOuterJoinSuite extends StreamTest with
StateStoreMetricsTest with
)
}
}
+
+ test("SPARK-27340 Windowed left out join with Alias on TimeWindow") {
Review comment:
super nit. `out` -> `outer`, Let's ignore for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28294: [SPARK-31519][SQL] Cast in having aggregate expressions returns the wrong result
AmplabJenkins removed a comment on pull request #28294: URL: https://github.com/apache/spark/pull/28294#issuecomment-620178022 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28294: [SPARK-31519][SQL] Cast in having aggregate expressions returns the wrong result
AmplabJenkins commented on pull request #28294: URL: https://github.com/apache/spark/pull/28294#issuecomment-620178022 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28294: [SPARK-31519][SQL] Cast in having aggregate expressions returns the wrong result
SparkQA commented on pull request #28294: URL: https://github.com/apache/spark/pull/28294#issuecomment-620176225 **[Test build #121903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121903/testReport)** for PR 28294 at commit [`18d857f`](https://github.com/apache/spark/commit/18d857fa993e2acd7abcb837700e6bc4f3511dd5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28294: [SPARK-31519][SQL] Cast in having aggregate expressions returns the wrong result
SparkQA removed a comment on pull request #28294: URL: https://github.com/apache/spark/pull/28294#issuecomment-619969970 **[Test build #121903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121903/testReport)** for PR 28294 at commit [`18d857f`](https://github.com/apache/spark/commit/18d857fa993e2acd7abcb837700e6bc4f3511dd5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620170846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620170846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
SparkQA commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620170277 **[Test build #121924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121924/testReport)** for PR 28373 at commit [`f4fe22d`](https://github.com/apache/spark/commit/f4fe22d1a4533ad1651501432b49e0fc47996e59). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28373: [SPARK-31580][BUILD][test-hive1.2] Upgrade Apache ORC to 1.5.10
dongjoon-hyun commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620168852 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
dongjoon-hyun commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620165939 cc @gatorsmile and @yhuai since we need to update ORC dependency in `branch-3.0`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
dongjoon-hyun commented on a change in pull request #28373:
URL: https://github.com/apache/spark/pull/28373#discussion_r416059695
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
##
@@ -540,6 +540,12 @@ abstract class OrcSuite extends OrcTest with
BeforeAndAfterAll {
}
}
}
+
+ test("SPARK-31580: Read a file written before ORC-569") {
Review comment:
This test case is in `OrcSuite` and will be tested in both
`OrcSourceSuite` and `HiveOrcSourceSuite`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
AmplabJenkins removed a comment on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620162918 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
AmplabJenkins commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620162918 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28373: [SPARK-31580][BUILD] Upgrade Apache ORC to 1.5.10
SparkQA commented on pull request #28373: URL: https://github.com/apache/spark/pull/28373#issuecomment-620162279 **[Test build #121923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121923/testReport)** for PR 28373 at commit [`f4fe22d`](https://github.com/apache/spark/commit/f4fe22d1a4533ad1651501432b49e0fc47996e59). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28194: [SPARK-31372][SQL][TEST] Display expression schema for double check.
AmplabJenkins removed a comment on pull request #28194: URL: https://github.com/apache/spark/pull/28194#issuecomment-620159417 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121911/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
srowen commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r416043932
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -124,38 +127,57 @@ private class ClientEndpoint(
}
}
- /* Find out driver status then exit the JVM */
+ /**
+ * Find out driver status then exit the JVM. If the waitAppCompletion is set
to true, monitors
+ * the application until it finishes, fails or is killed.
+ */
def pollAndReportStatus(driverId: String): Unit = {
// Since ClientEndpoint is the only RpcEndpoint in the process, blocking
the event loop thread
// is fine.
logInfo("... waiting before polling master for driver state")
Thread.sleep(5000)
logInfo("... polling master for driver state")
-val statusResponse =
-
activeMasterEndpoint.askSync[DriverStatusResponse](RequestDriverStatus(driverId))
-if (statusResponse.found) {
- logInfo(s"State of $driverId is ${statusResponse.state.get}")
- // Worker node, if present
- (statusResponse.workerId, statusResponse.workerHostPort,
statusResponse.state) match {
-case (Some(id), Some(hostPort), Some(DriverState.RUNNING)) =>
- logInfo(s"Driver running on $hostPort ($id)")
-case _ =>
- }
- // Exception, if present
- statusResponse.exception match {
-case Some(e) =>
- logError(s"Exception from cluster was: $e")
- e.printStackTrace()
- System.exit(-1)
-case _ =>
- System.exit(0)
+while (true) {
+ val statusResponse =
+
activeMasterEndpoint.askSync[DriverStatusResponse](RequestDriverStatus(driverId))
+ if (statusResponse.found) {
+logInfo(s"State of $driverId is ${statusResponse.state.get}")
+// Worker node, if present
+(statusResponse.workerId, statusResponse.workerHostPort,
statusResponse.state) match {
+ case (Some(id), Some(hostPort), Some(DriverState.RUNNING)) =>
+logInfo(s"Driver running on $hostPort ($id)")
+ case _ =>
+}
+// Exception, if present
+statusResponse.exception match {
+ case Some(e) =>
+logError(s"Exception from cluster was: $e")
+e.printStackTrace()
+System.exit(-1)
+ case _ =>
+if (!waitAppCompletion) {
+ logInfo(s"No exception found and waitAppCompletion is false, " +
+s"exiting spark-submit JVM.")
+ System.exit(0)
+} else if (statusResponse.state.get == DriverState.FINISHED ||
Review comment:
Just use a match statement to simplify the next 10 lines or so
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -124,38 +127,57 @@ private class ClientEndpoint(
}
}
- /* Find out driver status then exit the JVM */
+ /**
+ * Find out driver status then exit the JVM. If the waitAppCompletion is set
to true, monitors
+ * the application until it finishes, fails or is killed.
+ */
def pollAndReportStatus(driverId: String): Unit = {
// Since ClientEndpoint is the only RpcEndpoint in the process, blocking
the event loop thread
// is fine.
logInfo("... waiting before polling master for driver state")
Thread.sleep(5000)
logInfo("... polling master for driver state")
-val statusResponse =
-
activeMasterEndpoint.askSync[DriverStatusResponse](RequestDriverStatus(driverId))
-if (statusResponse.found) {
- logInfo(s"State of $driverId is ${statusResponse.state.get}")
- // Worker node, if present
- (statusResponse.workerId, statusResponse.workerHostPort,
statusResponse.state) match {
-case (Some(id), Some(hostPort), Some(DriverState.RUNNING)) =>
- logInfo(s"Driver running on $hostPort ($id)")
-case _ =>
- }
- // Exception, if present
- statusResponse.exception match {
-case Some(e) =>
- logError(s"Exception from cluster was: $e")
- e.printStackTrace()
- System.exit(-1)
-case _ =>
- System.exit(0)
+while (true) {
+ val statusResponse =
+
activeMasterEndpoint.askSync[DriverStatusResponse](RequestDriverStatus(driverId))
+ if (statusResponse.found) {
+logInfo(s"State of $driverId is ${statusResponse.state.get}")
+// Worker node, if present
+(statusResponse.workerId, statusResponse.workerHostPort,
statusResponse.state) match {
+ case (Some(id), Some(hostPort), Some(DriverState.RUNNING)) =>
+logInfo(s"Driver running on $hostPort ($id)")
+ case _ =>
+}
+// Exception, if present
+statusResponse.exception match {
+ case Some(e) =>
+logError(s"Exception from cluster was: $e")
+e.printStackTrace()
+System.exit(-1)
+ case _ =>
+if (!waitAppComple
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28194: [SPARK-31372][SQL][TEST] Display expression schema for double check.
AmplabJenkins removed a comment on pull request #28194: URL: https://github.com/apache/spark/pull/28194#issuecomment-620159404 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28194: [SPARK-31372][SQL][TEST] Display expression schema for double check.
AmplabJenkins commented on pull request #28194: URL: https://github.com/apache/spark/pull/28194#issuecomment-620159404 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28194: [SPARK-31372][SQL][TEST] Display expression schema for double check.
SparkQA commented on pull request #28194: URL: https://github.com/apache/spark/pull/28194#issuecomment-620158998 **[Test build #121911 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121911/testReport)** for PR 28194 at commit [`da0adba`](https://github.com/apache/spark/commit/da0adbab67b2805b947520d53b99069ce4b5c425). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
