[GitHub] [spark] MaxGekk commented on issue #26438: [SPARK-29408][SQL] Support sign before `interval` in interval literals
MaxGekk commented on issue #26438: [SPARK-29408][SQL] Support sign before `interval` in interval literals URL: https://github.com/apache/spark/pull/26438#issuecomment-552171915 While comparing to PostgreSQL, I have found it doesn't support '+' before interval: ```sql maxim=# select +interval '-1 month 1 day -1 second'; ERROR: operator does not exist: + interval LINE 1: select +interval '-1 month 1 day -1 second'; ^ HINT: No operator matches the given name and argument type. You might need to add an explicit type cast. ``` @dongjoon-hyun Shall I support only `-`? or `+` is ok too? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
AmplabJenkins commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552171866 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
AmplabJenkins commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552171867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113524/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
AmplabJenkins removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552171866 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
AmplabJenkins removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552171867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113524/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
SparkQA removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552164869 **[Test build #113524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113524/testReport)** for PR 26416 at commit [`cff732c`](https://github.com/apache/spark/commit/cff732c4920134f52b2c7700eeb66df86c3f9ad0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
SparkQA commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552171748 **[Test build #113524 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113524/testReport)** for PR 26416 at commit [`cff732c`](https://github.com/apache/spark/commit/cff732c4920134f52b2c7700eeb66df86c3f9ad0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26439: [SPARK-29801][ML] ML models unify toString method
AmplabJenkins removed a comment on issue #26439: [SPARK-29801][ML] ML models unify toString method URL: https://github.com/apache/spark/pull/26439#issuecomment-552171270 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26439: [SPARK-29801][ML] ML models unify toString method
AmplabJenkins removed a comment on issue #26439: [SPARK-29801][ML] ML models unify toString method URL: https://github.com/apache/spark/pull/26439#issuecomment-552171271 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18417/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26439: [SPARK-29801][ML] ML models unify toString method
AmplabJenkins commented on issue #26439: [SPARK-29801][ML] ML models unify toString method URL: https://github.com/apache/spark/pull/26439#issuecomment-552171270 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26439: [SPARK-29801][ML] ML models unify toString method
AmplabJenkins commented on issue #26439: [SPARK-29801][ML] ML models unify toString method URL: https://github.com/apache/spark/pull/26439#issuecomment-552171271 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18417/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26439: [SPARK-29801][ML] ML models unify toString method
SparkQA commented on issue #26439: [SPARK-29801][ML] ML models unify toString method URL: https://github.com/apache/spark/pull/26439#issuecomment-552171192 **[Test build #113528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113528/testReport)** for PR 26439 at commit [`536bfcc`](https://github.com/apache/spark/commit/536bfccedfb3339e21c86f08b77450b61c64d4b4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552171035 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552171036 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113523/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552171036 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113523/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552171035 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
SparkQA removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552164500 **[Test build #113523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113523/testReport)** for PR 26312 at commit [`05502f0`](https://github.com/apache/spark/commit/05502f0bdec7cff89481c946a600a0e1b1da792d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
SparkQA commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552170949 **[Test build #113523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113523/testReport)** for PR 26312 at commit [`05502f0`](https://github.com/apache/spark/commit/05502f0bdec7cff89481c946a600a0e1b1da792d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units
AmplabJenkins removed a comment on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units URL: https://github.com/apache/spark/pull/26455#issuecomment-552170427 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units
AmplabJenkins removed a comment on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units URL: https://github.com/apache/spark/pull/26455#issuecomment-552170431 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18416/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units
AmplabJenkins commented on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units URL: https://github.com/apache/spark/pull/26455#issuecomment-552170427 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units
AmplabJenkins commented on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units URL: https://github.com/apache/spark/pull/26455#issuecomment-552170431 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18416/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper
AmplabJenkins removed a comment on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper URL: https://github.com/apache/spark/pull/26397#issuecomment-552170339 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper
AmplabJenkins removed a comment on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper URL: https://github.com/apache/spark/pull/26397#issuecomment-552170340 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113521/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units
SparkQA commented on issue #26455: [SPARK-29819][SQL] Introduce an enum for interval units URL: https://github.com/apache/spark/pull/26455#issuecomment-552170330 **[Test build #113527 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113527/testReport)** for PR 26455 at commit [`1667a33`](https://github.com/apache/spark/commit/1667a330dbe5634b3ee7c476d87b5e93901cc159). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper
AmplabJenkins commented on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper URL: https://github.com/apache/spark/pull/26397#issuecomment-552170339 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper
AmplabJenkins commented on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper URL: https://github.com/apache/spark/pull/26397#issuecomment-552170340 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113521/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper
SparkQA commented on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper URL: https://github.com/apache/spark/pull/26397#issuecomment-552170229 **[Test build #113521 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113521/testReport)** for PR 26397 at commit [`698a657`](https://github.com/apache/spark/commit/698a6571e6f798b74d8469be32b41999c5397d60). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper
SparkQA removed a comment on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper URL: https://github.com/apache/spark/pull/26397#issuecomment-552163600 **[Test build #113521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113521/testReport)** for PR 26397 at commit [`698a657`](https://github.com/apache/spark/commit/698a6571e6f798b74d8469be32b41999c5397d60). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26456: [SPARK-29820][INFRA] Use GitHub Action Cache for `./.m2/repository/[com|org]`
dongjoon-hyun commented on issue #26456: [SPARK-29820][INFRA] Use GitHub Action Cache for `./.m2/repository/[com|org]` URL: https://github.com/apache/spark/pull/26456#issuecomment-552170004 Could you review this, @HyukjinKwon ? In these days, Maven download is very flaky. We need this patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
dongjoon-hyun commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552169915 Thank you. AppVeyor passed! Please ignore github action failure. It will be fixed when my another PR about GitHub Action Cache PR is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
maropu commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552169405 Can you file a new jira for this for better traceability? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26425: [SPARK-29789][SQL] should not parse the bucket column name when creating v2 tables
viirya commented on a change in pull request #26425: [SPARK-29789][SQL] should
not parse the bucket column name when creating v2 tables
URL: https://github.com/apache/spark/pull/26425#discussion_r344475154
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/expressions/expressions.scala
##
@@ -43,20 +43,22 @@ private[sql] object LogicalExpressions {
def reference(name: String): NamedReference =
FieldReference(parser.parseMultipartIdentifier(name))
+ def reference(nameParts: Seq[String]): NamedReference =
FieldReference(nameParts)
+
def apply(name: String, arguments: Expression*): Transform =
ApplyTransform(name, arguments)
- def bucket(numBuckets: Int, columns: String*): BucketTransform =
-BucketTransform(literal(numBuckets, IntegerType), columns.map(reference))
+ def bucket(numBuckets: Int, references: Array[NamedReference]):
BucketTransform =
+BucketTransform(literal(numBuckets, IntegerType), references)
Review comment:
For internal API, I think it is more convenient to use NamedReference
directly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kevinyu98 commented on a change in pull request #25573: [SPARK-28833][DOCS][SQL] Document ALTER VIEW command
kevinyu98 commented on a change in pull request #25573:
[SPARK-28833][DOCS][SQL] Document ALTER VIEW command
URL: https://github.com/apache/spark/pull/25573#discussion_r344475265
##
File path: docs/sql-ref-syntax-ddl-alter-view.md
##
@@ -19,4 +19,78 @@ license: |
limitations under the License.
---
-**This page is under construction**
+### Description
+The `ALTER VIEW` statement changes various auxiliary properties of a view.
+
+
+ Rename view
+Rename the existing view. If the view name already exists in the database, an
exception is thrown. This operation does
+support moving the views cross databases.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier RENAME TO viewIdentifier
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Set view properties
+Set one or more properties of an existing view. The properties are the key
value pairs. If the properties' keys exist,
+the values are replaced with the new values. If the properties' keys does not
exist, the key value pairs are added into
+the properties.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier SET TBLPROPERTIES (key1=val1, key2=val2, ...)
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Drop view properties
+Drop one or more properties of an existing view. If the specified keys do not
exist, an exception is thrown. Use
+`IF EXISTS` to avoid the exception.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier UNSET TBLPROPERTIES [IF EXISTS] (key1=val1,
key2=val2, ...)
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Alter View As Select
+`ALTER VIEW AS SELECT` statement changes the definition of a view, the
`select_statement` must valid, and the `VIEW`
+must exist.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier AS select_statement
+viewIdentifier:= [db_name.]view_name
+select_statement:= [select_statement](sql-ref-syntax-qry-select.html)
+{% endhighlight %}
+
+
+ Example
+{% highlight sql %}
+-- Rename only change the view name.
+-- The source and target databases of the view have to be the same, use
qualified or unqualified name for the target view
+ALTER VIEW tempdb.view1 RENAME TO view2
+{% endhighlight %}
+
+{% highlight sql %}
+-- Use `DESC TABLE EXTENDED tempdb.view1` before and after the `ALTER VIEW`
statement to verify the changes.
+ALTER VIEW tempdb.view1 SET TBLPROPERTIES ('propKey1' = "propVal1", 'propKey2'
= "propVal2" )
+{% endhighlight %}
+
+{% highlight sql %}
+-- Use `DESC TABLE EXTENDED tempdb.view1` before and after the `ALTER VIEW` to
verify the change.
+ALTER VIEW tempdb.view1 UNSET TBLPROPERTIES ('propKey1')
+{% endhighlight %}
+
+{% highlight sql %}
+-- Do the select on tempdb.view1 before and after the `ALTER VIEW` statement
to verify.
+ALTER VIEW tempdb.view1 AS SELECT * FROM tempdb.view2
+{% endhighlight %}
+
+ Related Statements
+[describe-table](sql-ref-syntax-aux-describe-table.html)
+[create-view](sql-ref-syntax-ddl-create-view.html)
+[drop-view](sql-ref-syntax-ddl-drop-view.html)
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kevinyu98 commented on a change in pull request #25573: [SPARK-28833][DOCS][SQL] Document ALTER VIEW command
kevinyu98 commented on a change in pull request #25573:
[SPARK-28833][DOCS][SQL] Document ALTER VIEW command
URL: https://github.com/apache/spark/pull/25573#discussion_r344475219
##
File path: docs/sql-ref-syntax-ddl-alter-view.md
##
@@ -19,4 +19,78 @@ license: |
limitations under the License.
---
-**This page is under construction**
+### Description
+The `ALTER VIEW` statement changes various auxiliary properties of a view.
+
+
+ Rename view
+Rename the existing view. If the view name already exists in the database, an
exception is thrown. This operation does
+support moving the views cross databases.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier RENAME TO viewIdentifier
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Set view properties
+Set one or more properties of an existing view. The properties are the key
value pairs. If the properties' keys exist,
+the values are replaced with the new values. If the properties' keys does not
exist, the key value pairs are added into
+the properties.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier SET TBLPROPERTIES (key1=val1, key2=val2, ...)
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Drop view properties
+Drop one or more properties of an existing view. If the specified keys do not
exist, an exception is thrown. Use
+`IF EXISTS` to avoid the exception.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier UNSET TBLPROPERTIES [IF EXISTS] (key1=val1,
key2=val2, ...)
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Alter View As Select
+`ALTER VIEW AS SELECT` statement changes the definition of a view, the
`select_statement` must valid, and the `VIEW`
+must exist.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier AS select_statement
+viewIdentifier:= [db_name.]view_name
+select_statement:= [select_statement](sql-ref-syntax-qry-select.html)
+{% endhighlight %}
+
+
+ Example
+{% highlight sql %}
+-- Rename only change the view name.
+-- The source and target databases of the view have to be the same, use
qualified or unqualified name for the target view
+ALTER VIEW tempdb.view1 RENAME TO view2
+{% endhighlight %}
+
+{% highlight sql %}
+-- Use `DESC TABLE EXTENDED tempdb.view1` before and after the `ALTER VIEW`
statement to verify the changes.
+ALTER VIEW tempdb.view1 SET TBLPROPERTIES ('propKey1' = "propVal1", 'propKey2'
= "propVal2" )
+{% endhighlight %}
+
+{% highlight sql %}
+-- Use `DESC TABLE EXTENDED tempdb.view1` before and after the `ALTER VIEW` to
verify the change.
+ALTER VIEW tempdb.view1 UNSET TBLPROPERTIES ('propKey1')
+{% endhighlight %}
+
+{% highlight sql %}
+-- Do the select on tempdb.view1 before and after the `ALTER VIEW` statement
to verify.
+ALTER VIEW tempdb.view1 AS SELECT * FROM tempdb.view2
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kevinyu98 commented on a change in pull request #25573: [SPARK-28833][DOCS][SQL] Document ALTER VIEW command
kevinyu98 commented on a change in pull request #25573:
[SPARK-28833][DOCS][SQL] Document ALTER VIEW command
URL: https://github.com/apache/spark/pull/25573#discussion_r344475211
##
File path: docs/sql-ref-syntax-ddl-alter-view.md
##
@@ -19,4 +19,78 @@ license: |
limitations under the License.
---
-**This page is under construction**
+### Description
+The `ALTER VIEW` statement changes various auxiliary properties of a view.
+
+
+ Rename view
+Rename the existing view. If the view name already exists in the database, an
exception is thrown. This operation does
+support moving the views cross databases.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier RENAME TO viewIdentifier
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Set view properties
+Set one or more properties of an existing view. The properties are the key
value pairs. If the properties' keys exist,
+the values are replaced with the new values. If the properties' keys does not
exist, the key value pairs are added into
+the properties.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier SET TBLPROPERTIES (key1=val1, key2=val2, ...)
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Drop view properties
+Drop one or more properties of an existing view. If the specified keys do not
exist, an exception is thrown. Use
+`IF EXISTS` to avoid the exception.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier UNSET TBLPROPERTIES [IF EXISTS] (key1=val1,
key2=val2, ...)
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Alter View As Select
+`ALTER VIEW AS SELECT` statement changes the definition of a view, the
`select_statement` must valid, and the `VIEW`
+must exist.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier AS select_statement
+viewIdentifier:= [db_name.]view_name
+select_statement:= [select_statement](sql-ref-syntax-qry-select.html)
+{% endhighlight %}
+
+
+ Example
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kevinyu98 commented on a change in pull request #25573: [SPARK-28833][DOCS][SQL] Document ALTER VIEW command
kevinyu98 commented on a change in pull request #25573:
[SPARK-28833][DOCS][SQL] Document ALTER VIEW command
URL: https://github.com/apache/spark/pull/25573#discussion_r344475200
##
File path: docs/sql-ref-syntax-ddl-alter-view.md
##
@@ -19,4 +19,78 @@ license: |
limitations under the License.
---
-**This page is under construction**
+### Description
+The `ALTER VIEW` statement changes various auxiliary properties of a view.
+
+
+ Rename view
+Rename the existing view. If the view name already exists in the database, an
exception is thrown. This operation does
+support moving the views cross databases.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier RENAME TO viewIdentifier
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Set view properties
+Set one or more properties of an existing view. The properties are the key
value pairs. If the properties' keys exist,
+the values are replaced with the new values. If the properties' keys does not
exist, the key value pairs are added into
+the properties.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier SET TBLPROPERTIES (key1=val1, key2=val2, ...)
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Drop view properties
+Drop one or more properties of an existing view. If the specified keys do not
exist, an exception is thrown. Use
+`IF EXISTS` to avoid the exception.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier UNSET TBLPROPERTIES [IF EXISTS] (key1=val1,
key2=val2, ...)
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Alter View As Select
+`ALTER VIEW AS SELECT` statement changes the definition of a view, the
`select_statement` must valid, and the `VIEW`
+must exist.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier AS select_statement
+viewIdentifier:= [db_name.]view_name
+select_statement:= [select_statement](sql-ref-syntax-qry-select.html)
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kevinyu98 commented on a change in pull request #25573: [SPARK-28833][DOCS][SQL] Document ALTER VIEW command
kevinyu98 commented on a change in pull request #25573:
[SPARK-28833][DOCS][SQL] Document ALTER VIEW command
URL: https://github.com/apache/spark/pull/25573#discussion_r344475176
##
File path: docs/sql-ref-syntax-ddl-alter-view.md
##
@@ -19,4 +19,78 @@ license: |
limitations under the License.
---
-**This page is under construction**
+### Description
+The `ALTER VIEW` statement changes various auxiliary properties of a view.
+
+
+ Rename view
+Rename the existing view. If the view name already exists in the database, an
exception is thrown. This operation does
+support moving the views cross databases.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier RENAME TO viewIdentifier
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Set view properties
+Set one or more properties of an existing view. The properties are the key
value pairs. If the properties' keys exist,
+the values are replaced with the new values. If the properties' keys does not
exist, the key value pairs are added into
+the properties.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier SET TBLPROPERTIES (key1=val1, key2=val2, ...)
+viewIdentifier:= [db_name.]view_name
+{% endhighlight %}
+
+
+ Drop view properties
+Drop one or more properties of an existing view. If the specified keys do not
exist, an exception is thrown. Use
+`IF EXISTS` to avoid the exception.
+# Syntax
+{% highlight sql %}
+ALTER VIEW viewIdentifier UNSET TBLPROPERTIES [IF EXISTS] (key1=val1,
key2=val2, ...)
Review comment:
@srowen you are right, will make change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26278: [DO-NOT-MERBE] Investigate Appveyor failure of master
HyukjinKwon commented on issue #26278: [DO-NOT-MERBE] Investigate Appveyor failure of master URL: https://github.com/apache/spark/pull/26278#issuecomment-552167981 Sure, sorry for slow progress : ( This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
HyukjinKwon commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552167967 Thanks @dongjoon-hyun for trying this out! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
AmplabJenkins removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552167841 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
AmplabJenkins commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552167842 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113516/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
AmplabJenkins commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552167841 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
AmplabJenkins removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552167842 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113516/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
SparkQA removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552157428 **[Test build #113516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113516/testReport)** for PR 26449 at commit [`8851f35`](https://github.com/apache/spark/commit/8851f3538f69cd6f80ed207d0a4bdfeefd848fef). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
SparkQA commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552167771 **[Test build #113516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113516/testReport)** for PR 26449 at commit [`8851f35`](https://github.com/apache/spark/commit/8851f3538f69cd6f80ed207d0a4bdfeefd848fef). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function
AmplabJenkins removed a comment on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function URL: https://github.com/apache/spark/pull/26429#issuecomment-552167627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113519/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function
AmplabJenkins removed a comment on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function URL: https://github.com/apache/spark/pull/26429#issuecomment-552167625 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function
AmplabJenkins commented on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function URL: https://github.com/apache/spark/pull/26429#issuecomment-552167627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113519/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function
AmplabJenkins commented on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function URL: https://github.com/apache/spark/pull/26429#issuecomment-552167625 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function
SparkQA commented on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function URL: https://github.com/apache/spark/pull/26429#issuecomment-552167544 **[Test build #113519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113519/testReport)** for PR 26429 at commit [`10925bf`](https://github.com/apache/spark/commit/10925bfb1537a86d4773e2788739037a164d6ed3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function
SparkQA removed a comment on issue #26429: [SPARK-29777][SparkR] SparkR::cleanClosure aggressively removes a function required by user function URL: https://github.com/apache/spark/pull/26429#issuecomment-552159377 **[Test build #113519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113519/testReport)** for PR 26429 at commit [`10925bf`](https://github.com/apache/spark/commit/10925bfb1537a86d4773e2788739037a164d6ed3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26441: [SPARK-29682][SQL] Resolve conflicting references in aggregate expressions
viirya commented on a change in pull request #26441: [SPARK-29682][SQL] Resolve conflicting references in aggregate expressions URL: https://github.com/apache/spark/pull/26441#discussion_r344474408 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -949,14 +949,19 @@ class Analyzer( if oldVersion.outputSet.intersect(conflictingAttributes).nonEmpty => (oldVersion, oldVersion.copy(serializer = oldVersion.serializer.map(_.newInstance( -// Handle projects that create conflicting aliases. case oldVersion @ Project(projectList, _) -if findAliases(projectList).intersect(conflictingAttributes).nonEmpty => Review comment: I am not sure why we only process aliases previously. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
AmplabJenkins commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166911 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
AmplabJenkins commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166914 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18415/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
AmplabJenkins removed a comment on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166911 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
AmplabJenkins removed a comment on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166914 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18415/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552166855 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113520/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552166854 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552166854 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552166855 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113520/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
SparkQA commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166859 **[Test build #113526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113526/testReport)** for PR 26457 at commit [`4020832`](https://github.com/apache/spark/commit/40208323f6dcec7b841f2c26ef4ae6711d4e323d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
SparkQA removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552159703 **[Test build #113520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113520/testReport)** for PR 26312 at commit [`27047da`](https://github.com/apache/spark/commit/27047dabf31ab92a5dbfc1ba3aa13fa6d9797b8f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
dongjoon-hyun commented on a change in pull request #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#discussion_r344474133 ## File path: R/README.md ## @@ -1,4 +1,4 @@ -# R on Spark +# R on Spark! Review comment: This is only for triggering AppVoyer and this will be reverted before merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
SparkQA commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552166770 **[Test build #113520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113520/testReport)** for PR 26312 at commit [`27047da`](https://github.com/apache/spark/commit/27047dabf31ab92a5dbfc1ba3aa13fa6d9797b8f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
AmplabJenkins removed a comment on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166574 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
AmplabJenkins commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166575 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18414/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
AmplabJenkins commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166574 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
AmplabJenkins removed a comment on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166575 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18414/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
SparkQA commented on issue #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457#issuecomment-552166511 **[Test build #113525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113525/testReport)** for PR 26457 at commit [`41a2b5a`](https://github.com/apache/spark/commit/41a2b5ac262ff759dd7b5c30b18fa022f8eaf111). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26185: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.2.4 for Scala 2.13.1
dongjoon-hyun commented on issue #26185: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.2.4 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26185#issuecomment-552166449 I made a new attempt with 4.3.0. - https://github.com/apache/spark/pull/26457 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26278: [DO-NOT-MERBE] Investigate Appveyor failure of master
dongjoon-hyun commented on issue #26278: [DO-NOT-MERBE] Investigate Appveyor failure of master URL: https://github.com/apache/spark/pull/26278#issuecomment-552166327 Let's continue to investigation on https://github.com/apache/spark/pull/26457 since `4.3.0` is released. I'll close this~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #26278: [DO-NOT-MERBE] Investigate Appveyor failure of master
dongjoon-hyun closed pull request #26278: [DO-NOT-MERBE] Investigate Appveyor failure of master URL: https://github.com/apache/spark/pull/26278 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1
dongjoon-hyun opened a new pull request #26457: [SPARK-29528][BUILD][test-maven] Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 URL: https://github.com/apache/spark/pull/26457 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
AmplabJenkins removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552166043 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26278: [DO-NOT-MERBE] Investigate Appveyor failure of master
dongjoon-hyun commented on issue #26278: [DO-NOT-MERBE] Investigate Appveyor failure of master URL: https://github.com/apache/spark/pull/26278#issuecomment-552166097 @HyukjinKwon . Is there any progress on this? We are still locked on `4.2.0`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
AmplabJenkins removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552166044 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113515/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
AmplabJenkins commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552166044 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113515/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
AmplabJenkins commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552166043 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
SparkQA removed a comment on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552155474 **[Test build #113515 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113515/testReport)** for PR 26449 at commit [`65e2a1c`](https://github.com/apache/spark/commit/65e2a1c09746bb64acdc1e77dae542b91bba045c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26405: [SPARK-29766][sql] Do metrics aggregation asynchronously in SQL listener.
dongjoon-hyun commented on a change in pull request #26405: [SPARK-29766][sql]
Do metrics aggregation asynchronously in SQL listener.
URL: https://github.com/apache/spark/pull/26405#discussion_r344473710
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
##
@@ -320,12 +321,18 @@ class SQLAppStatusListener(
private def onExecutionEnd(event: SparkListenerSQLExecutionEnd): Unit = {
val SparkListenerSQLExecutionEnd(executionId, time) = event
Option(liveExecutions.get(executionId)).foreach { exec =>
- exec.metricsValues = aggregateMetrics(exec)
exec.completionTime = Some(new Date(time))
- exec.endEvents += 1
update(exec)
Review comment:
As a side-effect, this PR seems to increase the number of invocation
`update(exec)` exactly twice. Is this okay?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values
SparkQA commented on issue #26449: [SPARK-29605][SQL][FOLLOWUP] Fix cast error when there are spaces between signs and values URL: https://github.com/apache/spark/pull/26449#issuecomment-552165960 **[Test build #113515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113515/testReport)** for PR 26449 at commit [`65e2a1c`](https://github.com/apache/spark/commit/65e2a1c09746bb64acdc1e77dae542b91bba045c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
HeartSaVioR commented on a change in pull request #26416:
[WIP][SPARK-29779][CORE] Compact old event log files and cleanup
URL: https://github.com/apache/spark/pull/26416#discussion_r344473273
##
File path:
core/src/main/scala/org/apache/spark/deploy/history/EventLogFileCompactor.scala
##
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.net.URI
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.io.{Codec, Source}
+
+import com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FileSystem, Path}
+import org.json4s.jackson.JsonMethods.parse
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.EVENT_LOG_ROLLING_MAX_FILES_TO_RETAIN
+import org.apache.spark.scheduler._
+import org.apache.spark.util.{JsonProtocol, Utils}
+
+class EventLogFileCompactor(
+sparkConf: SparkConf,
+hadoopConf: Configuration,
+fs: FileSystem) extends Logging {
+
+ private val maxFilesToRetain: Int =
sparkConf.get(EVENT_LOG_ROLLING_MAX_FILES_TO_RETAIN)
+
+ // FIXME: javadoc - caller should provide event log files (either compacted
or original)
+ // sequentially if the last event log file is already a compacted file,
everything
+ // will be skipped
+ def compact(eventLogFiles: Seq[FileStatus]): Seq[FileStatus] = {
+if (eventLogFiles.isEmpty) {
+ return Seq.empty[FileStatus]
+}
+
+// skip everything if the last file is already a compacted file
+if (EventLogFileWriter.isCompacted(eventLogFiles.last.getPath)) {
+ return Seq(eventLogFiles.last)
+}
+
+val (filesToCompact, filesToRetain) = findFilesToCompact(eventLogFiles)
+if (filesToCompact.isEmpty) {
+ filesToRetain
+} else {
+ // first pass
+ val bus = new ReplayListenerBus()
+
+ val builders = ServiceLoader.load(classOf[EventFilterBuilder],
+Utils.getContextOrSparkClassLoader).asScala
+ builders.foreach(bus.addListener)
+
+ filesToCompact.foreach { log =>
+Utils.tryWithResource(EventLogFileReader.openEventLog(log.getPath,
fs)) { in =>
+ bus.replay(in, log.getPath.getName)
+}
+ }
+
+ // second pass
+ val rewriter = new FilteredEventLogFileRewriter(sparkConf, hadoopConf,
+filesToCompact, fs, builders.map(_.createFilter()).toSeq)
+ rewriter.start()
+ rewriter.rewrite()
+ rewriter.stop()
+
+ // cleanup files which are replaced with new compacted file.
+ cleanupCompactedFiles(filesToCompact)
+
+ fs.getFileStatus(new Path(rewriter.logPath)) :: filesToRetain.toList
+}
+ }
+
+ private def cleanupCompactedFiles(files: Seq[FileStatus]): Unit = {
+files.foreach { file =>
+ try {
+fs.delete(file.getPath, true)
+ } catch {
+case _: IOException => logWarning(s"Failed to remove ${file.getPath} /
skip removing.")
+ }
+}
+ }
+
+ private def findFilesToCompact(
+ eventLogFiles: Seq[FileStatus]): (Seq[FileStatus], Seq[FileStatus]) = {
+val lastCompactedFileIdx = eventLogFiles.lastIndexWhere { fs =>
+ EventLogFileWriter.isCompacted(fs.getPath)
+}
+
+val files = if (lastCompactedFileIdx > -1) {
+ eventLogFiles.drop(lastCompactedFileIdx)
+} else {
+ eventLogFiles
+}
+
+if (files.length > maxFilesToRetain) {
+ (files.dropRight(maxFilesToRetain), files.takeRight(maxFilesToRetain))
+} else {
+ (Seq.empty, files)
+}
+ }
+}
+
+class FilteredEventLogFileRewriter(
+sparkConf: SparkConf,
+hadoopConf: Configuration,
+eventLogFiles: Seq[FileStatus],
+fs: FileSystem,
+filters: Seq[EventFilter]) extends Logging {
+
+ require(eventLogFiles.nonEmpty)
+
+ private val targetEventLogFilePath = eventLogFiles.last.getPath
+ private val logWriter: CompactedEventLogFileWriter = new
CompactedEventLogFileWriter(
+targetEventLogFilePath, "dummy", None,
targetEventLogFilePath.getParent.toUri,
+sparkConf, hadoopConf)
+
+ def logPath:
[GitHub] [spark] SparkQA commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
SparkQA commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552164869 **[Test build #113524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113524/testReport)** for PR 26416 at commit [`cff732c`](https://github.com/apache/spark/commit/cff732c4920134f52b2c7700eeb66df86c3f9ad0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552164556 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins removed a comment on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552164557 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18413/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552164557 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18413/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
AmplabJenkins commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552164556 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
SparkQA commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552164500 **[Test build #113523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113523/testReport)** for PR 26312 at commit [`05502f0`](https://github.com/apache/spark/commit/05502f0bdec7cff89481c946a600a0e1b1da792d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
AmplabJenkins removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552164333 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113522/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
AmplabJenkins removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552164332 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
SparkQA commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552164328 **[Test build #113522 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113522/testReport)** for PR 26416 at commit [`053a91c`](https://github.com/apache/spark/commit/053a91c2e367e8b71c62a57099bd259962dafb6a). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
AmplabJenkins commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552164332 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
SparkQA removed a comment on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552164009 **[Test build #113522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113522/testReport)** for PR 26416 at commit [`053a91c`](https://github.com/apache/spark/commit/053a91c2e367e8b71c62a57099bd259962dafb6a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
AmplabJenkins commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552164333 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113522/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file
viirya commented on issue #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file URL: https://github.com/apache/spark/pull/26312#issuecomment-552164087 @dongjoon-hyun Thanks for your review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
SparkQA commented on issue #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup URL: https://github.com/apache/spark/pull/26416#issuecomment-552164009 **[Test build #113522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113522/testReport)** for PR 26416 at commit [`053a91c`](https://github.com/apache/spark/commit/053a91c2e367e8b71c62a57099bd259962dafb6a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper
SparkQA commented on issue #26397: [SPARK-29755][CORE] Provide @JsonDeserialize for Option[Long] in LogInfo & AttemptInfoWrapper URL: https://github.com/apache/spark/pull/26397#issuecomment-552163600 **[Test build #113521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113521/testReport)** for PR 26397 at commit [`698a657`](https://github.com/apache/spark/commit/698a6571e6f798b74d8469be32b41999c5397d60). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
HeartSaVioR commented on a change in pull request #26416:
[WIP][SPARK-29779][CORE] Compact old event log files and cleanup
URL: https://github.com/apache/spark/pull/26416#discussion_r344472395
##
File path:
core/src/main/scala/org/apache/spark/deploy/history/EventLogFileCompactor.scala
##
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.IOException
+import java.net.URI
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+import scala.io.{Codec, Source}
+
+import com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, FileSystem, Path}
+import org.json4s.jackson.JsonMethods.parse
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.EVENT_LOG_ROLLING_MAX_FILES_TO_RETAIN
+import org.apache.spark.scheduler._
+import org.apache.spark.util.{JsonProtocol, Utils}
+
+class EventLogFileCompactor(
+sparkConf: SparkConf,
+hadoopConf: Configuration,
+fs: FileSystem) extends Logging {
+
+ private val maxFilesToRetain: Int =
sparkConf.get(EVENT_LOG_ROLLING_MAX_FILES_TO_RETAIN)
+
+ // FIXME: javadoc - caller should provide event log files (either compacted
or original)
+ // sequentially if the last event log file is already a compacted file,
everything
+ // will be skipped
+ def compact(eventLogFiles: Seq[FileStatus]): Seq[FileStatus] = {
+if (eventLogFiles.isEmpty) {
+ return Seq.empty[FileStatus]
+}
+
+// skip everything if the last file is already a compacted file
+if (EventLogFileWriter.isCompacted(eventLogFiles.last.getPath)) {
+ return Seq(eventLogFiles.last)
+}
+
+val (filesToCompact, filesToRetain) = findFilesToCompact(eventLogFiles)
+if (filesToCompact.isEmpty) {
+ filesToRetain
+} else {
+ // first pass
+ val bus = new ReplayListenerBus()
+
+ val builders = ServiceLoader.load(classOf[EventFilterBuilder],
+Utils.getContextOrSparkClassLoader).asScala
+ builders.foreach(bus.addListener)
+
+ filesToCompact.foreach { log =>
+Utils.tryWithResource(EventLogFileReader.openEventLog(log.getPath,
fs)) { in =>
+ bus.replay(in, log.getPath.getName)
+}
+ }
+
+ // second pass
+ val rewriter = new FilteredEventLogFileRewriter(sparkConf, hadoopConf,
+filesToCompact, fs, builders.map(_.createFilter()).toSeq)
+ rewriter.start()
+ rewriter.rewrite()
+ rewriter.stop()
+
+ // cleanup files which are replaced with new compacted file.
+ cleanupCompactedFiles(filesToCompact)
+
+ fs.getFileStatus(new Path(rewriter.logPath)) :: filesToRetain.toList
+}
+ }
+
+ private def cleanupCompactedFiles(files: Seq[FileStatus]): Unit = {
+files.foreach { file =>
+ try {
+fs.delete(file.getPath, true)
+ } catch {
+case _: IOException => logWarning(s"Failed to remove ${file.getPath} /
skip removing.")
+ }
+}
+ }
+
+ private def findFilesToCompact(
+ eventLogFiles: Seq[FileStatus]): (Seq[FileStatus], Seq[FileStatus]) = {
+val lastCompactedFileIdx = eventLogFiles.lastIndexWhere { fs =>
+ EventLogFileWriter.isCompacted(fs.getPath)
+}
+
+val files = if (lastCompactedFileIdx > -1) {
+ eventLogFiles.drop(lastCompactedFileIdx)
+} else {
+ eventLogFiles
+}
+
+if (files.length > maxFilesToRetain) {
+ (files.dropRight(maxFilesToRetain), files.takeRight(maxFilesToRetain))
+} else {
+ (Seq.empty, files)
+}
+ }
+}
+
+class FilteredEventLogFileRewriter(
+sparkConf: SparkConf,
+hadoopConf: Configuration,
+eventLogFiles: Seq[FileStatus],
+fs: FileSystem,
+filters: Seq[EventFilter]) extends Logging {
+
+ require(eventLogFiles.nonEmpty)
+
+ private val targetEventLogFilePath = eventLogFiles.last.getPath
+ private val logWriter: CompactedEventLogFileWriter = new
CompactedEventLogFileWriter(
+targetEventLogFilePath, "dummy", None,
targetEventLogFilePath.getParent.toUri,
+sparkConf, hadoopConf)
+
+ def logPath:
[GitHub] [spark] HeartSaVioR commented on a change in pull request #26416: [WIP][SPARK-29779][CORE] Compact old event log files and cleanup
HeartSaVioR commented on a change in pull request #26416:
[WIP][SPARK-29779][CORE] Compact old event log files and cleanup
URL: https://github.com/apache/spark/pull/26416#discussion_r344472347
##
File path:
core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
##
@@ -164,6 +164,7 @@ abstract class EventLogFileWriter(
object EventLogFileWriter {
// Suffix applied to the names of files still being written by applications.
val IN_PROGRESS = ".inprogress"
+ val COMPACTED = ".compact"
Review comment:
I followed the naming of extension where the extension of state store's
snapshotted file is ".snapshot". Let's hear more voices on this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
