[GitHub] [spark] AmplabJenkins removed a comment on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC
AmplabJenkins removed a comment on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC URL: https://github.com/apache/spark/pull/25336#issuecomment-519612931 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
SparkQA commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-519612956 **[Test build #108841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108841/testReport)** for PR 24892 at commit [`4f9606c`](https://github.com/apache/spark/commit/4f9606cb3cfd8c46e67f6c7e56c7233bb5f1c8dc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519613070 **[Test build #108833 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108833/testReport)** for PR 25198 at commit [`108b7f9`](https://github.com/apache/spark/commit/108b7f970b9bcbd15db91b5108c928b4aaafa3ff). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC
AmplabJenkins commented on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC URL: https://github.com/apache/spark/pull/25336#issuecomment-519612936 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108827/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC
AmplabJenkins commented on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC URL: https://github.com/apache/spark/pull/25336#issuecomment-519612931 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC
SparkQA removed a comment on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC URL: https://github.com/apache/spark/pull/25336#issuecomment-519525828 **[Test build #108827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108827/testReport)** for PR 25336 at commit [`dd6f092`](https://github.com/apache/spark/commit/dd6f092f5cbc94eacca91dc8b9caba6c6b01746e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-519612250 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13917/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
AmplabJenkins commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-519612245 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC
SparkQA commented on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC URL: https://github.com/apache/spark/pull/25336#issuecomment-519612199 **[Test build #108827 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108827/testReport)** for PR 25336 at commit [`dd6f092`](https://github.com/apache/spark/commit/dd6f092f5cbc94eacca91dc8b9caba6c6b01746e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable
zsxwing commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable URL: https://github.com/apache/spark/pull/25382#issuecomment-519611520 > ideally we should respect the nullability ... Agreed with this. However, I think making the file source report the correct nullability is hard. It's either be inferred or user specified, which is error-prone. As Spark right now doesn't validate data to make sure nullability is correct, it can lead to weird NPE when processing the bad data, or corrupted files when writing. It would be hard for a Spark user to debug such errors. I don't know why we added this behavior for batch queriers. Ping @liancheng to comment this since he added it in https://github.com/apache/spark/pull/6285/files#diff-3d26956194a9a58c7eca9b364395e0c2R249 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable
SparkQA commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable URL: https://github.com/apache/spark/pull/25382#issuecomment-519610100 **[Test build #108840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108840/testReport)** for PR 25382 at commit [`b4878b9`](https://github.com/apache/spark/commit/b4878b9ed278657e5cd725546324801b4b9965dd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable
AmplabJenkins removed a comment on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable URL: https://github.com/apache/spark/pull/25382#issuecomment-519609259 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13916/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable
AmplabJenkins commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable URL: https://github.com/apache/spark/pull/25382#issuecomment-519609259 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13916/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable
AmplabJenkins removed a comment on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable URL: https://github.com/apache/spark/pull/25382#issuecomment-519609249 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable
AmplabJenkins commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable URL: https://github.com/apache/spark/pull/25382#issuecomment-519609249 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
AmplabJenkins removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519606413 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108832/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
AmplabJenkins removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519606407 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
AmplabJenkins commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519606413 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108832/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
AmplabJenkins commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519606407 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
SparkQA removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519542639 **[Test build #108832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108832/testReport)** for PR 25391 at commit [`4d58719`](https://github.com/apache/spark/commit/4d58719934a1f9374d730f134df21b21058a1275). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
SparkQA commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519605691 **[Test build #108832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108832/testReport)** for PR 25391 at commit [`4d58719`](https://github.com/apache/spark/commit/4d58719934a1f9374d730f134df21b21058a1275). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
SparkQA commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519595380 **[Test build #108839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108839/testReport)** for PR 25201 at commit [`997f141`](https://github.com/apache/spark/commit/997f141154e1f11f14252f2df4fd33df44e276e1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519594587 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519594604 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13915/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519594587 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519594604 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13915/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE]
Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r312126840
##
File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
##
@@ -683,8 +702,7 @@ private[deploy] class Master(
if (app.coresLeft >= coresPerExecutor) {
// Filter out workers that don't have enough resources to launch an
executor
val usableWorkers = workers.toArray.filter(_.state ==
WorkerState.ALIVE)
- .filter(worker => worker.memoryFree >= app.desc.memoryPerExecutorMB
&&
-worker.coresFree >= coresPerExecutor)
+ .filter(canLaunchExecutor(_, app.desc))
.sortBy(_.coresFree).reverse
Review comment:
How about this way?
```
for (app <- waitingApps) {
...
val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)
.filter(canLaunchExecutor(_, app.desc))
.sortBy(_.coresFree).reverse
if (waitingApps.size == 1 && usableWorkers.isEmpty) {
logWarn("The app requires more resources(mem, core, accelerator) than
any of Workers could have.")
}
...
}
```
Telling "the Workers are not configured with the resources(I mean
accelerator) as an app requested" may require more changes. For example, you
may need to traversal workers again to judge whether it's due to resources(I
mean accelerator) or memory or cores. Or, you need to refactor
`canLaunchExecutor` to tell more details.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE]
Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r312126840
##
File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
##
@@ -683,8 +702,7 @@ private[deploy] class Master(
if (app.coresLeft >= coresPerExecutor) {
// Filter out workers that don't have enough resources to launch an
executor
val usableWorkers = workers.toArray.filter(_.state ==
WorkerState.ALIVE)
- .filter(worker => worker.memoryFree >= app.desc.memoryPerExecutorMB
&&
-worker.coresFree >= coresPerExecutor)
+ .filter(canLaunchExecutor(_, app.desc))
.sortBy(_.coresFree).reverse
Review comment:
How about this way?
```
for (app <- waitingApps) {
...
val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)
.filter(canLaunchExecutor(_, app.desc))
.sortBy(_.coresFree).reverse
if (waitingApps.size == 1 && usableWorkers.isEmpty) {
logWarn("The app requires more resources(mem, core, accelerator) than
any of Workers could have.")
}
...
}
```
Telling the Workers are not configured with the resources(I mean
accelerator) as an app requested may require more changes. For example, you may
need to traversal workers again to judge whether it's due to resources(I mean
accelerator) or memory or cores. Or, you need to refactor `canLaunchExecutor`
to tell more details.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE]
Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r312126840
##
File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
##
@@ -683,8 +702,7 @@ private[deploy] class Master(
if (app.coresLeft >= coresPerExecutor) {
// Filter out workers that don't have enough resources to launch an
executor
val usableWorkers = workers.toArray.filter(_.state ==
WorkerState.ALIVE)
- .filter(worker => worker.memoryFree >= app.desc.memoryPerExecutorMB
&&
-worker.coresFree >= coresPerExecutor)
+ .filter(canLaunchExecutor(_, app.desc))
.sortBy(_.coresFree).reverse
Review comment:
How about this way?
```
for (app <- waitingApps) {
...
val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)
.filter(canLaunchExecutor(_, app.desc))
.sortBy(_.coresFree).reverse
if (waitingApps.size == 1 && usableWorkers.isEmpty) {
logWarn("The app requires more resources(mem, core, accelerator) than
any of Workers could have .")
}
...
}
```
Telling the Workers are not configured with the resources as an app
requested may require more changes. For example, you may need to traversal
workers again to judge whether it's due to resources(I mean accelerator) or
memory or cores. Or, you need to refactor `canLaunchExecutor` to tell more
details.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths
SparkQA commented on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths URL: https://github.com/apache/spark/pull/25305#issuecomment-519589345 **[Test build #108838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108838/testReport)** for PR 25305 at commit [`b7a5ac5`](https://github.com/apache/spark/commit/b7a5ac549c26153a3aa8bf3e73bd415929a78a1f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths
AmplabJenkins removed a comment on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths URL: https://github.com/apache/spark/pull/25305#issuecomment-519588551 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths
AmplabJenkins removed a comment on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths URL: https://github.com/apache/spark/pull/25305#issuecomment-519588561 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13914/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths
AmplabJenkins commented on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths URL: https://github.com/apache/spark/pull/25305#issuecomment-519588561 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13914/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths
AmplabJenkins commented on issue #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths URL: https://github.com/apache/spark/pull/25305#issuecomment-519588551 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24498: [SPARK-27605][UI] Add new column "Partition ID" to the tasks table in stages page
SparkQA commented on issue #24498: [SPARK-27605][UI] Add new column "Partition ID" to the tasks table in stages page URL: https://github.com/apache/spark/pull/24498#issuecomment-519586212 **[Test build #108837 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108837/testReport)** for PR 24498 at commit [`308a5a9`](https://github.com/apache/spark/commit/308a5a91df86d9638479d12ef0e1d331a00b58cb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths
brkyvz commented on a change in pull request #25305: [SPARK-28572][SQL] Simple
analyzer checks for v2 table creation code paths
URL: https://github.com/apache/spark/pull/25305#discussion_r312122520
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##
@@ -236,6 +237,46 @@ case class PreprocessTableCreation(sparkSession:
SparkSession) extends Rule[Logi
c.copy(tableDesc = normalizedTable.copy(schema = reorderedSchema))
}
+
+case create: V2TableCreation =>
+ val schema = create.tableSchema
+ val partitioning = create.partitioning
+ val identifier = create.tableName
+ val isCaseSensitive =
sparkSession.sessionState.conf.caseSensitiveAnalysis
+ // Check that columns are not duplicated in the schema
+ val flattenedSchema = SchemaUtils.explodeNestedFieldNames(schema)
+ SchemaUtils.checkColumnNameDuplication(
+flattenedSchema,
+s"in the table definition of $identifier",
+isCaseSensitive)
+
+ // Check that columns are not duplicated in the partitioning statement
+ SchemaUtils.checkTransformDuplication(
+partitioning, "in the partitioning", isCaseSensitive)
+
+ if (schema.isEmpty) {
Review comment:
no. Existing data may exist for a table, and all that you may want is to
create a pointer to that table (Think of creating a pointer to a JDBC table in
the MetaStore)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server
AmplabJenkins removed a comment on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server URL: https://github.com/apache/spark/pull/25373#issuecomment-519585829 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108826/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server
AmplabJenkins removed a comment on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server URL: https://github.com/apache/spark/pull/25373#issuecomment-519585817 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server
AmplabJenkins commented on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server URL: https://github.com/apache/spark/pull/25373#issuecomment-519585829 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108826/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server
AmplabJenkins commented on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server URL: https://github.com/apache/spark/pull/25373#issuecomment-519585817 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC
MaxGekk commented on a change in pull request #25336: [SPARK-28017][SQL]
Support additional levels of truncations by DATE_TRUNC/TRUNC
URL: https://github.com/apache/spark/pull/25336#discussion_r312121856
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##
@@ -1471,14 +1471,24 @@ trait TruncInstant extends BinaryExpression with
ImplicitCastInputTypes {
@ExpressionDescription(
usage = """
_FUNC_(date, fmt) - Returns `date` with the time portion of the day
truncated to the unit specified by the format model `fmt`.
-`fmt` should be one of ["year", "", "yy", "mon", "month", "mm"]
+`fmt` should be one of ["week", "mon", "month", "mm", "quarter", "year",
"", "yy", "decade", "century", "millennium"]
""",
examples = """
Examples:
+ > SELECT _FUNC_('2019-08-04', 'week');
+ 2019-07-29
+ > SELECT _FUNC_('2019-08-04', 'quarter');
+ 2019-07-01
> SELECT _FUNC_('2009-02-12', 'MM');
2009-02-01
> SELECT _FUNC_('2015-10-27', 'YEAR');
2015-01-01
+ > SELECT _FUNC_('2015-10-27', 'DECADE');
+ 2010-01-01
+ > SELECT _FUNC_('1981-01-19', 'century');
+ 1901-01-01
+ > SELECT _FUNC_('1981-01-19', 'millennium');
+ 1001-01-01
Review comment:
Century (and millennium as well) starts from the first year, see
https://en.wikipedia.org/wiki/Century#Viewpoint_1:_Strict_usage and
https://en.wikipedia.org/wiki/Millennium#Viewpoint_1:_Strict_usage . Decades
are calculated as PostgreSQL does which follows common practice:
https://en.wikipedia.org/wiki/List_of_decades
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server
SparkQA removed a comment on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server URL: https://github.com/apache/spark/pull/25373#issuecomment-519516464 **[Test build #108826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108826/testReport)** for PR 25373 at commit [`3f21189`](https://github.com/apache/spark/commit/3f211898ffa1f6303751c2737e148c5f46b77266). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server
SparkQA commented on issue #25373: [SPARK-28527][SQL][TEST] Directly re-run all the tests in SQLQueryTestSuite via Thrift Server URL: https://github.com/apache/spark/pull/25373#issuecomment-519585045 **[Test build #108826 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108826/testReport)** for PR 25373 at commit [`3f21189`](https://github.com/apache/spark/commit/3f211898ffa1f6303751c2737e148c5f46b77266). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths
brkyvz commented on a change in pull request #25305: [SPARK-28572][SQL] Simple
analyzer checks for v2 table creation code paths
URL: https://github.com/apache/spark/pull/25305#discussion_r312120351
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -1193,3 +1211,17 @@ case class Deduplicate(
override def output: Seq[Attribute] = child.output
}
+
+/** A trait used for logical plan nodes that create or replace V2 table
definitions. */
+sealed trait V2TableCreation extends LogicalPlan {
Review comment:
doesn't need to be sealed. All the definitions were here anyway.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #25305: [SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths
brkyvz commented on a change in pull request #25305: [SPARK-28572][SQL] Simple
analyzer checks for v2 table creation code paths
URL: https://github.com/apache/spark/pull/25305#discussion_r312120029
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -1193,3 +1211,17 @@ case class Deduplicate(
override def output: Seq[Attribute] = child.output
}
+
+/** A trait used for logical plan nodes that create or replace V2 table
definitions. */
+sealed trait V2TableCreation extends LogicalPlan {
+ def tableName: Identifier
+ def partitioning: Seq[Transform]
+ def tableSchema: StructType
+
+ /**
+ * Creates a copy of this node with the new partitoning transforms. This
method is used to
+ * rewrite these nodes after normalizing the field references for these
transforms according to
+ * case sensitivity.
+ */
+ def withPartitioning(rewritten: Seq[Transform]): V2TableCreation
Review comment:
yup
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table
AmplabJenkins removed a comment on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table URL: https://github.com/apache/spark/pull/25375#issuecomment-519579070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13913/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table
AmplabJenkins removed a comment on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table URL: https://github.com/apache/spark/pull/25375#issuecomment-519579064 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table
AmplabJenkins commented on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table URL: https://github.com/apache/spark/pull/25375#issuecomment-519579064 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table
AmplabJenkins commented on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table URL: https://github.com/apache/spark/pull/25375#issuecomment-519579070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13913/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC
cloud-fan commented on a change in pull request #25336: [SPARK-28017][SQL]
Support additional levels of truncations by DATE_TRUNC/TRUNC
URL: https://github.com/apache/spark/pull/25336#discussion_r312113545
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##
@@ -1471,14 +1471,24 @@ trait TruncInstant extends BinaryExpression with
ImplicitCastInputTypes {
@ExpressionDescription(
usage = """
_FUNC_(date, fmt) - Returns `date` with the time portion of the day
truncated to the unit specified by the format model `fmt`.
-`fmt` should be one of ["year", "", "yy", "mon", "month", "mm"]
+`fmt` should be one of ["week", "mon", "month", "mm", "quarter", "year",
"", "yy", "decade", "century", "millennium"]
""",
examples = """
Examples:
+ > SELECT _FUNC_('2019-08-04', 'week');
+ 2019-07-29
+ > SELECT _FUNC_('2019-08-04', 'quarter');
+ 2019-07-01
> SELECT _FUNC_('2009-02-12', 'MM');
2009-02-01
> SELECT _FUNC_('2015-10-27', 'YEAR');
2015-01-01
+ > SELECT _FUNC_('2015-10-27', 'DECADE');
+ 2010-01-01
+ > SELECT _FUNC_('1981-01-19', 'century');
+ 1901-01-01
+ > SELECT _FUNC_('1981-01-19', 'millennium');
+ 1001-01-01
Review comment:
just for curiosity: why `century` and `millennium` need to add one year but
`DECADE` doesn't need to?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #25047: [SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
tgravescs commented on a change in pull request #25047: [SPARK-27371][CORE]
Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r312111378
##
File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
##
@@ -683,8 +702,7 @@ private[deploy] class Master(
if (app.coresLeft >= coresPerExecutor) {
// Filter out workers that don't have enough resources to launch an
executor
val usableWorkers = workers.toArray.filter(_.state ==
WorkerState.ALIVE)
- .filter(worker => worker.memoryFree >= app.desc.memoryPerExecutorMB
&&
-worker.coresFree >= coresPerExecutor)
+ .filter(canLaunchExecutor(_, app.desc))
.sortBy(_.coresFree).reverse
Review comment:
Right if something changes (other app, other workers, etc) it retries, but
if I'm the only app on the cluster its not clear why the app isn't launching.
The one thing I don't want is it to be to noisy though either. I thought about
that before making the comment, because like you said if its just out of
resources because other apps are running we don't really want to print
anything. I think for now we should just limit it to resources and perhaps
just say no Workers are configured with the resources you requested. If we can
do that without much performance impact lets do it. If not maybe we just file a
separate jira for it and look at it there
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #25375: [SPARK-28642][SQL] Hide credentials in show create table
wangyum commented on a change in pull request #25375: [SPARK-28642][SQL] Hide
credentials in show create table
URL: https://github.com/apache/spark/pull/25375#discussion_r312111387
##
File path: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
##
@@ -1029,6 +1030,32 @@ class JDBCSuite extends QueryTest
}
}
+ test("Hide credentials in show create table") {
+val password = "testPass"
Review comment:
Do you mean remove `password` declaration?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table
SparkQA commented on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table URL: https://github.com/apache/spark/pull/25375#issuecomment-519576722 **[Test build #108836 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108836/testReport)** for PR 25375 at commit [`20bf38a`](https://github.com/apache/spark/commit/20bf38a931820fbb48b8f75dae453c0fd4a28f5c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519571992 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108831/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519571985 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519571985 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519571992 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108831/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
SparkQA removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519532489 **[Test build #108831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108831/testReport)** for PR 25201 at commit [`dd5ce26`](https://github.com/apache/spark/commit/dd5ce263676c51a0f23db6bcc678539e07ba6f52). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
SparkQA commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519571575 **[Test build #108831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108831/testReport)** for PR 25201 at commit [`dd5ce26`](https://github.com/apache/spark/commit/dd5ce263676c51a0f23db6bcc678539e07ba6f52). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4]
AmplabJenkins removed a comment on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4] URL: https://github.com/apache/spark/pull/25392#issuecomment-519569119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13912/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4]
AmplabJenkins removed a comment on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4] URL: https://github.com/apache/spark/pull/25392#issuecomment-519569108 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4]
SparkQA commented on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4] URL: https://github.com/apache/spark/pull/25392#issuecomment-519570039 **[Test build #108835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108835/testReport)** for PR 25392 at commit [`9ead274`](https://github.com/apache/spark/commit/9ead27401c5b57c71bc46221cfdbc26745fa2a15). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4]
AmplabJenkins commented on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4] URL: https://github.com/apache/spark/pull/25392#issuecomment-519569119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13912/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4]
AmplabJenkins commented on issue #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4] URL: https://github.com/apache/spark/pull/25392#issuecomment-519569108 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum opened a new pull request #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4]
wangyum opened a new pull request #25392: [SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4] URL: https://github.com/apache/spark/pull/25392 ## What changes were proposed in this pull request? This PR is to port AGGREGATES.sql from PostgreSQL regression tests. https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/aggregates.sql#L607-L997 The expected results can be found in the link: https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/expected/aggregates.out#L1615-L2289 When porting the test cases, found five PostgreSQL specific features that do not exist in Spark SQL: [SPARK-27980](https://issues.apache.org/jira/browse/SPARK-27980): Ordered-Set Aggregate Functions [SPARK-28661](https://issues.apache.org/jira/browse/SPARK-28661): Hypothetical-Set Aggregate Functions [SPARK-28382](https://issues.apache.org/jira/browse/SPARK-28382): Array Functions: unnest [SPARK-28663](https://issues.apache.org/jira/browse/SPARK-28663): Aggregate Functions for Statistics [SPARK-28664](https://issues.apache.org/jira/browse/SPARK-28664): ORDER BY in aggregate function ## How was this patch tested? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE]
Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r312094305
##
File path: core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
##
@@ -48,21 +48,21 @@ private[spark] case class ResourceRequest(
discoveryScript: Option[String],
vendor: Option[String])
-private[spark] case class TaskResourceRequirement(resourceName: String,
amount: Int)
+private[spark] case class ResourceRequirement(resourceName: String, amount:
Int)
/**
* Case class representing allocated resource addresses for a specific
resource.
* Cluster manager uses the JSON serialization of this case class to pass
allocated resource info to
* driver and executors. See the ``--resourcesFile`` option there.
*/
-private[spark] case class ResourceAllocation(id: ResourceID, addresses:
Seq[String]) {
+@Evolving
+case class ResourceAllocation(id: ResourceID, addresses: Seq[String]) {
Review comment:
never mind.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()
SparkQA commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier() URL: https://github.com/apache/spark/pull/25235#issuecomment-519563522 **[Test build #108834 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108834/testReport)** for PR 25235 at commit [`42989e6`](https://github.com/apache/spark/commit/42989e611cdc8ab31e28315d19498b0883c8b45a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
Ngone51 commented on a change in pull request #25047: [SPARK-27371][CORE]
Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r312094080
##
File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
##
@@ -683,8 +702,7 @@ private[deploy] class Master(
if (app.coresLeft >= coresPerExecutor) {
// Filter out workers that don't have enough resources to launch an
executor
val usableWorkers = workers.toArray.filter(_.state ==
WorkerState.ALIVE)
- .filter(worker => worker.memoryFree >= app.desc.memoryPerExecutorMB
&&
-worker.coresFree >= coresPerExecutor)
+ .filter(canLaunchExecutor(_, app.desc))
.sortBy(_.coresFree).reverse
Review comment:
Actually, it does retry when other executors or drivers finish. But, we can
warn if executor or driver requires more resources than any of workers could
have. BTW, I'm thinking do we have the same issue for memory and cores ? For
example, a Worker has 10 cores at most while an executor ask for 20 cores ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()
AmplabJenkins removed a comment on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier() URL: https://github.com/apache/spark/pull/25235#issuecomment-519562628 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()
AmplabJenkins removed a comment on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier() URL: https://github.com/apache/spark/pull/25235#issuecomment-519562638 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13911/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()
AmplabJenkins commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier() URL: https://github.com/apache/spark/pull/25235#issuecomment-519562638 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13911/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
HyukjinKwon commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519562507 I am okay with this logging stuff though don't feel strongly about it. Let me leave it to @cloud-fan. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()
AmplabJenkins commented on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier() URL: https://github.com/apache/spark/pull/25235#issuecomment-519562628 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
HyukjinKwon commented on a change in pull request #25391:
[SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single
large unsplittable file
URL: https://github.com/apache/spark/pull/25391#discussion_r312092543
##
File path: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
##
@@ -215,8 +215,7 @@ class HadoopRDD[K, V](
val codecFactory = new CompressionCodecFactory(jobConf)
if (Utils.isFileSplittable(path, codecFactory)) {
logWarning(s"Loading one large file ${path.toString} with only one
partition, " +
- s"we can increase partition numbers by the `minPartitions`
argument in method " +
- "`sc.textFile`")
+ s"we can increase partition numbers for improving performance.")
Review comment:
I am okay with it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25289: [SPARK-27889][INFRA] Make development scripts under dev/ support Python 3
HyukjinKwon commented on a change in pull request #25289: [SPARK-27889][INFRA]
Make development scripts under dev/ support Python 3
URL: https://github.com/apache/spark/pull/25289#discussion_r312088928
##
File path: dev/github_jira_sync.py
##
@@ -122,13 +129,13 @@ def reset_pr_labels(pr_num, jira_components):
url = '%s/issues/%s/labels' % (GITHUB_API_BASE, pr_num)
labels = ', '.join(('"%s"' % c) for c in jira_components)
try:
-request = urllib2.Request(url, data='{"labels":[%s]}' % labels)
+request = Request(url, data=('{"labels":[%s]}' %
labels).encode('utf-8'))
Review comment:
Ah, hm, there are multiple places like this.
Okay, NVM. Components won't likely be out of ascii (which might be happen
during implicit conversion bytes -> unicodes in Python 2)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25289: [SPARK-27889][INFRA] Make development scripts under dev/ support Python 3
HyukjinKwon commented on a change in pull request #25289: [SPARK-27889][INFRA]
Make development scripts under dev/ support Python 3
URL: https://github.com/apache/spark/pull/25289#discussion_r312085007
##
File path: dev/github_jira_sync.py
##
@@ -122,13 +129,13 @@ def reset_pr_labels(pr_num, jira_components):
url = '%s/issues/%s/labels' % (GITHUB_API_BASE, pr_num)
labels = ', '.join(('"%s"' % c) for c in jira_components)
try:
-request = urllib2.Request(url, data='{"labels":[%s]}' % labels)
+request = Request(url, data=('{"labels":[%s]}' %
labels).encode('utf-8'))
Review comment:
@WeichenXu123, can use `u'{"labels":[%s]}' % labels).encode('utf-8')` so
that it's always an unicode in both Python 2 and Python 3, and always executes
same code path?
Currently,
- In Python 2, it will be bytes -> unicodes -> bytes (by implicit conversion)
- in Python 3, unicodes -> bytes.
After using `u`,
Both will do unicodes -> bytes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519552589 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519552598 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13910/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519552589 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519552598 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13910/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
ulysses-you commented on a change in pull request #25198: [SPARK-28443][SQL]
Spark sql add exception when create field type NullType
URL: https://github.com/apache/spark/pull/25198#discussion_r312077202
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##
@@ -2516,4 +2516,13 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-28443: fail the DDL command if it creates a table with null-type
columns") {
Review comment:
I forget that spark not support `create table t as select null as c` now.
Remove this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519549786 **[Test build #108833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108833/testReport)** for PR 25198 at commit [`108b7f9`](https://github.com/apache/spark/commit/108b7f970b9bcbd15db91b5108c928b4aaafa3ff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
ulysses-you commented on a change in pull request #25198: [SPARK-28443][SQL]
Spark sql add exception when create field type NullType
URL: https://github.com/apache/spark/pull/25198#discussion_r312056710
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##
@@ -2516,4 +2516,13 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-28443: fail the DDL command if it creates a table with null-type
columns") {
+withTable("t") {
+ val e = intercept[AnalysisException]{
+sql(s"CREATE TABLE t as SELECT NULL AS c")
Review comment:
Nested fields are tested in `CatalogSuite`. This UT is just to confirm that
cannot using CTAS with `NullType`. I have not seen other place confirm this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
AmplabJenkins removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519545253 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13909/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
AmplabJenkins removed a comment on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519545248 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
AmplabJenkins commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519545248 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
AmplabJenkins commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519545253 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13909/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
SparkQA commented on issue #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391#issuecomment-519542639 **[Test build #108832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108832/testReport)** for PR 25391 at commit [`4d58719`](https://github.com/apache/spark/commit/4d58719934a1f9374d730f134df21b21058a1275). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on issue #25289: [SPARK-27889][INFRA] Make development scripts under dev/ support Python 3
WeichenXu123 commented on issue #25289: [SPARK-27889][INFRA] Make development scripts under dev/ support Python 3 URL: https://github.com/apache/spark/pull/25289#issuecomment-519542648 @srowen Yes I test on both py2 and py3. Let's wait @HyukjinKwon and @dongjoon-hyun 's opinion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks when executors are lost
squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks when executors are lost URL: https://github.com/apache/spark/pull/24462#issuecomment-519542105 ok, I see what you're trying to do -- and yeah I don't think you can do it with the api we are proposing. That is a bummer, but we also need to try to draw a reasonable balance here, so I don't know if this case is really that compelling for extending the api. Are you sure your cache feature really saves you that much? You've still got to make a remote read for the index file This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WeichenXu123 opened a new pull request #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file
WeichenXu123 opened a new pull request #25391: [SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25391 ## What changes were proposed in this pull request? * Add log in `NewHadoopRDD` * Remove some words in logs which related to specific user API. ## How was this patch tested? Manual. Please review https://spark.apache.org/contributing.html before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] squito commented on issue #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter
squito commented on issue #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter URL: https://github.com/apache/spark/pull/25342#issuecomment-519537297 @gczsjdy > Will spill be supported in the series of PRs? No, spill is still to local disk. trying to generalize local spills was explicitly out of scope for now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
ulysses-you commented on a change in pull request #25198: [SPARK-28443][SQL]
Spark sql add exception when create field type NullType
URL: https://github.com/apache/spark/pull/25198#discussion_r312056710
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##
@@ -2516,4 +2516,13 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-28443: fail the DDL command if it creates a table with null-type
columns") {
+withTable("t") {
+ val e = intercept[AnalysisException]{
+sql(s"CREATE TABLE t as SELECT NULL AS c")
Review comment:
Nested fields are tested in `CatalogSuite`. This UT is just to confirm that
cannot using CTAS with `NullType`. I have not seen other place confirm this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
ulysses-you commented on a change in pull request #25198: [SPARK-28443][SQL]
Spark sql add exception when create field type NullType
URL: https://github.com/apache/spark/pull/25198#discussion_r312056710
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##
@@ -2516,4 +2516,13 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-28443: fail the DDL command if it creates a table with null-type
columns") {
+withTable("t") {
+ val e = intercept[AnalysisException]{
+sql(s"CREATE TABLE t as SELECT NULL AS c")
Review comment:
Nested fields are tested in `CatalogSuite`. This UT is just to confirm that
cannot using CTAS with `NullType`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #25349: [SPARK-28538][UI][WIP] Document SQL page
srowen commented on a change in pull request #25349: [SPARK-28538][UI][WIP] Document SQL page URL: https://github.com/apache/spark/pull/25349#discussion_r312056394 ## File path: docs/monitoring.md ## @@ -23,7 +23,7 @@ There are several ways to monitor Spark applications: web UIs, metrics, and exte # Web Interfaces -Every SparkContext launches a web UI, by default on port 4040, that +Every SparkContext launches a [Web UI](web-ui.html), by default on port 4040, that Review comment: I think it's in `_layouts/global.html`, but I am not sure this needs a top level menu item. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
SparkQA commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519532489 **[Test build #108831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108831/testReport)** for PR 25201 at commit [`dd5ce26`](https://github.com/apache/spark/commit/dd5ce263676c51a0f23db6bcc678539e07ba6f52). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #22997: SPARK-25999: make-distribution.sh failure with --r and -Phadoop-provided
srowen commented on issue #22997: SPARK-25999: make-distribution.sh failure with --r and -Phadoop-provided URL: https://github.com/apache/spark/pull/22997#issuecomment-519532289 See comments above. I think this breaks what `hadoop-provided` is for This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519531590 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins removed a comment on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519531598 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13908/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication .
AmplabJenkins commented on issue #25201: [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . URL: https://github.com/apache/spark/pull/25201#issuecomment-519531598 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13908/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
