[GitHub] [spark] SparkQA removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
SparkQA removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519372350 **[Test build #108796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108796/testReport)** for PR 25379 at commit [`c75b1dd`](https://github.com/apache/spark/commit/c75b1ddc411487fffbf54e36cbd2bcc8e25a0fc7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
AmplabJenkins removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519377128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108796/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
AmplabJenkins removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519377122 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
SparkQA commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519377040 **[Test build #108796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108796/testReport)** for PR 25379 at commit [`c75b1dd`](https://github.com/apache/spark/commit/c75b1ddc411487fffbf54e36cbd2bcc8e25a0fc7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
AmplabJenkins commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519377128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108796/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
AmplabJenkins commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519377122 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #25369: [SPARK-28638][WebUI] Task summary metrics are wrong when there are running tasks
shahidki31 commented on a change in pull request #25369: [SPARK-28638][WebUI]
Task summary metrics are wrong when there are running tasks
URL: https://github.com/apache/spark/pull/25369#discussion_r311719400
##
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##
@@ -156,7 +162,8 @@ private[spark] class AppStatusStore(
// cheaper for disk stores (avoids deserialization).
val count = {
Utils.tryWithResource(
-if (store.isInstanceOf[InMemoryStore]) {
+if (isInMemoryStore) {
Review comment:
@gengliangwang If I understand correctly, ~~#23008~~ #23088 is actually
fixing this issue. right ?(At least history server case).
Because, `count` is always filtering out the running tasks, as
`ExecutorRunTime` will be defined for only finished tasks. But `scan tasks`
contains all the tasks, including running tasks.
https://github.com/apache/spark/blob/a59fdc4b5783be591a236bfc60d1107caa818412/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala#L169
https://github.com/apache/spark/blob/a59fdc4b5783be591a236bfc60d1107caa818412/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala#L263-L265
So, reverting ~~23008~~ 23088 and hence this PR, would not fix the issue?
Please correct me if I am wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
SparkQA commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519372350 **[Test build #108796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108796/testReport)** for PR 25379 at commit [`c75b1dd`](https://github.com/apache/spark/commit/c75b1ddc411487fffbf54e36cbd2bcc8e25a0fc7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
AmplabJenkins removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519371872 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
AmplabJenkins removed a comment on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519371876 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13876/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
AmplabJenkins commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519371872 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
AmplabJenkins commented on issue #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#issuecomment-519371876 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13876/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
SparkQA commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-519370553 **[Test build #108795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108795/testReport)** for PR 24525 at commit [`4ff4112`](https://github.com/apache/spark/commit/4ff411209d973a19db33cce2d7878c0c18ab88b6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
AmplabJenkins removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-519370022 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13875/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
AmplabJenkins removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-519370020 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
AmplabJenkins commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-519370020 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
viirya commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-519370061 @cloud-fan @dongjoon-hyun Please let me know if you have more comments or thoughts on this? thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
AmplabJenkins commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-519370022 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13875/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519368280 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519368286 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108793/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519368280 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519368286 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108793/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
SparkQA removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519329542 **[Test build #108793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108793/testReport)** for PR 25333 at commit [`7baa349`](https://github.com/apache/spark/commit/7baa3492ecba858fe1258f9b8c5bb7d64edea03d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
SparkQA commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519367941 **[Test build #108793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108793/testReport)** for PR 25333 at commit [`7baa349`](https://github.com/apache/spark/commit/7baa3492ecba858fe1258f9b8c5bb7d64edea03d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
AmplabJenkins removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519366129 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
AmplabJenkins removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519366135 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108792/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
AmplabJenkins commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519366129 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
SparkQA removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519329534 **[Test build #108792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108792/testReport)** for PR 25377 at commit [`2ecaae0`](https://github.com/apache/spark/commit/2ecaae0e6aa825ef49e3feb0a262d8564b0b64ef). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
AmplabJenkins commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519366135 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108792/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
wangyum commented on a change in pull request #25379: [SPARK-28474][SQL] Hive
0.12 JDBC client can not handle binary type
URL: https://github.com/apache/spark/pull/25379#discussion_r311856992
##
File path:
sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java
##
@@ -195,7 +195,7 @@ public static TColumnValue toTColumnValue(Type type,
Object value) {
case DECIMAL_TYPE:
return stringValue(((HiveDecimal)value));
case BINARY_TYPE:
- return stringValue((String)value);
+ return stringValue(new String((byte[])value));
Review comment:
How about:
```stringValue(UTF8String.fromBytes((byte[])value).toString());```?
https://github.com/apache/spark/blob/b367b323d2d69c6fcb8d4c83d87813e3924b7dc8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L245
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
SparkQA commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519365756 **[Test build #108792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108792/testReport)** for PR 25377 at commit [`2ecaae0`](https://github.com/apache/spark/commit/2ecaae0e6aa825ef49e3feb0a262d8564b0b64ef). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shanyu commented on issue #22997: SPARK-25999: make-distribution.sh failure with --r and -Phadoop-provided
shanyu commented on issue #22997: SPARK-25999: make-distribution.sh failure with --r and -Phadoop-provided URL: https://github.com/apache/spark/pull/22997#issuecomment-519361866 Is there any way to reconsider this patch? thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shanyu commented on issue #22997: SPARK-25999: make-distribution.sh failure with --r and -Phadoop-provided
shanyu commented on issue #22997: SPARK-25999: make-distribution.sh failure
with --r and -Phadoop-provided
URL: https://github.com/apache/spark/pull/22997#issuecomment-519361762
I'm sorry this PR completely slipped out of my mind. Sorry for the late
reply.
I copied hadoop dependencies to
$SPARK_HOME/assembly/target/scala-${SCALA_VERSION}/jars, that's where these
jars are read during building R packages. This is not where we package the tar
ball. The folder where we create the dist tar ball is defined at the top of
this file:
DISTDIR="$SPARK_HOME/dist"
And we've done copying from the assembly folder to the dist folder earlier:
cp "$SPARK_HOME"/assembly/target/scala*/jars/* "$DISTDIR/jars/"
If we build with hadoop, then hadoop jars will end up in the assembly folder
first, then copied to dist folder. If we don't choose to build with hadoop,
then before building R packages, we copy the dependencies to assembly folder.
These jars won't get in the dist tar ball. I've tested this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519361621 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108791/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519361612 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519361612 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519361621 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108791/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
SparkQA removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519321171 **[Test build #108791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108791/testReport)** for PR 25198 at commit [`517a1e6`](https://github.com/apache/spark/commit/517a1e6e6014539fded96cf4b46353fc61ce1f1a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-519361215 **[Test build #108791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108791/testReport)** for PR 25198 at commit [`517a1e6`](https://github.com/apache/spark/commit/517a1e6e6014539fded96cf4b46353fc61ce1f1a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519357596 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108794/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519357592 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519357596 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108794/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519357592 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable
cloud-fan commented on issue #25382: [SPARK-28651][SS]Force streaming file source to be nullable URL: https://github.com/apache/spark/pull/25382#issuecomment-519357567 eventually we should allow data source to report its nullability, but Spark should add validation when read data to make sure the nullability is correct. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519357399 **[Test build #108794 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108794/testReport)** for PR 25383 at commit [`3af602e`](https://github.com/apache/spark/commit/3af602e58f7d6c6d579ca10672dfc87c07a92c5f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519345507 **[Test build #108794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108794/testReport)** for PR 25383 at commit [`3af602e`](https://github.com/apache/spark/commit/3af602e58f7d6c6d579ca10672dfc87c07a92c5f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gczsjdy commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks when executors are lost
gczsjdy commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks
when executors are lost
URL: https://github.com/apache/spark/pull/24462#issuecomment-519354047
@squito Index and data files are both stored on DFS, the difference is that:
data files are directly read from DFS, however, for index files, a reducer
fetches them from the executors('s cache) who wrote them, if there aren't
required index files in cache, they will be loaded from DFS. This approach
simulates the external shuffle service's cache, but instead of in another Java
process, it's in Executor.
This approach needs a reasonable place(and it's the coordinated map
executor) to cache index files. Returning a `None` location for mapper task
will make
- The no resubmit tasks' need satisfied
- But the cache feature not satisfied : (
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519345215 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13874/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins removed a comment on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519345210 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519345507 **[Test build #108794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108794/testReport)** for PR 25383 at commit [`3af602e`](https://github.com/apache/spark/commit/3af602e58f7d6c6d579ca10672dfc87c07a92c5f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519345215 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13874/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
AmplabJenkins commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-519345210 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng opened a new pull request #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
zhengruifeng opened a new pull request #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383 ## What changes were proposed in this pull request? Tree-based feature transformation is a widely used feature and already implemented in many famous libraries, like sklearn/xgboost/lightgbm/catboost. But is still missing in ML. The previous discussions and design doc can be found in [SPARK-13677](https://issues.apache.org/jira/browse/SPARK-13677), which is the only left subtask in 'GBT improvement umbrella' [SPARK-14047](https://issues.apache.org/jira/browse/SPARK-14047). This pr is to add tree-based feature transformation. ## How was this patch tested? existing and added suites This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #25117: [SPARK-28454][PYTHON] Validate LongType in `createDataFrame(verifySchema=True)`
HyukjinKwon closed pull request #25117: [SPARK-28454][PYTHON] Validate LongType in `createDataFrame(verifySchema=True)` URL: https://github.com/apache/spark/pull/25117 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25117: [SPARK-28454][PYTHON] Validate LongType in `createDataFrame(verifySchema=True)`
HyukjinKwon commented on issue #25117: [SPARK-28454][PYTHON] Validate LongType in `createDataFrame(verifySchema=True)` URL: https://github.com/apache/spark/pull/25117#issuecomment-519341802 Thanks for the contribution, @simplylizz This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25117: [SPARK-28454][PYTHON] Validate LongType in `createDataFrame(verifySchema=True)`
HyukjinKwon commented on issue #25117: [SPARK-28454][PYTHON] Validate LongType in `createDataFrame(verifySchema=True)` URL: https://github.com/apache/spark/pull/25117#issuecomment-519341521 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25253: [SPARK-28470][SQL] Cast to decimal throws ArithmeticException on overflow
HyukjinKwon commented on issue #25253: [SPARK-28470][SQL] Cast to decimal throws ArithmeticException on overflow URL: https://github.com/apache/spark/pull/25253#issuecomment-519340794 late LGTM too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25253: [SPARK-28470][SQL] Cast to decimal throws ArithmeticException on overflow
HyukjinKwon commented on a change in pull request #25253: [SPARK-28470][SQL]
Cast to decimal throws ArithmeticException on overflow
URL: https://github.com/apache/spark/pull/25253#discussion_r311836194
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -498,22 +499,34 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
b => x.numeric.asInstanceOf[Numeric[Any]].toInt(b).toByte
}
+ private val nullOnOverflow = SQLConf.get.decimalOperationsNullOnOverflow
+
/**
* Change the precision / scale in a given decimal to those set in
`decimalType` (if any),
* returning null if it overflows or modifying `value` in-place and
returning it if successful.
*
* NOTE: this modifies `value` in-place, so don't call it on external data.
*/
private[this] def changePrecision(value: Decimal, decimalType: DecimalType):
Decimal = {
-if (value.changePrecision(decimalType.precision, decimalType.scale)) value
else null
+if (value.changePrecision(decimalType.precision, decimalType.scale)) {
+ value
+} else {
Review comment:
ah, that's fine. no big deal.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
HyukjinKwon commented on a change in pull request #25379: [SPARK-28474][SQL]
Hive 0.12 JDBC client can not handle binary type
URL: https://github.com/apache/spark/pull/25379#discussion_r311835530
##
File path:
sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java
##
@@ -195,7 +195,7 @@ public static TColumnValue toTColumnValue(Type type,
Object value) {
case DECIMAL_TYPE:
return stringValue(((HiveDecimal)value));
case BINARY_TYPE:
- return stringValue((String)value);
+ return stringValue(new String((byte[])value));
Review comment:
String roundtrip with system default encoding (say, UTF-8) could be broken.
Can we add a test? If that works, then should be good to go.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence
AmplabJenkins removed a comment on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence URL: https://github.com/apache/spark/pull/25381#issuecomment-519339803 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108789/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default
AmplabJenkins removed a comment on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default URL: https://github.com/apache/spark/pull/24208#issuecomment-519339716 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence
AmplabJenkins removed a comment on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence URL: https://github.com/apache/spark/pull/25381#issuecomment-519339798 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence
AmplabJenkins commented on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence URL: https://github.com/apache/spark/pull/25381#issuecomment-519339803 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108789/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default
AmplabJenkins commented on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default URL: https://github.com/apache/spark/pull/24208#issuecomment-519339716 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default
AmplabJenkins removed a comment on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default URL: https://github.com/apache/spark/pull/24208#issuecomment-519339721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108790/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence
AmplabJenkins commented on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence URL: https://github.com/apache/spark/pull/25381#issuecomment-519339798 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default
AmplabJenkins commented on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default URL: https://github.com/apache/spark/pull/24208#issuecomment-519339721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108790/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence
SparkQA removed a comment on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence URL: https://github.com/apache/spark/pull/25381#issuecomment-519310920 **[Test build #108789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108789/testReport)** for PR 25381 at commit [`af17881`](https://github.com/apache/spark/commit/af178819eda2ac1bbd4ef6b999ee96499b7135ba). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence
SparkQA commented on issue #25381: [SPARK-28531][SQL] Improve Extract Python UDFs optimizer rule to enforce idempotence URL: https://github.com/apache/spark/pull/25381#issuecomment-519339663 **[Test build #108789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108789/testReport)** for PR 25381 at commit [`af17881`](https://github.com/apache/spark/commit/af178819eda2ac1bbd4ef6b999ee96499b7135ba). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default
SparkQA removed a comment on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default URL: https://github.com/apache/spark/pull/24208#issuecomment-519312548 **[Test build #108790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108790/testReport)** for PR 24208 at commit [`65f2eca`](https://github.com/apache/spark/commit/65f2eca35576d0fed398c7eb100a1b03249425a9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default
SparkQA commented on issue #24208: [SPARK-27272][CORE] Enable blacklisting of node/executor on fetch failures by default URL: https://github.com/apache/spark/pull/24208#issuecomment-519339316 **[Test build #108790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108790/testReport)** for PR 24208 at commit [`65f2eca`](https://github.com/apache/spark/commit/65f2eca35576d0fed398c7eb100a1b03249425a9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #25369: [SPARK-28638][WebUI] Task summary metrics are wrong when there are running tasks
shahidki31 commented on a change in pull request #25369: [SPARK-28638][WebUI]
Task summary metrics are wrong when there are running tasks
URL: https://github.com/apache/spark/pull/25369#discussion_r311719400
##
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##
@@ -156,7 +162,8 @@ private[spark] class AppStatusStore(
// cheaper for disk stores (avoids deserialization).
val count = {
Utils.tryWithResource(
-if (store.isInstanceOf[InMemoryStore]) {
+if (isInMemoryStore) {
Review comment:
@gengliangwang If I understand correctly, #23088 is actually fixing this
issue. right ?(At least history server case).
Because, `count` is always filtering out the running tasks, as
`ExecutorRunTime` will be defined for only finished tasks. But `scan tasks`
contains all the tasks, including running tasks.
https://github.com/apache/spark/blob/a59fdc4b5783be591a236bfc60d1107caa818412/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala#L169
https://github.com/apache/spark/blob/a59fdc4b5783be591a236bfc60d1107caa818412/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala#L263-L265
So, reverting 23088 and hence this PR, would not fix the issue? Please
correct me if I am wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
HyukjinKwon commented on a change in pull request #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#discussion_r311832287 ## File path: sql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/ColumnValue.java ## @@ -199,7 +199,7 @@ public static TColumnValue toTColumnValue(TypeDescriptor typeDescriptor, Object case DECIMAL_TYPE: return stringValue((HiveDecimal)value, typeDescriptor); case BINARY_TYPE: - return stringValue((String)value); + return stringValue(new String((byte[])value)); Review comment: Oops This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #25369: [SPARK-28638][WebUI] Task summary metrics are wrong when there are running tasks
gengliangwang commented on a change in pull request #25369:
[SPARK-28638][WebUI] Task summary metrics are wrong when there are running tasks
URL: https://github.com/apache/spark/pull/25369#discussion_r311832281
##
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##
@@ -156,7 +162,8 @@ private[spark] class AppStatusStore(
// cheaper for disk stores (avoids deserialization).
val count = {
Utils.tryWithResource(
-if (store.isInstanceOf[InMemoryStore]) {
+if (isInMemoryStore) {
Review comment:
@shahidki31 Do you mean #23088?
> as ExecutorRunTime will be defined for only finished tasks
That's not true. It is also defined in running tasks as I observed.
> reverting 23008 and hence this PR, would not fix the issue
I am not trying to revert anything. Just out of curiosity for the feature.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #25369: [SPARK-28638][WebUI] Task summary metrics are wrong when there are running tasks
gengliangwang commented on a change in pull request #25369:
[SPARK-28638][WebUI] Task summary metrics are wrong when there are running tasks
URL: https://github.com/apache/spark/pull/25369#discussion_r311832281
##
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##
@@ -156,7 +162,8 @@ private[spark] class AppStatusStore(
// cheaper for disk stores (avoids deserialization).
val count = {
Utils.tryWithResource(
-if (store.isInstanceOf[InMemoryStore]) {
+if (isInMemoryStore) {
Review comment:
@shahidki31 Do you mean #23088?
> as ExecutorRunTime will be defined for only finished tasks
That's not true. It is also defined in running tasks as I observed.
> reverting 23008 and hence this PR, would not fix the issue
I am not trying to revert anything. Just out of curiosity for the feature.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
HyukjinKwon closed pull request #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-519335752 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-519335758 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108788/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-519335752 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-519335758 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108788/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
wangyum commented on a change in pull request #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type URL: https://github.com/apache/spark/pull/25379#discussion_r311831678 ## File path: sql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/ColumnValue.java ## @@ -199,7 +199,7 @@ public static TColumnValue toTColumnValue(TypeDescriptor typeDescriptor, Object case DECIMAL_TYPE: return stringValue((HiveDecimal)value, typeDescriptor); case BINARY_TYPE: - return stringValue((String)value); + return stringValue(new String((byte[])value)); Review comment: This is for our v2.3.5. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
SparkQA removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-519296305 **[Test build #108788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108788/testReport)** for PR 25354 at commit [`4538721`](https://github.com/apache/spark/commit/45387211d054400017ac83627b7b40887f614d16). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
SparkQA commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-519335374 **[Test build #108788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108788/testReport)** for PR 25354 at commit [`4538721`](https://github.com/apache/spark/commit/45387211d054400017ac83627b7b40887f614d16). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
HyukjinKwon commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519334825 I am merging this because related linter checks passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer edited a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
beliefer edited a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-518917065 @dongjoon-hyun Could you check this PR? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25379: [SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type
HyukjinKwon commented on a change in pull request #25379: [SPARK-28474][SQL]
Hive 0.12 JDBC client can not handle binary type
URL: https://github.com/apache/spark/pull/25379#discussion_r311830843
##
File path:
sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java
##
@@ -195,7 +195,7 @@ public static TColumnValue toTColumnValue(Type type,
Object value) {
case DECIMAL_TYPE:
return stringValue(((HiveDecimal)value));
case BINARY_TYPE:
- return stringValue((String)value);
+ return stringValue(new String((byte[])value));
Review comment:
Out of curiosity, why we don't need to fix v2.3.5?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer edited a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
beliefer edited a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-518913939 @maropu Could you continue to take a look? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table
HyukjinKwon commented on issue #25375: [SPARK-28642][SQL] Hide credentials in show create table URL: https://github.com/apache/spark/pull/25375#issuecomment-519334321 Seems fine to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25375: [SPARK-28642][SQL] Hide credentials in show create table
HyukjinKwon commented on a change in pull request #25375: [SPARK-28642][SQL]
Hide credentials in show create table
URL: https://github.com/apache/spark/pull/25375#discussion_r311830600
##
File path: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
##
@@ -1029,6 +1030,32 @@ class JDBCSuite extends QueryTest
}
}
+ test("Hide credentials in show create table") {
+val password = "testPass"
Review comment:
Can we explicitly set the conf in the test?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25375: [SPARK-28642][SQL] Hide credentials in show create table
HyukjinKwon commented on a change in pull request #25375: [SPARK-28642][SQL]
Hide credentials in show create table
URL: https://github.com/apache/spark/pull/25375#discussion_r311830196
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##
@@ -1099,7 +1100,7 @@ case class ShowCreateTableCommand(table:
TableIdentifier) extends RunnableComman
private def showDataSourceTableOptions(metadata: CatalogTable, builder:
StringBuilder): Unit = {
builder ++= s"USING ${metadata.provider.get}\n"
-val dataSourceOptions = metadata.storage.properties.map {
+val dataSourceOptions = Utils.redact(metadata.storage.properties).map {
Review comment:
Seems `SQLConf.redactOptions` looks right one.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
AmplabJenkins removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519330617 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
AmplabJenkins removed a comment on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519330623 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13872/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519330632 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13873/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins removed a comment on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519330626 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
AmplabJenkins commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519330623 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13872/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519330632 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13873/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener'
AmplabJenkins commented on issue #25377: [MINOR][DOC] Fix a typo 'lister' -> 'listener' URL: https://github.com/apache/spark/pull/25377#issuecomment-519330617 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
AmplabJenkins commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519330626 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #25376: [SPARK-28395][FOLLOW-UP][SQL] Make spark.sql.function.preferIntegralDivision internal
HyukjinKwon closed pull request #25376: [SPARK-28395][FOLLOW-UP][SQL] Make spark.sql.function.preferIntegralDivision internal URL: https://github.com/apache/spark/pull/25376 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error
SparkQA commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-519329542 **[Test build #108793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108793/testReport)** for PR 25333 at commit [`7baa349`](https://github.com/apache/spark/commit/7baa3492ecba858fe1258f9b8c5bb7d64edea03d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
