[GitHub] cloud-fan closed pull request #23510: [SPARK-26590][CORE] make fetch-block-to-disk backward compatible
cloud-fan closed pull request #23510: [SPARK-26590][CORE] make fetch-block-to-disk backward compatible URL: https://github.com/apache/spark/pull/23510 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on issue #23510: [SPARK-26590][CORE] make fetch-block-to-disk backward compatible
cloud-fan commented on issue #23510: [SPARK-26590][CORE] make fetch-block-to-disk backward compatible URL: https://github.com/apache/spark/pull/23510#issuecomment-456704763 makes sense, I'm closing it This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
AmplabJenkins removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456704002 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101565/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
AmplabJenkins removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456703998 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
AmplabJenkins commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456703998 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
AmplabJenkins commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456704002 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101565/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC
AmplabJenkins removed a comment on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC URL: https://github.com/apache/spark/pull/23601#issuecomment-456703649 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC
AmplabJenkins removed a comment on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC URL: https://github.com/apache/spark/pull/23601#issuecomment-456703652 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7344/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC
AmplabJenkins commented on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC URL: https://github.com/apache/spark/pull/23601#issuecomment-456703649 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC
AmplabJenkins commented on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC URL: https://github.com/apache/spark/pull/23601#issuecomment-456703652 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7344/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
SparkQA commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456703514 **[Test build #101565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101565/testReport)** for PR 23260 at commit [`cc25c2d`](https://github.com/apache/spark/commit/cc25c2ddb30250e69980ae28599d9c514d0cf6a2). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
SparkQA removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456650364 **[Test build #101565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101565/testReport)** for PR 23260 at commit [`cc25c2d`](https://github.com/apache/spark/commit/cc25c2ddb30250e69980ae28599d9c514d0cf6a2). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
AmplabJenkins removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456702346 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101566/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
AmplabJenkins removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456702339 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
AmplabJenkins commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456702339 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
AmplabJenkins commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456702346 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101566/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
SparkQA removed a comment on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456651426 **[Test build #101566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101566/testReport)** for PR 23260 at commit [`e343df8`](https://github.com/apache/spark/commit/e343df845bd7959bd0e47ea93834c9b1bd51852b). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC
SparkQA commented on issue #23601: [WIP][SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC URL: https://github.com/apache/spark/pull/23601#issuecomment-456701943 **[Test build #101575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101575/testReport)** for PR 23601 at commit [`2ca90a7`](https://github.com/apache/spark/commit/2ca90a76473aa561c6f6473fae4a1210133a3d8f). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
SparkQA commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456701837 **[Test build #101566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101566/testReport)** for PR 23260 at commit [`e343df8`](https://github.com/apache/spark/commit/e343df845bd7959bd0e47ea93834c9b1bd51852b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan closed pull request #23566: [SPARK-26637][SQL] Makes GetArrayItem nullability more precise
cloud-fan closed pull request #23566: [SPARK-26637][SQL] Makes GetArrayItem nullability more precise URL: https://github.com/apache/spark/pull/23566 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on issue #23566: [SPARK-26637][SQL] Makes GetArrayItem nullability more precise
cloud-fan commented on issue #23566: [SPARK-26637][SQL] Makes GetArrayItem nullability more precise URL: https://github.com/apache/spark/pull/23566#issuecomment-456699355 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23117: [WIP][SPARK-7721][INFRA] Run and generate test coverage report from Python via Jenkins
HyukjinKwon commented on issue #23117: [WIP][SPARK-7721][INFRA] Run and generate test coverage report from Python via Jenkins URL: https://github.com/apache/spark/pull/23117#issuecomment-456694748 @shaneknapp, can you take a look for https://github.com/apache/spark/pull/23117#issuecomment-456083489? Looks coverage makes PySpark tests flaky but it at least passed once at https://github.com/apache/spark/pull/23117#issuecomment-456060314. I will fix them orthogonally. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon closed pull request #23624: Branch 2.2
HyukjinKwon closed pull request #23624: Branch 2.2 URL: https://github.com/apache/spark/pull/23624 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories
AmplabJenkins removed a comment on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories URL: https://github.com/apache/spark/pull/23623#issuecomment-456691182 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories
AmplabJenkins commented on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories URL: https://github.com/apache/spark/pull/23623#issuecomment-456691482 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories
AmplabJenkins removed a comment on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories URL: https://github.com/apache/spark/pull/23623#issuecomment-456691093 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] 136357477 opened a new pull request #23624: Branch 2.2
136357477 opened a new pull request #23624: Branch 2.2 URL: https://github.com/apache/spark/pull/23624 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories
AmplabJenkins commented on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories URL: https://github.com/apache/spark/pull/23623#issuecomment-456691182 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories
AmplabJenkins commented on issue #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories URL: https://github.com/apache/spark/pull/23623#issuecomment-456691093 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SongYadong opened a new pull request #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories
SongYadong opened a new pull request #23623: [SPARK-26698][CORE] Use ConfigEntry for hardcoded configs for memory and storage categories URL: https://github.com/apache/spark/pull/23623 ## What changes were proposed in this pull request? This PR makes hardcoded configs about spark memory and storage to use `ConfigEntry` and put them in the config package. ## How was this patch tested? Existing unit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] liupc commented on a change in pull request #23445: [SPARK-26530]Validate heartheat arguments in HeartbeatReceiver
liupc commented on a change in pull request #23445: [SPARK-26530]Validate heartheat arguments in HeartbeatReceiver URL: https://github.com/apache/spark/pull/23445#discussion_r250069316 ## File path: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ## @@ -79,6 +79,17 @@ private[spark] class HeartbeatReceiver(sc: SparkContext, clock: Clock) private val checkTimeoutIntervalMs = sc.conf.get(Network.NETWORK_TIMEOUT_INTERVAL) + private val executorHeartbeatIntervalMs = sc.conf.get(config.EXECUTOR_HEARTBEAT_INTERVAL) + + require(checkTimeoutIntervalMs < executorTimeoutMs, Review comment: Anyway, I will update as your suggestion for I can't find a better way to make this log more clear. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] gatorsmile commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
gatorsmile commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456687021 I think this is pretty serious. We might need a release of 2.4.1 and send a note to dev list. @rdblue Could you confirm the impact of this bug? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] liupc commented on a change in pull request #23445: [SPARK-26530]Validate heartheat arguments in HeartbeatReceiver
liupc commented on a change in pull request #23445: [SPARK-26530]Validate heartheat arguments in HeartbeatReceiver URL: https://github.com/apache/spark/pull/23445#discussion_r250068258 ## File path: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ## @@ -79,6 +79,17 @@ private[spark] class HeartbeatReceiver(sc: SparkContext, clock: Clock) private val checkTimeoutIntervalMs = sc.conf.get(Network.NETWORK_TIMEOUT_INTERVAL) + private val executorHeartbeatIntervalMs = sc.conf.get(config.EXECUTOR_HEARTBEAT_INTERVAL) + + require(checkTimeoutIntervalMs < executorTimeoutMs, Review comment: @srowen I really want to use .key strings directly, but the .key strings like NETWORK_TIMEOUT_INTERVAL may not be set and in this case it's actually the default value from STORAGE_BLOCKMANGER_TIMEOUTINTERVAL works. Only refer to NETWORK_TIMEOUT_INTERVAL in the message might make it confusing. I have no idea how to make this log mesage better. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon closed pull request #17190: [SPARK-19478][SS] JDBC Sink
HyukjinKwon closed pull request #17190: [SPARK-19478][SS] JDBC Sink URL: https://github.com/apache/spark/pull/17190 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
HyukjinKwon removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456682944 All the related codes at Parquet side are written by @rdblue. We can talk directly with the author as well. I think that's quicker way then iterating with Parquet JIRAs for this Spark issue specifically. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
HyukjinKwon commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456683715 I haven't filed yet for this bug at Parquet yet.. I want to talk with @rdblue first anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched
AmplabJenkins removed a comment on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched URL: https://github.com/apache/spark/pull/22806#issuecomment-456682989 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101560/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched
AmplabJenkins commented on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched URL: https://github.com/apache/spark/pull/22806#issuecomment-456682989 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101560/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched
SparkQA removed a comment on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched URL: https://github.com/apache/spark/pull/22806#issuecomment-456606337 **[Test build #101560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101560/testReport)** for PR 22806 at commit [`0610939`](https://github.com/apache/spark/commit/06109393a4edce1d5db7193288bbcb523f3eed28). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon edited a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
HyukjinKwon edited a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456682944 All the related codes at Parquet side are written by @rdblue. We can talk directly with the author as well. I think that's quicker way then iterating with Parquet JIRAs for this Spark issue specifically. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched
AmplabJenkins removed a comment on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched URL: https://github.com/apache/spark/pull/22806#issuecomment-456682986 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched
AmplabJenkins commented on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched URL: https://github.com/apache/spark/pull/22806#issuecomment-456682986 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon edited a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
HyukjinKwon edited a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456682944 All the related codes at Parquet side are written by @rdblue. We can talk with the author rather then iterating with Parquet JIRAs for this Spark issue specifically. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
HyukjinKwon commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456682944 All the codes at Parquet side are written by @rdblue. We can talk with the author rather then iterating with Parquet JIRAs for this Spark issue specifically. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched
SparkQA commented on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched URL: https://github.com/apache/spark/pull/22806#issuecomment-456682875 **[Test build #101560 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101560/testReport)** for PR 22806 at commit [`0610939`](https://github.com/apache/spark/commit/06109393a4edce1d5db7193288bbcb523f3eed28). * This patch **fails from timeout after a configured wait of `400m`**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #21977: [SPARK-25004][CORE] Add spark.executor.pyspark.memory limit.
HyukjinKwon commented on issue #21977: [SPARK-25004][CORE] Add spark.executor.pyspark.memory limit. URL: https://github.com/apache/spark/pull/21977#issuecomment-456682488 FWIW, I hope all reviewers here put some input for the JIRA. It's confusing to have both configurations and I think we should fix. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
cloud-fan commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456682532 is there a parquet JIRA to track this dictionary encoding bug? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
AmplabJenkins removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456681933 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7343/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
AmplabJenkins removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456681928 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
AmplabJenkins commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456681933 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7343/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
AmplabJenkins commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456681928 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
SparkQA commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456682062 **[Test build #101574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101574/testReport)** for PR 23622 at commit [`eb2eb2b`](https://github.com/apache/spark/commit/eb2eb2bd93394517716df9f47ab650078544e184). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell
AmplabJenkins removed a comment on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell URL: https://github.com/apache/spark/pull/23618#issuecomment-456679580 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell
AmplabJenkins removed a comment on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell URL: https://github.com/apache/spark/pull/23618#issuecomment-456679583 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101562/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell
AmplabJenkins commented on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell URL: https://github.com/apache/spark/pull/23618#issuecomment-456679580 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell
AmplabJenkins commented on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell URL: https://github.com/apache/spark/pull/23618#issuecomment-456679583 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101562/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell
SparkQA removed a comment on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell URL: https://github.com/apache/spark/pull/23618#issuecomment-456628026 **[Test build #101562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101562/testReport)** for PR 23618 at commit [`a66db05`](https://github.com/apache/spark/commit/a66db054b1f78301a5aad779a55a173c825c4fab). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
AmplabJenkins removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456679323 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7342/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
AmplabJenkins removed a comment on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456679319 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
AmplabJenkins commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456679323 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7342/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
AmplabJenkins commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456679319 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell
SparkQA commented on issue #23618: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell URL: https://github.com/apache/spark/pull/23618#issuecomment-456679173 **[Test build #101562 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101562/testReport)** for PR 23618 at commit [`a66db05`](https://github.com/apache/spark/commit/a66db054b1f78301a5aad779a55a173c825c4fab). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
SparkQA commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456679047 **[Test build #101573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101573/testReport)** for PR 23622 at commit [`9798dd4`](https://github.com/apache/spark/commit/9798dd4b1246fb50a3b4ec1bc0a9e57860020267). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
HyukjinKwon commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456678640 adding @rdblue as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon opened a new pull request #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
HyukjinKwon opened a new pull request #23622: [SPARK-26677][SQL] Disable
dictionary filtering by default at Parquet
URL: https://github.com/apache/spark/pull/23622
## What changes were proposed in this pull request?
### Problem
This is a correctness issue and should be backported as well. If we use
dictionary encoding as below, it hits a correctness issue as below:
```scala
// Repeat the values for dictionary encoding. PLAIN_DICTIONARY.
Seq(Some("A"), Some("A"),
None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/foo")
spark.read.parquet("/tmp/foo").where("NOT (value <=> 'A')").show()
```
```
+-+
|value|
+-+
+-+
```
Note that, if we don't use dictionary encoding it's fine. So it was
difficult to find the issue.
```scala
// It becomes PLAIN encoding.
Seq(Some("A"),
None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/bar")
spark.read.parquet("/tmp/bar").where("NOT (value <=> 'A')").show()
```
```
+-+
|value|
+-+
| null|
+-+
```
### How did it happen?
This is because Parquet side dictionary filter fails to handle `null`. The
former case hits here:
```java
Set dictSet = expandDictionary(meta);
if (dictSet != null && dictSet.size() == 1 && dictSet.contains(value))
{
return BLOCK_CANNOT_MATCH;
}
```
https://github.com/apache/parquet-mr/blob/apache-parquet-1.10.0/parquet-hadoop/src/main/java/org/apache/parquet/filter2/dictionarylevel/DictionaryFilter.java#L182
Given dictionary set does not contain `null` and `value` is `'A'` here. So,
it filters the row group out.
The latter case above works fine because it hits here:
```java
// if the chunk has non-dictionary pages, don't bother decoding the
// dictionary because the row group can't be eliminated.
if (hasNonDictionaryPages(meta)) {
return BLOCK_MIGHT_MATCH;
}
```
https://github.com/apache/parquet-mr/blob/apache-parquet-1.10.0/parquet-hadoop/src/main/java/org/apache/parquet/filter2/dictionarylevel/DictionaryFilter.java#L176
So, it does not filter the row group out.
Parquet predicate handles `null` too (see also
https://github.com/apache/parquet-mr/blob/apache-parquet-1.10.0/parquet-column/src/main/java/org/apache/parquet/filter2/predicate/Operators.java#L188).
Up to my knowledge, Parquet predicate is null-safe.
### How does this PR fix?
This PR explicitly disable dictionary encoding. However there's another
problem:
We should disable `parquet.filter.dictionary.enabled` but Parquet 1.10.x has
a mistake - the 'parquet.filter.stats.enabled' and
'parquet.filter.dictionary.enabled' were swapped mistakenly in Parquet side:
```java
useDictionaryFilter(conf.getBoolean(STATS_FILTERING_ENABLED, true));
useStatsFilter(conf.getBoolean(DICTIONARY_FILTERING_ENABLED, true));
```
https://github.com/apache/parquet-mr/blob/apache-parquet-1.10.0/parquet-hadoop/src/main/java/org/apache/parquet/HadoopReadOptions.java#L83-L84
This is fixed after 1.11. See PARQUET-1309.
Therefore, this PR explicitly disable `parquet.filter.stats.enabled` to
disable dictionary filtering if that's not set.
### User side workaround
They can explicitly disable `parquet.filter.stats.enabled` to disable
dictionary filtering.
### ETC
- This is quite a conservative fix. This should be backported to Spark 2.4.
- I only tested null-safe equality comparison but looks equality comparison
having related issues.
## How was this patch tested?
Unit tests were added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet
HyukjinKwon commented on issue #23622: [SPARK-26677][SQL] Disable dictionary filtering by default at Parquet URL: https://github.com/apache/spark/pull/23622#issuecomment-456678482 cc @cloud-fan and @gatorsmile - this is a correctness issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
AmplabJenkins removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456677201 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
AmplabJenkins removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456677206 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101570/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
AmplabJenkins removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456677081 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101569/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
AmplabJenkins commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456677206 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101570/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
SparkQA removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-45311 **[Test build #101570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101570/testReport)** for PR 21632 at commit [`6adeda8`](https://github.com/apache/spark/commit/6adeda8fab0738a241f9897a13a259b54f9caaa0). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
AmplabJenkins commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456677201 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
AmplabJenkins removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456677077 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
SparkQA commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456677071 **[Test build #101570 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101570/testReport)** for PR 21632 at commit [`6adeda8`](https://github.com/apache/spark/commit/6adeda8fab0738a241f9897a13a259b54f9caaa0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
AmplabJenkins commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456677081 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101569/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
AmplabJenkins commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456677077 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
SparkQA removed a comment on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456665451 **[Test build #101569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101569/testReport)** for PR 21632 at commit [`a993ce3`](https://github.com/apache/spark/commit/a993ce370b9f18611fa6d3de3b531b0889c6cbd1). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
SparkQA commented on issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees URL: https://github.com/apache/spark/pull/21632#issuecomment-456676896 **[Test build #101569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101569/testReport)** for PR 21632 at commit [`a993ce3`](https://github.com/apache/spark/commit/a993ce370b9f18611fa6d3de3b531b0889c6cbd1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read
AmplabJenkins removed a comment on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read URL: https://github.com/apache/spark/pull/23619#issuecomment-456674726 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7341/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read
AmplabJenkins removed a comment on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read URL: https://github.com/apache/spark/pull/23619#issuecomment-456674722 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes.
AmplabJenkins removed a comment on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes. URL: https://github.com/apache/spark/pull/23621#issuecomment-456674653 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7340/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes.
AmplabJenkins commented on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes. URL: https://github.com/apache/spark/pull/23621#issuecomment-456674653 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7340/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes.
AmplabJenkins removed a comment on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes. URL: https://github.com/apache/spark/pull/23621#issuecomment-456674651 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read
AmplabJenkins commented on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read URL: https://github.com/apache/spark/pull/23619#issuecomment-456674722 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes.
AmplabJenkins commented on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes. URL: https://github.com/apache/spark/pull/23621#issuecomment-456674651 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read
AmplabJenkins commented on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read URL: https://github.com/apache/spark/pull/23619#issuecomment-456674726 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7341/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes.
SparkQA commented on issue #23621: [SPARK-26697][CORE] Log local & remote block sizes. URL: https://github.com/apache/spark/pull/23621#issuecomment-456674475 **[Test build #101572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101572/testReport)** for PR 23621 at commit [`12fba6e`](https://github.com/apache/spark/commit/12fba6ee280108a5281bfbc1bf55377e656c13f2). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] squito opened a new pull request #23621: [SPARK-26697][CORE] Log local & remote block sizes.
squito opened a new pull request #23621: [SPARK-26697][CORE] Log local & remote block sizes. URL: https://github.com/apache/spark/pull/23621 ## What changes were proposed in this pull request? To help debugging failed or slow tasks, its really useful to know the size of the blocks getting fetched. Though that is available at the debug level, debug logs aren't on in general -- but there is already an info level log line that this augments a little. ## How was this patch tested? Ran very basic local-cluster mode app, looked at logs. Full suite via jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read
SparkQA commented on issue #23619: [SPARK-26695][SQL] data source v2 API refactor - continuous read URL: https://github.com/apache/spark/pull/23619#issuecomment-456673649 **[Test build #101571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101571/testReport)** for PR 23619 at commit [`8fe6366`](https://github.com/apache/spark/commit/8fe6366541261bffab9163341e968025432e2a8d). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD
AmplabJenkins commented on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD URL: https://github.com/apache/spark/pull/23559#issuecomment-456672921 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101563/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD
AmplabJenkins removed a comment on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD URL: https://github.com/apache/spark/pull/23559#issuecomment-456672919 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD
AmplabJenkins removed a comment on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD URL: https://github.com/apache/spark/pull/23559#issuecomment-456672921 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101563/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD
AmplabJenkins commented on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD URL: https://github.com/apache/spark/pull/23559#issuecomment-456672919 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public
AmplabJenkins removed a comment on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public URL: https://github.com/apache/spark/pull/23620#issuecomment-456672461 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public
AmplabJenkins commented on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public URL: https://github.com/apache/spark/pull/23620#issuecomment-456672716 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD
SparkQA removed a comment on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD URL: https://github.com/apache/spark/pull/23559#issuecomment-456646227 **[Test build #101563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101563/testReport)** for PR 23559 at commit [`ecad50d`](https://github.com/apache/spark/commit/ecad50d2e5d6ea19ace9a99452377e695f9eb119). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD
SparkQA commented on issue #23559: [SPARK-26630][SQL] Fix ClassCastException in TableReader while creating HadoopRDD URL: https://github.com/apache/spark/pull/23559#issuecomment-456672574 **[Test build #101563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101563/testReport)** for PR 23559 at commit [`ecad50d`](https://github.com/apache/spark/commit/ecad50d2e5d6ea19ace9a99452377e695f9eb119). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public
AmplabJenkins removed a comment on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public URL: https://github.com/apache/spark/pull/23620#issuecomment-456672400 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public
AmplabJenkins commented on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public URL: https://github.com/apache/spark/pull/23620#issuecomment-456672461 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public
AmplabJenkins commented on issue #23620: [SPARK-26696][SQL] Makes Dataset encoder public URL: https://github.com/apache/spark/pull/23620#issuecomment-456672400 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
