[GitHub] [spark] AmplabJenkins commented on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected
AmplabJenkins commented on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected URL: https://github.com/apache/spark/pull/26080#issuecomment-544105364 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17293/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected
AmplabJenkins removed a comment on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected URL: https://github.com/apache/spark/pull/26080#issuecomment-544105363 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected
AmplabJenkins removed a comment on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected URL: https://github.com/apache/spark/pull/26080#issuecomment-544105364 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17293/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected
AmplabJenkins commented on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected URL: https://github.com/apache/spark/pull/26080#issuecomment-544105363 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected
SparkQA commented on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected URL: https://github.com/apache/spark/pull/26080#issuecomment-544105296 **[Test build #112310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112310/testReport)** for PR 26080 at commit [`9efc8cf`](https://github.com/apache/spark/commit/9efc8cf1546de36fb5bc2393a82fcd6af71b8e03). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya closed pull request #26168: [SPARK-29512][SQL] REPAIR TABLE should look up catalog/table like v2 commands
viirya closed pull request #26168: [SPARK-29512][SQL] REPAIR TABLE should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26168 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #26168: [SPARK-29512][SQL] REPAIR TABLE should look up catalog/table like v2 commands
viirya commented on issue #26168: [SPARK-29512][SQL] REPAIR TABLE should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26168#issuecomment-544104764 Thanks! Merging to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
AmplabJenkins removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544104506 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112309/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
SparkQA removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544102583 **[Test build #112309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112309/testReport)** for PR 26172 at commit [`b3a26e1`](https://github.com/apache/spark/commit/b3a26e15ba1851669ffe5bf28e5bcb6f84a18316). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
AmplabJenkins removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544104505 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
AmplabJenkins commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544104505 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
AmplabJenkins commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544104506 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112309/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
SparkQA commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544104500 **[Test build #112309 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112309/testReport)** for PR 26172 at commit [`b3a26e1`](https://github.com/apache/spark/commit/b3a26e15ba1851669ffe5bf28e5bcb6f84a18316). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #26160: [SPARK-29498][SQL] CatalogTable to HiveTable should not change the table's ownership
yaooqinn commented on issue #26160: [SPARK-29498][SQL] CatalogTable to HiveTable should not change the table's ownership URL: https://github.com/apache/spark/pull/26160#issuecomment-544103892 Anyway, this change can keep the consistency of ownership between catalog table and hive table, so LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
AmplabJenkins removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544101243 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #25868: [SPARK-28527][SQL][TEST] Enable ThriftServerQueryTestSuite
wangyum commented on a change in pull request #25868: [SPARK-28527][SQL][TEST]
Enable ThriftServerQueryTestSuite
URL: https://github.com/apache/spark/pull/25868#discussion_r336722855
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
##
@@ -43,12 +45,12 @@ import org.apache.spark.sql.types._
* 2. Support DESC command.
* 3. Support SHOW command.
*/
-@Ignore
class ThriftServerQueryTestSuite extends SQLQueryTestSuite {
private var hiveServer2: HiveThriftServer2 = _
- override def beforeEach(): Unit = {
+ override def beforeAll(): Unit = {
+super.beforeAll()
Review comment:
Anyway, let's try to enable it: https://github.com/apache/spark/pull/26172
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
AmplabJenkins removed a comment on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544101245 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17292/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
SparkQA commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544102583 **[Test build #112309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112309/testReport)** for PR 26172 at commit [`b3a26e1`](https://github.com/apache/spark/commit/b3a26e15ba1851669ffe5bf28e5bcb6f84a18316). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #25868: [SPARK-28527][SQL][TEST] Enable ThriftServerQueryTestSuite
wangyum commented on a change in pull request #25868: [SPARK-28527][SQL][TEST]
Enable ThriftServerQueryTestSuite
URL: https://github.com/apache/spark/pull/25868#discussion_r336722831
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
##
@@ -43,12 +45,12 @@ import org.apache.spark.sql.types._
* 2. Support DESC command.
* 3. Support SHOW command.
*/
-@Ignore
class ThriftServerQueryTestSuite extends SQLQueryTestSuite {
private var hiveServer2: HiveThriftServer2 = _
- override def beforeEach(): Unit = {
+ override def beforeAll(): Unit = {
+super.beforeAll()
Review comment:
`pgSQL/text.sql` should fail if `spark.sql.hive.thriftServer.async` is
enabled: https://github.com/apache/spark/pull/25567#issuecomment-524507309
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
AmplabJenkins commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544101245 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17292/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
AmplabJenkins commented on issue #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172#issuecomment-544101243 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum opened a new pull request #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously
wangyum opened a new pull request #26172: [WIP][SPARK-29516][SQL][TEST] Test ThriftServerQueryTestSuite asynchronously URL: https://github.com/apache/spark/pull/26172 ### What changes were proposed in this pull request? This PR try to test `ThriftServerQueryTestSuite` in an asynchronous way. ### Why are the changes needed? The default value of `spark.sql.hive.thriftServer.async` is `true`. ### Does this PR introduce any user-facing change? No ### How was this patch tested? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544092020 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112307/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544092020 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112307/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544092015 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
SparkQA removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544066768 **[Test build #112307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112307/testReport)** for PR 26170 at commit [`94329b2`](https://github.com/apache/spark/commit/94329b2e48e93d395fa7460d0b6c6b5c31216118). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544092015 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544091805 **[Test build #112307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112307/testReport)** for PR 26170 at commit [`94329b2`](https://github.com/apache/spark/commit/94329b2e48e93d395fa7460d0b6c6b5c31216118). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
AmplabJenkins removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544081393 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112308/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
AmplabJenkins commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544081393 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112308/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
AmplabJenkins removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544081386 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
AmplabJenkins commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544081386 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
SparkQA commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544081367 **[Test build #112308 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112308/testReport)** for PR 26164 at commit [`631f3cb`](https://github.com/apache/spark/commit/631f3cba3509b00501724c505a18d695a6e2acfb). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
SparkQA removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544078485 **[Test build #112308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112308/testReport)** for PR 26164 at commit [`631f3cb`](https://github.com/apache/spark/commit/631f3cba3509b00501724c505a18d695a6e2acfb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
AmplabJenkins removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544078850 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17291/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
AmplabJenkins removed a comment on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544078842 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
AmplabJenkins commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544078842 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#discussion_r336720517 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SortExec.scala ## @@ -87,13 +95,13 @@ case class SortExec( } val pageSize = SparkEnv.get.memoryManager.pageSizeBytes -val sorter = UnsafeExternalRowSorter.create( +rowSorter = UnsafeExternalRowSorter.create( Review comment: Thanks, done in 631f3cb. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL]
Fix memory leak in SortMergeJoin
URL: https://github.com/apache/spark/pull/26164#discussion_r336720511
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
##
@@ -161,6 +162,10 @@ case class SortMergeJoinExec(
sqlContext.conf.sortMergeJoinExecBufferInMemoryThreshold
}
+ private def needEagerCleanup: Boolean = {
+sqlContext.conf.getConf(SORT_MERGE_JOIN_EXEC_EAGER_CLEANUP_RESOURCES)
Review comment:
Ah yes, thanks for reminding, done in 631f3cb
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
AmplabJenkins commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544078850 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17291/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL]
Fix memory leak in SortMergeJoin
URL: https://github.com/apache/spark/pull/26164#discussion_r336720485
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1437,6 +1437,14 @@ object SQLConf {
.intConf
.createWithDefault(SHUFFLE_SPILL_NUM_ELEMENTS_FORCE_SPILL_THRESHOLD.defaultValue.get)
+ val SORT_MERGE_JOIN_EXEC_EAGER_CLEANUP_RESOURCES =
+buildConf("spark.sql.sortMergeJoinExec.eagerCleanupResources")
+ .internal()
+ .doc("When true, the SortMergeJoinExec will trigger all upstream
resources cleanup right " +
+"after it finishes computing.")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
Just wants to add a config for protecting this core change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
SparkQA commented on issue #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin URL: https://github.com/apache/spark/pull/26164#issuecomment-544078485 **[Test build #112308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112308/testReport)** for PR 26164 at commit [`631f3cb`](https://github.com/apache/spark/commit/631f3cba3509b00501724c505a18d695a6e2acfb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL]
Fix memory leak in SortMergeJoin
URL: https://github.com/apache/spark/pull/26164#discussion_r336720473
##
File path: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
##
@@ -1040,3 +1044,49 @@ class JoinSuite extends QueryTest with
SharedSparkSession {
checkAnswer(df, Row(1, 2, 1, 2) :: Nil)
}
}
+
+class JoinWithResourceCleanSuite extends JoinSuite with BeforeAndAfterAll {
+ import testImplicits._
+ import scala.collection.mutable.ArrayBuffer
+
+ private def checkCleanupResourceTriggered(plan: SparkPlan) :
ArrayBuffer[SortExec] = {
+// Check cleanupResources are finally triggered in SortExec node
+val sorts = new ArrayBuffer[SortExec]()
+plan.foreachUp {
+ case s: SortExec => sorts += s
+ case _ =>
+}
+sorts.foreach { sort =>
+ val sortExec = spy(sort)
+ verify(sortExec, atLeastOnce).cleanupResources()
+ verify(sortExec.rowSorter, atLeastOnce).cleanupResources()
+}
+sorts
+ }
+
+ override def checkAnswer(df: => DataFrame, rows: Seq[Row]): Unit = {
+withSQLConf(
+ SQLConf.SORT_MERGE_JOIN_EXEC_EAGER_CLEANUP_RESOURCES.key -> "true") {
+ checkCleanupResourceTriggered(df.queryExecution.sparkPlan)
+ super.checkAnswer(df, rows)
+}
+ }
+
+ test("cleanupResource in code generation") {
+withSQLConf(
+ SQLConf.SORT_MERGE_JOIN_EXEC_EAGER_CLEANUP_RESOURCES.key -> "true",
+ SQLConf.SHUFFLE_PARTITIONS.key -> "1",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+ val df1 = spark.range(0, 10, 1, 2)
+ val df2 = spark.range(10).select($"id".as("b1"), (- $"id").as("b2"))
+ val res = df1.join(df2, $"id" === $"b1" && $"id" ===
$"b2").select($"b1", $"b2", $"id")
+
+ val sorts = checkCleanupResourceTriggered(res.queryExecution.sparkPlan)
Review comment:
Yeah, the first call is also want to collect all the SortExec to make sure
code generation takes effect.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL]
Fix memory leak in SortMergeJoin
URL: https://github.com/apache/spark/pull/26164#discussion_r336720439
##
File path: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
##
@@ -1040,3 +1044,49 @@ class JoinSuite extends QueryTest with
SharedSparkSession {
checkAnswer(df, Row(1, 2, 1, 2) :: Nil)
}
}
+
+class JoinWithResourceCleanSuite extends JoinSuite with BeforeAndAfterAll {
+ import testImplicits._
+ import scala.collection.mutable.ArrayBuffer
+
+ private def checkCleanupResourceTriggered(plan: SparkPlan) :
ArrayBuffer[SortExec] = {
Review comment:
Thanks, done in 631f3cb.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
xuanyuanking commented on a change in pull request #26164: [SPARK-21492][SQL]
Fix memory leak in SortMergeJoin
URL: https://github.com/apache/spark/pull/26164#discussion_r336720437
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SortExec.scala
##
@@ -181,4 +189,17 @@ case class SortExec(
|$sorterVariable.insertRow((UnsafeRow)${row.value});
""".stripMargin
}
+
+ /**
+ * In SortExec, we overwrites cleanupResources to close
UnsafeExternalRowSorter.
+ */
+ override protected[sql] def cleanupResources(): Unit = {
+super.cleanupResources()
Review comment:
Copy, done in 631f3cb.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
AmplabJenkins removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544071441 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112306/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
AmplabJenkins removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544071433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
AmplabJenkins commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544071441 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112306/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
AmplabJenkins commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544071433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
SparkQA removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544033241 **[Test build #112306 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112306/testReport)** for PR 25971 at commit [`b0ecff9`](https://github.com/apache/spark/commit/b0ecff9f8fc261aba0f3f4ab428bbeebb8895256). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
SparkQA commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544070857 **[Test build #112306 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112306/testReport)** for PR 25971 at commit [`b0ecff9`](https://github.com/apache/spark/commit/b0ecff9f8fc261aba0f3f4ab428bbeebb8895256). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #26143: [SPARK-29494][SQL] Fix for ArrayOutofBoundsException while converting string to timestamp
srowen commented on issue #26143: [SPARK-29494][SQL] Fix for ArrayOutofBoundsException while converting string to timestamp URL: https://github.com/apache/spark/pull/26143#issuecomment-544067825 Shoot, thanks for catching that @zsxwing . Looks like some test utility code isn't present in 2.4: ``` [error] /home/jenkins/workspace/spark-branch-2.4-test-maven-hadoop-2.7/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala:586: not found: value defaultZoneId [error] val time = DateTimeUtils.stringToTimestamp(s, defaultZoneId) [error] ^ [error] one error found [error] Compile failed at Oct 18, 2019 2:55:40 PM [15.246s] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544067276 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544067280 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17290/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544067280 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17290/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544067276 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544066768 **[Test build #112307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112307/testReport)** for PR 26170 at commit [`94329b2`](https://github.com/apache/spark/commit/94329b2e48e93d395fa7460d0b6c6b5c31216118). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cjn082030 commented on issue #26136: [SPARK-29488][WEBUI] In Web UI, stage page has js error when sort table.
cjn082030 commented on issue #26136: [SPARK-29488][WEBUI] In Web UI, stage page has js error when sort table. URL: https://github.com/apache/spark/pull/26136#issuecomment-544064476 ok. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp
AmplabJenkins removed a comment on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp URL: https://github.com/apache/spark/pull/26171#issuecomment-544055851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112305/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp
AmplabJenkins removed a comment on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp URL: https://github.com/apache/spark/pull/26171#issuecomment-544055847 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp
AmplabJenkins commented on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp URL: https://github.com/apache/spark/pull/26171#issuecomment-544055851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112305/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp
AmplabJenkins commented on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp URL: https://github.com/apache/spark/pull/26171#issuecomment-544055847 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp
SparkQA commented on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp URL: https://github.com/apache/spark/pull/26171#issuecomment-544055522 **[Test build #112305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112305/testReport)** for PR 26171 at commit [`4376b88`](https://github.com/apache/spark/commit/4376b88dfc1f278cc97fb8b162c4277bc98148ac). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp
SparkQA removed a comment on issue #26171: [SPARK-29494][SQL][2.4] Fix for ArrayOutofBoundsException while converting string to timestamp URL: https://github.com/apache/spark/pull/26171#issuecomment-543997698 **[Test build #112305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112305/testReport)** for PR 26171 at commit [`4376b88`](https://github.com/apache/spark/commit/4376b88dfc1f278cc97fb8b162c4277bc98148ac). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown
AmplabJenkins removed a comment on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown URL: https://github.com/apache/spark/pull/25955#issuecomment-544040377 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112304/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown
AmplabJenkins removed a comment on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown URL: https://github.com/apache/spark/pull/25955#issuecomment-544040370 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown
AmplabJenkins commented on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown URL: https://github.com/apache/spark/pull/25955#issuecomment-544040370 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown
AmplabJenkins commented on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown URL: https://github.com/apache/spark/pull/25955#issuecomment-544040377 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112304/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544040076 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544040083 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112303/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown
SparkQA removed a comment on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown URL: https://github.com/apache/spark/pull/25955#issuecomment-543986709 **[Test build #112304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112304/testReport)** for PR 25955 at commit [`e886e77`](https://github.com/apache/spark/commit/e886e773cddfd368836e4f419b5c8015d7ecdbd2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544040076 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown
SparkQA commented on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown URL: https://github.com/apache/spark/pull/25955#issuecomment-544040191 **[Test build #112304 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112304/testReport)** for PR 25955 at commit [`e886e77`](https://github.com/apache/spark/commit/e886e773cddfd368836e4f419b5c8015d7ecdbd2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544040083 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112303/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
SparkQA removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-543983512 **[Test build #112303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112303/testReport)** for PR 26085 at commit [`08c8fb2`](https://github.com/apache/spark/commit/08c8fb2d79f499b074578991dc217a61d1633bb6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
SparkQA commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544039535 **[Test build #112303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112303/testReport)** for PR 26085 at commit [`08c8fb2`](https://github.com/apache/spark/commit/08c8fb2d79f499b074578991dc217a61d1633bb6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
AmplabJenkins removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544033506 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
AmplabJenkins removed a comment on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544033516 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17289/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
AmplabJenkins commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544033506 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
AmplabJenkins commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544033516 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17289/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
SparkQA commented on issue #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint URL: https://github.com/apache/spark/pull/25971#issuecomment-544033241 **[Test build #112306 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112306/testReport)** for PR 25971 at commit [`b0ecff9`](https://github.com/apache/spark/commit/b0ecff9f8fc261aba0f3f4ab428bbeebb8895256). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
LantaoJin commented on a change in pull request #25971: [SPARK-29298][CORE]
Separate block manager heartbeat endpoint from driver endpoint
URL: https://github.com/apache/spark/pull/25971#discussion_r336714695
##
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterHeartbeatEndpoint.scala
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEnv, ThreadSafeRpcEndpoint}
+import org.apache.spark.storage.BlockManagerMessages.{BlockManagerHeartbeat,
RegisterBlockManager, RemoveExecutor, UpdateBlockInfo}
+
+/**
+ * Separate heartbeat out of BlockManagerMasterEndpoint due to performance
consideration.
+ */
+private[spark] class BlockManagerMasterHeartbeatEndpoint(
+override val rpcEnv: RpcEnv,
+isLocal: Boolean)
+ extends ThreadSafeRpcEndpoint with Logging {
+
+ // Mapping from block manager id to the block manager's information.
+ private val blockManagerLastSeen = new mutable.HashMap[BlockManagerId, Long]
Review comment:
No. we can not eliminate `blockManagerLastSeen ` for now at least. The
lastSeenMs is not used but the `blockManagerLastSeen ` map is used to store all
BlockManagerId for `def heartbeatReceived`. So the
```
private val blockManagerLastSeen = new mutable.HashMap[BlockManagerId,
Long]
```
can be changed to
```
private val blockManagers = new mutable.HashSet[BlockManagerId]
```
But I think `blockManagerLastSeen` structure is better and no need more
memory.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
LantaoJin commented on a change in pull request #25971: [SPARK-29298][CORE]
Separate block manager heartbeat endpoint from driver endpoint
URL: https://github.com/apache/spark/pull/25971#discussion_r336714695
##
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterHeartbeatEndpoint.scala
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEnv, ThreadSafeRpcEndpoint}
+import org.apache.spark.storage.BlockManagerMessages.{BlockManagerHeartbeat,
RegisterBlockManager, RemoveExecutor, UpdateBlockInfo}
+
+/**
+ * Separate heartbeat out of BlockManagerMasterEndpoint due to performance
consideration.
+ */
+private[spark] class BlockManagerMasterHeartbeatEndpoint(
+override val rpcEnv: RpcEnv,
+isLocal: Boolean)
+ extends ThreadSafeRpcEndpoint with Logging {
+
+ // Mapping from block manager id to the block manager's information.
+ private val blockManagerLastSeen = new mutable.HashMap[BlockManagerId, Long]
Review comment:
Yes. we could eliminate `blockManagerLastSeen ` for now at least. The only
useful code here is the return value of `def heartbeatReceived`. The lastSeen
it stored is no needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544028371 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112302/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544028364 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544028364 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544028371 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112302/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
SparkQA removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-543976965 **[Test build #112302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112302/testReport)** for PR 26170 at commit [`6a0cb1c`](https://github.com/apache/spark/commit/6a0cb1ce4e086c1eda95e97d4080c83b9f6d435a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-544027773 **[Test build #112302 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112302/testReport)** for PR 26170 at commit [`6a0cb1c`](https://github.com/apache/spark/commit/6a0cb1ce4e086c1eda95e97d4080c83b9f6d435a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
LantaoJin commented on a change in pull request #25971: [SPARK-29298][CORE]
Separate block manager heartbeat endpoint from driver endpoint
URL: https://github.com/apache/spark/pull/25971#discussion_r336714082
##
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterHeartbeatEndpoint.scala
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEnv, ThreadSafeRpcEndpoint}
+import org.apache.spark.storage.BlockManagerMessages.{BlockManagerHeartbeat,
RegisterBlockManager, RemoveExecutor, UpdateBlockInfo}
+
+/**
+ * Separate heartbeat out of BlockManagerMasterEndpoint due to performance
consideration.
+ */
+private[spark] class BlockManagerMasterHeartbeatEndpoint(
+override val rpcEnv: RpcEnv,
+isLocal: Boolean)
+ extends ThreadSafeRpcEndpoint with Logging {
+
+ // Mapping from block manager id to the block manager's information.
+ private val blockManagerLastSeen = new mutable.HashMap[BlockManagerId, Long]
Review comment:
Let me see. Does it have a bug in original code? If `def lastSeenMs` never
used is on purpose, why we need to do `updateLastSeenMs`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #25971: [SPARK-29298][CORE] Separate block manager heartbeat endpoint from driver endpoint
LantaoJin commented on a change in pull request #25971: [SPARK-29298][CORE]
Separate block manager heartbeat endpoint from driver endpoint
URL: https://github.com/apache/spark/pull/25971#discussion_r336714082
##
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterHeartbeatEndpoint.scala
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEnv, ThreadSafeRpcEndpoint}
+import org.apache.spark.storage.BlockManagerMessages.{BlockManagerHeartbeat,
RegisterBlockManager, RemoveExecutor, UpdateBlockInfo}
+
+/**
+ * Separate heartbeat out of BlockManagerMasterEndpoint due to performance
consideration.
+ */
+private[spark] class BlockManagerMasterHeartbeatEndpoint(
+override val rpcEnv: RpcEnv,
+isLocal: Boolean)
+ extends ThreadSafeRpcEndpoint with Logging {
+
+ // Mapping from block manager id to the block manager's information.
+ private val blockManagerLastSeen = new mutable.HashMap[BlockManagerId, Long]
Review comment:
Let me see. Does it have a bug in original code?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rdblue commented on a change in pull request #25651: [SPARK-28948][SQL] Support passing all Table metadata in TableProvider
rdblue commented on a change in pull request #25651: [SPARK-28948][SQL] Support
passing all Table metadata in TableProvider
URL: https://github.com/apache/spark/pull/25651#discussion_r336713473
##
File path:
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableProvider.java
##
@@ -36,26 +35,12 @@
public interface TableProvider {
/**
- * Return a {@link Table} instance to do read/write with user-specified
options.
+ * Return a {@link Table} instance with the given table options to do
read/write.
+ * Implementations should infer the table schema and partitioning.
*
* @param options the user-specified options that can identify a table, e.g.
file path, Kafka
*topic name, etc. It's an immutable case-insensitive
string-to-string map.
*/
+ // TODO: this should take a Map as table properties.
Review comment:
My point was that Spark needs to infer the partitioning of the table, not
exhaustively list directories. This can be done more quickly than in the
current implementation, by listing all files later and just getting the
directory structure for `inferPartititoning`.
The static cache I'm talking about is a cache of metastore connections, not
files. In this case, you could build your file list for a location and cache
that for some period of time, using it for partition and schema inference, as
well as for the `FileIndex` in the table you created. Caching would also help
consistency because the same files would be in all versions of the table loaded
within some period of time (and could be refreshed, of course). But, these
concerns shouldn't affect the API.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544023473 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544023484 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112301/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544023484 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112301/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544023473 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
SparkQA removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-543971350 **[Test build #112301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112301/testReport)** for PR 26085 at commit [`5aadd8f`](https://github.com/apache/spark/commit/5aadd8f089a00ceddc179804063cf058646f8a48). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
SparkQA commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544022786 **[Test build #112301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112301/testReport)** for PR 26085 at commit [`5aadd8f`](https://github.com/apache/spark/commit/5aadd8f089a00ceddc179804063cf058646f8a48). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544017780 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance
AmplabJenkins removed a comment on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544017786 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112300/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
