[GitHub] [spark] William1104 commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
William1104 commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r282959517
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala
##
@@ -361,4 +352,87 @@ class CachedTableSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
assert(spark.sharedState.cacheManager.lookupCachedData(samePlan).isDefined)
}
}
+
+ test("SPARK-27248 refreshTable should recreate cache with same cache name
and storage level") {
+
+// This section tests when a table is cached with its qualified name but
its is refreshed with
+// its unqualified name.
+withTempPath { path =>
+ withTempDatabase { db =>
+withTable(s"$db.cachedTable") {
+ withCache(s"$db.cachedTable") {
+
+// Create table 'cachedTable' in default db for testing purpose.
+sql(s"CREATE TABLE $db.cachedTable(key STRING) USING PARQUET
LOCATION '${path.toURI}'")
Review comment:
Hi @dongjoon-hyun , Sorry about this. The SQL is update to:
- s"CREATE TABLE $db.cachedTable AS SELECT 1 AS key"
- "CREATE TABLE cachedTable AS SELECT 1 AS key"
I also modified the function `withCache` in SQLTestUtils to avoid exception
masking. Before the modification, any exception throwing from finally block may
mask the exception throwing from try block. I hope the created function
`withFinallyBlock` does the right job.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491561206 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491561207 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105336/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491561207 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105336/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491561206 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556486 **[Test build #105336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105336/testReport)** for PR 24586 at commit [`e81cb1a`](https://github.com/apache/spark/commit/e81cb1a896cbaf53ce09b6402341ab26797279c9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491561130 **[Test build #105336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105336/testReport)** for PR 24586 at commit [`e81cb1a`](https://github.com/apache/spark/commit/e81cb1a896cbaf53ce09b6402341ab26797279c9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #24585: [Spark-27664][CORE] Performance issue while listing large number of files on an object store.
viirya commented on issue #24585: [Spark-27664][CORE] Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-491559791 Btw, I think it should be `[SQL]`, instead of `[CORE]`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24585: [Spark-27664][CORE] Performance issue while listing large number of files on an object store.
viirya commented on a change in pull request #24585: [Spark-27664][CORE]
Performance issue while listing large number of files on an object store.
URL: https://github.com/apache/spark/pull/24585#discussion_r283116822
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -585,6 +585,17 @@ object SQLConf {
.longConf
.createWithDefault(250 * 1024 * 1024)
+ val HIVE_FILESOURCE_FILE_CACHE_CONCURRENCY_LEVEL =
+buildConf("spark.sql.hive.filesourceFileCacheConcurrencyLevel")
+ .doc("Provide concurrency level for underlying guava cache. The default
is 1." +
+" Higher value may give better performance, when the number of " +
Review comment:
As it is set with default 1, `Higher value may give better performance`
sounds like we set it to lowest performance.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556414 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556415 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10608/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556486 **[Test build #105336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105336/testReport)** for PR 24586 at commit [`e81cb1a`](https://github.com/apache/spark/commit/e81cb1a896cbaf53ce09b6402341ab26797279c9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556414 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556415 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10608/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556081 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105335/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556080 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556081 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105335/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491556080 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491550609 **[Test build #105335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105335/testReport)** for PR 24586 at commit [`ea39ea2`](https://github.com/apache/spark/commit/ea39ea22aa5378fe1be88fc41a93547536efe63c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491555998 **[Test build #105335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105335/testReport)** for PR 24586 at commit [`ea39ea2`](https://github.com/apache/spark/commit/ea39ea22aa5378fe1be88fc41a93547536efe63c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24585: [Spark-27664][CORE] Performance issue while listing large number of files on an object store.
gatorsmile commented on a change in pull request #24585: [Spark-27664][CORE]
Performance issue while listing large number of files on an object store.
URL: https://github.com/apache/spark/pull/24585#discussion_r283115595
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -585,6 +585,17 @@ object SQLConf {
.longConf
.createWithDefault(250 * 1024 * 1024)
+ val HIVE_FILESOURCE_FILE_CACHE_CONCURRENCY_LEVEL =
+buildConf("spark.sql.hive.filesourceFileCacheConcurrencyLevel")
+ .doc("Provide concurrency level for underlying guava cache. The default
is 1." +
+" Higher value may give better performance, when the number of " +
+"entries in the cache is very large. This imposes a limit on the size
of single entry" +
Review comment:
Have you measured the perf overhead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #24585: [Spark-27664][CORE] Performance issue while listing large number of files on an object store.
gatorsmile commented on issue #24585: [Spark-27664][CORE] Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-491554884 @ScrapCodes Could you give a simple perf test case we can run? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
gatorsmile commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491554694 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491552753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105334/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491552752 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491552752 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491552753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105334/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491546723 **[Test build #105334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105334/testReport)** for PR 24586 at commit [`e012cd5`](https://github.com/apache/spark/commit/e012cd54c3705dbc4d0b0213ec3a5b752a5945bb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491552692 **[Test build #105334 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105334/testReport)** for PR 24586 at commit [`e012cd5`](https://github.com/apache/spark/commit/e012cd54c3705dbc4d0b0213ec3a5b752a5945bb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491550609 **[Test build #105335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105335/testReport)** for PR 24586 at commit [`ea39ea2`](https://github.com/apache/spark/commit/ea39ea22aa5378fe1be88fc41a93547536efe63c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491550508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10607/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491550507 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491550507 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491550508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10607/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
srowen commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283113973
##
File path:
mllib/src/test/scala/org/apache/spark/ml/tree/impl/RandomForestSuite.scala
##
@@ -505,11 +503,11 @@ class RandomForestSuite extends SparkFunSuite with
MLlibTestSparkContext {
val failString = s"Failed on test with:" +
s"numTrees=$numTrees, featureSubsetStrategy=$featureSubsetStrategy,"
+
s" numFeaturesPerNode=$numFeaturesPerNode, seed=$seed"
-val nodeStack = new mutable.ArrayStack[(Int, LearningNode)]
+val nodeStack = new mutable.ListBuffer[(Int, LearningNode)]
val topNodes: Array[LearningNode] = new Array[LearningNode](numTrees)
Range(0, numTrees).foreach { treeIndex =>
topNodes(treeIndex) = LearningNode.emptyNode(nodeIndex = 1)
- nodeStack.push((treeIndex, topNodes(treeIndex)))
+ nodeStack.+=:((treeIndex, topNodes(treeIndex)))
Review comment:
Oops, my search and replace didn't quite work. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
MaxGekk commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283113026
##
File path:
mllib/src/test/scala/org/apache/spark/ml/tree/impl/RandomForestSuite.scala
##
@@ -505,11 +503,11 @@ class RandomForestSuite extends SparkFunSuite with
MLlibTestSparkContext {
val failString = s"Failed on test with:" +
s"numTrees=$numTrees, featureSubsetStrategy=$featureSubsetStrategy,"
+
s" numFeaturesPerNode=$numFeaturesPerNode, seed=$seed"
-val nodeStack = new mutable.ArrayStack[(Int, LearningNode)]
+val nodeStack = new mutable.ListBuffer[(Int, LearningNode)]
val topNodes: Array[LearningNode] = new Array[LearningNode](numTrees)
Range(0, numTrees).foreach { treeIndex =>
topNodes(treeIndex) = LearningNode.emptyNode(nodeIndex = 1)
- nodeStack.push((treeIndex, topNodes(treeIndex)))
+ nodeStack.+=:((treeIndex, topNodes(treeIndex)))
Review comment:
Would you mind to use `prepend()` here and other places in the PR as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491547043 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10606/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491547040 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491547040 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491547043 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10606/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491546723 **[Test build #105334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105334/testReport)** for PR 24586 at commit [`e012cd5`](https://github.com/apache/spark/commit/e012cd54c3705dbc4d0b0213ec3a5b752a5945bb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` (it is in the PR description) which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated for each submission. For example, each app can specify a different upload path: s3://mybucket/app1, s3://mybucket/app2 ... I think @vanzin does not like that, subdirs is the design of choice for me so far, I dont get why this can be that bad. Or there is a misunderstanding here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` (it is in the PR description) which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated for each submission. For example, each app can specify a different upload path: s3://mybucket/app1, s3://mybucket/app2 ... I think @vanzin does not like that, subdirs is the design of choice for me so far, I dont get why this can be that bad. Or there is a misunderstanding anyway here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` (it is in the PR description) which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated for each submission. For example, each app can specify a different upload path: s3://mybucket/app1, s3://mybucket/app2 ... I think @vanzin does not like that, subdirs is the design of choice for me so far, I dont get why this can be that bad. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
srowen commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283112526
##
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##
@@ -468,21 +468,20 @@ private[spark] class DAGScheduler(
/** Find ancestor shuffle dependencies that are not registered in
shuffleToMapStage yet */
private def getMissingAncestorShuffleDependencies(
- rdd: RDD[_]): ArrayStack[ShuffleDependency[_, _, _]] = {
-val ancestors = new ArrayStack[ShuffleDependency[_, _, _]]
+ rdd: RDD[_]): ListBuffer[ShuffleDependency[_, _, _]] = {
+val ancestors = new ListBuffer[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
// We are manually maintaining a stack here to prevent StackOverflowError
// caused by recursively visiting
-val waitingForVisit = new ArrayStack[RDD[_]]
-waitingForVisit.push(rdd)
+val waitingForVisit = ListBuffer[RDD[_]](rdd)
while (waitingForVisit.nonEmpty) {
- val toVisit = waitingForVisit.pop()
+ val toVisit = waitingForVisit.remove(0)
if (!visited(toVisit)) {
visited += toVisit
getShuffleDependencies(toVisit).foreach { shuffleDep =>
if (!shuffleIdToMapStage.contains(shuffleDep.shuffleId)) {
-ancestors.push(shuffleDep)
-waitingForVisit.push(shuffleDep.rdd)
+ancestors.+=:(shuffleDep)
Review comment:
Yep got it now; I'm going with prepend.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` (it is in the PR description) which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated. For example: s3://mybucket/app1, s3://mybucket/app2 ... I think @vanzin does not like that, subdirs is the design of choice for me so far, I dont get why this can be that bad. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` (it is in the PR description) which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated. For example: s3://mybucket/app1, s3://mybucket/app2 ... I think @vanzin does not like that, subdirs is the design of choice for me so far. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated. For example: s3://mybucket/app1, s3://mybucket/app2 ... I think @vanzin does not like that, subdirs is the design of choice for me so far. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated. For example: s3://mybucket/app1, s3://mybucket/app2 ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated. For example: s3://mybucket/app1, s3://mybucket/app2 ... I think @vanzin does not like that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there is a property namely `spark.kubernetes.file.upload.path` which I introduced that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated. For example: /mybucket/app1 /mybucket/app2 ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there a property that can be used by the user to set where he wants to upload the artifacts. By specifying the full path which may have subdirs things can be isolated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
MaxGekk commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283111500
##
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##
@@ -468,21 +468,20 @@ private[spark] class DAGScheduler(
/** Find ancestor shuffle dependencies that are not registered in
shuffleToMapStage yet */
private def getMissingAncestorShuffleDependencies(
- rdd: RDD[_]): ArrayStack[ShuffleDependency[_, _, _]] = {
-val ancestors = new ArrayStack[ShuffleDependency[_, _, _]]
+ rdd: RDD[_]): ListBuffer[ShuffleDependency[_, _, _]] = {
+val ancestors = new ListBuffer[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
// We are manually maintaining a stack here to prevent StackOverflowError
// caused by recursively visiting
-val waitingForVisit = new ArrayStack[RDD[_]]
-waitingForVisit.push(rdd)
+val waitingForVisit = ListBuffer[RDD[_]](rdd)
while (waitingForVisit.nonEmpty) {
- val toVisit = waitingForVisit.pop()
+ val toVisit = waitingForVisit.remove(0)
if (!visited(toVisit)) {
visited += toVisit
getShuffleDependencies(toVisit).foreach { shuffleDep =>
if (!shuffleIdToMapStage.contains(shuffleDep.shuffleId)) {
-ancestors.push(shuffleDep)
-waitingForVisit.push(shuffleDep.rdd)
+ancestors.+=:(shuffleDep)
+waitingForVisit.+=:(shuffleDep.rdd)
Review comment:
The overhead shouldn't be so much. I would prefer `prepend(...)` for
readability.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
MaxGekk commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283111395
##
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##
@@ -468,21 +468,20 @@ private[spark] class DAGScheduler(
/** Find ancestor shuffle dependencies that are not registered in
shuffleToMapStage yet */
private def getMissingAncestorShuffleDependencies(
- rdd: RDD[_]): ArrayStack[ShuffleDependency[_, _, _]] = {
-val ancestors = new ArrayStack[ShuffleDependency[_, _, _]]
+ rdd: RDD[_]): ListBuffer[ShuffleDependency[_, _, _]] = {
+val ancestors = new ListBuffer[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
// We are manually maintaining a stack here to prevent StackOverflowError
// caused by recursively visiting
-val waitingForVisit = new ArrayStack[RDD[_]]
-waitingForVisit.push(rdd)
+val waitingForVisit = ListBuffer[RDD[_]](rdd)
while (waitingForVisit.nonEmpty) {
- val toVisit = waitingForVisit.pop()
+ val toVisit = waitingForVisit.remove(0)
if (!visited(toVisit)) {
visited += toVisit
getShuffleDependencies(toVisit).foreach { shuffleDep =>
if (!shuffleIdToMapStage.contains(shuffleDep.shuffleId)) {
-ancestors.push(shuffleDep)
-waitingForVisit.push(shuffleDep.rdd)
+ancestors.+=:(shuffleDep)
Review comment:
You can write:
```scala
shuffleDep +=: ancestors
```
Just in case, any operator ending in `:` is right-associative in Scala
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
srowen commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283111376
##
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##
@@ -468,21 +468,20 @@ private[spark] class DAGScheduler(
/** Find ancestor shuffle dependencies that are not registered in
shuffleToMapStage yet */
private def getMissingAncestorShuffleDependencies(
- rdd: RDD[_]): ArrayStack[ShuffleDependency[_, _, _]] = {
-val ancestors = new ArrayStack[ShuffleDependency[_, _, _]]
+ rdd: RDD[_]): ListBuffer[ShuffleDependency[_, _, _]] = {
+val ancestors = new ListBuffer[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
// We are manually maintaining a stack here to prevent StackOverflowError
// caused by recursively visiting
-val waitingForVisit = new ArrayStack[RDD[_]]
-waitingForVisit.push(rdd)
+val waitingForVisit = ListBuffer[RDD[_]](rdd)
while (waitingForVisit.nonEmpty) {
- val toVisit = waitingForVisit.pop()
+ val toVisit = waitingForVisit.remove(0)
if (!visited(toVisit)) {
visited += toVisit
getShuffleDependencies(toVisit).foreach { shuffleDep =>
if (!shuffleIdToMapStage.contains(shuffleDep.shuffleId)) {
-ancestors.push(shuffleDep)
-waitingForVisit.push(shuffleDep.rdd)
+ancestors.+=:(shuffleDep)
+waitingForVisit.+=:(shuffleDep.rdd)
Review comment:
I should have known.. it's a right associative operator. One is meant to
write `dependency.rdd +=: waitingForVisit`. I suppose it's kind of clear that
it's a prepend. Is it just clearer to use
`waitingForVisit.prepend(dependency.rdd)` and accept the very small overhead of
varargs? I think so
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
MaxGekk commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283110602
##
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##
@@ -468,21 +468,20 @@ private[spark] class DAGScheduler(
/** Find ancestor shuffle dependencies that are not registered in
shuffleToMapStage yet */
private def getMissingAncestorShuffleDependencies(
- rdd: RDD[_]): ArrayStack[ShuffleDependency[_, _, _]] = {
-val ancestors = new ArrayStack[ShuffleDependency[_, _, _]]
+ rdd: RDD[_]): ListBuffer[ShuffleDependency[_, _, _]] = {
+val ancestors = new ListBuffer[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
// We are manually maintaining a stack here to prevent StackOverflowError
// caused by recursively visiting
-val waitingForVisit = new ArrayStack[RDD[_]]
-waitingForVisit.push(rdd)
+val waitingForVisit = ListBuffer[RDD[_]](rdd)
while (waitingForVisit.nonEmpty) {
- val toVisit = waitingForVisit.pop()
+ val toVisit = waitingForVisit.remove(0)
if (!visited(toVisit)) {
visited += toVisit
getShuffleDependencies(toVisit).foreach { shuffleDep =>
if (!shuffleIdToMapStage.contains(shuffleDep.shuffleId)) {
-ancestors.push(shuffleDep)
-waitingForVisit.push(shuffleDep.rdd)
+ancestors.+=:(shuffleDep)
+waitingForVisit.+=:(shuffleDep.rdd)
Review comment:
It seems returned type confuses the compiler. Since the type is not
important here, you can cast it to `Unit` like:
```scala
(waitingForVisit.+=:(dependency.rdd)): Unit
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
srowen commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283110804
##
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##
@@ -506,17 +505,16 @@ private[spark] class DAGScheduler(
rdd: RDD[_]): HashSet[ShuffleDependency[_, _, _]] = {
val parents = new HashSet[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
-val waitingForVisit = new ArrayStack[RDD[_]]
-waitingForVisit.push(rdd)
+val waitingForVisit = ListBuffer[RDD[_]](rdd)
Review comment:
OK I think this 'optimization' does actually cause a problem. It's not happy
about the two unbound wildcard types involved in one expression now. Breaking
it back out ...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24584: [SPARK-27680][CORE][SQL][GRAPHX] Remove usage of Traversable
srowen commented on a change in pull request #24584:
[SPARK-27680][CORE][SQL][GRAPHX] Remove usage of Traversable
URL: https://github.com/apache/spark/pull/24584#discussion_r283110054
##
File path: core/src/main/scala/org/apache/spark/SparkConf.scala
##
@@ -168,6 +168,15 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable
with Logging with Seria
}
/** Set multiple parameters together */
+ def setAll(settings: Iterable[(String, String)]): SparkConf = {
+settings.foreach { case (k, v) => set(k, v) }
+this
+ }
+
+ /**
+ * Set multiple parameters together
+ */
+ @deprecated("Use setAll(Iterable) instead", "3.0.0")
Review comment:
Hm, is it even worth keeping and deprecating this and removing it later?
It'll be source compatible already and binary compatibility isn't guaranteed;
one would have to go out of one's way to call the deprecated method at all,
with a cast or with some odd custom collection
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24584: [SPARK-27680][CORE][SQL][GRAPHX] Remove usage of Traversable
srowen commented on a change in pull request #24584:
[SPARK-27680][CORE][SQL][GRAPHX] Remove usage of Traversable
URL: https://github.com/apache/spark/pull/24584#discussion_r283110129
##
File path: core/src/main/scala/org/apache/spark/SparkConf.scala
##
@@ -168,6 +168,15 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable
with Logging with Seria
}
/** Set multiple parameters together */
+ def setAll(settings: Iterable[(String, String)]): SparkConf = {
+settings.foreach { case (k, v) => set(k, v) }
+this
+ }
+
+ /**
+ * Set multiple parameters together
+ */
+ @deprecated("Use setAll(Iterable) instead", "3.0.0")
Review comment:
One reason I put this forth is that the `TraversableOnce` problem is harder;
its replacement `IterableOnce` doesn't exist in Scala 2.12. The same overload
strategy doesn't work out quite right for the places it's used (generic types
and function signatures make the overload ambiguous). I am considering
proposing just changing the type there for 3.0.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491535097 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105333/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491535094 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491535092 **[Test build #105333 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105333/testReport)** for PR 24586 at commit [`f70a5cc`](https://github.com/apache/spark/commit/f70a5ccc3df2da2dd5d10d4da020ee46b68c2d08). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491535097 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105333/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491535094 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491534816 **[Test build #105333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105333/testReport)** for PR 24586 at commit [`f70a5cc`](https://github.com/apache/spark/commit/f70a5ccc3df2da2dd5d10d4da020ee46b68c2d08). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491534711 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins removed a comment on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491534713 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10605/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
SparkQA commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491534816 **[Test build #105333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105333/testReport)** for PR 24586 at commit [`f70a5cc`](https://github.com/apache/spark/commit/f70a5ccc3df2da2dd5d10d4da020ee46b68c2d08). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491534711 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
AmplabJenkins commented on issue #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586#issuecomment-491534713 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10605/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
srowen commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283107229
##
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##
@@ -468,21 +468,20 @@ private[spark] class DAGScheduler(
/** Find ancestor shuffle dependencies that are not registered in
shuffleToMapStage yet */
private def getMissingAncestorShuffleDependencies(
- rdd: RDD[_]): ArrayStack[ShuffleDependency[_, _, _]] = {
-val ancestors = new ArrayStack[ShuffleDependency[_, _, _]]
+ rdd: RDD[_]): ListBuffer[ShuffleDependency[_, _, _]] = {
+val ancestors = new ListBuffer[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
// We are manually maintaining a stack here to prevent StackOverflowError
// caused by recursively visiting
-val waitingForVisit = new ArrayStack[RDD[_]]
-waitingForVisit.push(rdd)
+val waitingForVisit = ListBuffer[RDD[_]](rdd)
while (waitingForVisit.nonEmpty) {
- val toVisit = waitingForVisit.pop()
+ val toVisit = waitingForVisit.remove(0)
if (!visited(toVisit)) {
visited += toVisit
getShuffleDependencies(toVisit).foreach { shuffleDep =>
if (!shuffleIdToMapStage.contains(shuffleDep.shuffleId)) {
-ancestors.push(shuffleDep)
-waitingForVisit.push(shuffleDep.rdd)
+ancestors.+=:(shuffleDep)
Review comment:
"push" an element on the front. We can also call `prepend` which is clearer
but has some varargs overhead. This operator is weird and IntelliJ claims it
can't be invoked like `foo +=: bar`, but I wonder if it's mistaken. I don't so
much like this invocation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
srowen commented on a change in pull request #24586:
[SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that
will be removed in Scala 2.13 with work-alikes
URL: https://github.com/apache/spark/pull/24586#discussion_r283107192
##
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##
@@ -468,21 +468,20 @@ private[spark] class DAGScheduler(
/** Find ancestor shuffle dependencies that are not registered in
shuffleToMapStage yet */
private def getMissingAncestorShuffleDependencies(
- rdd: RDD[_]): ArrayStack[ShuffleDependency[_, _, _]] = {
-val ancestors = new ArrayStack[ShuffleDependency[_, _, _]]
+ rdd: RDD[_]): ListBuffer[ShuffleDependency[_, _, _]] = {
+val ancestors = new ListBuffer[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
// We are manually maintaining a stack here to prevent StackOverflowError
// caused by recursively visiting
-val waitingForVisit = new ArrayStack[RDD[_]]
-waitingForVisit.push(rdd)
+val waitingForVisit = ListBuffer[RDD[_]](rdd)
while (waitingForVisit.nonEmpty) {
- val toVisit = waitingForVisit.pop()
+ val toVisit = waitingForVisit.remove(0)
Review comment:
"pop" the first element
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen opened a new pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes
srowen opened a new pull request #24586: [SPARK-27682][CORE][GRAPHX][MLLIB] Replace use of collections and methods that will be removed in Scala 2.13 with work-alikes URL: https://github.com/apache/spark/pull/24586 ## What changes were proposed in this pull request? This replaces use of collection classes like `MutableList` and `ArrayStack` with workalikes that are available in 2.12, as they will be removed in 2.13. It also removes use of `.to[Collection]` as its uses was superfluous anyway. Removing `collection.breakOut` will have to wait until 2.13 ## How was this patch tested? Existing tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #24584: [SPARK-27680][CORE][SQL][GRAPHX] Remove usage of Traversable
srowen commented on issue #24584: [SPARK-27680][CORE][SQL][GRAPHX] Remove usage of Traversable URL: https://github.com/apache/spark/pull/24584#issuecomment-491532027 BTW we have a bigger related issue: `TraversableOnce` -> `IterableOnce`. This is much more widely used and in some key APIs. I may break that one out separately into a new JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
srowen commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491529548 If I understand this correctly, you're roughly arguing to not make subdirs of the user-supplied path, and @vanzin is. I'm slightly in favor of subdirs from what I know here, so the question was to you, what's the downside -- would you be strongly against it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491523244 @srowen I also felt missing the counterargument. @vanzin could you clarify what is the issue here so we can move forward? I also hope for @erikerlandson and @liyinan926 to comment here if there is a different perspective. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491523244 @srowen I also felt missing the counterargument. @vanzin could you clarify what is the issue here so we can move forward? I also hope for @erikerlandson and @liyinan926 to comment here if there is a different perspective. As a last resort btw and since the fix is trivial as stated by Marcelo and if the users complain we can always do the "fix" later, no? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491523244 @srowen I also felt missing the counterargument. @vanzin could you clarify what is the issue here so we can move forward? I also hope for @erikerlandson and @liyinan926 to comment here if there is a different perspective. As a last resort btw and since the fix is trivial as stated by Marcelo and if the users complain we can always do fix later, no? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491523244 @srowen I also felt missing the counterargument. @vanzin could you clarify what is the issue here so we can move forward? I also hope @erikerlandson and @liyinan926 to comment here if there is a different perspective. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491523244 @srowen I also felt missing the counterargument. @vanzin could you clarify what is the issue here so we can move forward? I also hope @erikerlandson and @liyinan926 to comment here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #23670: [SPARK-26601][SQL] Make broadcast-exchange thread pool configurable
HyukjinKwon commented on a change in pull request #23670: [SPARK-26601][SQL]
Make broadcast-exchange thread pool configurable
URL: https://github.com/apache/spark/pull/23670#discussion_r283096710
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##
@@ -132,6 +132,19 @@ object StaticSQLConf {
.intConf
.createWithDefault(1000)
+ val BROADCAST_EXCHANGE_MAX_THREAD_THREASHOLD =
+buildStaticConf("spark.sql.broadcastExchange.maxThreadThreshold")
+ .internal()
+ .doc("The maximum degree of parallelism to fetch and broadcast the
table. " +
+"If we encounter memory issue like frequently full GC or OOM when
broadcast table " +
+"we can decrease this number in order to reduce memory usage. " +
+"Notice the number should be carefully chosen since decreasing
parallelism might " +
+"cause longer waiting for other broadcasting. Also, increasing
parallelism may " +
+"cause memory problem.")
+ .intConf
+ .checkValue(thres => thres > 0 && thres <= 128, "The threshold should be
positive.")
Review comment:
Yea let's improve this one before getting this in
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
AmplabJenkins commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491510093 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105332/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
AmplabJenkins removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491510093 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105332/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
AmplabJenkins commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491510089 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
AmplabJenkins removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491510089 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
SparkQA removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491498561 **[Test build #105332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105332/testReport)** for PR 24576 at commit [`a5c2416`](https://github.com/apache/spark/commit/a5c2416807d720074958eab19e144c54d7ac45b2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
SparkQA commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491509936 **[Test build #105332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105332/testReport)** for PR 24576 at commit [`a5c2416`](https://github.com/apache/spark/commit/a5c2416807d720074958eab19e144c54d7ac45b2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24585: Performance issue while listing large number of files on an object store.
AmplabJenkins removed a comment on issue #24585: Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-491504997 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105331/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24585: Performance issue while listing large number of files on an object store.
AmplabJenkins removed a comment on issue #24585: Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-491504995 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24585: Performance issue while listing large number of files on an object store.
AmplabJenkins commented on issue #24585: Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-491504995 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24585: Performance issue while listing large number of files on an object store.
AmplabJenkins commented on issue #24585: Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-491504997 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105331/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24585: Performance issue while listing large number of files on an object store.
SparkQA removed a comment on issue #24585: Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-491493814 **[Test build #105331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105331/testReport)** for PR 24585 at commit [`4c94ce2`](https://github.com/apache/spark/commit/4c94ce2ca27e72d8385985becc61c2ae1896f42f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24585: Performance issue while listing large number of files on an object store.
SparkQA commented on issue #24585: Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-491504877 **[Test build #105331 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105331/testReport)** for PR 24585 at commit [`4c94ce2`](https://github.com/apache/spark/commit/4c94ce2ca27e72d8385985becc61c2ae1896f42f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
SparkQA commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491498561 **[Test build #105332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105332/testReport)** for PR 24576 at commit [`a5c2416`](https://github.com/apache/spark/commit/a5c2416807d720074958eab19e144c54d7ac45b2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
AmplabJenkins removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491498456 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10604/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
AmplabJenkins removed a comment on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491498454 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
AmplabJenkins commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491498454 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
AmplabJenkins commented on issue #24576: [SPARK-27671][SQL] Fix error when casting from a nested null in a struct URL: https://github.com/apache/spark/pull/24576#issuecomment-491498456 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10604/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24585: Performance issue while listing large number of files on an object store.
AmplabJenkins removed a comment on issue #24585: Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-491494178 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10603/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
