[GitHub] AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447546169 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100175/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447546166 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447546166 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
SparkQA removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535766 **[Test build #100175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100175/testReport)** for PR 23306 at commit [`e01779c`](https://github.com/apache/spark/commit/e01779cff25f5f465350f076d9d05c61e4931e71). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447546169 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100175/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
SparkQA commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447546127 **[Test build #100175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100175/testReport)** for PR 23306 at commit [`e01779c`](https://github.com/apache/spark/commit/e01779cff25f5f465350f076d9d05c61e4931e71). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
SparkQA removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545791 **[Test build #100179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100179/testReport)** for PR 23322 at commit [`8844f3d`](https://github.com/apache/spark/commit/8844f3de89fbdffe4e310db120b793de12f97928). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545862 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100179/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545860 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
SparkQA commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545858 **[Test build #100179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100179/testReport)** for PR 23322 at commit [`8844f3d`](https://github.com/apache/spark/commit/8844f3de89fbdffe4e310db120b793de12f97928). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545862 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100179/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545860 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
SparkQA commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545791 **[Test build #100179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100179/testReport)** for PR 23322 at commit [`8844f3d`](https://github.com/apache/spark/commit/8844f3de89fbdffe4e310db120b793de12f97928). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545750 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545751 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6157/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545750 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447545751 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6157/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] kiszk commented on a change in pull request #23278: [SPARK-24920][Core] Allow sharing Netty's memory pool allocators
kiszk commented on a change in pull request #23278: [SPARK-24920][Core] Allow
sharing Netty's memory pool allocators
URL: https://github.com/apache/spark/pull/23278#discussion_r241941523
##
File path:
common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##
@@ -95,6 +111,38 @@ public static String getRemoteAddress(Channel channel) {
return "";
}
+ /**
+ * Returns the default number of threads for both the Netty client and
server thread pools.
+ * If numUsableCores is 0, we will use Runtime get an approximate number of
available cores.
+ */
+ public static int defaultNumThreads(int numUsableCores) {
+final int availableCores;
+if (numUsableCores > 0) {
+ availableCores = numUsableCores;
+} else {
+ availableCores = Runtime.getRuntime().availableProcessors();
+}
+return Math.min(availableCores, MAX_DEFAULT_NETTY_THREADS);
+ }
+
+ /**
+ * Returns the lazily created shared pooled ByteBuf allocator for the
specified allowCache
+ * parameter value.
+ */
+ public static synchronized PooledByteBufAllocator
getSharedPooledByteBufAllocator(
Review comment:
If you will use volatile, this code must use two `volatile` instance
variables instead of one array. This is because `volatile
PooledByteBufAllocator _s[]` is effective only for `_s` rather than `_s[0],
_s[1], ...`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545453 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100178/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
SparkQA removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545327 **[Test build #100178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100178/testReport)** for PR 23243 at commit [`cc5fec2`](https://github.com/apache/spark/commit/cc5fec26ae1026fb7de16791c3cfb25582b235f9). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545452 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545453 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100178/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545452 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
SparkQA commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545451 **[Test build #100178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100178/testReport)** for PR 23243 at commit [`cc5fec2`](https://github.com/apache/spark/commit/cc5fec26ae1026fb7de16791c3cfb25582b235f9). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545254 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6156/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
SparkQA commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545327 **[Test build #100178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100178/testReport)** for PR 23243 at commit [`cc5fec2`](https://github.com/apache/spark/commit/cc5fec26ae1026fb7de16791c3cfb25582b235f9). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545250 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545254 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6156/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447545250 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-444769071 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] KeiichiHirobe commented on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore
KeiichiHirobe commented on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore URL: https://github.com/apache/spark/pull/23288#issuecomment-447544900 I agree with srowen's idea. Most cases, behaviour change makes no problem,I think. So, May I implement a new behavior, or should I wait for a moment? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
felixcheung commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-447544940 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544558 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100177/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
SparkQA commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544552 **[Test build #100177 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100177/testReport)** for PR 23322 at commit [`778acb4`](https://github.com/apache/spark/commit/778acb4781b25bffaa568055329d7103660b9572). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544556 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544558 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100177/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
SparkQA removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-44757 **[Test build #100177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100177/testReport)** for PR 23322 at commit [`778acb4`](https://github.com/apache/spark/commit/778acb4781b25bffaa568055329d7103660b9572). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544556 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
felixcheung commented on a change in pull request #23322: [SPARK-25922][K8]
Spark Driver/Executor spark-app-selector label mism…
URL: https://github.com/apache/spark/pull/23322#discussion_r241941126
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
##
@@ -60,6 +61,16 @@ private[spark] class KubernetesClusterSchedulerBackend(
removeExecutor(executorId, reason)
}
+ /**
+* Get an application ID associated with the job.
+* This returns the string value of spark.app.id if set, otherwise
+* the locally-generated ID from the superclass.
+* @return The application ID
+*/
+ override def applicationId(): String = {
+conf.getOption("spark.app.id").map(_.toString).getOrElse
{super.applicationId}
Review comment:
nit style: spacing around `{` `}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
felixcheung commented on a change in pull request #23322: [SPARK-25922][K8]
Spark Driver/Executor spark-app-selector label mism…
URL: https://github.com/apache/spark/pull/23322#discussion_r241941138
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
##
@@ -60,6 +61,16 @@ private[spark] class KubernetesClusterSchedulerBackend(
removeExecutor(executorId, reason)
}
+ /**
+* Get an application ID associated with the job.
+* This returns the string value of spark.app.id if set, otherwise
+* the locally-generated ID from the superclass.
+* @return The application ID
+*/
+ override def applicationId(): String = {
+conf.getOption("spark.app.id").map(_.toString).getOrElse
{super.applicationId}
Review comment:
actually, this should use `getOrElse(super.applicationId)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544405 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544405 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
felixcheung commented on a change in pull request #23322: [SPARK-25922][K8]
Spark Driver/Executor spark-app-selector label mism…
URL: https://github.com/apache/spark/pull/23322#discussion_r241941146
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
##
@@ -36,7 +36,7 @@ class KubernetesClusterSchedulerBackendSuite extends
SparkFunSuite with BeforeAn
private val requestExecutorsService = new DeterministicScheduler()
private val sparkConf = new SparkConf(false)
-.set("spark.executor.instances", "3")
+.set("spark.executor.instances", "3").set("spark.app.id",
TEST_SPARK_APP_ID)
Review comment:
test this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins removed a comment on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544406 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6155/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
SparkQA commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-44757 **[Test build #100177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100177/testReport)** for PR 23322 at commit [`778acb4`](https://github.com/apache/spark/commit/778acb4781b25bffaa568055329d7103660b9572). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
AmplabJenkins commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544406 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6155/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
felixcheung commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447544297 Jenkins, ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] weixiuli commented on a change in pull request #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
weixiuli commented on a change in pull request #23243: [SPARK-26288][CORE]add
initRegisteredExecutorsDB
URL: https://github.com/apache/spark/pull/23243#discussion_r241940942
##
File path: core/src/test/scala/org/apache/spark/deploy/worker/WorkerSuite.scala
##
@@ -243,4 +243,13 @@ class WorkerSuite extends SparkFunSuite with Matchers
with BeforeAndAfter {
ExecutorStateChanged("app1", 0, ExecutorState.EXITED, None, None))
assert(cleanupCalled.get() == value)
}
+ test("test initRegisteredExecutorsDB ") {
+val sparkConf = new SparkConf()
+Utils.loadDefaultSparkProperties(sparkConf)
+val securityManager = new SecurityManager(sparkConf)
+sparkConf.set(config.SHUFFLE_SERVICE_DB_ENABLED.key, "true")
+sparkConf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+sparkConf.set("spark.local.dir", "/tmp")
+val externalShuffleService = new ExternalShuffleService(sparkConf,
securityManager)
Review comment:
@dongjoon-hyun Thanks a lot,I have fixed scalastyle and removed the test
ablove yet.
There is a new test to test this patch.
Kindly review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] kiszk edited a comment on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore
kiszk edited a comment on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore URL: https://github.com/apache/spark/pull/23288#issuecomment-447541193 Good catch, could you please add test cases that throw this exception for a file and multiple files? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
AmplabJenkins removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447542191 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
AmplabJenkins removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447542192 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6154/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
AmplabJenkins commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447542192 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6154/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
AmplabJenkins commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447542191 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] kiszk edited a comment on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore
kiszk edited a comment on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore URL: https://github.com/apache/spark/pull/23288#issuecomment-447541193 Good catch, could you please add test cases that throw this exception? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
SparkQA commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447541968 **[Test build #100176 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100176/testReport)** for PR 23303 at commit [`a7fb7a6`](https://github.com/apache/spark/commit/a7fb7a67e1f4a683f60778f7f66b208ddf77609f). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] kiszk commented on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore
kiszk commented on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore URL: https://github.com/apache/spark/pull/23288#issuecomment-447541193 Good catch, could you please add a test case that throw this exception? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] kiszk edited a comment on issue #23171: [SPARK-26205][SQL] Optimize In for bytes, shorts, ints
kiszk edited a comment on issue #23171: [SPARK-26205][SQL] Optimize In for bytes, shorts, ints URL: https://github.com/apache/spark/pull/23171#issuecomment-447540932 Comment about JIT code generation for switch statement. Regardless of lookupswitch/tableswitch, JIT compiler generations native instructions suitable for data size/density (binary tree, table lookup, (or simple if-then?)). It would be good to see [this article](https://stackoverflow.com/questions/44988841/force-tableswitch-instead-of-lookupswitch). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] kiszk commented on issue #23171: [SPARK-26205][SQL] Optimize In for bytes, shorts, ints
kiszk commented on issue #23171: [SPARK-26205][SQL] Optimize In for bytes, shorts, ints URL: https://github.com/apache/spark/pull/23171#issuecomment-447540932 Comment about JIT code generation for switch statement. Regardless of lookupswitch/tableswitch, JIT compiler generations native instructions suitable for data size/density (simple if-then, binary tree, or table lookup). It would be good to see [this article](https://stackoverflow.com/questions/44988841/force-tableswitch-instead-of-lookupswitch). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on a change in pull request #23323: [SPARK-26372][SQL] Don't reuse value from previous row when parsing bad CSV input field
HyukjinKwon commented on a change in pull request #23323: [SPARK-26372][SQL]
Don't reuse value from previous row when parsing bad CSV input field
URL: https://github.com/apache/spark/pull/23323#discussion_r241939744
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
##
@@ -239,6 +239,7 @@ class UnivocityParser(
} catch {
case NonFatal(e) =>
badRecordException = badRecordException.orElse(Some(e))
+row(i) = null
Review comment:
nit: `row.setNullAt(i)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] xuanyuanking commented on issue #23299: [SPARK-26327][SQL][BACKPORT-2.3] Bug fix for `FileSourceScanExec` metrics update
xuanyuanking commented on issue #23299: [SPARK-26327][SQL][BACKPORT-2.3] Bug fix for `FileSourceScanExec` metrics update URL: https://github.com/apache/spark/pull/23299#issuecomment-447540460 Thanks Dongjoon. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] xuanyuanking commented on issue #23300: [SPARK-26327][SQL][BACKPORT-2.2] Bug fix for `FileSourceScanExec` metrics update
xuanyuanking commented on issue #23300: [SPARK-26327][SQL][BACKPORT-2.2] Bug fix for `FileSourceScanExec` metrics update URL: https://github.com/apache/spark/pull/23300#issuecomment-447540390 Thanks Dongjoon. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] asfgit closed pull request #23311: [SPARK-26362][CORE] Remove 'spark.driver.allowMultipleContexts' to disallow multiple creation of SparkContexts
asfgit closed pull request #23311: [SPARK-26362][CORE] Remove
'spark.driver.allowMultipleContexts' to disallow multiple creation of
SparkContexts
URL: https://github.com/apache/spark/pull/23311
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 696dafda6d1ec..09cc346db0ed2 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -64,9 +64,8 @@ import org.apache.spark.util.logging.DriverLogger
* Main entry point for Spark functionality. A SparkContext represents the
connection to a Spark
* cluster, and can be used to create RDDs, accumulators and broadcast
variables on that cluster.
*
- * Only one SparkContext may be active per JVM. You must `stop()` the active
SparkContext before
- * creating a new one. This limitation may eventually be removed; see
SPARK-2243 for more details.
- *
+ * @note Only one `SparkContext` should be active per JVM. You must `stop()`
the
+ * active `SparkContext` before creating a new one.
* @param config a Spark Config object describing the application
configuration. Any settings in
* this config overrides the default configs as well as system properties.
*/
@@ -75,14 +74,10 @@ class SparkContext(config: SparkConf) extends Logging {
// The call site where this SparkContext was constructed.
private val creationSite: CallSite = Utils.getCallSite()
- // If true, log warnings instead of throwing exceptions when multiple
SparkContexts are active
- private val allowMultipleContexts: Boolean =
-config.getBoolean("spark.driver.allowMultipleContexts", false)
-
// In order to prevent multiple SparkContexts from being active at the same
time, mark this
// context as having started construction.
// NOTE: this must be placed at the beginning of the SparkContext
constructor.
- SparkContext.markPartiallyConstructed(this, allowMultipleContexts)
+ SparkContext.markPartiallyConstructed(this)
val startTime = System.currentTimeMillis()
@@ -2392,7 +2387,7 @@ class SparkContext(config: SparkConf) extends Logging {
// In order to prevent multiple SparkContexts from being active at the same
time, mark this
// context as having finished construction.
// NOTE: this must be placed at the end of the SparkContext constructor.
- SparkContext.setActiveContext(this, allowMultipleContexts)
+ SparkContext.setActiveContext(this)
}
/**
@@ -2409,18 +2404,18 @@ object SparkContext extends Logging {
private val SPARK_CONTEXT_CONSTRUCTOR_LOCK = new Object()
/**
- * The active, fully-constructed SparkContext. If no SparkContext is
active, then this is `null`.
+ * The active, fully-constructed SparkContext. If no SparkContext is active,
then this is `null`.
*
- * Access to this field is guarded by SPARK_CONTEXT_CONSTRUCTOR_LOCK.
+ * Access to this field is guarded by `SPARK_CONTEXT_CONSTRUCTOR_LOCK`.
*/
private val activeContext: AtomicReference[SparkContext] =
new AtomicReference[SparkContext](null)
/**
- * Points to a partially-constructed SparkContext if some thread is in the
SparkContext
+ * Points to a partially-constructed SparkContext if another thread is in
the SparkContext
* constructor, or `None` if no SparkContext is being constructed.
*
- * Access to this field is guarded by SPARK_CONTEXT_CONSTRUCTOR_LOCK
+ * Access to this field is guarded by `SPARK_CONTEXT_CONSTRUCTOR_LOCK`.
*/
private var contextBeingConstructed: Option[SparkContext] = None
@@ -2428,24 +2423,16 @@ object SparkContext extends Logging {
* Called to ensure that no other SparkContext is running in this JVM.
*
* Throws an exception if a running context is detected and logs a warning
if another thread is
- * constructing a SparkContext. This warning is necessary because the
current locking scheme
+ * constructing a SparkContext. This warning is necessary because the
current locking scheme
* prevents us from reliably distinguishing between cases where another
context is being
* constructed and cases where another constructor threw an exception.
*/
- private def assertNoOtherContextIsRunning(
- sc: SparkContext,
- allowMultipleContexts: Boolean): Unit = {
+ private def assertNoOtherContextIsRunning(sc: SparkContext): Unit = {
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
Option(activeContext.get()).filter(_ ne sc).foreach { ctx =>
- val errMsg = "Only one SparkContext may be running in this JVM (see
SPARK-2243)." +
-" To ignore this error, set spark.driver.allowMultipleContexts =
true. " +
+
[GitHub] asfgit closed pull request #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
asfgit closed pull request #23294: [SPARK-26265][Core][Followup] Put freePage
into a finally block
URL: https://github.com/apache/spark/pull/23294
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
index fbba002f1f80f..7df8aafb2b674 100644
--- a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
+++ b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
@@ -262,36 +262,39 @@ private void advanceToNextPage() {
// reference to the page to free and free it after releasing the lock of
`MapIterator`.
MemoryBlock pageToFree = null;
- synchronized (this) {
-int nextIdx = dataPages.indexOf(currentPage) + 1;
-if (destructive && currentPage != null) {
- dataPages.remove(currentPage);
- pageToFree = currentPage;
- nextIdx --;
-}
-if (dataPages.size() > nextIdx) {
- currentPage = dataPages.get(nextIdx);
- pageBaseObject = currentPage.getBaseObject();
- offsetInPage = currentPage.getBaseOffset();
- recordsInPage = UnsafeAlignedOffset.getSize(pageBaseObject,
offsetInPage);
- offsetInPage += UnsafeAlignedOffset.getUaoSize();
-} else {
- currentPage = null;
- if (reader != null) {
-handleFailedDelete();
+ try {
+synchronized (this) {
+ int nextIdx = dataPages.indexOf(currentPage) + 1;
+ if (destructive && currentPage != null) {
+dataPages.remove(currentPage);
+pageToFree = currentPage;
+nextIdx--;
}
- try {
-Closeables.close(reader, /* swallowIOException = */ false);
-reader = spillWriters.getFirst().getReader(serializerManager);
-recordsInPage = -1;
- } catch (IOException e) {
-// Scala iterator does not handle exception
-Platform.throwException(e);
+ if (dataPages.size() > nextIdx) {
+currentPage = dataPages.get(nextIdx);
+pageBaseObject = currentPage.getBaseObject();
+offsetInPage = currentPage.getBaseOffset();
+recordsInPage = UnsafeAlignedOffset.getSize(pageBaseObject,
offsetInPage);
+offsetInPage += UnsafeAlignedOffset.getUaoSize();
+ } else {
+currentPage = null;
+if (reader != null) {
+ handleFailedDelete();
+}
+try {
+ Closeables.close(reader, /* swallowIOException = */ false);
+ reader = spillWriters.getFirst().getReader(serializerManager);
+ recordsInPage = -1;
+} catch (IOException e) {
+ // Scala iterator does not handle exception
+ Platform.throwException(e);
+}
}
}
- }
- if (pageToFree != null) {
-freePage(pageToFree);
+ } finally {
+if (pageToFree != null) {
+ freePage(pageToFree);
+}
}
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23311: [SPARK-26362][CORE] Remove 'spark.driver.allowMultipleContexts' to disallow multiple creation of SparkContexts
HyukjinKwon commented on issue #23311: [SPARK-26362][CORE] Remove 'spark.driver.allowMultipleContexts' to disallow multiple creation of SparkContexts URL: https://github.com/apache/spark/pull/23311#issuecomment-447540127 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
HyukjinKwon commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-447540018 tests were already passed for the same commit at https://github.com/apache/spark/pull/23294#issuecomment-446868590 Merged to master and branch-2.4. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
AmplabJenkins removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447539836 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100173/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
AmplabJenkins commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447539834 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
AmplabJenkins removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447539834 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
AmplabJenkins commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447539836 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100173/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
SparkQA removed a comment on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447531156 **[Test build #100173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100173/testReport)** for PR 23303 at commit [`385cbd3`](https://github.com/apache/spark/commit/385cbd349b26ef427097d75074693ccff1a19fbb). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes
SparkQA commented on issue #23303: [SPARK-26352][SQL] join reorder should not change the order of output attributes URL: https://github.com/apache/spark/pull/23303#issuecomment-447539799 **[Test build #100173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100173/testReport)** for PR 23303 at commit [`385cbd3`](https://github.com/apache/spark/commit/385cbd349b26ef427097d75074693ccff1a19fbb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] kiszk commented on a change in pull request #23171: [SPARK-26205][SQL] Optimize In for bytes, shorts, ints
kiszk commented on a change in pull request #23171: [SPARK-26205][SQL] Optimize
In for bytes, shorts, ints
URL: https://github.com/apache/spark/pull/23171#discussion_r241939270
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
##
@@ -335,6 +343,41 @@ case class In(value: Expression, list: Seq[Expression])
extends Predicate {
""".stripMargin)
}
+ private def genCodeWithSwitch(ctx: CodegenContext, ev: ExprCode): ExprCode =
{
+val (nullLiterals, nonNullLiterals) = list.partition {
+ case Literal(null, _) => true
+ case _ => false
+}
+val listGen = nonNullLiterals.map(_.genCode(ctx))
+val valueGen = value.genCode(ctx)
+
+val caseBranches = listGen.map(literal =>
+ s"""
+ |case ${literal.value}:
+ | ${ev.value} = true;
+ | break;
+ """.stripMargin)
+
+ev.copy(code =
+ code"""
+ |${valueGen.code}
+ |${CodeGenerator.JAVA_BOOLEAN} ${ev.isNull} = ${valueGen.isNull};
+ |${CodeGenerator.JAVA_BOOLEAN} ${ev.value} = false;
+ |if (!${valueGen.isNull}) {
+ | switch (${valueGen.value}) {
+ |${caseBranches.mkString("")}
+ |default:
+ | ${ev.isNull} = ${nullLiterals.nonEmpty};
+ | }
+ |}
+ """.stripMargin)
+ }
+
+ private def isSwitchCompatible: Boolean = list.forall {
Review comment:
Could you please take care of the following limitation of Java switch
statement, too?
> npairs pairs of signed 32-bit values
https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-6.html#jvms-6.5.lookupswitch
https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-6.html#jvms-6.5.tableswitch
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] viirya commented on issue #23271: [SPARK-26318][SQL] Enhance function merge performance in Row
viirya commented on issue #23271: [SPARK-26318][SQL] Enhance function merge performance in Row URL: https://github.com/apache/spark/pull/23271#issuecomment-447538351 And don't forget to update the PR title and description too. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] viirya commented on a change in pull request #23271: [SPARK-26318][SQL] Enhance function merge performance in Row
viirya commented on a change in pull request #23271: [SPARK-26318][SQL] Enhance
function merge performance in Row
URL: https://github.com/apache/spark/pull/23271#discussion_r241938910
##
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
##
@@ -57,9 +57,29 @@ object Row {
/**
* Merge multiple rows into a single row, one after another.
*/
+
+ @deprecated("This method is deprecated and will be removed in future
versions.", "3.0.0")
def merge(rows: Row*): Row = {
-// TODO: Improve the performance of this if used in performance critical
part.
-new GenericRow(rows.flatMap(_.toSeq).toArray)
+var fSize, sSize = rows.size
+var number = 0
+
+while (fSize != 0) {
+ fSize -= 1
+ number += rows(fSize).size
+}
+
+val container = Array.ofDim[Any](number)
+
+while (sSize != 0) {
+ sSize -= 1
+ var ssSize = rows(sSize).size
+ while (ssSize != 0) {
+number -= 1
+ssSize -= 1
+container(number) = rows(sSize)(ssSize)
+ }
+}
Review comment:
yea, it is not used before and is deprecated now. We don't need to do this
optimization.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535918 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535920 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6153/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535918 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535920 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6153/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
SparkQA commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535766 **[Test build #100175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100175/testReport)** for PR 23306 at commit [`e01779c`](https://github.com/apache/spark/commit/e01779cff25f5f465350f076d9d05c61e4931e71). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] rezasafi commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
rezasafi commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535703 Flaky test. Jenkins retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535494 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100171/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535493 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535493 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
SparkQA removed a comment on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447520824 **[Test build #100171 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100171/testReport)** for PR 23306 at commit [`e01779c`](https://github.com/apache/spark/commit/e01779cff25f5f465350f076d9d05c61e4931e71). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
AmplabJenkins commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535494 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100171/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system
SparkQA commented on issue #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306#issuecomment-447535448 **[Test build #100171 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100171/testReport)** for PR 23306 at commit [`e01779c`](https://github.com/apache/spark/commit/e01779cff25f5f465350f076d9d05c61e4931e71). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
AmplabJenkins removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-447535136 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
AmplabJenkins removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-447535137 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6152/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
AmplabJenkins commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-447535136 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
AmplabJenkins commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-447535137 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6152/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
SparkQA commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-447535123 **[Test build #100174 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100174/testReport)** for PR 23294 at commit [`630281b`](https://github.com/apache/spark/commit/630281b59503b88ed5ba14e1b8e5029e1589c478). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
HyukjinKwon commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-447534883 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23271: [SPARK-26318][SQL] Enhance function merge performance in Row
HyukjinKwon commented on issue #23271: [SPARK-26318][SQL] Enhance function merge performance in Row URL: https://github.com/apache/spark/pull/23271#issuecomment-447534828 Let's target to deprecate the API in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on a change in pull request #23271: [SPARK-26318][SQL] Enhance function merge performance in Row
HyukjinKwon commented on a change in pull request #23271: [SPARK-26318][SQL]
Enhance function merge performance in Row
URL: https://github.com/apache/spark/pull/23271#discussion_r241937089
##
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
##
@@ -57,9 +57,29 @@ object Row {
/**
* Merge multiple rows into a single row, one after another.
*/
+
+ @deprecated("This method is deprecated and will be removed in future
versions.", "3.0.0")
def merge(rows: Row*): Row = {
-// TODO: Improve the performance of this if used in performance critical
part.
-new GenericRow(rows.flatMap(_.toSeq).toArray)
+var fSize, sSize = rows.size
+var number = 0
+
+while (fSize != 0) {
+ fSize -= 1
+ number += rows(fSize).size
+}
+
+val container = Array.ofDim[Any](number)
+
+while (sSize != 0) {
+ sSize -= 1
+ var ssSize = rows(sSize).size
+ while (ssSize != 0) {
+number -= 1
+ssSize -= 1
+container(number) = rows(sSize)(ssSize)
+ }
+}
Review comment:
I would just leave the function as was. It will be removed in the next
release anyway.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] liyinan926 commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
liyinan926 commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447534192 Jenkins ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] suxingfate commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism…
suxingfate commented on issue #23322: [SPARK-25922][K8] Spark Driver/Executor spark-app-selector label mism… URL: https://github.com/apache/spark/pull/23322#issuecomment-447534051 Updated KubernetesClusterSchedulerBackendSuite to invoke override applicationId This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary to replace the duplicate code
AmplabJenkins commented on issue #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary to replace the duplicate code URL: https://github.com/apache/spark/pull/17654#issuecomment-447533253 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary to replace the duplicate code
AmplabJenkins removed a comment on issue #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary to replace the duplicate code URL: https://github.com/apache/spark/pull/17654#issuecomment-447533254 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100172/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary to replace the duplicate code
AmplabJenkins commented on issue #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary to replace the duplicate code URL: https://github.com/apache/spark/pull/17654#issuecomment-447533254 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100172/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary to replace the duplicate code
AmplabJenkins removed a comment on issue #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary to replace the duplicate code URL: https://github.com/apache/spark/pull/17654#issuecomment-447533253 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
