[GitHub] [spark] gatorsmile commented on issue #25533: [SPARK-28798][DOC][SQL]Document DROP TABLE/VIEW statement in SQL Reference.
gatorsmile commented on issue #25533: [SPARK-28798][DOC][SQL]Document DROP TABLE/VIEW statement in SQL Reference. URL: https://github.com/apache/spark/pull/25533#issuecomment-533707977 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25553: [SPARK-28797][DOC] Document DROP FUNCTION statement in SQL Reference.
gatorsmile commented on a change in pull request #25553: [SPARK-28797][DOC] Document DROP FUNCTION statement in SQL Reference. URL: https://github.com/apache/spark/pull/25553#discussion_r326801826 ## File path: docs/sql-ref-syntax-ddl-drop-function.md ## @@ -59,6 +59,18 @@ SHOW USER FUNCTIONS; | default.test_avg | +---+ +-- Create Temporary function `temp_avg` +CREATE TEMPORARY FUNCTION temp_avg as 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage'; + +-- List user functions +SHOW USER FUNCTIONS; + +---+ + | function | + +---+ + | default.test_avg | + | temp_avg | Review comment: Are the functions having the same name? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25590: [SPARK-28791] [DOC] Documentation for Alter table Command
SparkQA commented on issue #25590: [SPARK-28791] [DOC] Documentation for Alter table Command URL: https://github.com/apache/spark/pull/25590#issuecomment-533707720 **[Test build #111091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111091/testReport)** for PR 25590 at commit [`69ac8da`](https://github.com/apache/spark/commit/69ac8dadb45d12e743c7cd768251ff663ea453e3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25873: [WIP][SPARK-29192][TESTS] Extend BenchmarkBase to write JDK9+ results separately
SparkQA commented on issue #25873: [WIP][SPARK-29192][TESTS] Extend BenchmarkBase to write JDK9+ results separately URL: https://github.com/apache/spark/pull/25873#issuecomment-533707711 **[Test build #111090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111090/testReport)** for PR 25873 at commit [`941fcb7`](https://github.com/apache/spark/commit/941fcb773c5a6453e9883e3f63b5b21cea2f569f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25590: [SPARK-28791] [DOC] Documentation for Alter table Command
AmplabJenkins removed a comment on issue #25590: [SPARK-28791] [DOC] Documentation for Alter table Command URL: https://github.com/apache/spark/pull/25590#issuecomment-531891576 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #25590: [SPARK-28791] [DOC] Documentation for Alter table Command
gatorsmile commented on issue #25590: [SPARK-28791] [DOC] Documentation for Alter table Command URL: https://github.com/apache/spark/pull/25590#issuecomment-533706942 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25873: [WIP][SPARK-29192][TESTS] Extend BenchmarkBase to write JDK9+ results separately
dongjoon-hyun commented on issue #25873: [WIP][SPARK-29192][TESTS] Extend BenchmarkBase to write JDK9+ results separately URL: https://github.com/apache/spark/pull/25873#issuecomment-533706779 cc, @srowen and @wangyum . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #25873: [SPARK-29192][TESTS] Extend BenchmarkBase to write JDK9+ results separately
dongjoon-hyun opened a new pull request #25873: [SPARK-29192][TESTS] Extend BenchmarkBase to write JDK9+ results separately URL: https://github.com/apache/spark/pull/25873 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533705344 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533705349 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111086/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533705344 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533705349 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111086/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533662481 **[Test build #111086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111086/testReport)** for PR 22145 at commit [`e5cc9b9`](https://github.com/apache/spark/commit/e5cc9b9a36bb5365e3a75f28d51ee44b19f513e5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533704851 **[Test build #111086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111086/testReport)** for PR 22145 at commit [`e5cc9b9`](https://github.com/apache/spark/commit/e5cc9b9a36bb5365e3a75f28d51ee44b19f513e5). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jinxingwang commented on a change in pull request #25614: [SPARK-28887][K8S] Executor pod status fix
jinxingwang commented on a change in pull request #25614: [SPARK-28887][K8S]
Executor pod status fix
URL: https://github.com/apache/spark/pull/25614#discussion_r326796861
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala
##
@@ -176,6 +177,7 @@ private[spark] class ExecutorPodsLifecycleManager(
private def findExitCode(podState: FinalPodState): Int = {
podState.pod.getStatus.getContainerStatuses.asScala.find { containerStatus
=>
+ containerStatus.getName == DEFAULT_EXECUTOR_CONTAINER_NAME &
Review comment:
This make sure the exit code shows in the pod status is reflect from the
executor container.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in HashAggregateExec
viirya commented on a change in pull request #25830: [SPARK-29140][SQL] Handle
parameters having "array" of javaType properly in HashAggregateExec
URL: https://github.com/apache/spark/pull/25830#discussion_r326793862
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala
##
@@ -1021,13 +1021,28 @@ abstract class AggregationQuerySuite extends QueryTest
with SQLTestUtils with Te
test("SPARK-29122: hash-based aggregates for unfixed-length decimals in the
interpreter mode") {
withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false",
-SQLConf.CODEGEN_FACTORY_MODE.key ->
CodegenObjectFactoryMode.NO_CODEGEN.toString) {
+ SQLConf.CODEGEN_FACTORY_MODE.key ->
CodegenObjectFactoryMode.NO_CODEGEN.toString) {
Review comment:
Actually, I think previous is correct...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533698263 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16173/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533698255 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533698255 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533698213 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/16173/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533698263 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16173/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #25746: [WIP][SPARK-28292][SQL] Enable Injection of User-defined Hint
gatorsmile commented on issue #25746: [WIP][SPARK-28292][SQL] Enable Injection of User-defined Hint URL: https://github.com/apache/spark/pull/25746#issuecomment-533697880 We need to add a test case to ensure the other user-added hints can be added and recognized. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
dongjoon-hyun edited a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533693523 Right, it's impossible in Maven. So, we have been using this way. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533694088 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/16173/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
dongjoon-hyun edited a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533693523 Right. So, we have been using this way. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
dongjoon-hyun commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533693523 Right. So, we use this way. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in HashAggregateExec
SparkQA commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in HashAggregateExec URL: https://github.com/apache/spark/pull/25830#issuecomment-533693128 **[Test build #111089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111089/testReport)** for PR 25830 at commit [`28726da`](https://github.com/apache/spark/commit/28726dab8ec7e4e4b7bffc51e8feb5481fd12802). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
HeartSaVioR commented on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing URL: https://github.com/apache/spark/pull/25862#issuecomment-533693246 cc. @tdas @sameeragarwal as they've authored and reviewed 5adcc8 also cc. to @zsxwing @jose-torres @gaborgsomogyi This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
srowen commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533689498 Is it not possible to exclude a single suite? OK if this is necessary to achieve that, it's just a tag. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore
HeartSaVioR commented on a change in pull request #25811: [SPARK-29111][CORE]
Support snapshot/restore on KVStore
URL: https://github.com/apache/spark/pull/25811#discussion_r326781204
##
File path:
common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSnapshotter.java
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import com.google.common.io.ByteStreams;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+
+public class KVStoreSnapshotter {
+ private static final int MARKER_END_OF_TYPE = -2;
+ private static final int MARKER_END_OF_FILE = -1;
+
+ private final KVStoreSerializer serializer;
+
+ public KVStoreSnapshotter(KVStoreSerializer serializer) {
+this.serializer = serializer;
+ }
+
+ public void dump(KVStore store, File snapshotFile) throws Exception {
+DataOutputStream output = new DataOutputStream(new
FileOutputStream(snapshotFile));
+
+// store metadata if it exists
+Class metadataType = store.metadataType();
+if (metadataType != null) {
+ writeClassName(metadataType, output);
+ Object metadata = store.getMetadata(metadataType);
+ writeObject(metadata, output);
+ writeEndOfType(output);
+} else {
+ writeEndOfType(output);
+}
+
+Set> types = store.types();
+for (Class clazz : types) {
+ writeClassName(clazz, output);
+
+ KVStoreView view = store.view(clazz);
Review comment:
I'm not sure any implementations allow type with empty objects.
I see what you say - if there's some implementation allow the case like
types() returning Class but view(A.class) contains nothing, we don't provide
the way to only add type to KVStore. We may want to be clear in the interface
javadoc that "type with empty objects are ignored while recovering, so
implementations should not rely on this", as thinking theoretically, but I'm
afraid I might be over-thinking.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533688984 **[Test build #111088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111088/testReport)** for PR 22145 at commit [`e5cc9b9`](https://github.com/apache/spark/commit/e5cc9b9a36bb5365e3a75f28d51ee44b19f513e5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533688642 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer
HeartSaVioR edited a comment on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-533680034 The one of main goals in SPARK-28594 is limiting the overall size of log directory per application. (End users have been concerned about it.) That means, we should provide a way to roll the event log file within deterministic size, which is not applicable to roll file per lines. In the following patch I'll introduce max number of files (max file size is introduced in #25670 ) and clean up old event files via replacing these old files with snapshot file - so it'll take a snapshot for different purpose, though it also helps faster reading. Given two issues take a snapshot for different purposes, I'm kind of OK to go with different approaches and consolidate the approach later (assuming the snapshot file is compatible). One thing I might be concerning about is, we only talk about the new approach for in-memory store which Spark hides the implementation of KVStore via wrapping it with ElementTrackingStore. The change should be reflected to KVStore API so that caller side would deal with the way of snapshotting properly. (Now we only add some necessary methods in KVStore to snapshot from outside, but if we have both sync/async snapshot for KVStore, that should be reflected to the KVStore API.) To add some context on this, previously (in internal reviewing) I proposed snapshotting underlying LevelDB - archiving directory would just work - for LevelDB KVStore implementation and I was suggested to find a way to support snapshotting for all implementations of KVStore as we want to let users choose the implementation wisely (in SHS side). That's why current snapshot mechanism is based on KVStore interface. Once we respect the format of snapshot file, both sync/async snapshots would be compatible, but in same spirit, ideally we should support both approaches of snapshot smoothly, via KVStore interface level. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533684664 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533684664 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533684672 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111082/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533684672 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111082/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533637725 **[Test build #111082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111082/testReport)** for PR 22145 at commit [`e5cc9b9`](https://github.com/apache/spark/commit/e5cc9b9a36bb5365e3a75f28d51ee44b19f513e5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25805: [SPARK-29082][core] Skip delegation token generation if no credentials are available.
dongjoon-hyun commented on issue #25805: [SPARK-29082][core] Skip delegation token generation if no credentials are available. URL: https://github.com/apache/spark/pull/25805#issuecomment-533684351 Great! Thank you, @gaborgsomogyi and @squito ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533684270 **[Test build #111082 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111082/testReport)** for PR 22145 at commit [`e5cc9b9`](https://github.com/apache/spark/commit/e5cc9b9a36bb5365e3a75f28d51ee44b19f513e5). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
dongjoon-hyun edited a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533683187 cc @gengliangwang , @gatorsmile , @wangyum , @HyukjinKwon , @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
dongjoon-hyun commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533683187 cc @gengliangwang , @gatorsmile , @wangyum , @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
AmplabJenkins commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533682987 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
AmplabJenkins removed a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533682987 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
AmplabJenkins removed a comment on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533682992 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16172/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
AmplabJenkins commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533682992 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16172/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
SparkQA commented on issue #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872#issuecomment-533682666 **[Test build #111087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111087/testReport)** for PR 25872 at commit [`3d85e88`](https://github.com/apache/spark/commit/3d85e88871038b1c491540dbf1ee99e896225b02). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25829: [SPARK-29144][ML] Binarizer handle sparse vectors incorrectly with negative threshold
dongjoon-hyun edited a comment on issue #25829: [SPARK-29144][ML] Binarizer handle sparse vectors incorrectly with negative threshold URL: https://github.com/apache/spark/pull/25829#issuecomment-533682473 I tested this PR manually on JDK11. There is no problem for this PR~ We can ignore the above failure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25829: [SPARK-29144][ML] Binarizer handle sparse vectors incorrectly with negative threshold
dongjoon-hyun commented on issue #25829: [SPARK-29144][ML] Binarizer handle sparse vectors incorrectly with negative threshold URL: https://github.com/apache/spark/pull/25829#issuecomment-533682473 I test this PR manually on JDK11. There is no problem for this PR~ We can ignore the above failure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore
HeartSaVioR commented on a change in pull request #25811: [SPARK-29111][CORE]
Support snapshot/restore on KVStore
URL: https://github.com/apache/spark/pull/25811#discussion_r326773282
##
File path:
common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreSnapshotter.java
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.kvstore;
+
+import com.google.common.io.ByteStreams;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+
+public class KVStoreSnapshotter {
+ private static final int MARKER_END_OF_TYPE = -2;
+ private static final int MARKER_END_OF_FILE = -1;
+
+ private final KVStoreSerializer serializer;
+
+ public KVStoreSnapshotter(KVStoreSerializer serializer) {
+this.serializer = serializer;
+ }
+
+ public void dump(KVStore store, File snapshotFile) throws Exception {
Review comment:
Nice catch! Thanks for pointing out.
Maybe we need to have InputStream/OutputStream (or more specific types) as a
parameter instead of File so that it can be worked with any filesystem. kvstore
module doesn't have Hadoop dependency so it would be ideal to avoid depending
on Hadoop directly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite
dongjoon-hyun opened a new pull request #25872: [SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite URL: https://github.com/apache/spark/pull/25872 ### What changes were proposed in this pull request? `SQLQueryTestSuite` takes ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on issue #25805: [SPARK-29082][core] Skip delegation token generation if no credentials are available.
gaborgsomogyi commented on issue #25805: [SPARK-29082][core] Skip delegation token generation if no credentials are available. URL: https://github.com/apache/spark/pull/25805#issuecomment-533680397 We've found the issue with the help of @squito so going to file a PR soon with the required change... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer
HeartSaVioR commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-533680034 The one of main goals in SPARK-28594 is limiting the overall size of log directory per application. (End users have been concerned about it.) That means, we should provide a way to roll the event log file within deterministic size, which is not applicable to roll file per lines. In the following patch I'll introduce max number of files (max file size is introduced in #25670 ) and clean up old event files via replacing these old files with snapshot file - so it'll take a snapshot for different purpose, though it also helps faster reading. Given two issues take a snapshot for different purposes, I'm kind of OK to go with different approaches and consolidate the approach later (assuming the snapshot file is compatible). One thing I might be concerning about is, we only talk about the new approach for in-memory store which Spark hides the implementation of KVStore via wrapping it with ElementTrackingStore. The change should be reflected to KVStore API so that caller side would deal with the way of snapshotting properly. (Now we only add some necessary methods in KVStore to snapshot from outside, but if we have both sync/async snapshot for KVStore, that should be reflected to the KVStore API.) To add some context on this, previously (in internal reviewing) I proposed snapshotting underlying LevelDB - archiving directory would just work - for LevelDB KVStore implementation and I was suggested to find a way to support snapshotting for all implementations of KVStore. That's why current snapshot mechanism is based on KVStore interface. Once we respect the format of snapshot file, both sync/async snapshots would be compatible, but in same spirit, ideally we should support both approaches of snapshot smoothly, via KVStore interface level. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25829: [SPARK-29144][ML] Binarizer handle sparse vectors incorrectly with negative threshold
dongjoon-hyun commented on issue #25829: [SPARK-29144][ML] Binarizer handle sparse vectors incorrectly with negative threshold URL: https://github.com/apache/spark/pull/25829#issuecomment-533678570 @zhengruifeng and @srowen . The failure of `GitHub Action` seems due to the maven artifacts download. On top of that, there is a `GitHub Action` bug which sometimes it doesn't allow re-trigger because it thinks the test is still running. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533674225 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16171/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533674217 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533674225 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16171/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533674217 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533674102 > hmm: > > ``` > --smart/-S has been removed. Use +smart or -smart extension instead. > For example: pandoc -f markdown+smart -t markdown-smart. > Try pandoc --help for more information. > Error: processing vignette 'sparkr-vignettes.Rmd' failed with diagnostics: > pandoc document conversion failed with error 2 > Execution halted > ``` > > i also reverted the change to the k8s prb build until this is sorted. well, the build looks like it's going to pass on a different worker than where it failed. looking at the R packages installed on both machines, there are definitely differences in what's on each one but nothing looks to be missing on research-jenkins-worker-09. however, i am assuming that the error is caused by `rmarkdown`, and comparing versions across all ubuntu workers shows that 1.1 is installed on everything except amp-jenkins-staging-worker-02, which has 1.8. this is the worker that the successful k8s prb build ran on. before i go barking up that tree, i would like to confirm that my suspicions are correct. anyone? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533674175 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/16171/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533669447 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/16171/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25864: [SPARK-29139][CORE][TESTS] Increase timeout to wait for executor(s) to be up in SparkContextSuite
HeartSaVioR commented on issue #25864: [SPARK-29139][CORE][TESTS] Increase timeout to wait for executor(s) to be up in SparkContextSuite URL: https://github.com/apache/spark/pull/25864#issuecomment-533668827 Thanks all for reviewing and merging! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25870: [SPARK-27936][K8S] support python deps
AmplabJenkins removed a comment on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-533663338 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111078/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25870: [SPARK-27936][K8S] support python deps
AmplabJenkins removed a comment on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-533663330 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25870: [SPARK-27936][K8S] support python deps
AmplabJenkins commented on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-533663338 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111078/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25870: [SPARK-27936][K8S] support python deps
AmplabJenkins commented on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-533663330 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25870: [SPARK-27936][K8S] support python deps
SparkQA removed a comment on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-533605014 **[Test build #111078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111078/testReport)** for PR 25870 at commit [`40fe23c`](https://github.com/apache/spark/commit/40fe23c540a8db7dcea335457dc8fe5d3328f529). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25870: [SPARK-27936][K8S] support python deps
SparkQA commented on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-533662664 **[Test build #111078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111078/testReport)** for PR 25870 at commit [`40fe23c`](https://github.com/apache/spark/commit/40fe23c540a8db7dcea335457dc8fe5d3328f529). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533662481 **[Test build #111086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111086/testReport)** for PR 22145 at commit [`e5cc9b9`](https://github.com/apache/spark/commit/e5cc9b9a36bb5365e3a75f28d51ee44b19f513e5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #25708: [SPARK-28141][SQL] Support special date values
MaxGekk commented on issue #25708: [SPARK-28141][SQL] Support special date values URL: https://github.com/apache/spark/pull/25708#issuecomment-533661742 @dongjoon-hyun @maropu @HyukjinKwon Could take a look at this PR which is similar to already merged changes for the timestamp type. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp edited a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
shaneknapp edited a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533656763 hmm. i added the `--r` flag to `make-distribution.sh` for the master k8s build and it passed: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s/376/ the jdk11 build passed as well with the `--r` flag: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s-jdk11/29/ i'm going to take a look and see if there's anything missing on the ubuntu worker that the k8s prb job failed on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660660 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111085/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533661109 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660652 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660596 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16170/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660836 argh sorry, forgot to run the k8s build on a different worker than the one that failed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field`
AmplabJenkins removed a comment on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field` URL: https://github.com/apache/spark/pull/25871#issuecomment-533660483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16169/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660588 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field`
AmplabJenkins removed a comment on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field` URL: https://github.com/apache/spark/pull/25871#issuecomment-533660477 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660660 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111085/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660596 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16170/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660652 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field`
AmplabJenkins commented on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field` URL: https://github.com/apache/spark/pull/25871#issuecomment-533660483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16169/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660588 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field`
AmplabJenkins commented on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field` URL: https://github.com/apache/spark/pull/25871#issuecomment-533660477 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533660012 **[Test build #111085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111085/testReport)** for PR 22145 at commit [`e5cc9b9`](https://github.com/apache/spark/commit/e5cc9b9a36bb5365e3a75f28d51ee44b19f513e5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field`
SparkQA commented on issue #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field` URL: https://github.com/apache/spark/pull/25871#issuecomment-533660017 **[Test build #111084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111084/testReport)** for PR 25871 at commit [`53e57ef`](https://github.com/apache/spark/commit/53e57efd6d314c7fc2dd856408dd9876a186c665). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk opened a new pull request #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field`
MaxGekk opened a new pull request #25871: [SPARK-29190][SQL] Optimize `extract`/`date_part` for the milliseconds `field` URL: https://github.com/apache/spark/pull/25871 ### What changes were proposed in this pull request? Changed the `DateTimeUtils.getMilliseconds()` by avoiding the decimal division, and replacing it by setting scale and precision while converting microseconds to the decimal type. ### Why are the changes needed? This improves performance of `extract` and `date_part()` by more than **50 times**: Before: ``` Invoke extract for timestamp: Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative Invoke extract for timestamp: Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative cast to timestamp 397428 45 25.2 39.7 1.0X MILLISECONDS of timestamp 36723 36761 63 0.33672.3 0.0X ``` After: ``` Invoke extract for timestamp: Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative cast to timestamp 278284 6 36.0 27.8 1.0X MILLISECONDS of timestamp 592606 13 16.9 59.2 0.5X ``` ### Does this PR introduce any user-facing change? No ### How was this patch tested? By existing test suite - `DateExpressionsSuite` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533658558 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111083/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533658558 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111083/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533658397 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533658551 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533658397 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533658587 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533658414 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16168/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533658414 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16168/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533658551 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
SparkQA commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533657610 **[Test build #111083 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111083/testReport)** for PR 22145 at commit [`e5cc9b9`](https://github.com/apache/spark/commit/e5cc9b9a36bb5365e3a75f28d51ee44b19f513e5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
AmplabJenkins removed a comment on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533643446 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16167/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes
shaneknapp commented on issue #22145: [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes URL: https://github.com/apache/spark/pull/22145#issuecomment-533657315 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
