[GitHub] [spark] MaxGekk edited a comment on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
MaxGekk edited a comment on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-573391098 And in `pushedFilters`, I store only filters that refer to the schema, and the filters that I can handle. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
MaxGekk edited a comment on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-573390974 @cloud-fan The problem was in the line https://github.com/apache/spark/pull/26973/commits/f0aa0a88bfa0c87007f8781ba7fac8f9cd3057ba#diff-faa3cfad03552057c3cb431c5ce87f03L52 where I returned an empty array (https://github.com/apache/spark/commit/f0aa0a88bfa0c87007f8781ba7fac8f9cd3057ba#diff-44a98c4a53980cb04e57f0489b257a37L141). The comment for pushFilters says that I should return filters that should be evaluated after scanning https://github.com/apache/spark/blob/053dd858d38e6107bc71e0aa3a4954291b74f8c8/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownFilters.java#L31 I guessed if I consumed all filters then I should return nothing. But looking at ORC and Parquet, I noticed that they return all filters: https://github.com/apache/spark/blob/5114389aef2cacaacc82e6025696b33d6d20b2a6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala#L65 https://github.com/apache/spark/blob/5114389aef2cacaacc82e6025696b33d6d20b2a6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcScanBuilder.scala#L65 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
MaxGekk commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-573391098 And in `pushedFilters`, I store only filters that refers to the schema, and the filters that I can handle. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
MaxGekk commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-573390974 @cloud-fan The problem was in the line https://github.com/apache/spark/pull/26973/commits/f0aa0a88bfa0c87007f8781ba7fac8f9cd3057ba#diff-faa3cfad03552057c3cb431c5ce87f03L52 where I returned an empty array. The comment for pushFilters says that I should return filters that should be evaluated after scanning https://github.com/apache/spark/blob/053dd858d38e6107bc71e0aa3a4954291b74f8c8/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownFilters.java#L31 I guessed if I consumed all filters then I should return nothing. But looking at ORC and Parquet, I noticed that they return all filters: https://github.com/apache/spark/blob/5114389aef2cacaacc82e6025696b33d6d20b2a6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala#L65 https://github.com/apache/spark/blob/5114389aef2cacaacc82e6025696b33d6d20b2a6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcScanBuilder.scala#L65 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on issue #27008: [SPARK-30353][SQL] Add IsNotNull check in SimplifyBinaryComparison optimization
ulysses-you commented on issue #27008: [SPARK-30353][SQL] Add IsNotNull check in SimplifyBinaryComparison optimization URL: https://github.com/apache/spark/pull/27008#issuecomment-573390408 @cloud-fan thanks for merging. Thanks all ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2
cloud-fan commented on a change in pull request #26231: [SPARK-29572][SQL] add
v1 read fallback API in DS v2
URL: https://github.com/apache/spark/pull/26231#discussion_r365563231
##
File path:
sql/core/src/main/java/org/apache/spark/sql/connector/read/V1Scan.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import org.apache.spark.annotation.Unstable;
+import org.apache.spark.sql.SQLContext;
+import org.apache.spark.sql.sources.BaseRelation;
+import org.apache.spark.sql.sources.TableScan;
+
+/**
+ * A trait that should be implemented by V1 DataSources that would like to
leverage the DataSource
+ * V2 read code paths.
+ *
+ * This interface is designed to provide Spark DataSources time to migrate to
DataSource V2 and
+ * will be removed in a future Spark release.
+ *
+ * @since 3.0.0
+ */
+@Unstable
+public interface V1Scan extends Scan {
+
+ /**
+ * Creates an `BaseRelation` with `TableScan` that can scan data from
DataSource v1 to RDD[Row].
+ *
+ * @since 3.0.0
+ */
+ T toV1TableScan(SQLContext context);
Review comment:
We haven't marked `SQLContext` as deprecated yet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2
cloud-fan commented on a change in pull request #26231: [SPARK-29572][SQL] add
v1 read fallback API in DS v2
URL: https://github.com/apache/spark/pull/26231#discussion_r365563206
##
File path:
sql/core/src/main/java/org/apache/spark/sql/connector/read/V1Scan.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import org.apache.spark.annotation.Unstable;
+import org.apache.spark.sql.SQLContext;
+import org.apache.spark.sql.sources.BaseRelation;
+import org.apache.spark.sql.sources.TableScan;
+
+/**
+ * A trait that should be implemented by V1 DataSources that would like to
leverage the DataSource
+ * V2 read code paths.
+ *
+ * This interface is designed to provide Spark DataSources time to migrate to
DataSource V2 and
+ * will be removed in a future Spark release.
+ *
+ * @since 3.0.0
+ */
+@Unstable
+public interface V1Scan extends Scan {
Review comment:
The idea is the same with v1 write fallback API. The v1 write fallback API
also relies on the v2 API to config the write. It's better to leverage the v2
infra as much as we can. e.g. we may improve the v2 pushdown to push more
operators that v1 doesn't support.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
MaxGekk edited a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573389742 The difference between the master and this PR is some tests could put more log event in log appender like https://github.com/apache/spark/pull/27166/files#diff-5a20f7138b31f3c344a4af7ab6f426daL45 but log appender should be removed at the end of the test. The problem could happen if a log appender is not removed and continue gather other log events. I will add a check for maximum number of logged events in `LogAppender`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
MaxGekk commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573389742 The difference between the master and this PR is some tests could put more log event in log appender like https://github.com/apache/spark/pull/27166/files#diff-5a20f7138b31f3c344a4af7ab6f426daL45 but log appender should be removed at the end of the test. The problem could happen if a log appender is not removed and continue gather other log events. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26921: [SPARK-30282][SQL] Integrate V2 commands with UnresolvedV2Relation into new resolution framework
cloud-fan commented on issue #26921: [SPARK-30282][SQL] Integrate V2 commands with UnresolvedV2Relation into new resolution framework URL: https://github.com/apache/spark/pull/26921#issuecomment-573389585 SGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25024: [SPARK-27296][SQL] Allows Aggregator to be registered as a UDF
cloud-fan commented on issue #25024: [SPARK-27296][SQL] Allows Aggregator to be registered as a UDF URL: https://github.com/apache/spark/pull/25024#issuecomment-573389525 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #25024: [SPARK-27296][SQL] Allows Aggregator to be registered as a UDF
cloud-fan closed pull request #25024: [SPARK-27296][SQL] Allows Aggregator to be registered as a UDF URL: https://github.com/apache/spark/pull/25024 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27151: [SPARK-26218][SQL][Follow up] Fix the corner case when casting float to Integer.
cloud-fan commented on issue #27151: [SPARK-26218][SQL][Follow up] Fix the corner case when casting float to Integer. URL: https://github.com/apache/spark/pull/27151#issuecomment-573389447 We are talking about SQL semantic not IEEE floating number definition. For pgsql ``` cloud0fan=# SELECT CAST(CAST(2147483648 as FLOAT) as Int); ERROR: integer out of range ``` I think the fix makes sense. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
MaxGekk commented on a change in pull request #27166:
[SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in
tests
URL: https://github.com/apache/spark/pull/27166#discussion_r365562130
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerLoggingSuite.scala
##
@@ -39,45 +36,25 @@ class OptimizerLoggingSuite extends PlanTest {
ColumnPruning) :: Nil
}
- class MockAppender extends AppenderSkeleton {
-val loggingEvents = new ArrayBuffer[LoggingEvent]()
-
-override def append(loggingEvent: LoggingEvent): Unit = {
- if (loggingEvent.getRenderedMessage().contains("Applying Rule") ||
-loggingEvent.getRenderedMessage().contains("Result of Batch") ||
-loggingEvent.getRenderedMessage().contains("has no effect")) {
-loggingEvents.append(loggingEvent)
- }
-}
-
-override def close(): Unit = {}
-override def requiresLayout(): Boolean = false
- }
-
- private def withLogLevelAndAppender(level: Level, appender: Appender)(f: =>
Unit): Unit = {
Review comment:
This is removed because the method is not used in the test suite.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27008: [SPARK-30353][SQL] Add IsNotNull check in SimplifyBinaryComparison optimization
cloud-fan closed pull request #27008: [SPARK-30353][SQL] Add IsNotNull check in SimplifyBinaryComparison optimization URL: https://github.com/apache/spark/pull/27008 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27008: [SPARK-30353][SQL] Add IsNotNull check in SimplifyBinaryComparison optimization
cloud-fan commented on issue #27008: [SPARK-30353][SQL] Add IsNotNull check in SimplifyBinaryComparison optimization URL: https://github.com/apache/spark/pull/27008#issuecomment-573388716 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories
AmplabJenkins commented on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/27130#issuecomment-573388646 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21334/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3
AmplabJenkins removed a comment on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3 URL: https://github.com/apache/spark/pull/27169#issuecomment-573388635 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21333/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories
AmplabJenkins commented on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/27130#issuecomment-573388643 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories
AmplabJenkins removed a comment on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/27130#issuecomment-573388643 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3
AmplabJenkins removed a comment on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3 URL: https://github.com/apache/spark/pull/27169#issuecomment-573388633 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories
AmplabJenkins removed a comment on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/27130#issuecomment-573388646 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21334/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3
AmplabJenkins commented on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3 URL: https://github.com/apache/spark/pull/27169#issuecomment-573388635 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21333/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3
AmplabJenkins commented on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3 URL: https://github.com/apache/spark/pull/27169#issuecomment-573388633 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
MaxGekk commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573388568 Not sure how it is related to my changes: ``` == ERROR: test_memory_limit (pyspark.tests.test_worker.WorkerMemoryTest) -- Traceback (most recent call last): File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/tests/test_worker.py", line 193, in test_memory_limit actual = rdd.map(lambda _: getrlimit()).collect() File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/rdd.py", line 889, in collect sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.10.8.1-src.zip/py4j/java_gateway.py", line 1286, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.10.8.1-src.zip/py4j/protocol.py", line 328, in get_return_value format(target_id, ".", name), value) Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, amp-jenkins-worker-05.amp, executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories
SparkQA commented on issue #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/27130#issuecomment-573388533 **[Test build #116548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116548/testReport)** for PR 27130 at commit [`2cf3e26`](https://github.com/apache/spark/commit/2cf3e26b410f94a78be6e12021c9dc7418548cc7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3
SparkQA commented on issue #27169: [SPARK-30485][SQL] Remove SQL configs deprecated in Spark 2.1 and 2.3 URL: https://github.com/apache/spark/pull/27169#issuecomment-573388529 **[Test build #116547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116547/testReport)** for PR 27169 at commit [`ff26d47`](https://github.com/apache/spark/commit/ff26d477e7dada74bbffc60ec174346b84de910e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kevinyu98 commented on a change in pull request #27130: [SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories
kevinyu98 commented on a change in pull request #27130:
[SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with
subdirectories
URL: https://github.com/apache/spark/pull/27130#discussion_r365561742
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
##
@@ -170,4 +170,156 @@ class HiveOrcSourceSuite extends OrcSuite with
TestHiveSingleton {
test("SPARK-11412 read and merge orc schemas in parallel") {
testMergeSchemasInParallel(OrcFileOperator.readOrcSchemasInParallel)
}
+
+ test("SPARK-25993 CREATE EXTERNAL TABLE with subdirectories") {
+Seq(true, false).foreach { convertMetastore =>
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> s"$convertMetastore")
{
+withTempDir { dir =>
+ try {
+sql("USE default")
+sql(
+ """
+|CREATE EXTERNAL TABLE hive_orc(
Review comment:
@dongjoon-hyun Thanks for pointing out this. I was using other test cases
without thinking too much. I have changed the name. I also replaced the
hiveClient.runSqlHive for the insert stmt.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #22173: [SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests.
cloud-fan commented on issue #22173: [SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests. URL: https://github.com/apache/spark/pull/22173#issuecomment-573387824 Sounds good to me. We can explore different solutions after 3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] emaynardigs edited a comment on issue #27155: [SPARK-17636][SPARK-25557][SQL] Parquet and ORC predicate pushdown in nested fields
emaynardigs edited a comment on issue #27155: [SPARK-17636][SPARK-25557][SQL] Parquet and ORC predicate pushdown in nested fields URL: https://github.com/apache/spark/pull/27155#issuecomment-573387721 Yes, surely we should close one PR. The other one is inactive, fails tests, and doesn’t merge cleanly. This one has none of those issues and has more functionality. I don’t mind closing this one out if the other PR can get us to the same place just as quickly, but that seems like it would take more work at this point? Either way, let’s make sure there is an active PR for this issue which can be merged in. As I have no control over any other PR this is my submission towards that end. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
cloud-fan commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-573387725 How did you fix https://github.com/apache/spark/pull/27112/files#r365539740 ? I took a quick look and not sure what was wrong before. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] emaynardigs commented on issue #27155: [SPARK-17636][SPARK-25557][SQL] Parquet and ORC predicate pushdown in nested fields
emaynardigs commented on issue #27155: [SPARK-17636][SPARK-25557][SQL] Parquet and ORC predicate pushdown in nested fields URL: https://github.com/apache/spark/pull/27155#issuecomment-573387721 Yes, surely we should close one PR. The other one is inactive, fails, tests, and doesn’t merge cleanly. This one has none of those issues and has more functionality. I don’t mind closing this one out if the other PR can get us to the same place just as quickly, but that seems like it would take more work at this point? Either way, let’s make sure there is an active PR for this issue which can be merged in. As I have no control over any other PR this is my submission towards that end. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
MaxGekk commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-573387323 @HyukjinKwon @dongjoon-hyun @cloud-fan Could you review this PR, please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size
AmplabJenkins commented on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size URL: https://github.com/apache/spark/pull/26434#issuecomment-573386516 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21332/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size
AmplabJenkins commented on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size URL: https://github.com/apache/spark/pull/26434#issuecomment-573386514 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size
AmplabJenkins removed a comment on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size URL: https://github.com/apache/spark/pull/26434#issuecomment-573386516 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21332/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size
AmplabJenkins removed a comment on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size URL: https://github.com/apache/spark/pull/26434#issuecomment-573386514 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size
SparkQA commented on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size URL: https://github.com/apache/spark/pull/26434#issuecomment-573386433 **[Test build #116546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116546/testReport)** for PR 26434 at commit [`cee1c8c`](https://github.com/apache/spark/commit/cee1c8cb7b4c4714dd3d59fbaa693d1dab217767). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size
cloud-fan commented on issue #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size URL: https://github.com/apache/spark/pull/26434#issuecomment-573386247 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JkSelf opened a new pull request #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size
JkSelf opened a new pull request #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size URL: https://github.com/apache/spark/pull/26434 ### What changes were proposed in this pull request? This PR implement a rule `OptimizeSkewedPartitions` to optimize the skewed partition based on the partition data size. ### Why are the changes needed? To optimize the skewed partition in runtime based on AQE ### Does this PR introduce any user-facing change? No ### How was this patch tested? UT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size
cloud-fan closed pull request #26434: [SPARK-29544] [SQL] optimize skewed partition based on data size URL: https://github.com/apache/spark/pull/26434 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter URL: https://github.com/apache/spark/pull/27041#issuecomment-573385211 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116541/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter URL: https://github.com/apache/spark/pull/27041#issuecomment-573385210 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
SparkQA removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter URL: https://github.com/apache/spark/pull/27041#issuecomment-573377825 **[Test build #116541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116541/testReport)** for PR 27041 at commit [`6969ec2`](https://github.com/apache/spark/commit/6969ec290fc6981aeb39a3eeb2f0d8d878ef4b9e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter URL: https://github.com/apache/spark/pull/27041#issuecomment-573385211 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116541/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
AmplabJenkins commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter URL: https://github.com/apache/spark/pull/27041#issuecomment-573385210 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
SparkQA commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter URL: https://github.com/apache/spark/pull/27041#issuecomment-573385177 **[Test build #116541 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116541/testReport)** for PR 27041 at commit [`6969ec2`](https://github.com/apache/spark/commit/6969ec290fc6981aeb39a3eeb2f0d8d878ef4b9e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
AmplabJenkins removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573384519 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116543/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
AmplabJenkins removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573384517 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
AmplabJenkins commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573384517 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
SparkQA removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573379391 **[Test build #116543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116543/testReport)** for PR 27164 at commit [`5f37b64`](https://github.com/apache/spark/commit/5f37b64010bf669d1426db53e9e4c35770cf37b4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
AmplabJenkins commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573384519 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116543/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
AmplabJenkins removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573379469 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21329/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
SparkQA commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573384476 **[Test build #116543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116543/testReport)** for PR 27164 at commit [`5f37b64`](https://github.com/apache/spark/commit/5f37b64010bf669d1426db53e9e4c35770cf37b4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
AmplabJenkins removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573384282 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116544/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
AmplabJenkins commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573384282 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116544/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
AmplabJenkins removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573384217 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116542/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
AmplabJenkins removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573384279 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
AmplabJenkins commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573384279 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
AmplabJenkins removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573384213 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
SparkQA removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573379392 **[Test build #116544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116544/testReport)** for PR 27160 at commit [`f7d30f5`](https://github.com/apache/spark/commit/f7d30f5c3bd1b00f19de659c55b0b4ad8aa809ff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
SparkQA commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573384221 **[Test build #116544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116544/testReport)** for PR 27160 at commit [`f7d30f5`](https://github.com/apache/spark/commit/f7d30f5c3bd1b00f19de659c55b0b4ad8aa809ff). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
SparkQA removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573379393 **[Test build #116542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116542/testReport)** for PR 27166 at commit [`5ede347`](https://github.com/apache/spark/commit/5ede347f093e5e1b606f448b1c01578066bce75a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
AmplabJenkins commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573384217 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116542/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
AmplabJenkins commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573384213 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
SparkQA commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573384153 **[Test build #116542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116542/testReport)** for PR 27166 at commit [`5ede347`](https://github.com/apache/spark/commit/5ede347f093e5e1b606f448b1c01578066bce75a). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer
AmplabJenkins commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer URL: https://github.com/apache/spark/pull/27077#issuecomment-573381207 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer
AmplabJenkins removed a comment on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer URL: https://github.com/apache/spark/pull/27077#issuecomment-573381208 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21331/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer
AmplabJenkins removed a comment on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer URL: https://github.com/apache/spark/pull/27077#issuecomment-573381207 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer
AmplabJenkins commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer URL: https://github.com/apache/spark/pull/27077#issuecomment-573381208 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21331/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer
SparkQA commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer URL: https://github.com/apache/spark/pull/27077#issuecomment-573381164 **[Test build #116545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116545/testReport)** for PR 27077 at commit [`9d8b6ce`](https://github.com/apache/spark/commit/9d8b6cef5b5de527d6a3ffcf9d8de0f8ff99c407). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer
dongjoon-hyun commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer URL: https://github.com/apache/spark/pull/27077#issuecomment-573381092 cc @gatorsmile and @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer
dongjoon-hyun commented on a change in pull request #27077: [SPARK-30408][SQL]
Should not remove orderBy in sortBy clause in Optimizer
URL: https://github.com/apache/spark/pull/27077#discussion_r365556941
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateSortsSuite.scala
##
@@ -308,4 +308,27 @@ class EliminateSortsSuite extends PlanTest {
val correctAnswer = PushDownOptimizer.execute(noOrderByPlan.analyze)
comparePlans(optimized, correctAnswer)
}
+
+ test("should not remove orderBy in sortBy clause") {
Review comment:
@apapi . Please change like the following.
```scala
- test("should not remove orderBy in sortBy clause") {
+ test("SPARK-30408 should not remove orderBy in sortBy clause") {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer
dongjoon-hyun commented on a change in pull request #27077: [SPARK-30408][SQL]
Should not remove orderBy in sortBy clause in Optimizer
URL: https://github.com/apache/spark/pull/27077#discussion_r365556941
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateSortsSuite.scala
##
@@ -308,4 +308,27 @@ class EliminateSortsSuite extends PlanTest {
val correctAnswer = PushDownOptimizer.execute(noOrderByPlan.analyze)
comparePlans(optimized, correctAnswer)
}
+
+ test("should not remove orderBy in sortBy clause") {
Review comment:
Hi, @apapi . Please change like the following.
```scala
- test("should not remove orderBy in sortBy clause") {
+ test("SPARK-30408 should not remove orderBy in sortBy clause") {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer
dongjoon-hyun commented on issue #27077: [SPARK-30408][SQL] Should not remove orderBy in sortBy clause in Optimizer URL: https://github.com/apache/spark/pull/27077#issuecomment-573380891 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sddyljsx commented on issue #27135: [SPARK-30458][WEBUI] Fix Wrong Executor Computing Time in Time Line of Stage Page
sddyljsx commented on issue #27135: [SPARK-30458][WEBUI] Fix Wrong Executor Computing Time in Time Line of Stage Page URL: https://github.com/apache/spark/pull/27135#issuecomment-573380355 Thanks for review! @srowen @dongjoon-hyun @lw-lin @HyukjinKwon @gengliangwang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sddyljsx commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
sddyljsx commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573380206 Thanks! @srowen @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27135: [SPARK-30458][WEBUI] Fix Wrong Executor Computing Time in Time Line of Stage Page
dongjoon-hyun closed pull request #27135: [SPARK-30458][WEBUI] Fix Wrong Executor Computing Time in Time Line of Stage Page URL: https://github.com/apache/spark/pull/27135 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
AmplabJenkins removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573379465 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21330/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
AmplabJenkins removed a comment on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573379467 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
AmplabJenkins commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573379469 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21329/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
AmplabJenkins removed a comment on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573379464 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
dongjoon-hyun closed pull request #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
AmplabJenkins commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573379467 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
AmplabJenkins commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573379464 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
AmplabJenkins commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573379465 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21330/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
SparkQA commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573379393 **[Test build #116542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116542/testReport)** for PR 27166 at commit [`5ede347`](https://github.com/apache/spark/commit/5ede347f093e5e1b606f448b1c01578066bce75a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
SparkQA commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573379391 **[Test build #116543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116543/testReport)** for PR 27164 at commit [`5f37b64`](https://github.com/apache/spark/commit/5f37b64010bf669d1426db53e9e4c35770cf37b4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation
SparkQA commented on issue #27160: [SPARK-30478][CORE][DOCS] Fix Memory Package documentation URL: https://github.com/apache/spark/pull/27160#issuecomment-573379392 **[Test build #116544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116544/testReport)** for PR 27160 at commit [`f7d30f5`](https://github.com/apache/spark/commit/f7d30f5c3bd1b00f19de659c55b0b4ad8aa809ff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27160: [SPARK-30478][Core]Update the Doc of Memory Package
dongjoon-hyun commented on issue #27160: [SPARK-30478][Core]Update the Doc of Memory Package URL: https://github.com/apache/spark/pull/27160#issuecomment-573379133 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
AmplabJenkins removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573379079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21328/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
AmplabJenkins commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573379079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21328/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events
dongjoon-hyun commented on issue #27164: [SPARK-30479][SQL] Apply compaction of event log to SQL events URL: https://github.com/apache/spark/pull/27164#issuecomment-573379086 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
AmplabJenkins commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573379077 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
AmplabJenkins removed a comment on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573379077 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests
dongjoon-hyun commented on issue #27166: [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests URL: https://github.com/apache/spark/pull/27166#issuecomment-573379013 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27155: [SPARK-17636][SPARK-25557][SQL] Parquet and ORC predicate pushdown in nested fields
dongjoon-hyun commented on issue #27155: [SPARK-17636][SPARK-25557][SQL] Parquet and ORC predicate pushdown in nested fields URL: https://github.com/apache/spark/pull/27155#issuecomment-573378599 I'll leave this PR to @dbtsai . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27155: [SPARK-17636][SPARK-25557][SQL] Parquet and ORC predicate pushdown in nested fields
dongjoon-hyun commented on issue #27155: [SPARK-17636][SPARK-25557][SQL] Parquet and ORC predicate pushdown in nested fields URL: https://github.com/apache/spark/pull/27155#issuecomment-573378544 @emaynardigs . In this case, usually, we close the second PR (yours) because the original is still alive. You can retry this after the first one is closed. > Firstly, I should say that there are actually no commits here under the original author's ownership; the code has diverged to the point now where, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
dongjoon-hyun commented on a change in pull request #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter URL: https://github.com/apache/spark/pull/27041#discussion_r36097 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala ## @@ -177,9 +177,14 @@ private[hive] class HiveClientImpl( // has hive-site.xml. So, HiveConf will use that to override its default values. // 2: we set all spark confs to this hiveConf. // 3: we set all entries in config to this hiveConf. + +// not to loose command line overwritten properties Review comment: According to the context, do you mean `lose` instead of `loose`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter
AmplabJenkins removed a comment on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter URL: https://github.com/apache/spark/pull/27041#issuecomment-573377898 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
