[GitHub] [spark] SparkQA removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
SparkQA removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521857476 **[Test build #109170 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109170/testReport)** for PR 24903 at commit [`de5f671`](https://github.com/apache/spark/commit/de5f671dcaead11837ad027614cb2aa75315abb5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession
AmplabJenkins commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#issuecomment-521858429 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521858266 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109170/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521858257 **[Test build #109170 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109170/testReport)** for PR 24903 at commit [`de5f671`](https://github.com/apache/spark/commit/de5f671dcaead11837ad027614cb2aa75315abb5). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25436: [WIP]Support minPartitions for batch and Kafka source v1
AmplabJenkins removed a comment on issue #25436: [WIP]Support minPartitions for batch and Kafka source v1 URL: https://github.com/apache/spark/pull/25436#issuecomment-521858098 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521858263 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu opened a new pull request #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative execu
AngersZh opened a new pull request #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive before retry #24533 URL: https://github.com/apache/spark/pull/25469 ### What changes were proposed in this pull request? In pr #[24533](https://github.com/apache/spark/pull/24533/files) , it prevent retry to an removed Executor. In my test, I can't catch exceptions from ` new OneForOneBlockFetcher(client, appId, execId, blockIds, listener, transportConf, tempFileManager).start()` And I check the code carefully， method **start()** will handle exception of IOException in it's retry logical, won't throw it out. until it meet maxRetry times or meet exception that is not IOException. And if we meet the situation that when we fetch block , the executor is dead, when we rerun `RetryingBlockFetcher.BlockFetchStarter.createAndStart()` we may failed when we create a transport client to dead executor. it will throw a IOException. We should catch this IOException. ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25436: [WIP]Support minPartitions for batch and Kafka source v1
SparkQA removed a comment on issue #25436: [WIP]Support minPartitions for batch and Kafka source v1 URL: https://github.com/apache/spark/pull/25436#issuecomment-521857487 **[Test build #109163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109163/testReport)** for PR 25436 at commit [`d340d7d`](https://github.com/apache/spark/commit/d340d7d83504a847d9bb8caa5addd4e71a5af6bd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-521858440 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521858263 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive
SparkQA commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive before retry #24533 URL: https://github.com/apache/spark/pull/25469#issuecomment-521862739 **[Test build #109174 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109174/testReport)** for PR 25469 at commit [`e2dbe4b`](https://github.com/apache/spark/commit/e2dbe4bca387542e2043abc0801190531e805684). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25467: [SPARK-28634][yarn] Ignore kerberos login config in client mode AM.
AmplabJenkins removed a comment on issue #25467: [SPARK-28634][yarn] Ignore kerberos login config in client mode AM. URL: https://github.com/apache/spark/pull/25467#issuecomment-521862546 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521862762 **[Test build #109176 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109176/testReport)** for PR 24903 at commit [`c503f5d`](https://github.com/apache/spark/commit/c503f5d419323a16442f313857b030d49f28995b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML
SparkQA commented on issue #25383: [SPARK-13677][ML] Implement Tree-Based Feature Transformation for ML URL: https://github.com/apache/spark/pull/25383#issuecomment-521862787 **[Test build #109175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109175/testReport)** for PR 25383 at commit [`092e115`](https://github.com/apache/spark/commit/092e115e9fd344abc0031635ce79830789b5444b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25467: [SPARK-28634][yarn] Ignore kerberos login config in client mode AM.
AmplabJenkins removed a comment on issue #25467: [SPARK-28634][yarn] Ignore kerberos login config in client mode AM. URL: https://github.com/apache/spark/pull/25467#issuecomment-521862551 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109153/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25467: [SPARK-28634][yarn] Ignore kerberos login config in client mode AM.
SparkQA removed a comment on issue #25467: [SPARK-28634][yarn] Ignore kerberos login config in client mode AM. URL: https://github.com/apache/spark/pull/25467#issuecomment-521857551 **[Test build #109153 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109153/testReport)** for PR 25467 at commit [`1b11e43`](https://github.com/apache/spark/commit/1b11e43ece224efcc5cb6742c20ae565b5297fe2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
dilipbiswal commented on a change in pull request #24759:
[SPARK-27395][SQL][WIP] Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314567385
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
##
@@ -179,6 +182,22 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
extends TreeNode[PlanT
override def verboseString(maxFields: Int): String = simpleString(maxFields)
+ override def simpleStringWithNodeId(): String = {
+val tag = new TreeNodeTag[Int]("operatorId")
Review comment:
@cloud-fan Thanks.. will do.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP] Update the TODO with actual blocking JIRA IDs
HyukjinKwon edited a comment on issue #25415: [SPARK-28390][SQL][PYTHON][TESTS] [FOLLOW-UP] Update the TODO with actual blocking JIRA IDs URL: https://github.com/apache/spark/pull/25415#issuecomment-521426956 Yea that's fine to update comments. But @shivusondur can you confirm if you are unable to fix any test or some tests to have use in GROUP BY clause due to both JIRAs? If you can, let's add some tests and only comment out the other tests not working by both JIRAs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
cloud-fan commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715#issuecomment-521868936 let's discuss the perf problem later, and focus on the bug fix first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] hddong commented on a change in pull request #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples
hddong commented on a change in pull request #25412: [SPARK-28691][EXAMPLES]
Add Java/Scala DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#discussion_r314570912
##
File path:
examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKerberizedKafkaWordCount.java
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import scala.Tuple2;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.streaming.api.java.*;
+import org.apache.spark.streaming.kafka010.ConsumerStrategies;
+import org.apache.spark.streaming.kafka010.KafkaUtils;
+import org.apache.spark.streaming.kafka010.LocationStrategies;
+import org.apache.spark.streaming.Durations;
+
+/**
+ * Consumes messages from one or more topics in Kafka and does wordcount.
+ * Usage: JavaDirectKerberizedKafkaWordCount
+ *is a list of one or more Kafka brokers
+ *is a consumer group name to consume from topics
+ *is a list of one or more kafka topics to consume from
+ *
+ * Example:
+ *$ bin/run-example --files ${path}/kafka_jaas.conf \
+ * --driver-java-options
"-Djava.security.auth.login.config=${path}/kafka_jaas.conf" \
+ * --conf \
+ *
"spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_jaas.conf"
\
+ * streaming.JavaDirectKerberizedKafkaWordCount
broker1-host:port,broker2-host:port \
+ * consumer-group topic1,topic2
+ *
+ * kafka_jaas.conf can manually create, template as:
+ * KafkaClient {
+ * com.sun.security.auth.module.Krb5LoginModule required
+ * keyTab="${path_of_keytab}/kafka.service.keytab"
Review comment:
@gaborgsomogyi you are right, in my environment, keytab in same path of each
node. I think we can two use jaas files:
for driver `keyTab="${path_of_keytab}/kafka.service.keytab"`
for executor `keyTab="./kafka.service.keytab"`
and run-example as:
`bin/run-example --files`
`${path}/kafka_executor_jaas.conf,${kyetab_path}/kafka.service.keytab \`
` --driver-java-options
"-Djava.security.auth.login.config=${path}/kafka_driver_jaas.conf" \`
` --conf \`
`"spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_executor_jaas.conf"
\`
`streaming.JavaDirectKerberizedKafkaWordCount
broker1-host:port,broker2-host:port \`
` consumer-group topic1,topic2`
That will become more general. Do you have any succinct advice?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
AmplabJenkins removed a comment on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table URL: https://github.com/apache/spark/pull/25306#issuecomment-521870725 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table URL: https://github.com/apache/spark/pull/25306#issuecomment-521870725 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
AmplabJenkins removed a comment on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table URL: https://github.com/apache/spark/pull/25306#issuecomment-521870731 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14250/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table URL: https://github.com/apache/spark/pull/25306#issuecomment-521870731 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14250/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
cloud-fan commented on a change in pull request #24759: [SPARK-27395][SQL][WIP]
Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314572019
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##
@@ -127,10 +131,14 @@ class QueryExecution(
ReuseExchange(sparkSession.sessionState.conf),
ReuseSubquery(sparkSession.sessionState.conf))
- def simpleString: String = withRedaction {
+ def simpleString[T <: QueryPlan[T]]: String = withRedaction {
Review comment:
why do we need type parameter here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maryannxue opened a new pull request #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
maryannxue opened a new pull request #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE URL: https://github.com/apache/spark/pull/25471 ### What changes were proposed in this pull request? This PR changes subquery reuse in Adaptive Query Execution from compile-time static reuse to execution-time dynamic reuse. This PR adds a `ReuseAdaptiveSubquery` rule that applies to a query stage after it is created and before it is executed. The new dynamic reuse enables subqueries to be reused across all different subquery levels. ### Why are the changes needed? This is an improvement to the current subquery reuse in Adaptive Query Execution, which allows subquery reuse to happen in a lazy fashion as well as at different subquery levels. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Passed existing tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
dilipbiswal commented on a change in pull request #24759:
[SPARK-27395][SQL][WIP] Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314576009
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##
@@ -526,9 +531,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
extends Product {
* @param maxFields Maximum number of fields that will be converted to
strings.
* Any elements beyond the limit will be dropped.
*/
- def simpleString(maxFields: Int): String = {
-s"$nodeName ${argString(maxFields)}".trim
- }
+ def simpleString(maxFields: Int): String = s"$nodeName
${argString(maxFields)}".trim
+
+ def simpleStringWithNodeId(): String = simpleString(0)
Review comment:
@cloud-fan Good idea Wenchen. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
AmplabJenkins removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE URL: https://github.com/apache/spark/pull/25471#issuecomment-521875119 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
AmplabJenkins removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE URL: https://github.com/apache/spark/pull/25471#issuecomment-521875122 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14252/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
AmplabJenkins commented on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE URL: https://github.com/apache/spark/pull/25471#issuecomment-521875122 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14252/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
AmplabJenkins commented on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE URL: https://github.com/apache/spark/pull/25471#issuecomment-521875119 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available
shahidki31 commented on a change in pull request #25460:
[SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not
available
URL: https://github.com/apache/spark/pull/25460#discussion_r314577254
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala
##
@@ -72,7 +72,8 @@ case class HadoopFsRelation(
val compressionFactor = sqlContext.conf.fileCompressionFactor
val defaultSize = (location.sizeInBytes * compressionFactor).toLong
location match {
- case cfi: CatalogFileIndex if
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled =>
+ case cfi: CatalogFileIndex if
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled
+&& defaultSize == sqlContext.conf.defaultSizeInBytes =>
Review comment:
If the table statistics is available here,
https://github.com/apache/spark/blob/0ea8db9fd3d882140d8fa305dd69fc94db62cf8f/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L387-L390,
Then it should be available here too right?
https://github.com/apache/spark/blob/0ea8db9fd3d882140d8fa305dd69fc94db62cf8f/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala#L43-L45
So, ideally the flow shouldn't come here if the table statistics already
exists. That is why even after #24715, `location.sizeInBytes` is 8.0EB
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maryannxue commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution
maryannxue commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution URL: https://github.com/apache/spark/pull/25456#issuecomment-521877104 cc @cloud-fan @hvanhovell @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
AmplabJenkins removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE URL: https://github.com/apache/spark/pull/25471#issuecomment-521877944 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109181/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521880464 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109176/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
SparkQA removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521862762 **[Test build #109176 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109176/testReport)** for PR 24903 at commit [`c503f5d`](https://github.com/apache/spark/commit/c503f5d419323a16442f313857b030d49f28995b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521880460 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive
SparkQA commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive before retry #24533 URL: https://github.com/apache/spark/pull/25469#issuecomment-521880860 **[Test build #109174 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109174/testReport)** for PR 25469 at commit [`e2dbe4b`](https://github.com/apache/spark/commit/e2dbe4bca387542e2043abc0801190531e805684). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host
AmplabJenkins commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521882108 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host
AmplabJenkins commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521882113 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109169/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3
maropu commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521884890 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521884974 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109164/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3
AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521884680 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
SparkQA commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-521885024 **[Test build #109168 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109168/testReport)** for PR 25368 at commit [`45cbbd0`](https://github.com/apache/spark/commit/45cbbd04408251e14a9157d1a5b93ae6a8e91401). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521884969 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3
AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521884684 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109161/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
SparkQA removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521857496 **[Test build #109164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109164/testReport)** for PR 25423 at commit [`4d99b96`](https://github.com/apache/spark/commit/4d99b961e0f1f8ad983c041e8b4180cf94bfbba5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521884969 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-521885161 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-521885162 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109168/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3
SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521885021 **[Test build #109182 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109182/testReport)** for PR 25451 at commit [`f6c2f4a`](https://github.com/apache/spark/commit/f6c2f4acb6f9eb57eae23190b2be93002e1c0f6a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-521845038 Let's see how it goes. thanks guys! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] William1104 commented on issue #24912: [WIP][SPARK-28103][SQL] Fix constraints of a Union Logical Plan
William1104 commented on issue #24912: [WIP][SPARK-28103][SQL] Fix constraints of a Union Logical Plan URL: https://github.com/apache/spark/pull/24912#issuecomment-521844926 > @William1104 Is this still work in progress? Hi @venkata91, I am sorry that I have to put this work on hold at this moment. I hope I will work on it again some time later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-521844986 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available
maropu commented on a change in pull request #25460:
[SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not
available
URL: https://github.com/apache/spark/pull/25460#discussion_r314552291
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala
##
@@ -72,7 +72,8 @@ case class HadoopFsRelation(
val compressionFactor = sqlContext.conf.fileCompressionFactor
val defaultSize = (location.sizeInBytes * compressionFactor).toLong
location match {
- case cfi: CatalogFileIndex if
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled =>
+ case cfi: CatalogFileIndex if
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled
+&& defaultSize == sqlContext.conf.defaultSizeInBytes =>
Review comment:
btw, not `defaultSize == sqlContext.conf.defaultSizeInBytes` but
`location.sizeInBytes == sqlContext.conf.defaultSizeInBytes`?
See: https://github.com/apache/spark/pull/22502#discussion_r313938009
But, does this comparison work well even when
`sqlContext.conf.defaultSizeInBytes` changed by users?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available
maropu commented on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460#issuecomment-521847778 I feel ok to do so (I think we should fix the existing issue on master/2.4/2.3...) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries
maropu commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries URL: https://github.com/apache/spark/pull/25464#issuecomment-521852813 @gatorsmile @maryannxue We need this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries
AmplabJenkins commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries URL: https://github.com/apache/spark/pull/25464#issuecomment-521855487 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is
AmplabJenkins commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive before retry #24533 URL: https://github.com/apache/spark/pull/25469#issuecomment-521859543 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs
AmplabJenkins commented on issue #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs URL: https://github.com/apache/spark/pull/25465#issuecomment-521859617 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs
SparkQA commented on issue #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs URL: https://github.com/apache/spark/pull/25465#issuecomment-521859610 **[Test build #109154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109154/testReport)** for PR 25465 at commit [`648264e`](https://github.com/apache/spark/commit/648264e666f852eead6d148a66e477a0e8c63f2e). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521861460 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14233/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521861457 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25470: [SPARK-28751][Core] Imporve java serializer deserialization performance
AmplabJenkins removed a comment on issue #25470: [SPARK-28751][Core] Imporve java serializer deserialization performance URL: https://github.com/apache/spark/pull/25470#issuecomment-521861003 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is ali
cloud-fan commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive before retry #24533 URL: https://github.com/apache/spark/pull/25469#issuecomment-521861687 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
SparkQA commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521863526 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14243/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.
HyukjinKwon edited a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-521869683 cc @dongjoon-hyun, @cloud-fan and @gatorsmile as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
cloud-fan commented on a change in pull request #24759: [SPARK-27395][SQL][WIP]
Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314571122
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##
@@ -526,9 +531,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
extends Product {
* @param maxFields Maximum number of fields that will be converted to
strings.
* Any elements beyond the limit will be dropped.
*/
- def simpleString(maxFields: Int): String = {
-s"$nodeName ${argString(maxFields)}".trim
- }
+ def simpleString(maxFields: Int): String = s"$nodeName
${argString(maxFields)}".trim
+
+ def simpleStringWithNodeId(): String = simpleString(0)
Review comment:
I'd rather leave it abstract, to make sure it's implemented by `QueryPlan`
and we implement `Expression.simpleStringWithNodeId` by throwing exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-521869595 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14249/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ConeyLiu commented on a change in pull request #25470: [SPARK-28751][Core] Imporve java serializer deserialization performance
ConeyLiu commented on a change in pull request #25470: [SPARK-28751][Core]
Imporve java serializer deserialization performance
URL: https://github.com/apache/spark/pull/25470#discussion_r314571103
##
File path:
core/src/test/scala/org/apache/spark/serializer/JavaSerializerBenchmark.scala
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.serializer
+
+import java.nio.ByteBuffer
+
+import scala.reflect.ClassTag
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
+import org.apache.spark.internal.config._
+
+/**
+ * Benchmark for Java Serializer Deserialization use vs not use Class Resolve
Cache.
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt:
+ * bin/spark-submit --class --jars
+ * 2. build/sbt "core/test:runMain "
+ * 3. generate result:
+ * SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "core/test:runMain "
+ * Results will be written to
"benchmarks/JavaSerializerBenchmark-results.txt".
+ * }}}
+ */
+object JavaSerializerBenchmark extends BenchmarkBase {
+
+ val N = 10
+ override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
+val name = "Benchmark Java Serializer Deserialization use vs not use Class
Resolve Cache"
+runBenchmark(name) {
+ val benchmark = new Benchmark(name, N, 10, output = output)
+ Seq(true, false).foreach(useUnsafe => run(useUnsafe, benchmark))
+ benchmark.run()
+}
+ }
+
+ private def run(useCache: Boolean, benchmark: Benchmark): Unit = {
+def createCase[T: ClassTag](name: String, size: Int, gen: () => T): Unit =
{
+ lazy val ser = createSerializer(useCache)
+ val data: Array[ByteBuffer] = Array.fill(size)(ser.serialize(gen()))
+
+ benchmark.addCase(s"$name with cache:$useCache") { _ =>
+var i = 0
+var s: ByteBuffer = null
+while (i < size) {
+ s = data(i)
+ s.rewind()
+ ser.deserialize(s)
Review comment:
The `JavaSerializer` can provide threadsafe. I think that's why we use it in
closure, RPC, and others. To reuse `InputStream` may need to do some threadsafe
protection, such as `ThreadLocal`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] hddong commented on a change in pull request #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples
hddong commented on a change in pull request #25412: [SPARK-28691][EXAMPLES]
Add Java/Scala DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#discussion_r314570912
##
File path:
examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKerberizedKafkaWordCount.java
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import scala.Tuple2;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.streaming.api.java.*;
+import org.apache.spark.streaming.kafka010.ConsumerStrategies;
+import org.apache.spark.streaming.kafka010.KafkaUtils;
+import org.apache.spark.streaming.kafka010.LocationStrategies;
+import org.apache.spark.streaming.Durations;
+
+/**
+ * Consumes messages from one or more topics in Kafka and does wordcount.
+ * Usage: JavaDirectKerberizedKafkaWordCount
+ *is a list of one or more Kafka brokers
+ *is a consumer group name to consume from topics
+ *is a list of one or more kafka topics to consume from
+ *
+ * Example:
+ *$ bin/run-example --files ${path}/kafka_jaas.conf \
+ * --driver-java-options
"-Djava.security.auth.login.config=${path}/kafka_jaas.conf" \
+ * --conf \
+ *
"spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_jaas.conf"
\
+ * streaming.JavaDirectKerberizedKafkaWordCount
broker1-host:port,broker2-host:port \
+ * consumer-group topic1,topic2
+ *
+ * kafka_jaas.conf can manually create, template as:
+ * KafkaClient {
+ * com.sun.security.auth.module.Krb5LoginModule required
+ * keyTab="${path_of_keytab}/kafka.service.keytab"
Review comment:
@gaborgsomogyi you are right, in my environment, keytab in same path of each
node. I think we can two jaas file:
for driver `keyTab="${path_of_keytab}/kafka.service.keytab"`
for executor `keyTab="./kafka.service.keytab"`
and run-example as:
`bin/run-example --files`
`${path}/kafka_executor_jaas.conf,${kyetab_path}/kafka.service.keytab \`
` --driver-java-options
"-Djava.security.auth.login.config=${path}/kafka_driver_jaas.conf" \`
` --conf \`
`"spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_executor_jaas.conf"
\`
`streaming.JavaDirectKerberizedKafkaWordCount
broker1-host:port,broker2-host:port \`
` consumer-group topic1,topic2`
Do you have any succinct advice?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.
HyukjinKwon commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-521869683 cc @dongjoon-hyun and @cloud-fan as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
SparkQA commented on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE URL: https://github.com/apache/spark/pull/25471#issuecomment-521874340 **[Test build #109181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109181/testReport)** for PR 25471 at commit [`8ec528b`](https://github.com/apache/spark/commit/8ec528be4cbd2b109c45d2cf9e2e06bc339e743b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
dilipbiswal commented on a change in pull request #24759:
[SPARK-27395][SQL][WIP] Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314576051
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##
@@ -539,24 +544,27 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
extends Product {
override def toString: String = treeString
/** Returns a string representation of the nodes in this tree */
+ // final def treeString: String = treeString(verbose = true)
+
final def treeString: String = treeString(verbose = true)
final def treeString(
- verbose: Boolean,
- addSuffix: Boolean = false,
- maxFields: Int = SQLConf.get.maxToStringFields): String = {
+verbose: Boolean,
Review comment:
@cloud-fan Will fix.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
AmplabJenkins removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE URL: https://github.com/apache/spark/pull/25471#issuecomment-521877942 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
SparkQA removed a comment on issue #25471: [SPARK-28753][SQL] Dynamically reuse subqueries in AQE URL: https://github.com/apache/spark/pull/25471#issuecomment-521874340 **[Test build #109181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109181/testReport)** for PR 25471 at commit [`8ec528b`](https://github.com/apache/spark/commit/8ec528be4cbd2b109c45d2cf9e2e06bc339e743b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521880460 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521880464 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109176/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-521880352 **[Test build #109176 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109176/testReport)** for PR 24903 at commit [`c503f5d`](https://github.com/apache/spark/commit/c503f5d419323a16442f313857b030d49f28995b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates
maropu commented on a change in pull request #25410: [SPARK-28690][SQL] Add
`date_part` function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#discussion_r314579663
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -1409,48 +1409,7 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
* Create a Extract expression.
*/
override def visitExtract(ctx: ExtractContext): Expression = withOrigin(ctx)
{
-ctx.field.getText.toUpperCase(Locale.ROOT) match {
- case "MILLENNIUM" | "MILLENNIA" | "MIL" | "MILS" =>
-Millennium(expression(ctx.source))
- case "CENTURY" | "CENTURIES" | "C" | "CENT" =>
-Century(expression(ctx.source))
- case "DECADE" | "DECADES" | "DEC" | "DECS" =>
-Decade(expression(ctx.source))
- case "YEAR" | "Y" | "YEARS" | "YR" | "YRS" =>
-Year(expression(ctx.source))
- case "ISOYEAR" =>
-IsoYear(expression(ctx.source))
- case "QUARTER" | "QTR" =>
-Quarter(expression(ctx.source))
- case "MONTH" | "MON" | "MONS" | "MONTHS" =>
-Month(expression(ctx.source))
- case "WEEK" | "W" | "WEEKS" =>
-WeekOfYear(expression(ctx.source))
- case "DAY" | "D" | "DAYS" =>
-DayOfMonth(expression(ctx.source))
- case "DAYOFWEEK" =>
-DayOfWeek(expression(ctx.source))
- case "DOW" =>
-Subtract(DayOfWeek(expression(ctx.source)), Literal(1))
- case "ISODOW" =>
-Add(WeekDay(expression(ctx.source)), Literal(1))
- case "DOY" =>
-DayOfYear(expression(ctx.source))
- case "HOUR" | "H" | "HOURS" | "HR" | "HRS" =>
-Hour(expression(ctx.source))
- case "MINUTE" | "M" | "MIN" | "MINS" | "MINUTES" =>
-Minute(expression(ctx.source))
- case "SECOND" | "S" | "SEC" | "SECONDS" | "SECS" =>
-Second(expression(ctx.source))
- case "MILLISECONDS" | "MSEC" | "MSECS" | "MILLISECON" | "MSECONDS" |
"MS" =>
-Milliseconds(expression(ctx.source))
- case "MICROSECONDS" | "USEC" | "USECS" | "USECONDS" | "MICROSECON" |
"US" =>
-Microseconds(expression(ctx.source))
- case "EPOCH" =>
-Epoch(expression(ctx.source))
- case other =>
-throw new ParseException(s"Literals of type '$other' are currently not
supported.", ctx)
Review comment:
The exception changed from `ParseException` to `AnalysisException`? Can we
keep the current behaviour?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available
shahidki31 commented on a change in pull request #25460:
[SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not
available
URL: https://github.com/apache/spark/pull/25460#discussion_r314577254
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala
##
@@ -72,7 +72,8 @@ case class HadoopFsRelation(
val compressionFactor = sqlContext.conf.fileCompressionFactor
val defaultSize = (location.sizeInBytes * compressionFactor).toLong
location match {
- case cfi: CatalogFileIndex if
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled =>
+ case cfi: CatalogFileIndex if
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled
+&& defaultSize == sqlContext.conf.defaultSizeInBytes =>
Review comment:
If the table statistics is available here,
https://github.com/apache/spark/blob/0ea8db9fd3d882140d8fa305dd69fc94db62cf8f/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L387-L390,
Then it should be available here too right?
https://github.com/apache/spark/blob/0ea8db9fd3d882140d8fa305dd69fc94db62cf8f/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala#L43-L45
So, ideally the flow shouldn't come to the fallback logic, if the table
statistics already exists. That is why even after #24715,
`location.sizeInBytes` is 8.0EB
@cloud-fan Could you please give a reproducible test, where the issue can
happen?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host
AmplabJenkins removed a comment on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521882108 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host
AmplabJenkins removed a comment on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521882113 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109169/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession
AmplabJenkins commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#issuecomment-521884381 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession
SparkQA removed a comment on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#issuecomment-521857445 **[Test build #109155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109155/testReport)** for PR 25463 at commit [`8bc622f`](https://github.com/apache/spark/commit/8bc622f7332ad7970e54624b6d5b5aa184df3510). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession
SparkQA commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#issuecomment-521884185 **[Test build #109155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109155/testReport)** for PR 25463 at commit [`8bc622f`](https://github.com/apache/spark/commit/8bc622f7332ad7970e54624b6d5b5aa184df3510). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class DockerJDBCIntegrationSuite extends SharedSparkSession with Eventually ` * `class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with SharedSparkSession ` * `class OrcFilterSuite extends OrcTest with SharedSparkSession ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive
maropu commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive before retry #24533 URL: https://github.com/apache/spark/pull/25469#issuecomment-521885185 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession
SparkQA removed a comment on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#issuecomment-521858760 **[Test build #109173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109173/testReport)** for PR 25463 at commit [`8bc622f`](https://github.com/apache/spark/commit/8bc622f7332ad7970e54624b6d5b5aa184df3510). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-521885161 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
SparkQA removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-521857475 **[Test build #109168 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109168/testReport)** for PR 25368 at commit [`45cbbd0`](https://github.com/apache/spark/commit/45cbbd04408251e14a9157d1a5b93ae6a8e91401). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession
SparkQA commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#issuecomment-521885268 **[Test build #109173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109173/testReport)** for PR 25463 at commit [`8bc622f`](https://github.com/apache/spark/commit/8bc622f7332ad7970e54624b6d5b5aa184df3510). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class DockerJDBCIntegrationSuite extends SharedSparkSession with Eventually ` * `class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with SharedSparkSession ` * `class OrcFilterSuite extends OrcTest with SharedSparkSession ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs
AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-521885162 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109168/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521884974 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109164/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive
maropu commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive before retry #24533 URL: https://github.com/apache/spark/pull/25469#issuecomment-521885308 @AngersZh ok, the other prs hit the same errors, so they are not related to this pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available
wangyum commented on a change in pull request #25460:
[SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not
available
URL: https://github.com/apache/spark/pull/25460#discussion_r314552128
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala
##
@@ -72,7 +72,8 @@ case class HadoopFsRelation(
val compressionFactor = sqlContext.conf.fileCompressionFactor
val defaultSize = (location.sizeInBytes * compressionFactor).toLong
location match {
- case cfi: CatalogFileIndex if
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled =>
+ case cfi: CatalogFileIndex if
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled
+&& defaultSize == sqlContext.conf.defaultSizeInBytes =>
Review comment:
We can avoid this when constructing this `CatalogFileIndex`:
https://github.com/apache/spark/blob/c30b5297bc607ae33cc2fcf624b127942154e559/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L383-L387
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
HyukjinKwon commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521850418 Given https://issues.apache.org/jira/browse/SPARK-28750?focusedCommentId=16908587=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16908587, shall we go with build with Java 8 & test with Java 11 for now? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession
cloud-fan commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#issuecomment-521851980 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession
cloud-fan commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#issuecomment-521856533 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25467: [SPARK-28634][yarn] Ignore kerberos login config in client mode AM.
AmplabJenkins removed a comment on issue #25467: [SPARK-28634][yarn] Ignore kerberos login config in client mode AM. URL: https://github.com/apache/spark/pull/25467#issuecomment-521856005 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14222/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25403: [SPARK-28679][YARN] changes to setResourceInformation to handle empty resources and reflection error handling
SparkQA commented on issue #25403: [SPARK-28679][YARN] changes to setResourceInformation to handle empty resources and reflection error handling URL: https://github.com/apache/spark/pull/25403#issuecomment-521857488 **[Test build #109165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109165/testReport)** for PR 25403 at commit [`cfec179`](https://github.com/apache/spark/commit/cfec1798c134fc132be88ce693230b0a61c45bee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
