[GitHub] spark issue #20860: [SPARK-23743][SQL] Changed a comparison logic from conta...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20860 **[Test build #88538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88538/testReport)** for PR 20860 at commit [`2ea9b7a`](https://github.com/apache/spark/commit/2ea9b7a58279d0e5d7cdfad8d67ab9227983be1a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20541: [SPARK-23356][SQL]Pushes Project to both sides of Union ...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/20541 oh, I see, I fallback to the modification of the non-deterministic expression, and to keep the newly added test cases for a+1 and a+b, can you? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20851: [SPARK-23727][SQL] Support for pushing down filte...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20851#discussion_r176676405
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -353,6 +353,13 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_FILTER_PUSHDOWN_DATE_ENABLED =
buildConf("spark.sql.parquet.filterPushdown.date")
+.doc("If true, enables Parquet filter push-down optimization for Date.
" +
+ "This configuration only has an effect when
'spark.sql.parquet.filterPushdown' is enabled.")
+.internal()
+.booleanConf
+.createWithDefault(false)
--- End diff --
I am fine either way.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20885: [SPARK-23724][SPARK-23765][SQL] Line separator for the j...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20885 **[Test build #88539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88539/testReport)** for PR 20885 at commit [`d632706`](https://github.com/apache/spark/commit/d632706bf14c7a7c2688237e6dc552ca5aa9c98a). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20885: [SPARK-23724][SPARK-23765][SQL] Line separator for the j...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20885 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20885: [SPARK-23724][SPARK-23765][SQL] Line separator for the j...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20885 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88539/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20885: [SPARK-23724][SPARK-23765][SQL] Line separator for the j...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20885 **[Test build #88539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88539/testReport)** for PR 20885 at commit [`d632706`](https://github.com/apache/spark/commit/d632706bf14c7a7c2688237e6dc552ca5aa9c98a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user zuotingbing commented on the issue: https://github.com/apache/spark/pull/20864 @felixcheung Thanks. The test result is 0 failed, what is meaning of > This patch fails due to an unknown error code, -9 ? what should i do to fix the test fails? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20851: [SPARK-23727][SQL] Support for pushing down filte...
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/20851#discussion_r176660685
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -353,6 +353,13 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_FILTER_PUSHDOWN_DATE_ENABLED =
buildConf("spark.sql.parquet.filterPushdown.date")
+.doc("If true, enables Parquet filter push-down optimization for Date.
" +
+ "This configuration only has an effect when
'spark.sql.parquet.filterPushdown' is enabled.")
+.internal()
+.booleanConf
+.createWithDefault(false)
--- End diff --
@dongjoon-hyun @HyukjinKwon suggested such configuration could be better to
start with "false", but actually ORC does support Date push down by default,
any idea?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20836: SPARK-23685 : Fix for the Spark Structured Streaming Kaf...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/20836 I don't see the problem. You see an exception which tells exactly what can be done: > If you don't want your streaming query to fail on such cases, set the source option "failOnDataLoss" to "false". " Why do you want this change at all? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20836: SPARK-23685 : Fix for the Spark Structured Stream...
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/20836#discussion_r176656905
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -279,9 +279,8 @@ private[kafka010] case class InternalKafkaConsumer(
if (record.offset > offset) {
// This may happen when some records aged out but their offsets
already got verified
if (failOnDataLoss) {
- reportDataLoss(true, s"Cannot fetch records in [$offset,
${record.offset})")
--- End diff --
It's just breaks the whole concept. When `failOnDataLoss ` enabled this
exception means it should fail.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20860: [SPARK-23743][SQL] Changed a comparison logic from conta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20860 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20860: [SPARK-23743][SQL] Changed a comparison logic from conta...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20860 **[Test build #88538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88538/testReport)** for PR 20860 at commit [`2ea9b7a`](https://github.com/apache/spark/commit/2ea9b7a58279d0e5d7cdfad8d67ab9227983be1a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20860: [SPARK-23743][SQL] Changed a comparison logic from conta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20860 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1719/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20860: [SPARK-23743][SQL] Changed a comparison logic fro...
Github user jongyoul commented on a diff in the pull request:
https://github.com/apache/spark/pull/20860#discussion_r176654007
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
---
@@ -179,7 +179,7 @@ private[hive] class IsolatedClientLoader(
val isHadoopClass =
name.startsWith("org.apache.hadoop.") &&
!name.startsWith("org.apache.hadoop.hive.")
-name.contains("slf4j") ||
+name.startsWith("org.slf4j") ||
name.contains("log4j") ||
--- End diff --
Fixed
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20860: [SPARK-23743][SQL] Changed a comparison logic fro...
Github user jongyoul commented on a diff in the pull request:
https://github.com/apache/spark/pull/20860#discussion_r176652812
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
---
@@ -179,7 +179,7 @@ private[hive] class IsolatedClientLoader(
val isHadoopClass =
name.startsWith("org.apache.hadoop.") &&
!name.startsWith("org.apache.hadoop.hive.")
-name.contains("slf4j") ||
+name.startsWith("org.slf4j") ||
name.contains("log4j") ||
--- End diff --
Yes, It looks like a similar issue. I'll add it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the âhive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88537/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20864 **[Test build #88537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88537/testReport)** for PR 20864 at commit [`b177bf4`](https://github.com/apache/spark/commit/b177bf441a04c0a700e33d7e40c1b2408c3c0c3b). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
