[GitHub] [spark] AmplabJenkins removed a comment on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base
AmplabJenkins removed a comment on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base URL: https://github.com/apache/spark/pull/25090#issuecomment-510341195 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base
AmplabJenkins commented on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base URL: https://github.com/apache/spark/pull/25090#issuecomment-510341202 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107501/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base
AmplabJenkins commented on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base URL: https://github.com/apache/spark/pull/25090#issuecomment-510341195 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base
AmplabJenkins removed a comment on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base URL: https://github.com/apache/spark/pull/25090#issuecomment-510341202 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107501/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #25109: [SPARK-28343][SQL][TEST] PostgreSQL test should enable cartesian product
wangyum commented on issue #25109: [SPARK-28343][SQL][TEST] PostgreSQL test should enable cartesian product URL: https://github.com/apache/spark/pull/25109#issuecomment-510341125 @maropu I didn't enable `spark.sql.parser.ansi.enabled` because: ```sql spark-sql> set spark.sql.parser.ansi.enabled=true; spark.sql.parser.ansi.enabledtrue spark-sql> select 1 as false; Error in query: no viable alternative at input 'false'(line 1, pos 12) == SQL == select 1 as false ^^^ spark-sql> select 1 as minus; Error in query: no viable alternative at input 'minus'(line 1, pos 12) == SQL == select 1 as minus ^^^ ``` Is this we expected? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax
maropu commented on a change in pull request #25074: [SPARK-27924]Support ANSI
SQL Boolean-Predicate syntax
URL: https://github.com/apache/spark/pull/25074#discussion_r302374072
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/booleanExpressions.scala
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
+import org.apache.spark.sql.types.BooleanType
+
+/**
+ * String to indicate which boolean test selected.
+ */
+object BooleanTest {
+ val TRUE = "TRUE"
+ val FALSE = "FALSE"
+ val UNKNOWN = "UNKNOWN"
+
+ def calculate(input: Any, booleanValue: String): Boolean = {
+booleanValue match {
+ case TRUE => input == true
+ case FALSE => input == false
+ case UNKNOWN => input == null
+ case _ => throw new AnalysisException("Boolean test value must be one of
TRUE, " +
+"FALSE and UNKNOWN.")
+}
+ }
+}
+
+/**
+ * Test the value of an expression is true, false, or unknown.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, booleanValue) - Returns true if `expr` equals
booleanValue, " +
+"or false otherwise.",
+ arguments = """
+Arguments:
+ * expr - a boolean expression
+ * booleanValue - a boolean value represented by a string. booleanValue
must be one
+ of TRUE, FALSE and UNKNOWN.
+ """,
+ examples = """
+Examples:
+> SELECT _FUNC_(1 > 2, true);
+ false
+> SELECT _FUNC_(2 > 1, true);
+ true
+ """)
+case class BooleanTest(child: Expression, booleanValue: String)
Review comment:
As another design choice, how about handling`IS UNKNOWN` just as `IS NULL`,
then changing `booleanValue: String` into `booleanValue: Boolean`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base
SparkQA removed a comment on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base URL: https://github.com/apache/spark/pull/25090#issuecomment-510303053 **[Test build #107501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107501/testReport)** for PR 25090 at commit [`a09df4b`](https://github.com/apache/spark/commit/a09df4b5dc90c93f05d68fd6695ccb2de663895c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25109: [SPARK-28343][SQL][TEST] PostgreSQL test should enable cartesian product
AmplabJenkins commented on issue #25109: [SPARK-28343][SQL][TEST] PostgreSQL test should enable cartesian product URL: https://github.com/apache/spark/pull/25109#issuecomment-510340877 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25109: [SPARK-28343][SQL][TEST] PostgreSQL test should enable cartesian product
AmplabJenkins commented on issue #25109: [SPARK-28343][SQL][TEST] PostgreSQL test should enable cartesian product URL: https://github.com/apache/spark/pull/25109#issuecomment-510340883 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12635/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base
SparkQA commented on issue #25090: [SPARK-28278][SQL][PYTHON][TESTS] Convert and port 'except-all.sql' into UDF test base URL: https://github.com/apache/spark/pull/25090#issuecomment-510340755 **[Test build #107501 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107501/testReport)** for PR 25090 at commit [`a09df4b`](https://github.com/apache/spark/commit/a09df4b5dc90c93f05d68fd6695ccb2de663895c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
dongjoon-hyun commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-510340656 Thanks a lot! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum opened a new pull request #25109: [SPARK-28343][SQL][TEST] PostgreSQL test should enable cartesian product
wangyum opened a new pull request #25109: [SPARK-28343][SQL][TEST] PostgreSQL test should enable cartesian product URL: https://github.com/apache/spark/pull/25109 ## What changes were proposed in this pull request? This pr enables cartesian product for PostgreSQL test. ## How was this patch tested? manual tests: Run `test.sql` in [pgSQL](https://github.com/apache/spark/tree/master/sql/core/src/test/resources/sql-tests/inputs/pgSQL) directory and in [inputs](https://github.com/apache/spark/tree/master/sql/core/src/test/resources/sql-tests/inputs) directory: ```sql cat < test.sql create or replace temporary view t1 as select * from (values(1), (2)) as v (val); create or replace temporary view t2 as select * from (values(2), (1)) as v (val); select t1.*, t2.* from t1 join t2; EOF ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510339651 **[Test build #107508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107508/testReport)** for PR 24637 at commit [`a453d8c`](https://github.com/apache/spark/commit/a453d8c84644b1edf38ff90821b0dfff85f50c2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once
AmplabJenkins removed a comment on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once URL: https://github.com/apache/spark/pull/25108#issuecomment-510339208 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510339171 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once
SparkQA commented on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once URL: https://github.com/apache/spark/pull/25108#issuecomment-510339633 **[Test build #107507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107507/testReport)** for PR 25108 at commit [`c725002`](https://github.com/apache/spark/commit/c7250023de9613276954989c23168920616ad0f1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
HyukjinKwon commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-510339534 It actually passed in my local as well.. but seems dependent on specific Python version or OS specific ... ``` Expected "32.6[64]", but got "32.6[7]" Result did not match for query #1SELECT udf(avg(a)) AS avg_32 FROM aggtest WHERE a < 100 ``` I think it's https://docs.python.org/3/tutorial/floatingpoint.html .. Let me just cast it into int as I worked around in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510339180 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12634/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
dongjoon-hyun commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-510339155 Ya. Please check locally first because the above report is on JDK11~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510339171 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once
AmplabJenkins commented on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once URL: https://github.com/apache/spark/pull/25108#issuecomment-510339214 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12633/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
dongjoon-hyun edited a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-510339155 Thanks. Ya. Please check locally first because the above report is on JDK11~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once
AmplabJenkins commented on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once URL: https://github.com/apache/spark/pull/25108#issuecomment-510339208 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510339180 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12634/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
HyukjinKwon commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-510339061 Will take a look and revert it's going to take longer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
HyukjinKwon opened a new pull request #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069 ## What changes were proposed in this pull request? This PR adds some tests converted from `pgSQL/aggregates_part1.sql'` to test UDFs. Please see contribution guide of this umbrella ticket - [SPARK-27921](https://issues.apache.org/jira/browse/SPARK-27921). This PR also contains two minor fixes: 1. Change name of Scala UDF from `UDF:name(...)` to `name(...)` to be consistent with Python' 2. Fix Scala UDF at `IntegratedUDFTestUtils.scala ` to handle `null` in strings. Diff comparing to 'pgSQL/aggregates_part1.sql' ```diff diff --git a/sql/core/src/test/resources/sql-tests/results/pgSQL/aggregates_part1.sql.out b/sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-aggregates_part1.sql.out index 51ca1d55869..124fdd6416e 100644 --- a/sql/core/src/test/resources/sql-tests/results/pgSQL/aggregates_part1.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-aggregates_part1.sql.out @@ -3,7 +3,7 @@ -- !query 0 -SELECT avg(four) AS avg_1 FROM onek +SELECT avg(udf(four)) AS avg_1 FROM onek -- !query 0 schema struct -- !query 0 output @@ -11,15 +11,15 @@ struct -- !query 1 -SELECT avg(a) AS avg_32 FROM aggtest WHERE a < 100 +SELECT udf(avg(a)) AS avg_32 FROM aggtest WHERE a < 100 -- !query 1 schema -struct +struct -- !query 1 output 32.664 -- !query 2 -select CAST(avg(b) AS Decimal(10,3)) AS avg_107_943 FROM aggtest +select CAST(avg(udf(b)) AS Decimal(10,3)) AS avg_107_943 FROM aggtest -- !query 2 schema struct -- !query 2 output @@ -27,285 +27,286 @@ struct -- !query 3 -SELECT sum(four) AS sum_1500 FROM onek +SELECT sum(udf(four)) AS sum_1500 FROM onek -- !query 3 schema -struct +struct -- !query 3 output -1500 +1500.0 -- !query 4 -SELECT sum(a) AS sum_198 FROM aggtest +SELECT udf(sum(a)) AS sum_198 FROM aggtest -- !query 4 schema -struct +struct -- !query 4 output 198 -- !query 5 -SELECT sum(b) AS avg_431_773 FROM aggtest +SELECT udf(udf(sum(b))) AS avg_431_773 FROM aggtest -- !query 5 schema -struct +struct -- !query 5 output 431.77260909229517 -- !query 6 -SELECT max(four) AS max_3 FROM onek +SELECT udf(max(four)) AS max_3 FROM onek -- !query 6 schema -struct +struct -- !query 6 output 3 -- !query 7 -SELECT max(a) AS max_100 FROM aggtest +SELECT max(udf(a)) AS max_100 FROM aggtest -- !query 7 schema -struct +struct -- !query 7 output -100 +56 -- !query 8 -SELECT max(aggtest.b) AS max_324_78 FROM aggtest +SELECT CAST(udf(udf(max(aggtest.b))) AS int) AS max_324_78 FROM aggtest -- !query 8 schema -struct +struct -- !query 8 output -324.78 +324 -- !query 9 -SELECT stddev_pop(b) FROM aggtest +SELECT CAST(stddev_pop(udf(b)) AS int) FROM aggtest -- !query 9 schema -struct +struct -- !query 9 output -131.10703231895047 +131 -- !query 10 -SELECT stddev_samp(b) FROM aggtest +SELECT udf(stddev_samp(b)) FROM aggtest -- !query 10 schema -struct +struct -- !query 10 output 151.38936080399804 -- !query 11 -SELECT var_pop(b) FROM aggtest +SELECT CAST(var_pop(udf(b)) as int) FROM aggtest -- !query 11 schema -struct +struct -- !query 11 output -17189.053923482323 +17189 -- !query 12 -SELECT var_samp(b) FROM aggtest +SELECT udf(var_samp(b)) FROM aggtest -- !query 12 schema -struct +struct -- !query 12 output 22918.738564643096 -- !query 13 -SELECT stddev_pop(CAST(b AS Decimal(38,0))) FROM aggtest +SELECT udf(stddev_pop(CAST(b AS Decimal(38,0 FROM aggtest -- !query 13 schema -struct +struct -- !query 13 output 131.18117242958306 -- !query 14 -SELECT stddev_samp(CAST(b AS Decimal(38,0))) FROM aggtest +SELECT stddev_samp(CAST(udf(b) AS Decimal(38,0))) FROM aggtest -- !query 14 schema -struct +struct -- !query 14 output 151.47497042966097 -- !query 15 -SELECT var_pop(CAST(b AS Decimal(38,0))) FROM aggtest +SELECT udf(var_pop(CAST(b AS Decimal(38,0 FROM aggtest -- !query 15 schema -struct +struct -- !query 15 output 17208.5 -- !query 16 -SELECT var_samp(CAST(b AS Decimal(38,0))) FROM aggtest +SELECT var_samp(udf(CAST(b AS Decimal(38,0 FROM aggtest -- !query 16 schema -struct +struct -- !query 16 output 22944.6668 -- !query 17 -SELECT var_pop(1.0), var_samp(2.0) +SELECT udf(var_pop(1.0)),
[GitHub] [spark] HyukjinKwon closed pull request #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
HyukjinKwon closed pull request #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once
HyukjinKwon commented on issue #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once URL: https://github.com/apache/spark/pull/25108#issuecomment-510338860 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
dongjoon-hyun edited a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-510338586 Could you investigate this? For the others, this commit is still on testings. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
dongjoon-hyun commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-510338469 @HyukjinKwon . This `udf/pgSQL/udf-aggregates_part1.sql - Regular Python UDF` failed three times consecutively. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-jdk-11-ubuntu-testing/1122/testReport/org.apache.spark.sql/SQLQueryTestSuite/udf_pgSQL_udf_aggregates_part1_sql___Regular_Python_UDF/history/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax
maropu commented on a change in pull request #25074: [SPARK-27924]Support ANSI
SQL Boolean-Predicate syntax
URL: https://github.com/apache/spark/pull/25074#discussion_r302371689
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/booleanExpressions.scala
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
+import org.apache.spark.sql.types.BooleanType
+
Review comment:
How about moving this into `predicates.scala`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
dongjoon-hyun commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-510338586 Could you investigate this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once
HyukjinKwon opened a new pull request #25108: [SPARK-28321][SQL] 0-args Java UDF should not be called only once URL: https://github.com/apache/spark/pull/25108 ## What changes were proposed in this pull request? 0-args Java UDF alone calls the function even before making it as an expression. It causes that the function always returns the same value. Seems like a mistake. ## How was this patch tested? Unit test was added This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
SparkQA commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query URL: https://github.com/apache/spark/pull/25107#issuecomment-510336280 **[Test build #107505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107505/testReport)** for PR 25107 at commit [`1898674`](https://github.com/apache/spark/commit/189867428b8851da5242765641c2967608f1a833). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510336289 I made a PR to you which reverts all the other benchmark result to the master and adds the newly added test case result. - https://github.com/viirya/spark-1/pull/7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510336276 **[Test build #107506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107506/testReport)** for PR 24637 at commit [`456e6d2`](https://github.com/apache/spark/commit/456e6d2a8f67786105f0b4363e9e3213df8fd554). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
AmplabJenkins removed a comment on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query URL: https://github.com/apache/spark/pull/25107#issuecomment-510335921 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12631/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
AmplabJenkins commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query URL: https://github.com/apache/spark/pull/25107#issuecomment-510335917 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
AmplabJenkins commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query URL: https://github.com/apache/spark/pull/25107#issuecomment-510335921 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12631/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510335903 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12632/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510335897 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510335897 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510335903 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12632/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25104: [SPARK-28341][SQL] remove session catalog config
AmplabJenkins commented on issue #25104: [SPARK-28341][SQL] remove session catalog config URL: https://github.com/apache/spark/pull/25104#issuecomment-510335322 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107498/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25041: [SPARK-28133][SQL] Add acosh/asinh/atanh functions to SQL
maropu commented on issue #25041: [SPARK-28133][SQL] Add acosh/asinh/atanh functions to SQL URL: https://github.com/apache/spark/pull/25041#issuecomment-510335390 PostgreSQL implementation for asinh: https://github.com/postgres/postgres/blob/b5810de3f4eb0e7b242d31db93a46957d56ea8b4/src/backend/utils/adt/float.c#L2537 It just checks if the result is valid by `check_float8_val(const float8 val, const bool inf_is_valid, const bool zero_is_valid)`: https://github.com/postgres/postgres/blob/b5810de3f4eb0e7b242d31db93a46957d56ea8b4/src/backend/utils/adt/float.c#L2547 For asinh, inf_is_valid=true, so infinity is valid for asinh output. Anyway, +1 for @dongjoon-hyun and I think `-Infinity` is more suitable for the output. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25104: [SPARK-28341][SQL] remove session catalog config
AmplabJenkins removed a comment on issue #25104: [SPARK-28341][SQL] remove session catalog config URL: https://github.com/apache/spark/pull/25104#issuecomment-510335312 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25104: [SPARK-28341][SQL] remove session catalog config
AmplabJenkins commented on issue #25104: [SPARK-28341][SQL] remove session catalog config URL: https://github.com/apache/spark/pull/25104#issuecomment-510335312 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
cloud-fan commented on a change in pull request #25107: [SPARK-28344][SQL]
detect ambiguous self-join and fail the query
URL: https://github.com/apache/spark/pull/25107#discussion_r302368897
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##
@@ -66,30 +66,24 @@ class QueryExecution(
lazy val withCachedData: LogicalPlan = {
assertAnalyzed()
assertSupported()
-sparkSession.sharedState.cacheManager.useCachedData(analyzed)
+sparkSession.sharedState.cacheManager.useCachedData(analyzed.clone())
Review comment:
I'll move this out as a separate PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25104: [SPARK-28341][SQL] remove session catalog config
AmplabJenkins removed a comment on issue #25104: [SPARK-28341][SQL] remove session catalog config URL: https://github.com/apache/spark/pull/25104#issuecomment-510335322 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107498/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan opened a new pull request #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
cloud-fan opened a new pull request #25107: [SPARK-28344][SQL] detect ambiguous
self-join and fail the query
URL: https://github.com/apache/spark/pull/25107
## What changes were proposed in this pull request?
This is an alternative solution of
https://github.com/apache/spark/pull/24442 . It fails the query if ambiguous
self join is detected, instead of trying to de-ambiguous. The problem is that,
it's hard to come up with a reasonable rule to de-ambiguous, the rule proposed
by #24442 is mostly a heuristic.
### background of the self-join problem:
This is a long-standing bug and I've seen many people complaining about it
in JIRA/dev list.
A typical example:
```
val df1 = …
val df2 = df1.filter(...)
df1.join(df2, df1("a") > df2("a")) // returns empty result
```
The root cause is, `Dataset.apply` is so powerful that users think it
returns a column reference which can point to the column of the Dataset at
anywhere. This is not true in many cases. `Dataset.apply` returns an
`AttributeReference` . Different Datasets may share the same
`AttributeReference`. In the example above, `df2` adds a Filter operator above
the logical plan of `df1`, and the Filter operator reserves the output
`AttributeReference` of its child. This means, `df1("a")` is exactly the same
as `df2("a")`, and `df1("a") > df2("a")` always evaluates to false.
### The rule to detect ambiguous column reference caused by self join:
We can reuse the infra in #24442 :
1. each Dataset has a globally unique id.
2. the `AttributeReference` returned by `Dataset.apply` carries the ID and
column position(e.g. 3rd column of the Dataset) via metadata.
3. the logical plan of a `Dataset` carries the ID via `TreeNodeTag`
When self-join happens, the analyzer asks the right side plan of join to
re-generate output attributes with new exprIds. Based on it, a simple rule to
detect ambiguous self join is:
1. find all column references (i.e. `AttributeReference`s with Dataset ID
and col position) in the root node of a query plan.
2. for each column reference, traverse the query plan tree, find a sub-plan
that carries Dataset ID and the ID is the same as the one in the column
reference.
3. get the corresponding output attribute of the sub-plan by the col
position in the column reference.
4. if the corresponding output attribute has a different exprID than the
column reference, then it means this sub-plan is in the right side of a
self-join and has regenerated its output attributes. This is an ambiguous self
join because the column reference points to a table being self-joined.
## How was this patch tested?
existing tests and new test cases
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25104: [SPARK-28341][SQL] remove session catalog config
SparkQA removed a comment on issue #25104: [SPARK-28341][SQL] remove session catalog config URL: https://github.com/apache/spark/pull/25104#issuecomment-510295336 **[Test build #107498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107498/testReport)** for PR 25104 at commit [`cba0e25`](https://github.com/apache/spark/commit/cba0e25b707698aa7e529bd12ae13b147b198c46). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25104: [SPARK-28341][SQL] remove session catalog config
SparkQA commented on issue #25104: [SPARK-28341][SQL] remove session catalog config URL: https://github.com/apache/spark/pull/25104#issuecomment-510334756 **[Test build #107498 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107498/testReport)** for PR 25104 at commit [`cba0e25`](https://github.com/apache/spark/commit/cba0e25b707698aa7e529bd12ae13b147b198c46). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class CatalogManager(conf: SQLConf, val v2SessionCatalog: TableCatalog) ` * `class NoopV2SessionCatalog extends TableCatalog ` * `class V2SessionCatalog(catalog: SessionCatalog, conf: SQLConf) extends TableCatalog ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax
AmplabJenkins removed a comment on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax URL: https://github.com/apache/spark/pull/25074#issuecomment-510334269 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax
AmplabJenkins removed a comment on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax URL: https://github.com/apache/spark/pull/25074#issuecomment-510334273 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107500/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax
AmplabJenkins commented on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax URL: https://github.com/apache/spark/pull/25074#issuecomment-510334273 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107500/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax
AmplabJenkins commented on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax URL: https://github.com/apache/spark/pull/25074#issuecomment-510334269 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax
SparkQA removed a comment on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax URL: https://github.com/apache/spark/pull/25074#issuecomment-510297148 **[Test build #107500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107500/testReport)** for PR 25074 at commit [`c22f12d`](https://github.com/apache/spark/commit/c22f12db4bd6646fb13110d89e461217cc4e6a1c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax
SparkQA commented on issue #25074: [SPARK-27924]Support ANSI SQL Boolean-Predicate syntax URL: https://github.com/apache/spark/pull/25074#issuecomment-510333889 **[Test build #107500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107500/testReport)** for PR 25074 at commit [`c22f12d`](https://github.com/apache/spark/commit/c22f12db4bd6646fb13110d89e461217cc4e6a1c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint
AmplabJenkins commented on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint URL: https://github.com/apache/spark/pull/25071#issuecomment-510333758 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107496/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint
AmplabJenkins removed a comment on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint URL: https://github.com/apache/spark/pull/25071#issuecomment-510333758 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107496/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint
AmplabJenkins commented on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint URL: https://github.com/apache/spark/pull/25071#issuecomment-510333757 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint
AmplabJenkins removed a comment on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint URL: https://github.com/apache/spark/pull/25071#issuecomment-510333757 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #25083: [SPARK-28311][SQL] Fix STS OpenSession failed return wrong origin PROTOCOL_VERSION
AngersZh commented on issue #25083: [SPARK-28311][SQL] Fix STS OpenSession failed return wrong origin PROTOCOL_VERSION URL: https://github.com/apache/spark/pull/25083#issuecomment-510333514 @srowen @dongjoon-hyun I have write some description about this PR and put some my test result with this pr. If you have some good idea about how to write unit test of this pr. Give me a little guidance. Thanks a lot. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint
SparkQA commented on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint URL: https://github.com/apache/spark/pull/25071#issuecomment-51097 **[Test build #107496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107496/testReport)** for PR 25071 at commit [`8a60a22`](https://github.com/apache/spark/commit/8a60a22bf8526c23b4cb3daa273d22d90fb156c5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint
SparkQA removed a comment on issue #25071: [SPARK-28292][SQL] Enable inject user-defined Hint URL: https://github.com/apache/spark/pull/25071#issuecomment-510290440 **[Test build #107496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107496/testReport)** for PR 25071 at commit [`8a60a22`](https://github.com/apache/spark/commit/8a60a22bf8526c23b4cb3daa273d22d90fb156c5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users
SparkQA commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users URL: https://github.com/apache/spark/pull/24996#issuecomment-510333278 **[Test build #107504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107504/testReport)** for PR 24996 at commit [`f42e3b5`](https://github.com/apache/spark/commit/f42e3b5a4e41b1068d891f0c467cda93cd0d0b0c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune
unnecessary nested fields from Generate in explode
URL: https://github.com/apache/spark/pull/24637#discussion_r302367039
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1589,6 +1589,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val NESTED_PRUNING_ON_EXPRESSIONS =
+buildConf("spark.sql.optimizer.expression.nestedPruning.enabled")
+.internal()
Review comment:
Oops, thanks for the catch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
AmplabJenkins removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510332229 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
AmplabJenkins removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510332233 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107497/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
AmplabJenkins commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510332229 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
AmplabJenkins commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510332233 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107497/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users
HeartSaVioR commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users URL: https://github.com/apache/spark/pull/24996#issuecomment-510331849 retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
SparkQA commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510331819 **[Test build #107497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107497/testReport)** for PR 25105 at commit [`afb2862`](https://github.com/apache/spark/commit/afb2862daa708ff5652f6d02b25d91f42351162c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
SparkQA removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510295333 **[Test build #107497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107497/testReport)** for PR 25105 at commit [`afb2862`](https://github.com/apache/spark/commit/afb2862daa708ff5652f6d02b25d91f42351162c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25106: [SPARK-28345][SQL][Python] PythonUDF predicate should be able to pushdown to join
AmplabJenkins commented on issue #25106: [SPARK-28345][SQL][Python] PythonUDF predicate should be able to pushdown to join URL: https://github.com/apache/spark/pull/25106#issuecomment-510331429 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12630/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
AmplabJenkins removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510331374 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25106: [SPARK-28345][SQL][Python] PythonUDF predicate should be able to pushdown to join
AmplabJenkins commented on issue #25106: [SPARK-28345][SQL][Python] PythonUDF predicate should be able to pushdown to join URL: https://github.com/apache/spark/pull/25106#issuecomment-510331426 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
AmplabJenkins commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510331374 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25106: [SPARK-28345][SQL][Python] PythonUDF predicate should be able to pushdown to join
AmplabJenkins removed a comment on issue #25106: [SPARK-28345][SQL][Python] PythonUDF predicate should be able to pushdown to join URL: https://github.com/apache/spark/pull/25106#issuecomment-510331429 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12630/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
AmplabJenkins commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510331377 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107499/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
AmplabJenkins removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510331377 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107499/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25106: [SPARK-28345][SQL][Python] PythonUDF predicate should be able to pushdown to join
AmplabJenkins removed a comment on issue #25106: [SPARK-28345][SQL][Python] PythonUDF predicate should be able to pushdown to join URL: https://github.com/apache/spark/pull/25106#issuecomment-510331426 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
SparkQA removed a comment on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510297132 **[Test build #107499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107499/testReport)** for PR 25105 at commit [`f90660a`](https://github.com/apache/spark/commit/f90660a537ea63ff4aa7ca49fc2cf3947bd8d9aa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests
SparkQA commented on issue #25105: [SPARK-28342][SQL][TESTS] Replace REL_12_BETA1 to REL_12_BETA2 in PostgresSQL SQL tests URL: https://github.com/apache/spark/pull/25105#issuecomment-510331019 **[Test build #107499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107499/testReport)** for PR 25105 at commit [`f90660a`](https://github.com/apache/spark/commit/f90660a537ea63ff4aa7ca49fc2cf3947bd8d9aa). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510330614 Weirdly. The benchmark suite failed with the exact same reason at the first run. I'm running the second run. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya opened a new pull request #25106: [SPARK-28345][SQL][Python] PythonUDF predicate should be able to pushdown to join
viirya opened a new pull request #25106: [SPARK-28345][SQL][Python] PythonUDF
predicate should be able to pushdown to join
URL: https://github.com/apache/spark/pull/25106
## What changes were proposed in this pull request?
A `Filter` predicate using `PythonUDF` can't be push down into join
condition, currently. A predicate like that should be able to push down to join
condition. For `PythonUDF`s that can't be evaluated in join condition,
`PullOutPythonUDFInJoinCondition` will pull them out later.
An example like:
```scala
val pythonTestUDF = TestPythonUDF(name = "udf")
val left = Seq((1, 2), (2, 3)).toDF("a", "b")
val right = Seq((1, 2), (3, 4)).toDF("c", "d")
val df = left.crossJoin(right).where(pythonTestUDF($"a") ===
pythonTestUDF($"c"))
```
Query plan before the PR:
```
== Physical Plan ==
*(3) Project [a#2121, b#2122, c#2132, d#2133]
+- *(3) Filter (pythonUDF0#2142 = pythonUDF1#2143)
+- BatchEvalPython [udf(a#2121), udf(c#2132)], [pythonUDF0#2142,
pythonUDF1#2143]
+- BroadcastNestedLoopJoin BuildRight, Cross
:- *(1) Project [_1#2116 AS a#2121, _2#2117 AS b#2122]
: +- LocalTableScan [_1#2116, _2#2117]
+- BroadcastExchange IdentityBroadcastMode
+- *(2) Project [_1#2127 AS c#2132, _2#2128 AS d#2133]
+- LocalTableScan [_1#2127, _2#2128]
```
Query plan after the PR:
```
== Physical Plan ==
*(3) Project [a#2121, b#2122, c#2132, d#2133]
+- *(3) BroadcastHashJoin [pythonUDF0#2142], [pythonUDF0#2143], Cross,
BuildRight
:- BatchEvalPython [udf(a#2121)], [pythonUDF0#2142]
: +- *(1) Project [_1#2116 AS a#2121, _2#2117 AS b#2122]
: +- LocalTableScan [_1#2116, _2#2117]
+- BroadcastExchange HashedRelationBroadcastMode(List(input[2, string,
true]))
+- BatchEvalPython [udf(c#2132)], [pythonUDF0#2143]
+- *(2) Project [_1#2127 AS c#2132, _2#2128 AS d#2133]
+- LocalTableScan [_1#2127, _2#2128]
```
After this PR, the join can use `BroadcastHashJoin`, instead of
`BroadcastNestedLoopJoin`.
## How was this patch tested?
Added tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users
AmplabJenkins removed a comment on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users URL: https://github.com/apache/spark/pull/24996#issuecomment-510329849 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107502/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users
AmplabJenkins commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users URL: https://github.com/apache/spark/pull/24996#issuecomment-510329849 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107502/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users
AmplabJenkins removed a comment on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users URL: https://github.com/apache/spark/pull/24996#issuecomment-510329842 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users
AmplabJenkins commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users URL: https://github.com/apache/spark/pull/24996#issuecomment-510329842 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users
SparkQA removed a comment on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users URL: https://github.com/apache/spark/pull/24996#issuecomment-510307714 **[Test build #107502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107502/testReport)** for PR 24996 at commit [`f42e3b5`](https://github.com/apache/spark/commit/f42e3b5a4e41b1068d891f0c467cda93cd0d0b0c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users
SparkQA commented on issue #24996: [SPARK-28199][SS] Move Trigger implementations to Triggers.scala and avoid exposing these to the end users URL: https://github.com/apache/spark/pull/24996#issuecomment-510329656 **[Test build #107502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107502/testReport)** for PR 24996 at commit [`f42e3b5`](https://github.com/apache/spark/commit/f42e3b5a4e41b1068d891f0c467cda93cd0d0b0c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25041: [SPARK-28133][SQL] Add acosh/asinh/atanh functions to SQL
maropu commented on a change in pull request #25041: [SPARK-28133][SQL] Add acosh/asinh/atanh functions to SQL URL: https://github.com/apache/spark/pull/25041#discussion_r302362970 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala ## @@ -287,6 +287,30 @@ case class Cos(child: Expression) extends UnaryMathExpression(math.cos, "COS") """) case class Cosh(child: Expression) extends UnaryMathExpression(math.cosh, "COSH") +@ExpressionDescription( + usage = """ +_FUNC_(expr) - Returns inverse hyperbolic cosine of `expr`. + """, + arguments = """ +Arguments: + * expr - hyperbolic angle + """, + examples = """ +Examples: + > SELECT _FUNC_(1); + 0.0 + > SELECT _FUNC_(0); + NaN Review comment: This difference has already be covered in https://issues.apache.org/jira/browse/SPARK-27923? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itsvikramagr edited a comment on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation
itsvikramagr edited a comment on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-510327508 @HeartSaVioR, @gaborgsomogyi - thanks again for such a thorough review. I will soon resolve all your comments. Meanwhile, I was doing the performance analysis for various use-cases. While for complete output mode, I was able to ingest 2x-3x data than memory based backend in 1 executor and 5G container memory setup, I was not seeing encouraging results when the outputMode was "Append". I found that there is some memory leak (similar to one reported [here](https://github.com/facebook/rocksdb/issues/3216)) I am digging deeper to find the root cause. The fix might be to tune some rocksdb config or fall back to use an older version of rocksdb. (As @skonto commented earlier, code path for deletion of older states might need refactoring to get best out of rocksdb) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-510327508 @HeartSaVioR, @gaborgsomogyi - thanks again for such a thorough review. I will soon resolve all your comments. Meanwhile, I was doing the performance analysis for various use-cases. While for complete output mode, I was able to ingest 2x-3x data than memory based backend in 1 executor and 5G container memory setup, I was not seeing encouraging results when the outputMode was "Append". I found that there is some memory leak (similar to one reported [here](https://github.com/facebook/rocksdb/issues/3216)) I am digging deeper to find the root cause. It might be some rocksdb config tunning or falling back to use an older version of rocksdb. (As @skonto commented earlier, code path for deletion of older states might need refactoring to get best out of rocksdb) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
maropu commented on a change in pull request #25001: [SPARK-28083][SQL] Support
LIKE ... ESCAPE syntax
URL: https://github.com/apache/spark/pull/25001#discussion_r302360877
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
##
@@ -103,13 +103,16 @@ abstract class StringRegexExpression extends
BinaryExpression
Use RLIKE to match with standard regular expressions.
""",
since = "1.0.0")
-case class Like(left: Expression, right: Expression) extends
StringRegexExpression {
+case class Like(left: Expression, right: Expression, escapeCharOpt:
Option[String] = None)
+ extends StringRegexExpression {
+
+ private lazy val escapeStr = escapeCharOpt.getOrElse("\\")
- override def escape(v: String): String = StringUtils.escapeLikeRegex(v)
+ override def escape(v: String): String = StringUtils.escapeLikeRegex(v,
escapeStr)
override def matches(regex: Pattern, str: String): Boolean =
regex.matcher(str).matches()
- override def toString: String = s"$left LIKE $right"
+ override def toString: String = s"$left LIKE $right ESCAPE $escapeStr"
Review comment:
This is an optional feature, so I think its better to hide this if
escapeCharOpt = None.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510325348 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510325350 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107495/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510325350 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107495/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510325348 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
