Repository: spark
Updated Branches:
refs/heads/master f341e1c8a -> bbdf1de84
[SPARK-3371][SQL] Renaming a function expression with group by gives error
The following code gives error.
```
sqlContext.registerFunction("len", (s: String) => s.length)
sqlContext.sql("select len(foo) as a, count(1) from t1 group by
len(foo)").collect()
```
Because SQl parser creates the aliases to the functions in grouping expressions
with generated alias names. So if user gives the alias names to the functions
inside projection then it does not match the generated alias name of grouping
expression.
This kind of queries are working in Hive.
So the fix I have given that if user provides alias to the function in
projection then don't generate alias in grouping expression,use the same alias.
Author: ravipesala <[email protected]>
Closes #2511 from ravipesala/SPARK-3371 and squashes the following commits:
9fb973f [ravipesala] Removed aliases to grouping expressions.
f8ace79 [ravipesala] Fixed the testcase issue
bad2fd0 [ravipesala] SPARK-3371 : Fixed Renaming a function expression with
group by gives error
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bbdf1de8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bbdf1de8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bbdf1de8
Branch: refs/heads/master
Commit: bbdf1de84ffdd3bd172f17975d2f1422a9bcf2c6
Parents: f341e1c
Author: ravipesala <[email protected]>
Authored: Wed Oct 1 23:53:21 2014 -0700
Committer: Michael Armbrust <[email protected]>
Committed: Wed Oct 1 23:53:21 2014 -0700
----------------------------------------------------------------------
.../main/scala/org/apache/spark/sql/catalyst/SqlParser.scala | 2 +-
.../src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala | 5 +++++
2 files changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/bbdf1de8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
----------------------------------------------------------------------
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
index 862f787..2633633 100755
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
@@ -166,7 +166,7 @@ class SqlParser extends StandardTokenParsers with
PackratParsers {
val withFilter = f.map(f => Filter(f, base)).getOrElse(base)
val withProjection =
g.map {g =>
- Aggregate(assignAliases(g), assignAliases(p), withFilter)
+ Aggregate(g, assignAliases(p), withFilter)
}.getOrElse(Project(assignAliases(p), withFilter))
val withDistinct = d.map(_ =>
Distinct(withProjection)).getOrElse(withProjection)
val withHaving = h.map(h => Filter(h,
withDistinct)).getOrElse(withDistinct)
http://git-wip-us.apache.org/repos/asf/spark/blob/bbdf1de8/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
index fdf3a22..6fb6cb8 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
@@ -680,4 +680,9 @@ class SQLQuerySuite extends QueryTest with
BeforeAndAfterAll {
sql("SELECT CAST(TRUE AS STRING), CAST(FALSE AS STRING) FROM testData
LIMIT 1"),
("true", "false") :: Nil)
}
+
+ test("SPARK-3371 Renaming a function expression with group by gives error") {
+ registerFunction("len", (s: String) => s.length)
+ checkAnswer(
+ sql("SELECT len(value) as temp FROM testData WHERE key = 1 group by
len(value)"), 1)}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
