[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002986#comment-15002986 ] Xiao Li commented on SPARK-11637: - The fix is ready. Will submit a PR soon. > Alias do not work with udf with * parameter > --- > > Key: SPARK-11637 > URL: https://issues.apache.org/jira/browse/SPARK-11637 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.0, 1.5.1, 1.5.2 >Reporter: Pierre Borckmans > > In Spark < 1.5.0, this used to work : > {code:java|title=Spark <1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > res2: org.apache.spark.sql.DataFrame = [x: int] > {code} > From Spark 1.5.0+, it fails: > {code:java|title=Spark>=1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > org.apache.spark.sql.AnalysisException: unresolved operator 'Project > ['hash(*) AS x#1]; > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37) > at > org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:174) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49) > at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103) > ... > {code} > This is not specific to the `hash` udf. It also applies to user defined > functions. > The `*` seems to be the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003483#comment-15003483 ] Apache Spark commented on SPARK-11637: -- User 'gatorsmile' has created a pull request for this issue: https://github.com/apache/spark/pull/9683 > Alias do not work with udf with * parameter > --- > > Key: SPARK-11637 > URL: https://issues.apache.org/jira/browse/SPARK-11637 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.0, 1.5.1, 1.5.2 >Reporter: Pierre Borckmans > > In Spark < 1.5.0, this used to work : > {code:java|title=Spark <1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > res2: org.apache.spark.sql.DataFrame = [x: int] > {code} > From Spark 1.5.0+, it fails: > {code:java|title=Spark>=1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > org.apache.spark.sql.AnalysisException: unresolved operator 'Project > ['hash(*) AS x#1]; > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37) > at > org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:174) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49) > at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103) > ... > {code} > This is not specific to the `hash` udf. It also applies to user defined > functions. > The `*` seems to be the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003536#comment-15003536 ] Xiao Li commented on SPARK-11637: - https://github.com/apache/spark/pull/9343 has already fixed the problem. You can verify your issue using the latest master build or the nearly released 1.6 build. > Alias do not work with udf with * parameter > --- > > Key: SPARK-11637 > URL: https://issues.apache.org/jira/browse/SPARK-11637 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.0, 1.5.1, 1.5.2 >Reporter: Pierre Borckmans > > In Spark < 1.5.0, this used to work : > {code:java|title=Spark <1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > res2: org.apache.spark.sql.DataFrame = [x: int] > {code} > From Spark 1.5.0+, it fails: > {code:java|title=Spark>=1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > org.apache.spark.sql.AnalysisException: unresolved operator 'Project > ['hash(*) AS x#1]; > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37) > at > org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:174) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49) > at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103) > ... > {code} > This is not specific to the `hash` udf. It also applies to user defined > functions. > The `*` seems to be the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002460#comment-15002460 ] Xiao Li commented on SPARK-11637: - After using hiveContext, I can reproduce your problem: Exception in thread "main" org.apache.spark.sql.AnalysisException: unresolved operator 'Project ['hash(*) AS x#4]; > Alias do not work with udf with * parameter > --- > > Key: SPARK-11637 > URL: https://issues.apache.org/jira/browse/SPARK-11637 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.0, 1.5.1, 1.5.2 >Reporter: Pierre Borckmans > > In Spark < 1.5.0, this used to work : > {code:java|title=Spark <1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > res2: org.apache.spark.sql.DataFrame = [x: int] > {code} > From Spark 1.5.0+, it fails: > {code:java|title=Spark>=1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > org.apache.spark.sql.AnalysisException: unresolved operator 'Project > ['hash(*) AS x#1]; > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37) > at > org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:174) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49) > at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103) > ... > {code} > This is not specific to the `hash` udf. It also applies to user defined > functions. > The `*` seems to be the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1577#comment-1577 ] Xiao Li commented on SPARK-11637: - In 1.4.1, it works well. > Alias do not work with udf with * parameter > --- > > Key: SPARK-11637 > URL: https://issues.apache.org/jira/browse/SPARK-11637 > Project: Spark > Issue Type: Bug >Affects Versions: 1.5.0, 1.5.1, 1.5.2 >Reporter: Pierre Borckmans > > In Spark < 1.5.0, this used to work : > {code:java|title=Spark <1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > res2: org.apache.spark.sql.DataFrame = [x: int] > {code} > From Spark 1.5.0+, it fails: > {code:java|title=Spark>=1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > org.apache.spark.sql.AnalysisException: unresolved operator 'Project > ['hash(*) AS x#1]; > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37) > at > org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:174) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49) > at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103) > ... > {code} > This is not specific to the `hash` udf. It also applies to user defined > functions. > The `*` seems to be the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1522#comment-1522 ] Xiao Li commented on SPARK-11637: - In 1.5.1, the output of your query is: ''' Exception in thread "main" org.apache.spark.sql.AnalysisException: invalid expression hash(*); ''' I will try to investigate this issue. > Alias do not work with udf with * parameter > --- > > Key: SPARK-11637 > URL: https://issues.apache.org/jira/browse/SPARK-11637 > Project: Spark > Issue Type: Bug >Affects Versions: 1.5.0, 1.5.1, 1.5.2 >Reporter: Pierre Borckmans > > In Spark < 1.5.0, this used to work : > {code:java|title=Spark <1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > res2: org.apache.spark.sql.DataFrame = [x: int] > {code} > From Spark 1.5.0+, it fails: > {code:java|title=Spark>=1.5.0|borderStyle=solid} > scala> sqlContext.sql("select hash(*) as x from T") > org.apache.spark.sql.AnalysisException: unresolved operator 'Project > ['hash(*) AS x#1]; > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37) > at > org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:174) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49) > at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103) > ... > {code} > This is not specific to the `hash` udf. It also applies to user defined > functions. > The `*` seems to be the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org