[
https://issues.apache.org/jira/browse/SPARK-21691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124532#comment-16124532
]
Anton Okolnychyi commented on SPARK-21691:
------------------------------------------
The issue is related to `Project \[\*\]` but not to `Limit`. You will always
have this exception if you have a non-top level `Project \[\*\]` in your
logical plan. For instance, the following query will also produce the same
exception:
{noformat}
spark.sql("select * from (select * from (values 0, 1)) as
v").queryExecution.logical.canonicalized
{noformat}
In the failing example from the ticket description, the non-canonicalized
logical plan looks like:
{noformat}
'GlobalLimit 1
+- 'LocalLimit 1
+- 'Project [*]
+- 'SubqueryAlias __auto_generated_subquery_name
+- 'UnresolvedInlineTable [col1], [List(0), List(1)]
{noformat}
Once Spark tries to canonicalize it and processes `LocalLimit 1`, it will get
all attributes by calling `children.flatMap(_.output)`, which triggers the
problem. `Project#output` will try to convert its project list to attributes,
which will fail for `UnresolvedStar` with the aforementioned exception.
I see that `UnresolvedRelation` and `UnresolvedInlineTable` return Nil as
output. Therefore, one option to fix this problem is to return `Nil` as output
from `Project` if it is unresolved.
{noformat}
override def output: Seq[Attribute] = if (resolved)
projectList.map(_.toAttribute) else Nil
{noformat}
I can fix it once we agree on a solution.
> Accessing canonicalized plan for query with limit throws exception
> ------------------------------------------------------------------
>
> Key: SPARK-21691
> URL: https://issues.apache.org/jira/browse/SPARK-21691
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Bjoern Toldbod
>
> Accessing the logical, canonicalized plan fails for queries with limits.
> The following demonstrates the issue:
> {code:java}
> val session = SparkSession.builder.master("local").getOrCreate()
> // This works
> session.sql("select * from (values 0,
> 1)").queryExecution.logical.canonicalized
> // This fails
> session.sql("select * from (values 0, 1) limit
> 1").queryExecution.logical.canonicalized
> {code}
> The message in the thrown exception is somewhat confusing (or at least not
> directly related to the limit):
> "Invalid call to toAttribute on unresolved object, tree: *"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
