Anton Okolnychyi commented on SPARK-21691:

The issue is related to `Project \[\*\]` but not to `Limit`. You will always 
have this exception if you have a non-top level `Project \[\*\]` in your 
logical plan. For instance, the following query will also produce the same 

spark.sql("select * from (select * from (values 0, 1)) as 

In the failing example from the ticket description, the non-canonicalized 
logical plan looks like:

'GlobalLimit 1
+- 'LocalLimit 1
   +- 'Project [*]
      +- 'SubqueryAlias __auto_generated_subquery_name
         +- 'UnresolvedInlineTable [col1], [List(0), List(1)]

Once Spark tries to canonicalize it and processes `LocalLimit 1`, it will get 
all attributes by calling `children.flatMap(_.output)`, which triggers the 
problem. `Project#output` will try to convert its project list to attributes, 
which will fail for `UnresolvedStar` with the aforementioned exception.

I see that `UnresolvedRelation` and `UnresolvedInlineTable` return Nil as 
output. Therefore, one option to fix this problem is to return `Nil` as output 
from `Project` if it is unresolved.

override def output: Seq[Attribute] = if (resolved) 
projectList.map(_.toAttribute) else Nil

I can fix it once we agree on a solution.

> Accessing canonicalized plan for query with limit throws exception
> ------------------------------------------------------------------
>                 Key: SPARK-21691
>                 URL: https://issues.apache.org/jira/browse/SPARK-21691
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Bjoern Toldbod
> Accessing the logical, canonicalized plan fails for queries with limits.
> The following demonstrates the issue:
> {code:java}
> val session = SparkSession.builder.master("local").getOrCreate()
> // This works
> session.sql("select * from (values 0, 
> 1)").queryExecution.logical.canonicalized
> // This fails
> session.sql("select * from (values 0, 1) limit 
> 1").queryExecution.logical.canonicalized
> {code}
> The message in the thrown exception is somewhat confusing (or at least not 
> directly related to the limit):
> "Invalid call to toAttribute on unresolved object, tree: *"

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to