[ 
https://issues.apache.org/jira/browse/SPARK-61615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145597#comment-18145597
 ] 

Shivadarshan edited comment on SPARK-61615 at 11/16/26, 12:28 PM:
------------------------------------------------------------------

Hi [~sarutak], are you planning to work on this one? If not, I'd be happy to 
take it. I can reproduce it on master (63c95a05ab7) with a correlated subquery 
in the FILTER that refers to the scalar subquery's own table:
{code:sql}
CREATE OR REPLACE TEMPORARY VIEW t1(a, b) AS VALUES (1, 10), (2, 20), (3, 30);
CREATE OR REPLACE TEMPORARY VIEW t2(a, b, c) AS VALUES (1, 100, 1), (1, 200, 
2), (2, 300, 3);
CREATE OR REPLACE TEMPORARY VIEW t3(c, d) AS VALUES (1, 5), (3, 7), (10, 1);

SELECT a,
  (SELECT count(*) FILTER (WHERE EXISTS (SELECT 1 FROM t3 WHERE t3.c = t2.c))
   FROM t2 WHERE t2.a = t1.a) AS cnt
FROM t1;
-- [INTERNAL_ERROR] Cannot generate code for expression: outer(c#x)
{code}
The same happens with a correlated IN ({{{}t2.c IN (SELECT c FROM t3 WHERE t3.d 
< t2.b){}}}, {{{}Cannot generate code for expression: outer(b#x){}}}). A 
correlated scalar subquery in the FILTER fails in the optimizer with 
INTERNAL_ERROR. The same FILTER outside a correlated scalar subquery works, and 
so does an uncorrelated IN in the FILTER.


was (Author: JIRAUSER314571):
Hi [~sarutak] , are you planning to work on this one? If not, I'd be happy to 
take it. Could you share the query that hits the INTERNAL_ERROR, or is it the 
FILTER (WHERE EXISTS/IN ...) case inside an aggregate in a correlated scalar 
subquery?

> Correlated subquery in the FILTER clause of an aggregate function in a 
> correlated scalar subquery fails with INTERNAL_ERROR
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61615
>                 URL: https://issues.apache.org/jira/browse/SPARK-61615
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.1.0, 4.0.0, 4.2.0, 4.3.0, 4.4.0
>            Reporter: Kousuke Saruta
>            Priority: Major
>




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to