[jira] [Updated] (SPARK-29145) Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-10-23 Thread Dongjoon Hyun (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-29145:
--
Affects Version/s: 2.1.3

> Spark SQL cannot handle "NOT IN" condition when using "JOIN"
> 
>
> Key: SPARK-29145
> URL: https://issues.apache.org/jira/browse/SPARK-29145
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.3, 2.2.3, 2.3.4, 2.4.4
>Reporter: Dezhi Cai
>Priority: Minor
>
> sample sql: 
> {code}
> spark.range(10).createOrReplaceTempView("A")
> spark.range(10).createOrReplaceTempView("B")
> spark.range(10).createOrReplaceTempView("C")
> sql("""select * from A inner join B on A.id=B.id and A.id NOT IN (select id 
> from C)""")
> {code}
>  
> {code}
> org.apache.spark.sql.AnalysisException: Table or view not found: C; line 1 
> pos 74;
> 'Project [*]
> +- 'Join Inner, ((id#0L = id#2L) AND NOT id#0L IN (list#6 []))
>:  +- 'Project ['id]
>: +- 'UnresolvedRelation [C]
>:- SubqueryAlias `a`
>:  +- Range (0, 10, step=1, splits=Some(12))
>+- SubqueryAlias `b`
>   +- Range (0, 10, step=1, splits=Some(12))
>   at 
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:94)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:89)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:155)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:154)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:154)
>   at scala.collection.immutable.List.foreach(List.scala:392)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:154)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:89)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis$(CheckAnalysis.scala:86)
>   at 
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:120)
> ...
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29145) Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-10-23 Thread Dongjoon Hyun (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-29145:
--
Affects Version/s: 2.2.3

> Spark SQL cannot handle "NOT IN" condition when using "JOIN"
> 
>
> Key: SPARK-29145
> URL: https://issues.apache.org/jira/browse/SPARK-29145
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.2.3, 2.3.4, 2.4.4
>Reporter: Dezhi Cai
>Priority: Minor
>
> sample sql: 
> {code}
> spark.range(10).createOrReplaceTempView("A")
> spark.range(10).createOrReplaceTempView("B")
> spark.range(10).createOrReplaceTempView("C")
> sql("""select * from A inner join B on A.id=B.id and A.id NOT IN (select id 
> from C)""")
> {code}
>  
> {code}
> org.apache.spark.sql.AnalysisException: Table or view not found: C; line 1 
> pos 74;
> 'Project [*]
> +- 'Join Inner, ((id#0L = id#2L) AND NOT id#0L IN (list#6 []))
>:  +- 'Project ['id]
>: +- 'UnresolvedRelation [C]
>:- SubqueryAlias `a`
>:  +- Range (0, 10, step=1, splits=Some(12))
>+- SubqueryAlias `b`
>   +- Range (0, 10, step=1, splits=Some(12))
>   at 
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:94)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:89)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:155)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:154)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:154)
>   at scala.collection.immutable.List.foreach(List.scala:392)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:154)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:89)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis$(CheckAnalysis.scala:86)
>   at 
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:120)
> ...
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29145) Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-10-22 Thread Dongjoon Hyun (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun updated SPARK-29145:
--
Affects Version/s: 2.3.4

> Spark SQL cannot handle "NOT IN" condition when using "JOIN"
> 
>
> Key: SPARK-29145
> URL: https://issues.apache.org/jira/browse/SPARK-29145
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.4, 2.4.4
>Reporter: Dezhi Cai
>Priority: Minor
>
> sample sql: 
> {code}
> spark.range(10).createOrReplaceTempView("A")
> spark.range(10).createOrReplaceTempView("B")
> spark.range(10).createOrReplaceTempView("C")
> sql("""select * from A inner join B on A.id=B.id and A.id NOT IN (select id 
> from C)""")
> {code}
>  
> {code}
> org.apache.spark.sql.AnalysisException: Table or view not found: C; line 1 
> pos 74;
> 'Project [*]
> +- 'Join Inner, ((id#0L = id#2L) AND NOT id#0L IN (list#6 []))
>:  +- 'Project ['id]
>: +- 'UnresolvedRelation [C]
>:- SubqueryAlias `a`
>:  +- Range (0, 10, step=1, splits=Some(12))
>+- SubqueryAlias `b`
>   +- Range (0, 10, step=1, splits=Some(12))
>   at 
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:94)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:89)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:155)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:154)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:154)
>   at scala.collection.immutable.List.foreach(List.scala:392)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:154)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:89)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis$(CheckAnalysis.scala:86)
>   at 
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:120)
> ...
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29145) Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-09-18 Thread Hyukjin Kwon (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-29145:
-
Description: 
sample sql: 

{code}
spark.range(10).createOrReplaceTempView("A")
spark.range(10).createOrReplaceTempView("B")
spark.range(10).createOrReplaceTempView("C")
sql("""select * from A inner join B on A.id=B.id and A.id NOT IN (select id 
from C)""")
{code}
 
{code}
org.apache.spark.sql.AnalysisException: Table or view not found: C; line 1 pos 
74;
'Project [*]
+- 'Join Inner, ((id#0L = id#2L) AND NOT id#0L IN (list#6 []))
   :  +- 'Project ['id]
   : +- 'UnresolvedRelation [C]
   :- SubqueryAlias `a`
   :  +- Range (0, 10, step=1, splits=Some(12))
   +- SubqueryAlias `b`
  +- Range (0, 10, step=1, splits=Some(12))

  at 
org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at 
org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:94)
  at 
org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:89)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:155)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:154)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:154)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:154)
  at 
org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:89)
  at 
org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis$(CheckAnalysis.scala:86)
  at 
org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:120)
...
{code}
 

  was:
sample sql: 

select * from A inner join B on A.id=B.id and A.id NOT IN (select id from C)

spark sql throw exception: table or view `C` not found

 

 


> Spark SQL cannot handle "NOT IN" condition when using "JOIN"
> 
>
> Key: SPARK-29145
> URL: https://issues.apache.org/jira/browse/SPARK-29145
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.4.4
>Reporter: Dezhi Cai
>Priority: Minor
>
> sample sql: 
> {code}
> spark.range(10).createOrReplaceTempView("A")
> spark.range(10).createOrReplaceTempView("B")
> spark.range(10).createOrReplaceTempView("C")
> sql("""select * from A inner join B on A.id=B.id and A.id NOT IN (select id 
> from C)""")
> {code}
>  
> {code}
> org.apache.spark.sql.AnalysisException: Table or view not found: C; line 1 
> pos 74;
> 'Project [*]
> +- 'Join Inner, ((id#0L = id#2L) AND NOT id#0L IN (list#6 []))
>:  +- 'Project ['id]
>: +- 'UnresolvedRelation [C]
>:- SubqueryAlias `a`
>:  +- Range (0, 10, step=1, splits=Some(12))
>+- SubqueryAlias `b`
>   +- Range (0, 10, step=1, splits=Some(12))
>   at 
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:94)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:89)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:155)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:154)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:154)
>   at scala.collection.immutable.List.foreach(List.scala:392)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:154)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:89)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis$(CheckAnalysis.scala:86)
>   at 
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:120)
> ...
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org