Nicholas Chammas created SPARK-23945:

             Summary: Column.isin() should accept a single-column DataFrame as 
                 Key: SPARK-23945
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.3.0
            Reporter: Nicholas Chammas

In SQL you can filter rows based on the result of a subquery:

FROM table1
    SELECT name
    FROM table2
In the Spark DataFrame API, the equivalent would probably look like this:
However, .isin() currently [only accepts a local list of 

I imagine making this enhancement would happen as part of a larger effort to 
support correlated subqueries in the DataFrame API.

Or perhaps there is no plan to support this style of query in the DataFrame 
API, and queries like this should instead be written in a different way? How 
would we write a query like the one I have above in the DataFrame API, without 
needing to collect values locally for the NOT IN filter?


This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to