Re: [PySPark] How to check if value of one column is in array of another column

[Sean Owen]

I think you want array_contains:
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.array_contains.html

On Tue, Jan 17, 2023 at 4:18 PM Oliver Ruebenacker <
oliv...@broadinstitute.org> wrote:

>
>  Hello,
>
>   I have data originally stored as JSON. Column gene contains a string,
> column nearest an array of strings. How can I check whether the value of
> gene is an element of the array of nearest?
>
>   I tried: genes_joined.gene.isin(genes_joined.nearest)
>
>   But I get an error that says:
>
> pyspark.sql.utils.AnalysisException: cannot resolve '(gene IN (nearest))'
> due to data type mismatch: Arguments must be same type but were: string !=
> array;
>
>   How do I do this? Thanks!
>
>  Best, Oliver
>
> --
> Oliver Ruebenacker, Ph.D. (he)
> Senior Software Engineer, Knowledge Portal Network , 
> Flannick
> Lab , Broad Institute
> 
>

[PySPark] How to check if value of one column is in array of another column

[Oliver Ruebenacker]

 Hello,

  I have data originally stored as JSON. Column gene contains a string,
column nearest an array of strings. How can I check whether the value of
gene is an element of the array of nearest?

  I tried: genes_joined.gene.isin(genes_joined.nearest)

  But I get an error that says:

pyspark.sql.utils.AnalysisException: cannot resolve '(gene IN (nearest))'
due to data type mismatch: Arguments must be same type but were: string !=
array;

  How do I do this? Thanks!

 Best, Oliver

-- 
Oliver Ruebenacker, Ph.D. (he)
Senior Software Engineer, Knowledge Portal Network
, Flannick
Lab , Broad Institute