[
https://issues.apache.org/jira/browse/SPARK-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
RaviShankar KS updated SPARK-10967:
-----------------------------------
Description:
We notice that the join conditions are not working as expected in the case of
nested columns being compared.
Consider below example for two data frames d5 and d5_opp :
d5.printSchema
root
|-- key: integer (nullable = false)
|-- value: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- col1: string (nullable = true)
| | |-- col2: string (nullable = true)
|-- value1: struct (nullable = false)
| |-- col1: string (nullable = false)
| |-- col2: string (nullable = false)
d5_opp.printSchema
root
|-- key: integer (nullable = false)
|-- value: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- col2: string (nullable = true)
| | |-- col1: string (nullable = true)
|-- value1: struct (nullable = false)
| |-- col2: string (nullable = false)
| |-- col1: string (nullable = false)
> Incorrect Join behavior in filter conditions
> --------------------------------------------
>
> Key: SPARK-10967
> URL: https://issues.apache.org/jira/browse/SPARK-10967
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 1.4.1
> Environment: Ubuntu on AWS
> Reporter: RaviShankar KS
> Assignee: Josh Rosen
> Labels: sql, union
> Fix For: 1.5.0
>
>
> We notice that the join conditions are not working as expected in the case of
> nested columns being compared.
> Consider below example for two data frames d5 and d5_opp :
> d5.printSchema
> root
> |-- key: integer (nullable = false)
> |-- value: array (nullable = true)
> | |-- element: struct (containsNull = true)
> | | |-- col1: string (nullable = true)
> | | |-- col2: string (nullable = true)
> |-- value1: struct (nullable = false)
> | |-- col1: string (nullable = false)
> | |-- col2: string (nullable = false)
> d5_opp.printSchema
> root
> |-- key: integer (nullable = false)
> |-- value: array (nullable = true)
> | |-- element: struct (containsNull = true)
> | | |-- col2: string (nullable = true)
> | | |-- col1: string (nullable = true)
> |-- value1: struct (nullable = false)
> | |-- col2: string (nullable = false)
> | |-- col1: string (nullable = false)
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
