[ https://issues.apache.org/jira/browse/SPARK-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
RaviShankar KS updated SPARK-10967: ----------------------------------- Description: We notice that the join conditions are not working as expected in the case of nested columns being compared. Consider below example for two data frames d5 and d5_opp : d5.printSchema root |-- key: integer (nullable = false) |-- value: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- col1: string (nullable = true) | | |-- col2: string (nullable = true) |-- value1: struct (nullable = false) | |-- col1: string (nullable = false) | |-- col2: string (nullable = false) d5_opp.printSchema root |-- key: integer (nullable = false) |-- value: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- col2: string (nullable = true) | | |-- col1: string (nullable = true) |-- value1: struct (nullable = false) | |-- col2: string (nullable = false) | |-- col1: string (nullable = false) > Incorrect Join behavior in filter conditions > -------------------------------------------- > > Key: SPARK-10967 > URL: https://issues.apache.org/jira/browse/SPARK-10967 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL > Affects Versions: 1.4.1 > Environment: Ubuntu on AWS > Reporter: RaviShankar KS > Assignee: Josh Rosen > Labels: sql, union > Fix For: 1.5.0 > > > We notice that the join conditions are not working as expected in the case of > nested columns being compared. > Consider below example for two data frames d5 and d5_opp : > d5.printSchema > root > |-- key: integer (nullable = false) > |-- value: array (nullable = true) > | |-- element: struct (containsNull = true) > | | |-- col1: string (nullable = true) > | | |-- col2: string (nullable = true) > |-- value1: struct (nullable = false) > | |-- col1: string (nullable = false) > | |-- col2: string (nullable = false) > d5_opp.printSchema > root > |-- key: integer (nullable = false) > |-- value: array (nullable = true) > | |-- element: struct (containsNull = true) > | | |-- col2: string (nullable = true) > | | |-- col1: string (nullable = true) > |-- value1: struct (nullable = false) > | |-- col2: string (nullable = false) > | |-- col1: string (nullable = false) > -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org