[
https://issues.apache.org/jira/browse/SPARK-20913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028163#comment-16028163
]
Saurabh commented on SPARK-20913:
---------------------------------
I have 2 dataframes...
df_1 has schema (a,b,c,d)
& df_2 has schema(c,d,e,f,a)
Then I do a join on these 2 dataframes
join_df = df_1.join(df_2,'c')
Now I have join_df with schema (a,b,c,d,d,e,f,a)
So when I do the operation join_df.dropDuplicates() I get Reference 'd' is
ambiguous, could be d#4 or d#5
Pls note renaming with alias option is not possible as my actual dataframe has
over 40columns.
So how should I drop the duplicate columns ??
> Reference is ambiguous exception during dropDuplicates
> ------------------------------------------------------
>
> Key: SPARK-20913
> URL: https://issues.apache.org/jira/browse/SPARK-20913
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.6.0
> Environment: Spark 1.6
> Reporter: Saurabh
> Labels: PySpark
> Original Estimate: 216h
> Remaining Estimate: 216h
>
> I'm getting an error when I try to do .dropDuplicates, its giving u"Reference
> 'xcol14x' is ambiguous exception.
> Changing the alias cannot be a solution as , if I have 40 column in each
> dataframe its merely impossible to change the name of each.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
