[
https://issues.apache.org/jira/browse/SPARK-16896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413271#comment-15413271
]
Hyukjin Kwon edited comment on SPARK-16896 at 8/9/16 9:47 AM:
--------------------------------------------------------------
[~srowen] Oh, I am sorry, I coundn't see the comment above. Please ignore the
comment I just left. I thought you left the comment for mine.
was (Author: hyukjin.kwon):
Sean, If my understanding is correct, we have tried to resembles {{read.csv}}
in R.
For example, we renamed `deimilter` to `sep` and `charset` to `encoding` to
match up with R in SPARK-12668
Also, assuming from the PR I tried
(https://github.com/apache/spark/pull/11457), I assume it'd be better if both
`read.csv` in R and `read.csv` in Spark are similar.
I didn't want to say we must match but it'd be nicer if we resembles the
behaviour in R. This is because also we have a good reference here in R.
> Loading csv with duplicate column names
> ---------------------------------------
>
> Key: SPARK-16896
> URL: https://issues.apache.org/jira/browse/SPARK-16896
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Aseem Bansal
>
> It would be great if the library allows us to load csv with duplicate column
> names. I understand that having duplicate columns in the data is odd but
> sometimes we get data that has duplicate columns. Getting upstream data like
> that can happen. We may choose to ignore them but currently there is no way
> to drop those as we are not able to load them at all. Currently as a
> pre-processing I loaded the data into R, changed the column names and then
> make a fixed version with which Spark Java API can work.
> But if talk about other options, e.g. R has read.csv which automatically
> takes care of such situation by appending a number to the column name.
> Also case sensitivity in column names can also cause problems. I mean if we
> have columns like
> ColumnName, columnName
> I may want to have them as separate. But the option to do this is not
> documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
