Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/17105
@felixcheung Ahh, it seems that we have some conflicting design issues.
1. From the test in collect() and crossJoin, it seems to allow dup names in
SparkDataFrame by design:
```
# collect() correctly handles multiple columns with same name
df <- createDataFrame(list(list(1, 2)), schema = c("name", "name"))
ldf <- collect(df)
expect_equal(names(ldf), c("name", "name"))
```
2. However, it seems that the `mutate` method does not allow dup names:
```
# Check if there is any duplicated column name in the DataFrame
dfCols <- columns(x)
if (length(unique(dfCols)) != length(dfCols)) {
stop("Error: found duplicated column name in the DataFrame")
}
```
Wonder if you know the reasoning for such conflicting design? I think it's
best to not allow dup names as it does not work with Spark SQL (which does not
allow dup names). For example, we can not even extract the columns which will
report error:
```
l <- list(list(1, 2), list(3, 4))
df <- createDataFrame(l, c("a", "a"))
df$a
```
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
