[GitHub] spark issue #17105: [SPARK-19773][SparkR] SparkDataFrame should not allow du...
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/17105 @felixcheung Thanks for the clarification. I will close this then. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17105: [SPARK-19773][SparkR] SparkDataFrame should not allow du...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17105 so tl; dr; I think we should support duplicated name like everything else in Spark does. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17105: [SPARK-19773][SparkR] SparkDataFrame should not allow du...
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17105
@actuaryzhang there's a bit of a history about this... but long story
short, Spark does support DataFrame with multiple columns having the same name,
for example
```
# in pyspark
>>> from pyspark.sql import Row
>>> from pyspark.sql.types import *
>>> data = [(1, 2, 'Foo')]
>>> df = spark.createDataFrame(data, ("key", "key", "value"))
>>> df
DataFrame[key: bigint, key: bigint, value: string]
```
And each column will get a unique id, so underneath the cover they are not
actually "duplicating".
You could in fact get columns with same name when doing a self-join, for
instance.
Now the reason why you are getting an error with `df$a = df$a * 2.0` is
because "a" is not a full unique id. You get the same in python
```
>>> df.select(col("key"))
...
raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u"Reference 'key' is ambiguous, could
be: key#0L, key#1L.;"
```
And in R, `df$a` is essentially a shortcut to that and so it will also fail
similarly.
As for why it is disallowed in `mutate` - it is just a factor of the
implementation. I think we could potentially implement it to support duplicated
names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17105: [SPARK-19773][SparkR] SparkDataFrame should not allow du...
Github user actuaryzhang commented on the issue:
https://github.com/apache/spark/pull/17105
@felixcheung Ahh, it seems that we have some conflicting design issues.
1. From the test in collect() and crossJoin, it seems to allow dup names in
SparkDataFrame by design:
```
# collect() correctly handles multiple columns with same name
df <- createDataFrame(list(list(1, 2)), schema = c("name", "name"))
ldf <- collect(df)
expect_equal(names(ldf), c("name", "name"))
```
2. However, it seems that the `mutate` method does not allow dup names:
```
# Check if there is any duplicated column name in the DataFrame
dfCols <- columns(x)
if (length(unique(dfCols)) != length(dfCols)) {
stop("Error: found duplicated column name in the DataFrame")
}
```
Wonder if you know the reasoning for such conflicting design? I think it's
best to not allow dup names as it does not work with Spark SQL (which does not
allow dup names). For example, we can not even extract the columns which will
report error:
```
l <- list(list(1, 2), list(3, 4))
df <- createDataFrame(l, c("a", "a"))
df$a
```
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17105: [SPARK-19773][SparkR] SparkDataFrame should not allow du...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17105 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73609/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17105: [SPARK-19773][SparkR] SparkDataFrame should not allow du...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17105 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17105: [SPARK-19773][SparkR] SparkDataFrame should not allow du...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17105 **[Test build #73609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73609/testReport)** for PR 17105 at commit [`df61c70`](https://github.com/apache/spark/commit/df61c70dcb2c46e7dcbe3caf2d708692075d05b3). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17105: [SPARK-19773][SparkR] SparkDataFrame should not allow du...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17105 **[Test build #73609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73609/testReport)** for PR 17105 at commit [`df61c70`](https://github.com/apache/spark/commit/df61c70dcb2c46e7dcbe3caf2d708692075d05b3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17105: [SPARK-19773][SparkR] SparkDataFrame should not allow du...
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/17105 @felixcheung --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
