Github user olarayej commented on the pull request:
https://github.com/apache/spark/pull/11336#issuecomment-190867380
SparkR doesn't support operations between columns from different DataFrame
objects. Yet you can do:
```
c1 <- df1$c1
c2 <- df2$c2
c3 < - c1 + c2
```
c3 can't be used at all. See examples below:
```
## Create two DataFrames from Iris
> irisDF <- createDataFrame(sqlContext, iris)
> irisDF2 <- createDataFrame(sqlContext, iris)
## Create Column x, adding two Columns in two DataFrame's
> x <- irisDF$Sepal_Length + irisDF2$Sepal_Length
## You can't use Column x as a predicate
> irisDF[x > 0, ]
16/03/01 11:04:19 ERROR RBackendHandler: filter on 76 failed
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
org.apache.spark.sql.AnalysisException: resolved attribute(s)
Sepal_Length#20 missing from
Sepal_Length#15,Petal_Width#18,Sepal_Width#16,Petal_Length#17,Species#19 in
operator !Filter ((Sepal_Length#15 + Sepal_Length#20) > 0.0);
## You can't select Column x either
> select(irisDF, x)
16/03/01 11:04:43 ERROR RBackendHandler: select on 76 failed
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
org.apache.spark.sql.AnalysisException: resolved attribute(s)
Sepal_Length#20 missing from
Sepal_Length#15,Petal_Width#18,Sepal_Width#16,Petal_Length#17,Species#19 in
operator !Project [(Sepal_Length#15 + Sepal_Length#20) AS (Sepal_Length +
Sepal_Length)#25];
> select(irisDF2, x)
16/03/01 11:04:45 ERROR RBackendHandler: select on 91 failed
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
org.apache.spark.sql.AnalysisException: resolved attribute(s)
Sepal_Length#15 missing from
Sepal_Length#20,Sepal_Width#21,Species#24,Petal_Width#23,Petal_Length#22 in
operator !Project [(Sepal_Length#15 + Sepal_Length#20) AS (Sepal_Length +
Sepal_Length)#26];
```
In my opinion, we should throw an error if the user is trying to operate on
Columns coming from different DataFrames.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
