[
https://issues.apache.org/jira/browse/SPARK-16506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-16506:
---------------------------------
Labels: bug bulk-closed dataframe error join joins sql (was: bug dataframe
error join joins sql)
> Subsequent dataframe join dont work
> -----------------------------------
>
> Key: SPARK-16506
> URL: https://issues.apache.org/jira/browse/SPARK-16506
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.2
> Reporter: Tiago Albineli Motta
> Priority: Minor
> Labels: bug, bulk-closed, dataframe, error, join, joins, sql
>
> Here is the example code:
> {quote}
> import sql.implicits._
> val objs = sc.parallelize(Seq(("1", "um"), ("2", "dois"), ("3",
> "tres"))).toDF.selectExpr("_1 as id", "_2 as name")
>
> val rawj = sc.parallelize(Seq(("1", "2"), ("1", "3"), ("2", "3"),
> ("2", "1"))).toDF.selectExpr("_1 as id1", "_2 as id2")
>
> val join1 = rawj.join(objs, objs("id") === rawj("id1"))
> .withColumnRenamed("id", "anything")
>
> println("works...")
> val join2a = join1.join(objs, 'id2 === 'id )
> join2a.show()
>
> println("works...")
> val join2b = objs.join(join1, objs("id") === join1("id2"))
> join2b.show()
>
> println("do not works...")
> val join2c = join1.join(objs, join1("id2") === objs("id") )
> join2c.show()
> {quote}
> Fisrt two joins work. But the last one gave me this error:
> {quote}
> Exception in thread "main" org.apache.spark.sql.AnalysisException: resolved
> attribute(s) id#2 missing from anything#8,name#14,name#3,id1#6,id2#7,id#13 in
> operator !Join Inner, Some((id2#7 = id#2));
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:38)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:183)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:50)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:105)
> {quote}
> Without the first column rename, the error happens in silence since the join
> get empty:
> {quote}
> import sql.implicits._
> val objs = sc.parallelize(Seq(("1", "um"), ("2", "dois"), ("3",
> "tres"))).toDF.selectExpr("_1 as id", "_2 as name")
>
> val rawj = sc.parallelize(Seq(("1", "2"), ("1", "3"), ("2", "3"),
> ("2", "1"))).toDF.selectExpr("_1 as id1", "_2 as id2")
>
> val join1 = rawj.join(objs, objs("id") === rawj("id1"))
>
> println("do not works...")
> val join2c = join1.join(objs, join1("id2") === objs("id") )
> join2c.show()
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
