Yun Zhao created SPARK-10838:
--------------------------------
Summary: Repeat to join one DataFrame twice，there will be
AnalysisException.
Key: SPARK-10838
URL: https://issues.apache.org/jira/browse/SPARK-10838
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.4.1
Reporter: Yun Zhao
The detail of exception is:
{quote}
Exception in thread "main" org.apache.spark.sql.AnalysisException: resolved
attribute(s) col_a#1 missing from col_a#0,col_b#2,col_a#3,col_b#4 in operator
!Join Inner, Some((col_b#2 = col_a#1));
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:154)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:49)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:44)
at
org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:908)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:132)
at
org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$logicalPlanToDataFrame(DataFrame.scala:154)
at org.apache.spark.sql.DataFrame.join(DataFrame.scala:554)
at org.apache.spark.sql.DataFrame.join(DataFrame.scala:521)
{quote}
The related codes are：
{quote}
object DFJoinTest extends App {
case class Foo(col_a: String)
case class Bar(col_a: String, col_b: String)
val sc = new SparkContext(new
SparkConf().setMaster("local").setAppName("DFJoinTest"))
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
val df1 = sc.parallelize(Array("1")).map(_.split(",")).map(p =>
Foo(p(0))).toDF()
val df2 = sc.parallelize(Array("1,1")).map(_.split(",")).map(p => Bar(p(0),
p(1))).toDF()
val df3 = df1.join(df2, df1("col_a") === df2("col_a")).select(df1("col_a"),
$"col_b")
df3.join(df2, df3("col_b") === df2("col_a")).show()
// val df4 = df2.as("df4")
// df3.join(df4, df3("col_b") === df4("col_a")).show()
// df3.join(df2.as("df4"), df3("col_b") === $"df4.col_a").show()
sc.stop()
}
{quote}
When uses
{quote}
val df4 = df2.as("df4")
df3.join(df4, df3("col_b") === df4("col_a")).show()
{quote}
there's errors,but when uses
{quote}
df3.join(df2.as("df4"), df3("col_b") === $"df4.col_a").show()
{quote}
it's normal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
