Re: Spark SQL: what does an exclamation mark mean in the plan?
Hi, Michael, Thank you again! Just found the functions that generate the ! mark /** * A prefix string used when printing the plan. * * We use "!" to indicate an invalid plan, and "'" to indicate an unresolved plan. */ protected def statePrefix = if (missingInput.nonEmpty && children.nonEmpty) "!" else "" override def simpleString: String = statePrefix + super.simpleString Xiao Li 2015-10-19 11:16 GMT-07:00 Michael Armbrust : > It means that there is an invalid attribute reference (i.e. a #n where the > attribute is missing from the child operator). > > On Sun, Oct 18, 2015 at 11:38 PM, Xiao Li wrote: > >> Hi, all, >> >> After turning on the trace, I saw a strange exclamation mark in >> the intermediate plans. This happened in catalyst analyzer. >> >> Join Inner, Some((col1#0 = col1#6)) >> Project [col1#0,col2#1,col3#2,col2_alias#24,col3#2 AS col3_alias#13] >> Project [col1#0,col2#1,col3#2,col2#1 AS col2_alias#24] >>LogicalRDD [col1#0,col2#1,col3#2], MapPartitionsRDD[1] at >> createDataFrame at SimpleApp.scala:32 >> Aggregate [col1#6], [col1#6,count(col1#6) AS count(col1)#5L] >> *!Project [col1#6,col2#7,col3#8,col2_alias#24,col3#8 AS col3_alias#4]* >>Project [col1#6,col2#7,col3#8,col2#7 AS col2_alias#3] >> LogicalRDD [col1#6,col2#7,col3#8], MapPartitionsRDD[1] at >> createDataFrame at SimpleApp.scala:32 >> >> Could anybody give me a hint why there exists a !(exclamation mark) >> before the node name (Project)? This ! mark does not disappear in the >> subsequent query plan. >> >> Thank you! >> >> Xiao Li >> > >
Re: Spark SQL: what does an exclamation mark mean in the plan?
It means that there is an invalid attribute reference (i.e. a #n where the attribute is missing from the child operator). On Sun, Oct 18, 2015 at 11:38 PM, Xiao Li wrote: > Hi, all, > > After turning on the trace, I saw a strange exclamation mark in > the intermediate plans. This happened in catalyst analyzer. > > Join Inner, Some((col1#0 = col1#6)) > Project [col1#0,col2#1,col3#2,col2_alias#24,col3#2 AS col3_alias#13] > Project [col1#0,col2#1,col3#2,col2#1 AS col2_alias#24] >LogicalRDD [col1#0,col2#1,col3#2], MapPartitionsRDD[1] at > createDataFrame at SimpleApp.scala:32 > Aggregate [col1#6], [col1#6,count(col1#6) AS count(col1)#5L] > *!Project [col1#6,col2#7,col3#8,col2_alias#24,col3#8 AS col3_alias#4]* >Project [col1#6,col2#7,col3#8,col2#7 AS col2_alias#3] > LogicalRDD [col1#6,col2#7,col3#8], MapPartitionsRDD[1] at > createDataFrame at SimpleApp.scala:32 > > Could anybody give me a hint why there exists a !(exclamation mark) before > the node name (Project)? This ! mark does not disappear in the subsequent > query plan. > > Thank you! > > Xiao Li >
Spark SQL: what does an exclamation mark mean in the plan?
Hi, all, After turning on the trace, I saw a strange exclamation mark in the intermediate plans. This happened in catalyst analyzer. Join Inner, Some((col1#0 = col1#6)) Project [col1#0,col2#1,col3#2,col2_alias#24,col3#2 AS col3_alias#13] Project [col1#0,col2#1,col3#2,col2#1 AS col2_alias#24] LogicalRDD [col1#0,col2#1,col3#2], MapPartitionsRDD[1] at createDataFrame at SimpleApp.scala:32 Aggregate [col1#6], [col1#6,count(col1#6) AS count(col1)#5L] *!Project [col1#6,col2#7,col3#8,col2_alias#24,col3#8 AS col3_alias#4]* Project [col1#6,col2#7,col3#8,col2#7 AS col2_alias#3] LogicalRDD [col1#6,col2#7,col3#8], MapPartitionsRDD[1] at createDataFrame at SimpleApp.scala:32 Could anybody give me a hint why there exists a !(exclamation mark) before the node name (Project)? This ! mark does not disappear in the subsequent query plan. Thank you! Xiao Li