Thanks for the extra info. At a quick glance the query plan looks fine to me. The class IntegerType does build a type tag.... I wonder if you are seeing the Scala issue manifest in some new way. We will attempt to reproduce locally.
On Tue, Jul 15, 2014 at 1:41 PM, gorenuru <goren...@gmail.com> wrote: > Just my "few cents" on this. > > I having the same problems with v 1.0.1 but this bug is sporadic and looks > like is relayed to object initialization. > > Even more, i'm not using any SQL or something. I just have utility class > like this: > > object DataTypeDescriptor { > type DataType = String > > val BOOLEAN = "BOOLEAN" > val STRING = "STRING" > val TIMESTAMP = "TIMESTAMP" > val LONG = "LONG" > val INT = "INT" > val SHORT = "SHORT" > val BYTE = "BYTE" > val DECIMAL = "DECIMAL" > val DOUBLE = "DOUBLE" > val FLOAT = "FLOAT" > > def $$(name: String, format: Option[String] = None) = > DataTypeDescriptor(name, format) > > private lazy val nativeTypes: Map[String, NativeType] = Map( > BOOLEAN -> BooleanType, STRING -> StringType, TIMESTAMP -> > TimestampType, LONG -> LongType, INT -> IntegerType, > SHORT -> ShortType, BYTE -> ByteType, DECIMAL -> DecimalType, DOUBLE -> > DoubleType, FLOAT -> FloatType > ) > > lazy val defaultValues: Map[String, Any] = Map( > BOOLEAN -> false, STRING -> "", TIMESTAMP -> null, LONG -> 0L, INT -> > 0, > SHORT -> 0.toShort, BYTE -> 0.toByte, > DECIMAL -> BigDecimal(0d), DOUBLE -> 0d, FLOAT -> 0f > ) > > def apply(dataType: String): DataTypeDescriptor = { > DataTypeDescriptor(dataType.toUpperCase, None) > } > > def apply(dataType: SparkDataType): DataTypeDescriptor = { > nativeTypes > .find { case (_, descriptor) => descriptor == dataType } > .map { case (name, descriptor) => DataTypeDescriptor(name, None) } > .get > } > > ..... > > and some test that check each of this methods. > > The problem is that this test fails randomly with this error. > > P.S.: I did not have this problem in Spark 1.0.0 > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Error-while-running-Spark-SQL-join-when-using-Spark-1-0-1-tp9776p9817.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >