Thanks for the extra info. At a quick glance the query plan looks fine to
me. The class IntegerType does build a type tag.... I wonder if you are
seeing the Scala issue manifest in some new way. We will attempt to
reproduce locally.
On Tue, Jul 15, 2014 at 1:41 PM, gorenuru <goren...@gmail.com> wrote:
> Just my "few cents" on this.
>
> I having the same problems with v 1.0.1 but this bug is sporadic and looks
> like is relayed to object initialization.
>
> Even more, i'm not using any SQL or something. I just have utility class
> like this:
>
> object DataTypeDescriptor {
> type DataType = String
>
> val BOOLEAN = "BOOLEAN"
> val STRING = "STRING"
> val TIMESTAMP = "TIMESTAMP"
> val LONG = "LONG"
> val INT = "INT"
> val SHORT = "SHORT"
> val BYTE = "BYTE"
> val DECIMAL = "DECIMAL"
> val DOUBLE = "DOUBLE"
> val FLOAT = "FLOAT"
>
> def $$(name: String, format: Option[String] = None) =
> DataTypeDescriptor(name, format)
>
> private lazy val nativeTypes: Map[String, NativeType] = Map(
> BOOLEAN -> BooleanType, STRING -> StringType, TIMESTAMP ->
> TimestampType, LONG -> LongType, INT -> IntegerType,
> SHORT -> ShortType, BYTE -> ByteType, DECIMAL -> DecimalType, DOUBLE ->
> DoubleType, FLOAT -> FloatType
> )
>
> lazy val defaultValues: Map[String, Any] = Map(
> BOOLEAN -> false, STRING -> "", TIMESTAMP -> null, LONG -> 0L, INT ->
> 0,
> SHORT -> 0.toShort, BYTE -> 0.toByte,
> DECIMAL -> BigDecimal(0d), DOUBLE -> 0d, FLOAT -> 0f
> )
>
> def apply(dataType: String): DataTypeDescriptor = {
> DataTypeDescriptor(dataType.toUpperCase, None)
> }
>
> def apply(dataType: SparkDataType): DataTypeDescriptor = {
> nativeTypes
> .find { case (_, descriptor) => descriptor == dataType }
> .map { case (name, descriptor) => DataTypeDescriptor(name, None) }
> .get
> }
>
> .....
>
> and some test that check each of this methods.
>
> The problem is that this test fails randomly with this error.
>
> P.S.: I did not have this problem in Spark 1.0.0
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Error-while-running-Spark-SQL-join-when-using-Spark-1-0-1-tp9776p9817.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
