Hi Keith & gorenuru,
This patch (https://github.com/apache/spark/pull/1423) solves the
errors for me in my local tests. If possible, can you guys test this
out to see if it solves your test programs?
Thanks,
Zongheng
On Tue, Jul 15, 2014 at 3:08 PM, Zongheng Yang <zonghen...@gmail.com> wrote:
> - user@incubator
>
> Hi Keith,
>
> I did reproduce this using local-cluster[2,2,1024], and the errors
> look almost the same. Just wondering, despite the errors did your
> program output any result for the join? On my machine, I could see the
> correct output.
>
> Zongheng
>
> On Tue, Jul 15, 2014 at 1:46 PM, Michael Armbrust
> <mich...@databricks.com> wrote:
>> Thanks for the extra info. At a quick glance the query plan looks fine to
>> me. The class IntegerType does build a type tag.... I wonder if you are
>> seeing the Scala issue manifest in some new way. We will attempt to
>> reproduce locally.
>>
>>
>> On Tue, Jul 15, 2014 at 1:41 PM, gorenuru <goren...@gmail.com> wrote:
>>>
>>> Just my "few cents" on this.
>>>
>>> I having the same problems with v 1.0.1 but this bug is sporadic and looks
>>> like is relayed to object initialization.
>>>
>>> Even more, i'm not using any SQL or something. I just have utility class
>>> like this:
>>>
>>> object DataTypeDescriptor {
>>> type DataType = String
>>>
>>> val BOOLEAN = "BOOLEAN"
>>> val STRING = "STRING"
>>> val TIMESTAMP = "TIMESTAMP"
>>> val LONG = "LONG"
>>> val INT = "INT"
>>> val SHORT = "SHORT"
>>> val BYTE = "BYTE"
>>> val DECIMAL = "DECIMAL"
>>> val DOUBLE = "DOUBLE"
>>> val FLOAT = "FLOAT"
>>>
>>> def $$(name: String, format: Option[String] = None) =
>>> DataTypeDescriptor(name, format)
>>>
>>> private lazy val nativeTypes: Map[String, NativeType] = Map(
>>> BOOLEAN -> BooleanType, STRING -> StringType, TIMESTAMP ->
>>> TimestampType, LONG -> LongType, INT -> IntegerType,
>>> SHORT -> ShortType, BYTE -> ByteType, DECIMAL -> DecimalType, DOUBLE
>>> ->
>>> DoubleType, FLOAT -> FloatType
>>> )
>>>
>>> lazy val defaultValues: Map[String, Any] = Map(
>>> BOOLEAN -> false, STRING -> "", TIMESTAMP -> null, LONG -> 0L, INT ->
>>> 0,
>>> SHORT -> 0.toShort, BYTE -> 0.toByte,
>>> DECIMAL -> BigDecimal(0d), DOUBLE -> 0d, FLOAT -> 0f
>>> )
>>>
>>> def apply(dataType: String): DataTypeDescriptor = {
>>> DataTypeDescriptor(dataType.toUpperCase, None)
>>> }
>>>
>>> def apply(dataType: SparkDataType): DataTypeDescriptor = {
>>> nativeTypes
>>> .find { case (_, descriptor) => descriptor == dataType }
>>> .map { case (name, descriptor) => DataTypeDescriptor(name, None) }
>>> .get
>>> }
>>>
>>> .....
>>>
>>> and some test that check each of this methods.
>>>
>>> The problem is that this test fails randomly with this error.
>>>
>>> P.S.: I did not have this problem in Spark 1.0.0
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Error-while-running-Spark-SQL-join-when-using-Spark-1-0-1-tp9776p9817.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>>
