Github user dongjoon-hyun commented on the issue:
    @gatorsmile and @HyukjinKwon .
    Two failures are due to limitation of the current JSON data source 
implementation. Here, we can see that the test suite correctly tests the target 
data source.
    1. `resolveRelation for a FileFormat DataSource without userSchema scan 
filesystem only once`
      For Json source, the statistic count becomes 2.
    2. `Pre insert nullability check (MapType)`
      Since Json source save as string, it raises ClassCastException when the 
given user or table schema is different.
    scala> (Tuple1(Map(1 -> (null: Integer))) :: 
     |-- a: struct (nullable = true)
     |    |-- 1: string (nullable = true)
    scala> (Tuple1(Map(1 -> (null: Integer))) :: 
    18/03/02 21:13:49 WARN HiveExternalCatalog: Couldn't find corresponding 
Hive SerDe for data source provider json. Persisting data source table 
`default`.`map` into Hive metastore in Spark SQL specific format, which is NOT 
compatible with Hive.
     |-- a: struct (nullable = true)
     |    |-- 1: string (nullable = true)
    scala> spark.table("map").printSchema
     |-- a: map (nullable = true)
     |    |-- key: integer
     |    |-- value: integer (valueContainsNull = true)
    scala> spark.table("map").show
    18/03/02 21:14:12 ERROR Executor: Exception in task 0.0 in stage 10.0 (TID 
    java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String 
cannot be cast to java.lang.Integer
            at scala.runtime.BoxesRunTime.unboxToInt(
    For JSON format, could you confirm this, @HyukjinKwon ?


To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to