[ https://issues.apache.org/jira/browse/SPARK-27233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-27233. ---------------------------------- Resolution: Duplicate > Schema of ArrayType change after saveAsTable and read > ------------------------------------------------------ > > Key: SPARK-27233 > URL: https://issues.apache.org/jira/browse/SPARK-27233 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.0 > Environment: Spark 2.4 > Scala 2.11 > Reporter: Kritsada Limpawatkul > Priority: Major > > This is code for reproducing. > {code:java} > val testTable = "testseq" > val df = Seq(Some(Seq(1L,2L,3L)), None).toDF("seq") > sqlContext.sql(s"DROP TABLE IF EXISTS $testTable") > df.write.format("parquet").saveAsTable(testTable) > val res = sqlContext.table(testTable) > assert(df.schema === res.schema){code} > My code is trying to save dataframe with array type and read from that table. > After checking schema, I found that schema has change. > This is the assertion message. > {code:java} > Expected :StructType(StructField(seq,ArrayType(LongType,true),true)) > Actual :StructType(StructField(seq,ArrayType(LongType,false),true)){code} > containsNull in ArrayType changes from false to true after reading from the > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org