[
https://issues.apache.org/jira/browse/SPARK-47029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18010698#comment-18010698
]
Kevin Beyer commented on SPARK-47029:
-------------------------------------
The root cause of this issue is the described in SPARK-527772: default values
definitions on complex types are not passed to the json writer vai the query
attribute metadata.
> ALTER COLUMN DROP DEFAULT test fails with JSON data sources
> -----------------------------------------------------------
>
> Key: SPARK-47029
> URL: https://issues.apache.org/jira/browse/SPARK-47029
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.5.0, 4.0.0
> Reporter: Mark Jarvin
> Priority: Major
>
> Enabling the JSON data source causes a test case to fail:
> {code:java}
> [info] - SPARK-39557 INSERT INTO statements with tables with map defaults ***
> FAILED *** (1 second, 498 milliseconds)
> [info] Results do not match for query:
> [info] Timezone:
> sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
> [info] Timezone Env:
> [info]
> [info] == Parsed Logical Plan ==
> [info] 'UnresolvedRelation [t], [], false
> [info]
> [info] == Analyzed Logical Plan ==
> [info] i: int, s:
> struct<x:array<struct<a:int,b:int>>,y:array<map<string,boolean>>>, t:
> array<map<string,boolean>>
> [info] SubqueryAlias spark_catalog.default.t
> [info] +- Relation spark_catalog.default.t[i#13929,s#13930,t#13931] json
> [info]
> [info] == Optimized Logical Plan ==
> [info] Relation spark_catalog.default.t[i#13929,s#13930,t#13931] json
> [info]
> [info] == Physical Plan ==
> [info] FileScan json spark_catalog.default.t[i#13929,s#13930,t#13931]
> Batched: false, DataFilters: [], Format: JSON, Location: InMemoryFileIndex(1
> paths)[file:/home/mark.jarvin/photon/spark/sql/core/spark-warehouse/org.apach...,
> PartitionFilters: [], PushedFilters: [], ReadSchema:
> struct<i:int,s:struct<x:array<struct<a:int,b:int>>,y:array<map<string,boolean>>>,t:array<map<stri...
> [info]
> [info] == Results ==
> [info]
> [info] == Results ==
> [info] !== Correct Answer - 4 ==
> == Spark Answer - 4 ==
> [info] !struct<>
>
> struct<i:int,s:struct<x:array<struct<a:int,b:int>>,y:array<map<string,boolean>>>,t:array<map<string,boolean>>>
> [info] ![1,[List([1,2]),List(Map(def -> false, jkl -> true))],List(Map(xyz
> -> true))] [1,[ArraySeq([1,2]),ArraySeq(Map(def -> false, jkl ->
> true))],ArraySeq(Map(xyz -> true))]
> [info] ![2,null,List(Map(xyz -> true))]
> [2,[ArraySeq([1,2]),ArraySeq(Map(def -> false, jkl ->
> true))],ArraySeq(Map(xyz -> true))]
> [info] ![3,[List([3,4]),List(Map(mno -> false, pqr -> true))],List(Map(xyz
> -> true))] [3,[ArraySeq([3,4]),ArraySeq(Map(mno -> false, pqr ->
> true))],ArraySeq(Map(xyz -> true))]
> [info] ![4,[List([3,4]),List(Map(mno -> false, pqr -> true))],List(Map(xyz
> -> true))] [4,[ArraySeq([3,4]),ArraySeq(Map(mno -> false, pqr ->
> true))],ArraySeq(Map(xyz -> true))] (QueryTest.scala:267){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
