[
https://issues.apache.org/jira/browse/SPARK-60940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60940:
-----------------------------------
Labels: pull-request-available (was: )
> Vectorized Parquet/ORC readers return wrong values for map column DEFAULTs
> ---------------------------------------------------------------------------
>
> Key: SPARK-60940
> URL: https://issues.apache.org/jira/browse/SPARK-60940
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Hemanth Boyina
> Priority: Major
> Labels: pull-request-available
>
> When a column with a MAP DEFAULT is added to a table with ALTER TABLE ADD
> COLUMN, files written before the column existed do not contain it, so the
> vectorized Parquet and ORC readers fill in the default for every row with
> WritableColumnVector.appendObjects.
>
> repro steps:
>
>
>
>
>
> CREATE TABLE t (id INT) USING PARQUET;
>
>
> INSERT INTO t VALUES (1), (2), (3), (4), (5);
>
>
> ALTER TABLE t ADD COLUMN m MAP<STRING, INT> DEFAULT map('a', 1, 'b', 2);
>
>
> SELECT id, size(m), map_entries(m) FROM t ORDER BY id;
>
>
>
>
>
> Every row should be \{a=1, b=2}, but some rows come back as empty maps.
> The same happens with ORC.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
