[
https://issues.apache.org/jira/browse/SPARK-49743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikhil Sheoran updated SPARK-49743:
-----------------------------------
Description:
The `OptimizeCsvJsonExprs` rule can potentially change the schema of the
underlying `StructField` if there are differences in the field used to access
the struct vs the field in the underlying struct.
This surfaces as a correctness issue where instead of picking the values for
the corresponding column we end up returning NULL.
A simple example query is:
{code:java}
SELECT
from_json('[{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT, b:
INT>>').a,
from_json('[{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT, b:
INT>>').A
FROM
range(3) as t{code}
Here, the result is `[0], [1], [2]` for `a` but `[null], [null], [null]` for
`A`. Since struct field accessor is case-insensitive, the result should had
been `[0], [1], [2]` for both.
was:
The `OptimizeCsvJsonExprs` rule can potentially change the schema of the
underlying `StructField` if there are differences in the field used to access
the struct vs the field in the underlying struct.
This surfaces as a correctness issue where instead of picking the values for
the corresponding column we end up returning NULL.
A simple example query is:
{code:java}
SELECT
from_json('[\{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT, b:
INT>>').a,
from_json('[\{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT, b:
INT>>').A
FROM
range(3) as t{code}
Here, the result is `[0], [1], [2]` for `a` but `[null], [null], [null]` for
`A`. Since struct field accessor is case-insensitive, the result should had
been `[0], [1], [2]` for both.
> OptimizeCsvJsonExpr should not change the schema of underlying StructType in
> GetArrayStructFields
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-49743
> URL: https://issues.apache.org/jira/browse/SPARK-49743
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.2
> Reporter: Nikhil Sheoran
> Priority: Major
> Labels: pull-request-available
>
> The `OptimizeCsvJsonExprs` rule can potentially change the schema of the
> underlying `StructField` if there are differences in the field used to access
> the struct vs the field in the underlying struct.
> This surfaces as a correctness issue where instead of picking the values for
> the corresponding column we end up returning NULL.
>
> A simple example query is:
> {code:java}
> SELECT
> from_json('[{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT,
> b: INT>>').a,
> from_json('[{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT,
> b: INT>>').A
> FROM
> range(3) as t{code}
>
>
> Here, the result is `[0], [1], [2]` for `a` but `[null], [null], [null]` for
> `A`. Since struct field accessor is case-insensitive, the result should had
> been `[0], [1], [2]` for both.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
