[
https://issues.apache.org/jira/browse/SPARK-61500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar Yu updated SPARK-61500:
------------------------------
Description:
ColumnarArray.get() and ColumnarBatchRow.get() do not consistently honor null
flags when reading values from column vectors.
For primitive-backed types, writing a value such as 42 into a slot and then
marking it null can cause get() to return 42 even though isNullAt() returns
true. For NullType vectors, both getters throw an unsupported data type
exception instead of returning null.
Enable null handling in ColumnarArray.get() and check isNullAt() before type
dispatch in ColumnarBatchRow.get(). This makes their null handling consistent
with UnsafeArrayData.get(), UnsafeRow.get(), and ColumnarRow.get().
SPARK-53434 addressed the same issue in ColumnarRow.get().
was:
ColumnarArray.get(ordinal, dataType) does not check whether the element is
null before reading its value. For primitive-backed types, it can return
the underlying stored value even when isNullAt(ordinal) returns true.
For example:
{code:scala}
val vector = new OnHeapColumnVector(1, IntegerType)
try {
vector.putInt(0, 42)
vector.putNull(0)
val array = new ColumnarArray(vector, 0, 1)
assert(array.isNullAt(0))
array.get(0, IntegerType) // Returns 42; expected null.
} finally {
vector.close()
}
{code}
The same issue affects OffHeapColumnVector.
Additionally, get(ordinal, NullType) throws an unsupported-data-type
exception instead of returning null.
Enable null handling in the SpecializedGettersReader.read() call used by
ColumnarArray.get(), consistent with UnsafeArrayData.get() and
UnsafeRow.get().
> Handle null values in ColumnarArray.get() and ColumnarBatchRow.get()
> --------------------------------------------------------------------
>
> Key: SPARK-61500
> URL: https://issues.apache.org/jira/browse/SPARK-61500
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.4
> Reporter: Wechar Yu
> Priority: Major
>
> ColumnarArray.get() and ColumnarBatchRow.get() do not consistently honor null
> flags when reading values from column vectors.
> For primitive-backed types, writing a value such as 42 into a slot and then
> marking it null can cause get() to return 42 even though isNullAt() returns
> true. For NullType vectors, both getters throw an unsupported data type
> exception instead of returning null.
> Enable null handling in ColumnarArray.get() and check isNullAt() before type
> dispatch in ColumnarBatchRow.get(). This makes their null handling consistent
> with UnsafeArrayData.get(), UnsafeRow.get(), and ColumnarRow.get().
> SPARK-53434 addressed the same issue in ColumnarRow.get().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]