Max Gekk created SPARK-59571:
--------------------------------
Summary: Cover every TIME precision, null pattern and interval
sign in ArrowCachedBatchSerializerSuite
Key: SPARK-59571
URL: https://issues.apache.org/jira/browse/SPARK-59571
Project: Spark
Issue Type: Sub-task
Components: SQL, Tests
Affects Versions: 4.3.0
Reporter: Max Gekk
h3. What
Add two tests to \{{ArrowCachedBatchSerializerSuite}} for the Arrow cache
serializer (SPARK-57268) over the \{{TIME}} and \{{INTERVAL DAY TO SECOND}}
types:
* \{{TIME(p)}} for p in 0, 3, 6 and 9, each at three null patterns (every 31st
row null, no nulls, all nulls), a thousand rows in one partition. Each case
checks that the cached frame answers what the uncached one did, that the cached
\{{InMemoryRelation}} keeps \{{TimeType(p)}} with the declared precision, and
that the serializer's own columnar read path yields a \{{TimeNanoVector}} per
batch whose row and null counts add up to the input's. The values are truncated
to the declared precision so a precision loss could not hide behind a value the
type would round anyway.
* \{{INTERVAL DAY TO SECOND}} with whole microseconds of both signs, at the
same three null patterns, checking the same three things against a
\{{DurationVector}}.
h3. Why
The suite today round-trips \{{TIME}} once, at precision 6, over two non-null
rows, and checks its \{{LongColumnStats}}. Every precision of \{{TIME}} is
written to the same \{{TimeNanoVector}}, and the precision travels only in the
Arrow field metadata, so a regression that dropped it on the way back through
the cache would leave the values intact and fail no existing test. The all-null
column takes a branch of its own in the vector (no validity buffer), and
negative intervals are not covered at all. \{{TIME}} is about to be enabled by
default, and the Arrow cache is one of the paths it takes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
