sepuri sai krishna created SPARK-60326:
------------------------------------------
Summary: ORC filter pushdown drops rows for TIMESTAMP literals
with sub-millisecond digits
Key: SPARK-60326
URL: https://issues.apache.org/jira/browse/SPARK-60326
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.3, 4.2.0
Reporter: sepuri sai krishna
ORC filter pushdown drops rows when a TIMESTAMP filter literal has
sub-millisecond digits. With the default configuration:
{code:sql}
CREATE TABLE t USING orc AS SELECT 1 AS id, TIMESTAMP'2024-06-01
12:00:00.123456' AS ts;
SELECT count(*) FROM t WHERE ts = TIMESTAMP'2024-06-01 12:00:00.123456'; --
0, expected 1
SELECT count(*) FROM t WHERE ts <= TIMESTAMP'2024-06-01 12:00:00.123456'; --
0, expected 1
SELECT count(*) FROM t WHERE ts >= TIMESTAMP'2024-06-01 12:00:00.123456'; -- 1
CREATE TABLE m USING orc AS SELECT id, TIMESTAMP'2024-06-01 12:00:00.123' AS ts
FROM range(10);
SELECT count(*) FROM m WHERE ts != TIMESTAMP'2024-06-01 12:00:00.123456'; --
0, expected 10
SELECT count(*) FROM m WHERE ts < TIMESTAMP'2024-06-01 12:00:00.123001'; --
0, expected 10
SET spark.sql.orc.filterPushdown=false; -- all of the above are then correct
{code}
Same for {{<=>}}, {{<}} and {{IN}}, on 4.0.3, 4.2.0 (official binaries) and
master, with the V1 and V2 readers, the vectorized reader on and off, and
{{spark.sql.orc.impl=hive}}. The nanosecond timestamp types on master
(SPARK-57823) lose rows in the same way.
The {{SearchArgument}} that Spark builds has the exact literal, and the ORC
file statistics are exact too: reading the file with the ORC library and that
{{SearchArgument}} returns the row. But Spark does not give the
{{SearchArgument}} to the reader. {{OrcFileFormat}} and
{{OrcPartitionReaderFactory}} store it in the Hadoop conf with
{{OrcInputFormat.setSearchArgument}}, which serializes it with Kryo, and the
reader decodes it with {{OrcInputFormat.buildOptions}}. Kryo's default
serializer for {{java.sql.Timestamp}} only keeps milliseconds, so the reader
gets {{2024-06-01 12:00:00.123}}:
* {{ts = .123456}} and {{ts <= .123456}} become {{ts = .123}} and {{ts <=
.123}}, which skip a stripe or row group whose minimum is {{.123456}};
* {{ts != .123456}} becomes {{ts != .123}}, which skips a stripe or row group
whose values are all {{.123}}.
{{>}} and {{>=}} only get wider. A literal in the middle of a row group gives
correct results, so it only shows when the literal is within a millisecond of
the minimum or maximum of a row group, stripe or file, e.g. a file of one
value, or a lookup of the first event of a file.
Spark builds the {{SearchArgument}} on the executor right before it creates the
reader, so it can pass it to {{Reader.Options.searchArgument}} directly. With
{{spark.sql.orc.impl=hive}}, Hive's {{OrcInputFormat}} reads the
{{SearchArgument}} from the conf ({{sarg.pushdown}}) with Kryo, so that one has
to be handled differently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
