Yaoxuan Wu created FLINK-41359:
----------------------------------
Summary: ORC format: filter push-down on a DOUBLE column drops
rows when the column holds NaN (orc-core 1.5.6, ORC-629)
Key: FLINK-41359
URL: https://issues.apache.org/jira/browse/FLINK-41359
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 2.2.1, 2.3.0
Reporter: Yaoxuan Wu
{code:sql}
SET 'execution.runtime-mode' = 'batch';
SET 'parallelism.default' = '1';
CREATE TABLE o1 (x DOUBLE) WITH ('connector' = 'filesystem', 'path' =
'file:///tmp/o1', 'format' = 'orc');
CREATE TABLE o2 (x DOUBLE) WITH ('connector' = 'filesystem', 'path' =
'file:///tmp/o2', 'format' = 'orc');
INSERT INTO o1 SELECT CAST(s AS DOUBLE) FROM (VALUES ('NaN'), ('6.5'),
('-9.0')) AS v(s);
INSERT INTO o2 SELECT CAST(s AS DOUBLE) FROM (VALUES ('3.0'), ('NaN')) AS v(s);
SELECT x FROM o1; -- NaN, 6.5, -9.0
SELECT x FROM o1 WHERE x <= 1.5e0; -- expected -9.0; actual: no row
SELECT x FROM o1 WHERE x = -9.0e0; -- expected -9.0; actual: no row
SELECT x FROM o2 WHERE x <> 3.0e0; -- expected NaN; actual: no row
{code}
Same output on 2.0.1, 2.2.1, 2.3.0 and 2.4-SNAPSHOT (each with its
flink-sql-orc jar). The same data in {{'format' = 'json'}} returns the expected
rows. Rows are dropped when a filter on a DOUBLE column (FLOAT uses the same
statistics) is pushed into the ORC reader and a stripe of that column holds NaN:
* NaN is the first value of the stripe: its statistics are min = max = NaN, and
range / equality predicates on the column skip the stripe ({{o1}});
* NaN comes later: the statistics leave it out (min = max = 3.0 for {{o2}}), so
{{x <> 3.0}}, pushed as NOT (x = 3.0), skips the NaN row.
Cause: Flink bundles orc-core 1.5.6 ({{orc.version}} in the root pom.xml of
release-2.3.0 and master). {{OrcFilters}} maps DOUBLE / FLOAT to
{{PredicateLeaf.Type.FLOAT}} and {{OrcShimV200.createRecordReader}} passes the
predicates to ORC with {{options.searchArgument(...)}}, so ORC's row-group
pruning decides which rows are read. In ORC 1.5.6,
{{ColumnStatisticsImpl.DoubleStatisticsImpl.updateDouble}} keeps min = max =
NaN when NaN is the first value and never records a later NaN:
{code:java}
if (!hasMinimum) {
hasMinimum = true;
minimum = value;
maximum = value;
} else if (value < minimum) {
minimum = value;
} else if (value > maximum) {
maximum = value;
}
{code}
and {{RecordReaderImpl.evaluatePredicateProto}} evaluates the predicate against
these min / max without a NaN check. This is ORC-629 ("PPD: Floating point NaN
is not transitive across comparisons"), fixed in ORC 1.6.4 / 1.7.0, where
predicate evaluation returns YES_NO(_NULL) when the statistics' sum is NaN; it
was not backported to 1.5.x.
Possible fixes: upgrade ORC (see FLINK-36652 / FLINK-27805 / FLINK-23551), or
do not push predicates on FLOAT / DOUBLE columns into ORC while orc 1.5.x is
used.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
