Tim Armstrong created IMPALA-7360:
-------------------------------------
Summary: Avro scanner sometimes skips blocks when skip marker is
on HDFS block boundary
Key: IMPALA-7360
URL: https://issues.apache.org/jira/browse/IMPALA-7360
Project: IMPALA
Issue Type: Bug
Components: Backed
Affects Versions: Impala 2.12.0, Impala 3.0, Impala 2.11.0, Impala 2.10.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong
The Avro changes in IMPALA-3905 introduced a correctness bug. You can hit it
organically if you have a large avro file where the 16 byte sync marker
straddles a block boundary. In that case the block after the sync marker may
not be scanned, resulting in a few records missing.
It's possible to reproduce on our test data by tweaking max_scan_range_length
until you find a value where count(*) returns fewer results.
{code}
[localhost:21000] default> set max_scan_range_length=256k; select count(*) from
tpch_avro_snap.lineitem;
MAX_SCAN_RANGE_LENGTH set to 256k
Query: select count(*) from tpch_avro_snap.lineitem
Query submitted at: 2018-07-26 10:08:21 (Coordinator:
http://tarmstrong-box:25000)
Query progress can be monitored at:
http://tarmstrong-box:25000/query_plan?query_id=5142ec7a702e67ac:b6882a6f00000000
+----------+
| count(*) |
+----------+
| 6001215 |
+----------+
Fetched 1 row(s) in 6.77s
[localhost:21000] default> set max_scan_range_length=255k; select count(*) from
tpch_avro_snap.lineitem;
MAX_SCAN_RANGE_LENGTH set to 255k
Query: select count(*) from tpch_avro_snap.lineitem
Query submitted at: 2018-07-26 10:08:31 (Coordinator:
http://tarmstrong-box:25000)
Query progress can be monitored at:
http://tarmstrong-box:25000/query_plan?query_id=3d40e63dacaac65b:99d17eaf00000000
+----------+
| count(*) |
+----------+
| 6000679 |
+----------+
Fetched 1 row(s) in 1.33s
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
