Zoltán Borók-Nagy created IMPALA-15435:
------------------------------------------
Summary: [DOCS] Document turning off block location loading
(impala.preload-block-locations-for-scheduling)
Key: IMPALA-15435
URL: https://issues.apache.org/jira/browse/IMPALA-15435
Project: IMPALA
Issue Type: Documentation
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
IMPALA-14138 lets admins turn off block location loading for block-based
filesystems through Hadoop configuration. Nothing in docs/ mentions it (git
grep -i preload docs/ is empty). These are core-site.xml keys, not gflags, so
they don't show up in --help or /varz either; only /hadoop-varz shows the value
in effect.
Add a concept — impala_scalability.xml, or next to "Block Location Tracking" in
impala_config_performance.xml — covering:
- The keys: impala.preload-block-locations-for-scheduling (all filesystems),
...scheme.<scheme> and ...authority.<authority>. The authority is the bare URI
authority, such as mycluster or nn-host:8020. The default is true, and a false
at any level wins.
- Affected schemes: hdfs, ofs, o3fs, alluxio. Object stores never load block
locations.
- The keys are read once at JVM startup, so a restart is needed.
- A warning that catalogd and every coordinator must agree. With catalogd off
and a coordinator on, files arrive with no blocks and get skipped, so queries
can silently return too few rows. The opposite mismatch hits a Preconditions
check.
- Effects when off: no locality and no short-circuit reads, every scan range
remote, synthetic splits (Parquet follows PARQUET_OBJECT_STORE_SPLIT_SIZE,
other formats the filesystem's default block size), the loading pool sized by
max_nonhdfs_partitions_parallel_load, and much cheaper table loading for large
tables. Point at the data
cache for remote reads.
- When to use it: Impala not co-located with HDFS/Ozone, for example in
containers.
- For Iceberg tables in this mode, SHOW FILES shows 'N/A' in the EC Policy
column (IMPALA-15291), and erasure-coding detection (ALLOW_ERASURE_CODED_FILES,
the EC scan counters) doesn't apply to those files.
Also update:
- impala_parquet_object_store_split_size.xml: it says the option controls split
sizes "on non-block based stores, e.g. S3, ADLS", which is now incomplete — it
also applies to HDFS and Ozone when preloading is off.
- impala_config_performance.xml, section block_location_tracking: "Mandatory:
Block Location Tracking ... Impala will not start unless this setting is
enabled" is obsolete (a Hadoop 2 claim) and now contradicts a supported
opt-out. Say that locality only matters when Impala runs next to the DataNodes.
The same page links to that section
as "Optional: Block Location Tracking", so the two titles already disagree.
- Optional: one-line xrefs from impala_ozone.xml and impala_data_cache.xml, and
a short explanation of the EC Policy column values (NONE, the policy name,
MIXED, N/A) in the SHOW FILES usage notes in impala_show.xml.
The default is unchanged, so no incompatible-changes entry is needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
