[
https://issues.apache.org/jira/browse/IMPALA-15435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noémi Pap-Takács resolved IMPALA-15435.
---------------------------------------
Resolution: Fixed
> [DOCS] Document turning off block location loading
> (impala.preload-block-locations-for-scheduling)
> --------------------------------------------------------------------------------------------------
>
> Key: IMPALA-15435
> URL: https://issues.apache.org/jira/browse/IMPALA-15435
> Project: IMPALA
> Issue Type: Documentation
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> IMPALA-14138 lets admins turn off block location loading for block-based
> filesystems through Hadoop configuration. Nothing in docs/ mentions it (git
> grep -i preload docs/ is empty). These are core-site.xml keys, not gflags, so
> they don't show up in --help or /varz either; only /hadoop-varz shows the
> value in effect.
> Add a concept — impala_scalability.xml, or next to "Block Location Tracking"
> in impala_config_performance.xml — covering:
> - The keys: impala.preload-block-locations-for-scheduling (all filesystems),
> ...scheme.<scheme> and ...authority.<authority>. The authority is the bare
> URI authority, such as mycluster or nn-host:8020. The default is true, and a
> false at any level wins.
> - Affected schemes: hdfs, ofs, o3fs, alluxio. Object stores never load block
> locations.
> - The keys are read once at JVM startup, so a restart is needed.
> - A warning that catalogd and every coordinator must agree. With catalogd off
> and a coordinator on, files arrive with no blocks and get skipped, so queries
> can silently return too few rows. The opposite mismatch hits a Preconditions
> check.
> - Effects when off: no locality and no short-circuit reads, every scan range
> remote, synthetic splits (Parquet follows PARQUET_OBJECT_STORE_SPLIT_SIZE,
> other formats the filesystem's default block size), the loading pool sized by
> max_nonhdfs_partitions_parallel_load, and much cheaper table loading for
> large tables. Point at the data
> cache for remote reads.
> - When to use it: Impala not co-located with HDFS/Ozone, for example in
> containers.
> - For Iceberg tables in this mode, SHOW FILES shows 'N/A' in the EC Policy
> column (IMPALA-15291), and erasure-coding detection
> (ALLOW_ERASURE_CODED_FILES, the EC scan counters) doesn't apply to those
> files.
> Also update:
> - impala_parquet_object_store_split_size.xml: it says the option controls
> split sizes "on non-block based stores, e.g. S3, ADLS", which is now
> incomplete — it also applies to HDFS and Ozone when preloading is off.
> - impala_config_performance.xml, section block_location_tracking: "Mandatory:
> Block Location Tracking ... Impala will not start unless this setting is
> enabled" is obsolete (a Hadoop 2 claim) and now contradicts a supported
> opt-out. Say that locality only matters when Impala runs next to the
> DataNodes. The same page links to that section
> as "Optional: Block Location Tracking", so the two titles already disagree.
> - Optional: one-line xrefs from impala_ozone.xml and impala_data_cache.xml,
> and a short explanation of the EC Policy column values (NONE, the policy
> name, MIXED, N/A) in the SHOW FILES usage notes in impala_show.xml.
> The default is unchanged, so no incompatible-changes entry is needed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
