[
https://issues.apache.org/jira/browse/IMPALA-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18037963#comment-18037963
]
Sanjana Malhotra commented on IMPALA-14521:
-------------------------------------------
Draft content:
h2.
h2. Limiting File Metadata Fetched in Catalog Requests
To prevent Catalog Service (catalogd) Out-of-Memory (OOM) errors when fetching
metadata for tables with millions of files, the new configuration flag
{{catalog_partial_fetch_max_files}} has been introduced.
This flag limits the maximum number of file descriptors returned in a single
catalog fetch response.
* *Default Value:* 1,000,000 files.
* {*}Behavior{*}: If a request exceeds this limit, Catalogd truncates the
response at the partition level. The Impala coordinator then automatically
sends subsequent requests to fetch the remaining metadata, and it detects any
version changes to force a query replan, ensuring metadata consistency.
> Document for catalog_partial_fetch_max_files
> --------------------------------------------
>
> Key: IMPALA-14521
> URL: https://issues.apache.org/jira/browse/IMPALA-14521
> Project: IMPALA
> Issue Type: Documentation
> Reporter: Quanlong Huang
> Assignee: Sanjana Malhotra
> Priority: Major
>
> catalog_partial_fetch_max_files is a startup flag of catalogd added in
> IMPALA-11402. We need to document it somewhere.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
