[
https://issues.apache.org/jira/browse/IMPALA-10214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572642#comment-17572642
]
Michael Smith edited comment on IMPALA-10214 at 8/4/22 6:09 PM:
----------------------------------------------------------------
Local file handles are already cached because Ozone shares the HDFS
implementation.
The file handle cache now supports local file handles, as well as remote
HDFS/S3/ABFS handles. In evaluating Ozone, [~weichiu] identified that memory
used by the file handle cache was substantially larger than HDFS when using
remote handles. The original measurements behind setting
{{max_cached_file_handles=20000}} may not work for Ozone; we should measure to
determine a similar memory cap or see if there are ways Ozone could reduce
handle size.
was (Author: JIRAUSER288956):
Local file handles are already cached because Ozone shares the HDFS
implementation.
The file handle cache now supports local file handles, as well as remote
HDFS/S3/ABFS handles. In evaluating Ozone, [~weichiu] identified that memory
used by the file handle cache was substantially larger than HDFS when using
remote handles. The original measurements behind setting
{{max_cached_file_handles=20000}} may not work for Ozone; we should measure to
determine a similar memory cap.
> Ozone support for file handle cache
> -----------------------------------
>
> Key: IMPALA-10214
> URL: https://issues.apache.org/jira/browse/IMPALA-10214
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Sahil Takiar
> Priority: Major
>
> This is dependent on the Ozone input streams supporting the {{CanUnbuffer}}
> interface first (last I checked, the input streams don't implement the
> interface).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
