Tim Armstrong created IMPALA-7380:
-------------------------------------
Summary: Untracked memory for file metadata like AvroHeader for
sequence scanners accumulate throughout query
Key: IMPALA-7380
URL: https://issues.apache.org/jira/browse/IMPALA-7380
Project: IMPALA
Issue Type: Sub-task
Components: Backend
Reporter: Tim Armstrong
HdfsScanNodeBase maintains a map of per-file metadata objects for use by
different scan ranges from the same file, e.g. AvroFileHeader. These are not
cleaned up until the end of the query.
Note that because of IMPALA-6932 this doesn't necessarily increase peak memory
significantly (because the headers are all accumulated during the
header-parsing phase anyway).
We should track the number of scanners remaining for each file and delete the
headers when we no longer need them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
