Daniel Becker created IMPALA-13988:
--------------------------------------
Summary: Take Parquet page size into account when estimating
memory for HDFS WRITER
Key: IMPALA-13988
URL: https://issues.apache.org/jira/browse/IMPALA-13988
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Daniel Becker
The Iceberg table properties 'write.parquet.page-size-bytes' and
'write.parquet.dict-size-bytes' allow setting the size of Parquet pages. These
page sizes are not taken into account when estimating the memory of the Hdfs
writers, so if they are set to a large value, the query may fail with a
MemoryLimitExceeded error.
Note that before IMPALA-13963, we always incorrectly reserved a default-sized
buffer, so we didn't trigger the memory limit if we didn't actually write more
than the default page size, but if we did, Impala crashed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
