jhkcool created IMPALA-11810:
--------------------------------
Summary: The mem estimate is incorrect in HdfsScanNode
Key: IMPALA-11810
URL: https://issues.apache.org/jira/browse/IMPALA-11810
Project: IMPALA
Issue Type: Improvement
Components: fe
Affects Versions: Impala 4.1.1, Impala 4.2.0
Reporter: jhkcool
About perInstanceMemEstimate calculate int below method:
{code:java}
@Override
public void computeNodeResourceProfile(TQueryOptions queryOptions) {
...
long avgScanRangeBytes = (long) Math.ceil(sumValues(totalBytesPerFs_) /
(double) scanRangeSize);
...
}{code}
All table data file sizes are used in the calculation of scan hdfs memory
consumption, it is unreasonable. Because not all data in the table needs to be
read into the memory, but only the materialized fields are read, so only the
data file size occupied by the materialized field needs to be calculated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
