[
https://issues.apache.org/jira/browse/IMPALA-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Sinha closed IMPALA-1108.
------------------------------
Resolution: Fixed
> Impala should check the number of opened files/partition during insert
> ----------------------------------------------------------------------
>
> Key: IMPALA-1108
> URL: https://issues.apache.org/jira/browse/IMPALA-1108
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 1.4
> Reporter: Alan Choi
> Priority: Minor
> Labels: ramp-up
>
> For insert, when Impala is inserting into a huge number of partition, Impala
> might be opening too many files. HDFS will return an error, but the error is
> incomprehensible as "Error(12): Cannot allocate memory".
> We can do better to improve the error message. Here are two suggestions:
> 1. During planning, if there's stats, we know how many partitions are being
> inserted per Impalad. Based on that, we can determine if we'll be opening too
> many files. Either return an error or a warning message.
> 2. During query execution, keep track of the number of files opened for read
> and write. If we're opening too many files for write, abort the query and
> returns a proper error message.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
