[
https://issues.apache.org/jira/browse/IMPALA-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631597#comment-16631597
]
Ahshan edited comment on IMPALA-7641 at 9/28/18 9:53 AM:
---------------------------------------------------------
[~jeszyb] - you are right , but i don't see any parquet file size greater than
40MB
hence still can't understand on the mem_limit requiring more memory as It needs
to process more than 100G of data.
Our default memory limit is set to 20G. I believe that setting is per data
node. We have 18 data nodes, so assuming the data resides on all of them, we
would have access to 360G for this query
was (Author: ahshan...@gmail.com):
[~jeszyb] - you are right , but i don't see any parquet file size greater than
40MB
but, still can't understand on the mem_limit requiring more memory as It needs
to process more than 100G of data.
Our default memory limit is set to 20G. I believe that setting is per data
node. We have 18 data nodes, so assuming the data resides on all of them, we
would have access to 360G for this query
> Memory Limit Exceeded
> ---------------------
>
> Key: IMPALA-7641
> URL: https://issues.apache.org/jira/browse/IMPALA-7641
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.6.4
> Reporter: Ahshan
> Priority: Minor
> Labels: memory
> Attachments: profile(8).txt
>
>
> We are using CDH distribution with impala version -impalad version
> 2.6.0-cdh5.8.2 RELEASE
>
> As per my understanding memory requirement is of 288 MB and we have an total
> of 18 Impala Daemons which sum upto 5184MB of total memory consumption
> considering the above details, it should not lead to an memory issue when the
> MEM_LIMIT is set to 20GB
> Hence , Could you please let us know the cause of memory limit exceeding
> select * from emp_sales where job_id = 55451 and uploaded_month = 201808
> limit 1
> +-----------------------------------------------------------+
> |Explain String|
> +-----------------------------------------------------------+
> |Estimated Per-Host Requirements: Memory=288.00MB VCores=1|
> | |
> |01:EXCHANGE [UNPARTITIONED]|
> | |limit: 1|
> | | |
> |00:SCAN HDFS [fenet5.hmig_os_changes_details_malicious]|
> |partitions=1/25 files=3118 size=110.01GB|
> |predicates: job_id = 55451|
> |limit: 1|
> +-----------------------------------------------------------+
> WARNINGS:
> Memory limit exceeded
> HdfsParquetScanner::ReadDataPage() failed to allocate 269074889 bytes for
> dictionary.
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 257.23 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.63 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.27 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.39 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 16.09 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=19.74 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.74 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.74 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 15.20 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=19.64 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.64 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.64 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 14.61 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=19.64 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.64 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.64 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 257.11 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=19.47 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.47 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.47 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.51 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=19.24 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.24 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.24 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.32 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=19.24 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.24 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.24 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.73 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=19.49 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.49 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.49 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.29 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=19.49 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.49 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.49 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 256.61 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: memory limit exceeded.
> Limit=20.00 GB Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 256.05 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=19.69 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=19.69 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.69 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.35 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=17.97 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=17.97 GB
> HDFS_SCAN_NODE (id=0): Consumption=17.72 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 1.02 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=17.63 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=17.63 GB
> HDFS_SCAN_NODE (id=0): Consumption=17.63 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 1.01 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=16.94 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.94 GB
> HDFS_SCAN_NODE (id=0): Consumption=16.68 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 88.00 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=16.61 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.61 GB
> HDFS_SCAN_NODE (id=0): Consumption=16.36 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.23 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=16.30 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.30 GB
> HDFS_SCAN_NODE (id=0): Consumption=16.30 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB
> Consumption=8.02 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=8.02 GB
> HDFS_SCAN_NODE (id=0): Consumption=8.02 GB
> Block Manager: Limit=16.00 GB Consumption=0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
