[ https://issues.apache.org/jira/browse/IMPALA-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahshan updated IMPALA-7641: --------------------------- Epic Color: ghx-label-2 (was: ghx-label-3) > Memory Limit Exceeded > --------------------- > > Key: IMPALA-7641 > URL: https://issues.apache.org/jira/browse/IMPALA-7641 > Project: IMPALA > Issue Type: Bug > Components: Backend > Affects Versions: Impala 2.6.4 > Reporter: Ahshan > Priority: Blocker > Labels: memory > Attachments: profile(8).txt > > > We are using CDH distribution with impala version -impalad version > 2.6.0-cdh5.8.2 RELEASE > > As per my understanding memory requirement is of 288 MB and we have an total > of 18 Impala Daemons which sum upto 5184MB of total memory consumption > considering the above details, it should not lead to an memory issue when the > MEM_LIMIT is set to 20GB > Hence , Could you please let us know the cause of memory limit exceeding > select * from emp_sales where job_id = 55451 and uploaded_month = 201808 > limit 1 > +-----------------------------------------------------------+ > |Explain String| > +-----------------------------------------------------------+ > |Estimated Per-Host Requirements: Memory=288.00MB VCores=1| > | | > |01:EXCHANGE [UNPARTITIONED]| > | |limit: 1| > | | | > |00:SCAN HDFS [fenet5.hmig_os_changes_details_malicious]| > |partitions=1/25 files=3118 size=110.01GB| > |predicates: job_id = 55451| > |limit: 1| > +-----------------------------------------------------------+ > WARNINGS: > Memory limit exceeded > HdfsParquetScanner::ReadDataPage() failed to allocate 269074889 bytes for > dictionary. > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 257.23 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=20.00 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB > HDFS_SCAN_NODE (id=0): Consumption=20.00 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 255.63 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=20.00 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB > HDFS_SCAN_NODE (id=0): Consumption=20.00 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 255.27 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=20.00 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB > HDFS_SCAN_NODE (id=0): Consumption=20.00 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 255.39 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=20.00 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB > HDFS_SCAN_NODE (id=0): Consumption=20.00 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 16.09 KB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=19.74 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.74 GB > HDFS_SCAN_NODE (id=0): Consumption=19.74 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 15.20 KB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=19.64 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.64 GB > HDFS_SCAN_NODE (id=0): Consumption=19.64 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 14.61 KB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=19.64 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.64 GB > HDFS_SCAN_NODE (id=0): Consumption=19.64 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 257.11 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=19.47 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.47 GB > HDFS_SCAN_NODE (id=0): Consumption=19.47 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 255.51 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=19.24 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.24 GB > HDFS_SCAN_NODE (id=0): Consumption=19.24 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 255.32 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=19.24 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.24 GB > HDFS_SCAN_NODE (id=0): Consumption=19.24 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 255.73 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=19.49 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.49 GB > HDFS_SCAN_NODE (id=0): Consumption=19.49 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 255.29 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=19.49 GB > Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.49 GB > HDFS_SCAN_NODE (id=0): Consumption=19.49 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 256.61 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: memory limit exceeded. > Limit=20.00 GB Consumption=20.00 GB > Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=20.00 GB > HDFS_SCAN_NODE (id=0): Consumption=20.00 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 256.05 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=19.69 GB > Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=19.69 GB > HDFS_SCAN_NODE (id=0): Consumption=19.69 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 255.35 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=17.97 GB > Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=17.97 GB > HDFS_SCAN_NODE (id=0): Consumption=17.72 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 1.02 KB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=17.63 GB > Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=17.63 GB > HDFS_SCAN_NODE (id=0): Consumption=17.63 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 1.01 KB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=16.94 GB > Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.94 GB > HDFS_SCAN_NODE (id=0): Consumption=16.68 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 88.00 KB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=16.61 GB > Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.61 GB > HDFS_SCAN_NODE (id=0): Consumption=16.36 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > HDFS_SCAN_NODE (id=0) could not allocate 255.23 MB without exceeding limit. > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=16.30 GB > Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.30 GB > HDFS_SCAN_NODE (id=0): Consumption=16.30 GB > DataStreamSender: Consumption=1.45 KB > Block Manager: Limit=16.00 GB Consumption=0 > Memory Limit Exceeded > Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB > Consumption=8.02 GB > Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=8.02 GB > HDFS_SCAN_NODE (id=0): Consumption=8.02 GB > Block Manager: Limit=16.00 GB Consumption=0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org