Thanks Murtadha! The problem solved. However, increasing the number of cores didn't help to improve the performance of that query.
On Sun, Jan 28, 2018 at 5:05 PM, Murtadha Hubail <[email protected]> wrote: > Hi Rana, > > The memory used for query processing is automatically calculated as > follows: > JVM Max Memory - storage.buffercache.size - storage.memorycomponent. > globalbudget > > The documentation defaults for these parameters are outdated. The default > value for storage.buffercache.size is (JVM Max Memory / 4) and it's the > same for storage.memorycomponent.globalbudget. Since your dataset is > already loaded, you could reduce the budget of > storage.memorycomponent.globalbudget. > In addition, if I recall correctly, your dataset size is way smaller than > what's allocated for the buffer cache, so you might want to reduce the > buffer cache budget. That should give you more than enough memory to > execute on 39 cores. > > Cheers, > Murtadha > > On 01/29/2018, 3:30 AM, "Mike Carey" <[email protected]> wrote: > > + dev > > > On 1/28/18 3:37 PM, Rana Alotaibi wrote: > > Hi all, > > > > I would like to make AsterixDB utilizes all available CPU cores (39) > > that I have for the following query: > > > > USE mimiciii; > > SET `compiler.parallelism` "39"; > > SET `compiler.sortmemory` "128MB"; > > SET `compiler.joinmemory` "265MB"; > > SELECT P.SUBJECT_ID > > FROM LABITEMS I, PATIENTS P, P.ADMISSIONS A, A.LABEVENTS E > > WHERE E.ITEMID/*+bcast*/=I.ITEMID AND > > E.FLAG = 'abnormal' AND > > I.FLUID='Blood' AND > > I.LABEL='Haptoglobin' > > > > > > The total memory size that I have is 125GB(57GB for the AsterixDB > > buffer cache). By running the above query, I got the following error: > > > > "msg": "HYR0009: Job requirement (memory: 10705403904 bytes, CPU > > cores: 39) exceeds capacity (memory: 3258744832 bytes, CPU cores: > 39)" > > > > How can I change this capacity default configuration? I'm looking > into > > this page : https://asterixdb.apache.org/docs/0.9.2/ncservice.html . > > Could you please point me to the appropriate configuration parameter? > > > > Thanks > > -- Rana > > > > > > > > > > > > >
