Re: [OmniOS-discuss] Omnios: can't start as many kvm's as I thought due to memory pressure

[pieter van puymbroeck]

Will try it tonight or tomorrow and provide feedback. 
Thanks for the tip.
P.

Verstuurd vanaf mijn iPhone

> Op 9 mrt. 2017 om 09:55 heeft Jan Vlach  het volgende 
> geschreven:
> 
> Hi Pieter,
> 
> Can you make your swapfile 96G or larger? 
> 
> If I recall correctly, KVM needs all it's RAM backed up by swap, because
> of fragmentation and other reasons. 
> 
> Jan
> 
> 
> 
>> On Thu, Mar 09, 2017 at 06:34:11AM +, pieter van puymbroeck wrote:
>> Hello Jan,
>> 
>> $ swap -sh   
>> total: 52G allocated + 100M reserved = 53G used, 8.8G available
>> $
>> 
>> This is when 7 machines are running. 
>> I face the issue when starting the 8th one.
>> 
>> Br
>> Pieter
>> 
>> 
>> Verstuurd vanaf mijn iPhone
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Omnios: can't start as many kvm's as I thought due to memory pressure

[Jan Vlach]

Hi Pieter,

Can you make your swapfile 96G or larger? 

If I recall correctly, KVM needs all it's RAM backed up by swap, because
of fragmentation and other reasons. 

Jan



On Thu, Mar 09, 2017 at 06:34:11AM +, pieter van puymbroeck wrote:
> Hello Jan,
> 
> $ swap -sh   
> total: 52G allocated + 100M reserved = 53G used, 8.8G available
> $
> 
> This is when 7 machines are running. 
> I face the issue when starting the 8th one.
> 
> Br
> Pieter
> 
> 
> Verstuurd vanaf mijn iPhone
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Omnios: can't start as many kvm's as I thought due to memory pressure

[pieter van puymbroeck]

Hello Jan,

$ swap -sh   
total: 52G allocated + 100M reserved = 53G used, 8.8G available
$

This is when 7 machines are running. 
I face the issue when starting the 8th one.

Br
Pieter


Verstuurd vanaf mijn iPhone

> Op 8 mrt. 2017 om 22:29 heeft Jan Vlach  het volgende 
> geschreven:
> 
> Hi Pieter,
> 
> how much disk space do you have allocated for swap?
> 
> Jan
> 
>> On Wed, Mar 08, 2017 at 07:32:23PM +, pieter van puymbroeck wrote:
>> Hi,
>> 
>> 
>> I want to start 12 kvm machines in my global zone and I end up in an memlock 
>> issue after the 6th machine:
>> 
>> 
>> qemu_mlock: have only locked 0 of 7516192768 bytes; still trying…
>> 
>> 
>> The details:
>> 
>> 
>> # echo "::memstat" | mdb -k
>> 
>> Page SummaryPagesMB  %Tot
>> 
>>      
>> 
>> Kernel1462114  57115%
>> 
>> Boot pages166 00%
>> 
>> ZFS File Data  143529   5600%
>> 
>> Anon27477   1070%
>> 
>> Exec and libs1244 40%
>> 
>> Page cache   4911190%
>> 
>> Free (cachelist) 3597140%
>> 
>> Free (freelist)  29809914116444   95%
>> 
>> 
>> Total31452952122863
>> 
>> Physical 31452951122863
>> 
>> #
>> 
>> 
>> My pagesize is:
>> 
>> 
>> # pagesize
>> 
>> 4096
>> 
>> #
>> 
>> 
>> 
>> so 31452952 pages = (122863*1024*1024) bytes / 4096 (bytes)
>> 
>> thus: 31452952 / 256 = 122863MB
>> 
>> 
>> That's correct. So pages to MB we can divide by 256. MB to pages, multiply 
>> with 256.
>> 
>> 
>> I want 8GB per vm and I want 12vm's. This is 96GB (98304MB)
>> 
>> -> needs 98304 * 256 = 25165824 pages.
>> 
>> 
>> 
>> 31452952 - 25165824 = 6287128 left (or 24559 MB ~ 23.9GB )
>> 
>> 
>> I reserved (in /etc/system ) 15 GB for zfs arc
>> 
>> 15GB = 15360 MB = 3932160 pages
>> 
>> 
>> So this gives 6287128 - 3932160 = 2354968 pages left (or 9199MB ~ 8.98 GB)
>> 
>> 
>> -> this should be sufficient for omnios to run. Unless I'm not aware of 
>> other things, which is very well possible.
>> 
>> 
>> So remember
>> 
>> 
>> Page SummaryPagesMB  %Tot
>> 
>>      
>> 
>> ...
>> 
>> Free (cachelist) 3597140%
>> 
>> Free (freelist)  29809914116444   95%
>> 
>> 
>> Total31452952122863
>> 
>> Physical 31452951122863
>> 
>> #
>> 
>> 
>> Let's boot one kvm with 8GB ram for the vm. We expect to see (8GB *1024) * 
>> 256 = 2097152 pages be eaten up. So we should end up with 31452952 - 2097152 
>> = 29355800 pages.
>> 
>> OR if we are eating from the "Free" part, we should end up with 29809914 - 
>> 2097152 = 27712762 pages (or 108252MB ~105.71 GB)
>> 
>> 
>> Let's confirm:
>> 
>> 
>> Before boot:
>> 
>> # mdb -ke 'availrmem/D ; pages_pp_maximum/D'
>> 
>> availrmem:
>> 
>> availrmem:  29845807
>> 
>> pages_pp_maximum:
>> 
>> pages_pp_maximum:   1216998
>> 
>> #
>> 
>> 
>> Boot the machine.
>> 
>> 
>> # echo "::memstat" | mdb -k
>> 
>> Page SummaryPagesMB  %Tot
>> 
>>      
>> 
>> Kernel1472257  57515%
>> 
>> Boot pages166 00%
>> 
>> ZFS File Data  149321   5830%
>> 
>> Anon  2139596  83577%
>> 
>> Exec and libs1549 60%
>> 
>> Page cache   4955190%
>> 
>> Free (cachelist) 3445130%
>> 
>> Free (freelist)  27681663108131   88% <<<- 27712762 
>> pages
>> 
>> 
>> Total31452952122863
>> 
>> Physical 31452951122863
>> 
>> # mdb -ke 'availrmem/D ; pages_pp_maximum/D'
>> 
>> availrmem:
>> 
>> availrmem:  26621718
>> 
>> pages_pp_maximum:
>> 
>> pages_pp_maximum:   1216998
>> 
>> #
>> 
>> 
>> 
>> look there. We expected to end up with 29355800 pages or 27712762 free pages.
>> 
>> 
>> If we check in the available memory 29845807 - 26621718 = 3224089 pages were 
>> consumed (12594.09MB ~ 12.29GB )
>> 
>> Double check: 29809914 - 27681663 = 2128251 pages consumed ( 8313.48MB ~ 
>> 8.11GB )
>> 
>> And this for a machine of 8GB.
>> 
>> 
>> Let's do a test with a vm with 4GB allocated ( 1048576 pages )
>> 
>> 
>> 
>> Before boot:
>> 
>> # echo "::memstat" | mdb -k
>> 
>> Page SummaryPagesMB  %Tot
>> 
>>      
>> 
>> Kernel

Re: [OmniOS-discuss] Omnios: can't start as many kvm's as I thought due to memory pressure

[Jan Vlach]

Hi Pieter,

how much disk space do you have allocated for swap?

Jan

On Wed, Mar 08, 2017 at 07:32:23PM +, pieter van puymbroeck wrote:
> Hi,
> 
> 
> I want to start 12 kvm machines in my global zone and I end up in an memlock 
> issue after the 6th machine:
> 
> 
> qemu_mlock: have only locked 0 of 7516192768 bytes; still trying…
> 
> 
> The details:
> 
> 
> # echo "::memstat" | mdb -k
> 
> Page SummaryPagesMB  %Tot
> 
>      
> 
> Kernel1462114  57115%
> 
> Boot pages166 00%
> 
> ZFS File Data  143529   5600%
> 
> Anon27477   1070%
> 
> Exec and libs1244 40%
> 
> Page cache   4911190%
> 
> Free (cachelist) 3597140%
> 
> Free (freelist)  29809914116444   95%
> 
> 
> Total31452952122863
> 
> Physical 31452951122863
> 
> #
> 
> 
> My pagesize is:
> 
> 
> # pagesize
> 
> 4096
> 
> #
> 
> 
> 
> so 31452952 pages = (122863*1024*1024) bytes / 4096 (bytes)
> 
> thus: 31452952 / 256 = 122863MB
> 
> 
> That's correct. So pages to MB we can divide by 256. MB to pages, multiply 
> with 256.
> 
> 
> I want 8GB per vm and I want 12vm's. This is 96GB (98304MB)
> 
> -> needs 98304 * 256 = 25165824 pages.
> 
> 
> 
> 31452952 - 25165824 = 6287128 left (or 24559 MB ~ 23.9GB )
> 
> 
> I reserved (in /etc/system ) 15 GB for zfs arc
> 
> 15GB = 15360 MB = 3932160 pages
> 
> 
> So this gives 6287128 - 3932160 = 2354968 pages left (or 9199MB ~ 8.98 GB)
> 
> 
> -> this should be sufficient for omnios to run. Unless I'm not aware of other 
> things, which is very well possible.
> 
> 
> So remember
> 
> 
> Page SummaryPagesMB  %Tot
> 
>      
> 
> ...
> 
> Free (cachelist) 3597140%
> 
> Free (freelist)  29809914116444   95%
> 
> 
> Total31452952122863
> 
> Physical 31452951122863
> 
> #
> 
> 
> Let's boot one kvm with 8GB ram for the vm. We expect to see (8GB *1024) * 
> 256 = 2097152 pages be eaten up. So we should end up with 31452952 - 2097152 
> = 29355800 pages.
> 
> OR if we are eating from the "Free" part, we should end up with 29809914 - 
> 2097152 = 27712762 pages (or 108252MB ~105.71 GB)
> 
> 
> Let's confirm:
> 
> 
> Before boot:
> 
> # mdb -ke 'availrmem/D ; pages_pp_maximum/D'
> 
> availrmem:
> 
> availrmem:  29845807
> 
> pages_pp_maximum:
> 
> pages_pp_maximum:   1216998
> 
> #
> 
> 
> Boot the machine.
> 
> 
> # echo "::memstat" | mdb -k
> 
> Page SummaryPagesMB  %Tot
> 
>      
> 
> Kernel1472257  57515%
> 
> Boot pages166 00%
> 
> ZFS File Data  149321   5830%
> 
> Anon  2139596  83577%
> 
> Exec and libs1549 60%
> 
> Page cache   4955190%
> 
> Free (cachelist) 3445130%
> 
> Free (freelist)  27681663108131   88% <<<- 27712762 
> pages
> 
> 
> Total31452952122863
> 
> Physical 31452951122863
> 
> # mdb -ke 'availrmem/D ; pages_pp_maximum/D'
> 
> availrmem:
> 
> availrmem:  26621718
> 
> pages_pp_maximum:
> 
> pages_pp_maximum:   1216998
> 
> #
> 
> 
> 
> look there. We expected to end up with 29355800 pages or 27712762 free pages.
> 
> 
> If we check in the available memory 29845807 - 26621718 = 3224089 pages were 
> consumed (12594.09MB ~ 12.29GB )
> 
> Double check: 29809914 - 27681663 = 2128251 pages consumed ( 8313.48MB ~ 
> 8.11GB )
> 
> And this for a machine of 8GB.
> 
> 
> Let's do a test with a vm with 4GB allocated ( 1048576 pages )
> 
> 
> 
> Before boot:
> 
> # echo "::memstat" | mdb -k
> 
> Page SummaryPagesMB  %Tot
> 
>      
> 
> Kernel1476857  57685%
> 
> Boot pages166 00%
> 
> ZFS File Data  176243   6881%
> 
> Anon  2139416  83577%
> 
> Exec and libs1549 60%
> 
> Page cache   4961190%
> 
> Free (cachelist) 3440130%
> 
> Free (freelist)  27650320108009   88%
> 
> 
> Total31452952122863
>