Thanks for the information Alon! That is exactly the information I wanted.
Your theory of deferred memory usage pattern might be the reason for
browsers reporting used memory differently.
It is unfortunate that we will not be able to use PThreads in our main Wasm
because of this limitation, as we have lot of JS running alongside Wasm.
Any rough timeline on when we can expect ALLOW_MEMORY_GROWTH to work with
PTHREADS?
Also about checking the memory usage in devtools, I am using Chrome's task
manager as well as Activity Monitor (both on Mac) to check the webpage's
memory footprint. At both the places, the 2GB reserved memory is not
getting reflected. Maybe I am missing on checking other relevant fields.
But that should be fine, as I got the required information from you.
Appreciate the help Alon!
Thanks,
Prashanth Nethi
On Friday, August 28, 2020 at 12:23:24 AM UTC+5:30 alon...@gmail.com wrote:
> On Thu, Aug 27, 2020 at 10:28 AM Prashanth Nethi <prashan...@gmail.com>
> wrote:
>
>> My bad Alon! I will try to elaborate the scenario.I am trying to
>> understand the implications of switching off ALLOW_MEMORY_GROWTH in our
>> project. (which would be the case if we want PTHREADS enabled).
>>
>> The question is around, what if we set INITIAL_VALUE value to max value
>> (2GB). Does that mean when WASM is instantiated with INITIAL_VALUE=2048MB,
>> 2GB is reserved right upfront, even if not required right away? If yes,
>> does that mean this will reduce the usable JS heap size (by 2GB), right
>> from the beginning?
>>
>
> Yes, exactly. An initial value of X means X is allocated right from the
> start. Yes, this reduces available memory for other things, which can have
> downsides.
>
>
>> When I instantiate WASM (in my test app) with an INITIAL_VALUE=2000MB and
>> check for the memory that specific webpage is taking, I see that page does
>> not take 2GB but a lot lesser.
>>
>
> How are you measuring that?
>
> It's possible the browser allocates that memory via calloc() or such, and
> maybe the OS doesn't actually use any physical memory until those pages are
> touched, though. So maybe only virtual memory is used initially. (But even
> that can cause problems on 32 bit due to address space limits.)
>
> Measuring via browser devtools should report the full 2GB is used
> immediately.
>
>
>> It is when I start acquiring more memory, the memory usage goes up until
>> it hits the 2GB limit. Surprisingly this is the same behaviour I see with
>> ALLOW_MEMORY_GROWTH =1, USE_PTHREADS=0 (i.e. with PThreads disabled). So
>> trying to understand the dynamics and come up with the recommendation on
>> whether to enable or not enable PTHREADS in our app. FYI. The app has the
>> requirement to load on various browsers and devices, with Chrome and
>> Chromebook being our majority targets.
>>
>>
>> Regards,
>> Prashanth Nethi
>>
>> On Thursday, August 27, 2020 at 10:07:55 PM UTC+5:30 alon...@gmail.com
>> wrote:
>>
>>> On Thu, Aug 27, 2020 at 8:47 AM Prashanth Nethi <prashan...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Alon! That explains it! Yeah I should have thought a little
>>>> deeper.
>>>>
>>>> I am just posting my follow up question in case you did not get a
>>>> chance to look at it.
>>>>
>>>> One follow up question. May be a dumb one. What could be the potential
>>>> problems with ALLOW_MEMORY_GROWTH missing in PThreads mode? I see that
>>>> when
>>>> the Wasm is instantiated, the overall memory that the Chrome tab was
>>>> taking
>>>> was similar to the one taken by the WASM built with ALLOW_MEMORY_GROWTH.
>>>> Is
>>>> it that, we will not be able to instantiate WASM on low end devices if
>>>> built with ALLOW_MEMORY_GROWTH=0?
>>>>
>>>>
>>> I'm not sure what you're asking here?
>>>
>>> In general, not having memory growth enabled means that memory can't
>>> grow. So if you need more than the initial value, the program will hit a
>>> problem. I don't think there's anything special to pthreads in that case.
>>> (The reverse, having growth *enabled*, does have downsides for pthreads as
>>> the JS use of memory becomes somewhat slower.)
>>>
>>> Regards,
>>>> Prashanth Nethi
>>>>
>>>> On Thursday, August 27, 2020 at 2:37:07 AM UTC+5:30 alon...@gmail.com
>>>> wrote:
>>>>
>>>>> My guess is that's because of the behavior of std::vector and how it
>>>>> resizes. Over those appends it will malloc and free repeatedly and that
>>>>> may
>>>>> cause fragmentation that prevents a final larger size, which must be a
>>>>> single contiguous region. The second version allocates many smaller ones,
>>>>> not a single contiguous region.
>>>>>
>>>>> - Alon
>>>>>
>>>>>
>>>>> On Tue, Aug 25, 2020 at 11:24 PM Prashanth Nethi <prashan...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Alon! So here is something very weird. I could get the memory
>>>>>> usage go all the way to 2GB when I changed my testing code. This was my
>>>>>> original test code. So basically I was just adding elements to
>>>>>> std::vector
>>>>>> infinitely.
>>>>>>
>>>>>> class TestClass{
>>>>>> private:
>>>>>> int t = 0;
>>>>>> };
>>>>>>
>>>>>> struct Data {
>>>>>> int t;
>>>>>> TestClass obj;
>>>>>> };
>>>>>>
>>>>>> typedef std::vector<Data> Vec;
>>>>>>
>>>>>> Vec someVec;
>>>>>>
>>>>>> using namespace std;
>>>>>>
>>>>>> int main() {
>>>>>> printf("hello, world!\n");
>>>>>>
>>>>>> while(1){
>>>>>> Data data;
>>>>>> someVec.push_back(data);
>>>>>> }
>>>>>>
>>>>>> return 0;
>>>>>> }
>>>>>>
>>>>>> With this code, the WASM memory was going all the way to 1GB.
>>>>>>
>>>>>> But when I changed the code to this, where I am writing some value
>>>>>> after acquiring memory, then I am able to see the memory usage go all
>>>>>> the
>>>>>> way up to 2 GB. Could this be a bug? I am on emscripten 2.0.
>>>>>>
>>>>>>
>>>>>> int main() {
>>>>>> printf("hello, world!\n");
>>>>>> char *p = nullptr;
>>>>>> int byteSize = 50 * 1024 * 1024;
>>>>>> while(1){
>>>>>> p = new char(byteSize);
>>>>>> p[byteSize] = 20;
>>>>>> }
>>>>>> return 0;
>>>>>> }
>>>>>>
>>>>>> Also It is very encouraging to see that 4GB is considered for
>>>>>> PThreads as well! Thanks.
>>>>>>
>>>>>> One follow up question. May be a dumb one. What could be the
>>>>>> potential problems with ALLOW_MEMORY_GROWTH missing in PThreads mode? I
>>>>>> see
>>>>>> that when the Wasm is instantiated, the overall memory that the Chrome
>>>>>> tab
>>>>>> was taking was similar to the one taken by the WASM built with
>>>>>> ALLOW_MEMORY_GROWTH. Is it that, we will not be able to instantiate WASM
>>>>>> on
>>>>>> low end devices if built with ALLOW_MEMORY_GROWTH=0?
>>>>>>
>>>>>> Greatly appreciate your help Alon!
>>>>>>
>>>>>> Thanks,
>>>>>> Prashanth Nethi
>>>>>>
>>>>>> On Saturday, August 22, 2020 at 1:50:57 AM UTC+5:30 alon...@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>> I think you can do any number up to 2GB, including 2GB - 64Kb. So
>>>>>>> the limit isn't 1GB, unless you see that on some specific browser?
>>>>>>> Could be
>>>>>>> a bug.
>>>>>>>
>>>>>>> It should soon be possible to do up to 4GB for the initial memory
>>>>>>> (without growth), thanks to a spec change,
>>>>>>> https://github.com/WebAssembly/spec/pull/1174
>>>>>>>
>>>>>>> On Thu, Aug 20, 2020 at 8:10 AM Prashanth Nethi <
>>>>>>> prashan...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am currently building WASM with the following flags, to enable
>>>>>>>> PThreads in Wasm.
>>>>>>>> -s USING_PTHREADS=1 -s INITIAL_MEMORY=1999MB -s MAXIMUM_MEMORY=2GB.
>>>>>>>>
>>>>>>>> This works wonderfully for our use cases! In fact we are able to
>>>>>>>> get 2x performance in some cases!
>>>>>>>>
>>>>>>>> When I checked the max memory that the Wasm could use, with
>>>>>>>> PThreads enabled, it got capped at 1 GB. I am seeing that when the
>>>>>>>> WASM is
>>>>>>>> built with ALLOW_MEMORY_GROWTH, the Wasm can use upto 2GB. I know that
>>>>>>>> ALLOW_MEMORY_GROWTH with USE_PTHREADS is discouraged so can't look at
>>>>>>>> that
>>>>>>>> as a possible solution.
>>>>>>>>
>>>>>>>> Is there anyway I can get Wasm to use 2GB (or even potentially 4GB
>>>>>>>> in the future) with PThreads enabled? Is it that I am missing using
>>>>>>>> some
>>>>>>>> configuration options?
>>>>>>>>
>>>>>>>> I am really hoping there is a way to increase the WASM cap to 2GB,
>>>>>>>> as using PThreads, solves our use cases in a big way.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Prashanth Nethi
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "emscripten-discuss" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to emscripten-disc...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/730a6796-5b14-4a9e-a1d8-298415c67cd1n%40googlegroups.com
>>>>>>>>
>>>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/730a6796-5b14-4a9e-a1d8-298415c67cd1n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "emscripten-discuss" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to emscripten-disc...@googlegroups.com.
>>>>>>
>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/86a9fc74-2036-4749-8212-29f6802615d0n%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/86a9fc74-2036-4749-8212-29f6802615d0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "emscripten-discuss" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to emscripten-disc...@googlegroups.com.
>>>>
>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/emscripten-discuss/9ddd4487-ff48-4b05-a138-900103ec2a4dn%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/emscripten-discuss/9ddd4487-ff48-4b05-a138-900103ec2a4dn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "emscripten-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to emscripten-disc...@googlegroups.com.
>>
> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/emscripten-discuss/f197c8df-ba90-4f68-9018-3590303302ean%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/emscripten-discuss/f197c8df-ba90-4f68-9018-3590303302ean%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/emscripten-discuss/a03fe2fa-6203-4342-8e1f-49edaeeac272n%40googlegroups.com.
