Re: WASM built with USING_PTHREADS=1 can't go beyond 1GB

[Alon Zakai]

On Fri, Aug 28, 2020 at 6:17 AM Prashanth Nethi 
wrote:

> Thanks for the information Alon! That is exactly the information I wanted.
> Your theory of deferred memory usage pattern might be the reason for
> browsers reporting used memory differently.
>
> It is unfortunate that we will not be able to use PThreads in our main
> Wasm because of this limitation, as we have lot of JS running alongside
> Wasm. Any rough timeline on when we can expect ALLOW_MEMORY_GROWTH to work
> with PTHREADS?
>

It already works, but memory access from JS is somewhat slower. In most
cases you won't notice that, though - unless you've already tested and see
overhead? If so that could be useful to mention to the standards bodies
that are considering a spec change that could improve this, it could
increase the priority.

- Alon


> Also about checking the memory usage in devtools, I am using Chrome's task
> manager as well as Activity Monitor (both on Mac) to check the webpage's
> memory footprint. At both the places, the 2GB reserved memory is not
> getting reflected. Maybe I am missing on checking other relevant fields.
> But that should be fine, as I got the required information from you.
>
> Appreciate the help Alon!
>
> Thanks,
> Prashanth Nethi
>
> On Friday, August 28, 2020 at 12:23:24 AM UTC+5:30 alon...@gmail.com
> wrote:
>
>> On Thu, Aug 27, 2020 at 10:28 AM Prashanth Nethi 
>> wrote:
>>
>>> My bad Alon! I will try to elaborate the scenario.I am trying to
>>> understand the implications of switching off ALLOW_MEMORY_GROWTH in our
>>> project. (which would be the case if we want  PTHREADS enabled).
>>>
>>> The question is around, what if we set INITIAL_VALUE value to max value
>>> (2GB).  Does that mean when WASM is instantiated with INITIAL_VALUE=2048MB,
>>> 2GB is reserved right upfront, even if not required right away? If yes,
>>> does that mean this will reduce the usable JS heap size (by 2GB), right
>>> from the beginning?
>>>
>>
>> Yes, exactly. An initial value of X means X is allocated right from the
>> start. Yes, this reduces available memory for other things, which can have
>> downsides.
>>
>>
>>> When I instantiate WASM (in my test app) with an INITIAL_VALUE=2000MB
>>> and check for the memory that specific webpage is taking, I see that page
>>> does not take 2GB but a lot lesser.
>>>
>>
>> How are you measuring that?
>>
>> It's possible the browser allocates that memory via calloc() or such, and
>> maybe the OS doesn't actually use any physical memory until those pages are
>> touched, though. So maybe only virtual memory is used initially. (But even
>> that can cause problems on 32 bit due to address space limits.)
>>
>> Measuring via browser devtools should report the full 2GB is used
>> immediately.
>>
>>
>>> It is when I start acquiring more memory, the memory usage goes up until
>>> it hits the 2GB limit. Surprisingly this is the same behaviour I see with
>>> ALLOW_MEMORY_GROWTH =1, USE_PTHREADS=0 (i.e. with PThreads disabled). So
>>> trying to understand the dynamics and come up with the recommendation on
>>> whether to enable or not enable PTHREADS in our app. FYI. The app has the
>>> requirement to load on various browsers and devices, with Chrome and
>>> Chromebook being our majority targets.
>>>
>>>
>>> Regards,
>>> Prashanth Nethi
>>>
>>> On Thursday, August 27, 2020 at 10:07:55 PM UTC+5:30 alon...@gmail.com
>>> wrote:
>>>
 On Thu, Aug 27, 2020 at 8:47 AM Prashanth Nethi 
 wrote:

> Thanks Alon! That explains it! Yeah I should have thought a little
> deeper.
>
> I am just posting my follow up question in case you did not get a
> chance to look at it.
>
> One follow up question. May be a dumb one. What could be the potential
> problems with ALLOW_MEMORY_GROWTH missing in PThreads mode? I see that 
> when
> the Wasm is instantiated, the overall memory that the Chrome tab was 
> taking
> was similar to the one taken by the WASM built with ALLOW_MEMORY_GROWTH. 
> Is
> it that, we will not be able to instantiate WASM on low end devices if
> built with ALLOW_MEMORY_GROWTH=0?
>
>
 I'm not sure what you're asking here?

 In general, not having memory growth enabled means that memory can't
 grow. So if you need more than the initial value, the program will hit a
 problem. I don't think there's anything special to pthreads in that case.
 (The reverse, having growth *enabled*, does have downsides for pthreads as
 the JS use of memory becomes somewhat slower.)

 Regards,
> Prashanth Nethi
>
> On Thursday, August 27, 2020 at 2:37:07 AM UTC+5:30 alon...@gmail.com
> wrote:
>
>> My guess is that's because of the behavior of std::vector and how it
>> resizes. Over those appends it will malloc and free repeatedly and that 
>> may
>> cause fragmentation that prevents a final larger size, which must be a
>> single contiguous region. The second version allocates 

Re: WASM built with USING_PTHREADS=1 can't go beyond 1GB

[Prashanth Nethi]

Thanks for the information Alon! That is exactly the information I wanted. 
Your theory of deferred memory usage pattern might be the reason for 
browsers reporting used memory differently.

It is unfortunate that we will not be able to use PThreads in our main Wasm 
because of this limitation, as we have lot of JS running alongside Wasm. 
Any rough timeline on when we can expect ALLOW_MEMORY_GROWTH to work with 
PTHREADS?

Also about checking the memory usage in devtools, I am using Chrome's task 
manager as well as Activity Monitor (both on Mac) to check the webpage's 
memory footprint. At both the places, the 2GB reserved memory is not 
getting reflected. Maybe I am missing on checking other relevant fields. 
But that should be fine, as I got the required information from you.

Appreciate the help Alon!

Thanks,
Prashanth Nethi

On Friday, August 28, 2020 at 12:23:24 AM UTC+5:30 alon...@gmail.com wrote:

> On Thu, Aug 27, 2020 at 10:28 AM Prashanth Nethi  
> wrote:
>
>> My bad Alon! I will try to elaborate the scenario.I am trying to 
>> understand the implications of switching off ALLOW_MEMORY_GROWTH in our 
>> project. (which would be the case if we want  PTHREADS enabled).
>>
>> The question is around, what if we set INITIAL_VALUE value to max value 
>> (2GB).  Does that mean when WASM is instantiated with INITIAL_VALUE=2048MB, 
>> 2GB is reserved right upfront, even if not required right away? If yes, 
>> does that mean this will reduce the usable JS heap size (by 2GB), right 
>> from the beginning?
>>
>
> Yes, exactly. An initial value of X means X is allocated right from the 
> start. Yes, this reduces available memory for other things, which can have 
> downsides.
>
>
>> When I instantiate WASM (in my test app) with an INITIAL_VALUE=2000MB and 
>> check for the memory that specific webpage is taking, I see that page does 
>> not take 2GB but a lot lesser.
>>
>
> How are you measuring that?
>
> It's possible the browser allocates that memory via calloc() or such, and 
> maybe the OS doesn't actually use any physical memory until those pages are 
> touched, though. So maybe only virtual memory is used initially. (But even 
> that can cause problems on 32 bit due to address space limits.)
>
> Measuring via browser devtools should report the full 2GB is used 
> immediately.
>  
>
>> It is when I start acquiring more memory, the memory usage goes up until 
>> it hits the 2GB limit. Surprisingly this is the same behaviour I see with  
>> ALLOW_MEMORY_GROWTH =1, USE_PTHREADS=0 (i.e. with PThreads disabled). So 
>> trying to understand the dynamics and come up with the recommendation on 
>> whether to enable or not enable PTHREADS in our app. FYI. The app has the 
>> requirement to load on various browsers and devices, with Chrome and 
>> Chromebook being our majority targets.
>>
>>
>> Regards,
>> Prashanth Nethi
>>
>> On Thursday, August 27, 2020 at 10:07:55 PM UTC+5:30 alon...@gmail.com 
>> wrote:
>>
>>> On Thu, Aug 27, 2020 at 8:47 AM Prashanth Nethi  
>>> wrote:
>>>
 Thanks Alon! That explains it! Yeah I should have thought a little 
 deeper.

 I am just posting my follow up question in case you did not get a 
 chance to look at it.

 One follow up question. May be a dumb one. What could be the potential 
 problems with ALLOW_MEMORY_GROWTH missing in PThreads mode? I see that 
 when 
 the Wasm is instantiated, the overall memory that the Chrome tab was 
 taking 
 was similar to the one taken by the WASM built with ALLOW_MEMORY_GROWTH. 
 Is 
 it that, we will not be able to instantiate WASM on low end devices if 
 built with ALLOW_MEMORY_GROWTH=0?


>>> I'm not sure what you're asking here?
>>>
>>> In general, not having memory growth enabled means that memory can't 
>>> grow. So if you need more than the initial value, the program will hit a 
>>> problem. I don't think there's anything special to pthreads in that case. 
>>> (The reverse, having growth *enabled*, does have downsides for pthreads as 
>>> the JS use of memory becomes somewhat slower.)
>>>
>>> Regards,
 Prashanth Nethi

 On Thursday, August 27, 2020 at 2:37:07 AM UTC+5:30 alon...@gmail.com 
 wrote:

> My guess is that's because of the behavior of std::vector and how it 
> resizes. Over those appends it will malloc and free repeatedly and that 
> may 
> cause fragmentation that prevents a final larger size, which must be a 
> single contiguous region. The second version allocates many smaller ones, 
> not a single contiguous region.
>
> - Alon
>
>
> On Tue, Aug 25, 2020 at 11:24 PM Prashanth Nethi  
> wrote:
>
>> Thanks Alon! So here is something very weird. I could get the memory 
>> usage go all the way to 2GB when I changed my testing code. This was my 
>> original test code. So basically I was just adding elements to 
>> std::vector 
>> infinitely.
>>
>> class 

Re: .emscripten config not picked up

[r0l...@freemail.hu]

Already posted a few times but all of them gets deleted automatically. 
Testing if this one will remain...

s...@google.com a következőt írta (2020. augusztus 27., csütörtök, 21:23:05 
UTC+2):

> This is because emsdk doesn't read the config file at all, it only writes 
> it during `activate`.
>
> I think that is doing on is that emsdk is not aware that you have 
> installed binaryen or llvm so it doesn't think those tool are part of the 
> config. Tools have to be "activate"d before emsdk will consider them part 
> of the config I believe.
>
> If you are building your own tools and writing your own `.emscripten` file 
> I suggest you put the config file in the `emscripten` directory itself.
> This is the first place emscripten will look for the config file (alongside 
> emcc).
>
> Then you should be able to simply add the emscripten directory to your 
> PATH and avoid emsdk_env completely. (since you are not really using the 
> emsdk at this point).
>
> Should we look into the reasons why "it unfortunately cannot be built from 
> source"?  Is it a netbsd specific thing?   
>
> cheers,
> sam
>
>
>
> On Wed, Aug 26, 2020 at 8:55 AM r0l...@freemail.hu  
> wrote:
>
>>
>> Hi All,
>>
>> I've just installed and activated emsdk 1.39.8 on my NetBSD box and as it 
>> unfortunately cannot be built from source I built binaryen and fastcomp 
>> separately so that I can simply set up my .emscripten config so that 
>> LLVM_ROOT and BINARYEN_ROOT point to those directories. So the relevant 
>> paths look like this in emsdk/.emscripten:
>>
>> LLVM_ROOT = '/home/r0ller/fastcomp/emscripten-fastcomp/build'
>> BINARYEN_ROOT = ' /home/r0ller/binaryen'
>> EMSCRIPTEN_ROOT = emsdk_path + '/upstream/emscripten'
>>
>> However, no matter what I set up for LLVM_ROOT or BINARYEN_ROOT (paths 
>> ending in /bin or not as above), when issuing 'source emsdk_env.sh', it 
>> spits out only one environment variable that gets set up:
>>
>> EMSDK = /home/r0ller/emsdk
>>
>> What am I doing wrong? By the way, is there any way to build 1.39.8 from 
>> source like certain tagged versions?
>>
>> Thanks,
>> r0ller
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "emscripten-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to emscripten-disc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/emscripten-discuss/c5e813b3-37f7-4b30-9bd0-9e4fbac1cdecn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/9e2709fb-01da-4682-b529-484a6c7c7bb1n%40googlegroups.com.

Re: .emscripten config not picked up

[r0l...@freemail.hu]

Hi Sam,

Thanks for the explanation. Actually, that's what I concluded myself so I 
just wrote my own .emscripten config (LLVM_ROOT path above was actually not 
correct as I had to point it to the directory where the package manager 
installed llvm) and copied it to my home and to the emsdk directory as well 
which kicked the whole stuff in but as you mentioned I don't need to use 
emsdk_env.sh at all. Now even my QtCreator recognises emscripten and can 
compile qt programs to wasm :)

Concerning source compilation of emscripten on netbsd: it's definitely 
easier to install emscripten that way since one does not need to build 
binaryen (or fastcomp) separately. So usually I use emscripten versions 
which emsdk can install from source but now I wanted to use it with Qt 
which is bound to a certain version e.g. qt5.15 is bound to emscripten 
1.39.8, see:

https://doc.qt.io/qt-5/wasm.html

However, 1.39.8 cannot be installed from source via emsdk so that's where 
my whole story began... Though it worked out in the end fine, it would have 
been easier to simply issue an emsdk install command instead of cloning and 
compiling binaryen separately. Nevertheless, most stuff can be installed 
via the netbsd package manager like llvm, lld, node, etc. so it was only 
binaryen that added an extra turn to the screw.

Thanks again for your answer!

Best regards,
r0ller
s...@google.com a következőt írta (2020. augusztus 27., csütörtök, 21:23:05 
UTC+2):

> This is because emsdk doesn't read the config file at all, it only writes 
> it during `activate`.
>
> I think that is doing on is that emsdk is not aware that you have 
> installed binaryen or llvm so it doesn't think those tool are part of the 
> config. Tools have to be "activate"d before emsdk will consider them part 
> of the config I believe.
>
> If you are building your own tools and writing your own `.emscripten` file 
> I suggest you put the config file in the `emscripten` directory itself.
> This is the first place emscripten will look for the config file (alongside 
> emcc).
>
> Then you should be able to simply add the emscripten directory to your 
> PATH and avoid emsdk_env completely. (since you are not really using the 
> emsdk at this point).
>
> Should we look into the reasons why "it unfortunately cannot be built from 
> source"?  Is it a netbsd specific thing?   
>
> cheers,
> sam
>
>
>
> On Wed, Aug 26, 2020 at 8:55 AM r0l...@freemail.hu  
> wrote:
>
>>
>> Hi All,
>>
>> I've just installed and activated emsdk 1.39.8 on my NetBSD box and as it 
>> unfortunately cannot be built from source I built binaryen and fastcomp 
>> separately so that I can simply set up my .emscripten config so that 
>> LLVM_ROOT and BINARYEN_ROOT point to those directories. So the relevant 
>> paths look like this in emsdk/.emscripten:
>>
>> LLVM_ROOT = '/home/r0ller/fastcomp/emscripten-fastcomp/build'
>> BINARYEN_ROOT = ' /home/r0ller/binaryen'
>> EMSCRIPTEN_ROOT = emsdk_path + '/upstream/emscripten'
>>
>> However, no matter what I set up for LLVM_ROOT or BINARYEN_ROOT (paths 
>> ending in /bin or not as above), when issuing 'source emsdk_env.sh', it 
>> spits out only one environment variable that gets set up:
>>
>> EMSDK = /home/r0ller/emsdk
>>
>> What am I doing wrong? By the way, is there any way to build 1.39.8 from 
>> source like certain tagged versions?
>>
>> Thanks,
>> r0ller
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "emscripten-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to emscripten-disc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/emscripten-discuss/c5e813b3-37f7-4b30-9bd0-9e4fbac1cdecn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/fb0f4607-a889-4288-a157-083f34863a11n%40googlegroups.com.

Re: .emscripten config not picked up

[r0l...@freemail.hu]

I realized in the meantime that LLVM_ROOT was not set correctly as it 
should point to the package bin directory of llvm but it does not help as 
the config file is simply ignored. So I decided to create my own script 
that sets the environment variables via exports and that worked out.

r0l...@freemail.hu a következőt írta (2020. augusztus 26., szerda, 17:55:49 
UTC+2):

>
> Hi All,
>
> I've just installed and activated emsdk 1.39.8 on my NetBSD box and as it 
> unfortunately cannot be built from source I built binaryen and fastcomp 
> separately so that I can simply set up my .emscripten config so that 
> LLVM_ROOT and BINARYEN_ROOT point to those directories. So the relevant 
> paths look like this in emsdk/.emscripten:
>
> LLVM_ROOT = '/home/r0ller/fastcomp/emscripten-fastcomp/build'
> BINARYEN_ROOT = ' /home/r0ller/binaryen'
> EMSCRIPTEN_ROOT = emsdk_path + '/upstream/emscripten'
>
> However, no matter what I set up for LLVM_ROOT or BINARYEN_ROOT (paths 
> ending in /bin or not as above), when issuing 'source emsdk_env.sh', it 
> spits out only one environment variable that gets set up:
>
> EMSDK = /home/r0ller/emsdk
>
> What am I doing wrong? By the way, is there any way to build 1.39.8 from 
> source like certain tagged versions?
>
> Thanks,
> r0ller
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/7952b879-7806-4b17-ac9f-e517500b7506n%40googlegroups.com.