Hi Endre, We are facing (many years later!) very similar problems to yours, in a cross-platform engine targeting the browser via Emscripten.
For example, in a somewhat large scene, the Emscripten heap may report a total 500mb, but the browser process is consuming 2+GB. Spikes in memory appear to coincide with mesh and texture processing in third party libraries like "assimp," which we run in web workers; whereas our own code manages memory more carefully in pools/arenas, and the heap tooling and OS metrics agree. I'm guessing it's caused by many small heap allocations made by the codec libraries, but haven't been able to confirm that yet. Have you figured out the root cause and/or found any way to mitigate these issues? Thank you! -James On Wednesday, September 18, 2013 at 11:54:07 PM UTC-7, Endre Kolláth wrote: > > Ok thanks, we'll try it. > > > On 18 September 2013 22:48, Alon Zakai <[email protected] <javascript:>> > wrote: > >> I don't know what those chrome extensions measure. Perhaps JS heap for >> them does not include typed arrays, which could explain why the heap is not >> always bigger than the typed array. >> >> Chrome has a heap profiler, so you should be able to see what the heap >> elements actually are. >> >> There is no switch to change memory increase rate, but see enlargeMemory >> in src/preamble.js, easy to change. >> >> - Alon >> >> >> >> On Mon, Sep 16, 2013 at 9:07 AM, Endre Kolláth <[email protected] >> <javascript:>> wrote: >> >>> Yes, ALLOW_MEMORY_GROW is defined to 1. >>> >>> A special offline build is created with detailed export/import infos to >>> console with emscripten memory usage and JS heap size (for chrome). >>> This will write lines to the console like this: >>> Parsing of mesh finished. Em: 14 / 16 Mb JS: 40 / 61 Mb >>> For Em, DYNAMICTOP and TOTAL_MEMORY is used, >>> for JS, window.performance.memory.usedJSHeapSize and totalJSHeapSize is >>> used (for Chrome only) >>> >>> I wanted to attach a big object file, but 8Mb size limit is added to >>> this message board. >>> You can experiment with importing/exporting big objects. >>> Now browser crash is not frequent, and mostly happen for Windows only. >>> >>> There are some strange things, sometimes JS heap size is smaller then Em >>> memory size. How can this happen? >>> >>> > you need to consider memory like any C/C++ app - emscripten compiled >>> code uses the same amount of memory as C/C++ code natively does, structure >>> layout is the same, etc >>> >>> What I found is that Emscripten is used to allocate powers of 2 size >>> memory for heap simulating. So, after 256 Mb it will need 512 Mb, and after >>> 1024 Mb etc.. And I think it allocates the new one and copies the old one >>> to it, so it needs very big continuous memory blocks at a time. So it may >>> be true that amount of memory is the same but the distribution is not. >>> While the desktop build allocates more smaller blocks, emscripten tries to >>> allocate very big ones. >>> >>> A simple workaround might be to grow memory at a smaller rate, not to >>> duplicate it always. Is there any switch or compile option to set this? >>> >>> Regards, >>> Endre >>> >>> >>> On 14 September 2013 02:02, Alon Zakai <[email protected] <javascript:> >>> > wrote: >>> >>>> Ok, then yes, looks like it just runs out of memory when it enlarges >>>> the heap. To confirm, do you have the heap growth option on? >>>> >>>> If that's it, then you need to consider memory like any C/C++ app - >>>> emscripten compiled code uses the same amount of memory as C/C++ code >>>> natively does, structure layout is the same, etc. >>>> >>>> - Alon >>>> >>>> >>>> >>>> On Fri, Sep 13, 2013 at 1:18 AM, Endre Kolláth <[email protected] >>>> <javascript:>> wrote: >>>> >>>>> For Firefox, it's not crashes, what's happening is this: >>>>> On the console, at some point: >>>>> >>>>> uncaught exception: out of memory >>>>> >>>>> Then everything goes wrong, like >>>>> >>>>> Error: WebGL: readPixels: buffer too small >>>>> TypeError: FUNCTION_TABLE[HEAP32[((HEAP32[(r2 >> 2)] + 16) >> 2)]] >>>>> is not a function >>>>> >>>>> etc... >>>>> and soon after it will be unresponsive (it really crashes). >>>>> >>>>> >>>>> For Chrome, it says: >>>>> Aw, Snap! >>>>> Something went wrong....etc. >>>>> >>>>> The console window says: >>>>> Inspected target crashed. >>>>> >>>>> That's all. >>>>> There's no reported crashes in Chrome nor in Firefox. >>>>> >>>>> >>>>> Endre >>>>> >>>>> >>>>> >>>>> On 13 September 2013 07:02, Alon Zakai <[email protected] >>>>> <javascript:>> wrote: >>>>> >>>>>> There are many types of crashes - in firefox and chrome, you can see >>>>>> browser details (about:crashes and chrome://crashes respectively). Links >>>>>> to >>>>>> those could help see if this is indeed a memory issue, or perhaps large >>>>>> meshes trigger some other crash. >>>>>> >>>>>> Are you using the growable heap option? Then the heap can grow, but >>>>>> even so, it should never crash the browser - the browser should report >>>>>> OOM >>>>>> when the array is attempted to be allocated. >>>>>> >>>>>> If you can make a testcase that shows the problem, that would be >>>>>> helpful, we can investigate here and then file bugs on the browsers as >>>>>> necessary. >>>>>> >>>>>> - Alon >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 12, 2013 at 2:06 AM, Endre Kolláth <[email protected] >>>>>> <javascript:>> wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> We are developing a browser-based sculpting software. We use C++ and >>>>>>> emscripten to compile it to js from the beginning. You can try it free >>>>>>> at >>>>>>> www.leopoly.com >>>>>>> >>>>>>> However, as the project growing larger and larger, we have to face >>>>>>> some "bugs" where the browser crashes (Chrome and FF too). We figured >>>>>>> out >>>>>>> the the reason might be the large memory usage of the js engine. In the >>>>>>> browser the problem arises when we import or export large meshes. In >>>>>>> these >>>>>>> operations there are lots of memory allocations and deallocations, std >>>>>>> container creations, growings and destroys etc. All these happen >>>>>>> between >>>>>>> two animation frames. >>>>>>> >>>>>>> There are some possibilities we think of: >>>>>>> 1) The many memory operations fragmentate the large typed array used >>>>>>> to simulate the heap, then it needs to grow and grow, then lastly the >>>>>>> Js >>>>>>> engine runs out of memory. >>>>>>> 2) Even if the typed array can "shrink", I think that the browser >>>>>>> garbage collector is not working between two animation frames. >>>>>>> What do you suggest, how can we work around this problem? We may use >>>>>>> some shared buffers (create custom allocators or something else), may >>>>>>> distribute the processing between many frames, but all that will >>>>>>> require a >>>>>>> lot of work and make the code very ugly and hardly readable. Is there >>>>>>> any >>>>>>> other tricks? Can emscripten defragment the memory when we need? >>>>>>> Also, there are some other strange things about memory. Sometimes we >>>>>>> found that even with low memory usage in JS (we check that in Chrome >>>>>>> only >>>>>>> with window.performance.use/totalJSHeapSize), the browser memory usage >>>>>>> is >>>>>>> very large. For example with 60-80 Mbs of JS Heap size the browser >>>>>>> normally >>>>>>> runs with approx 150 Mb, but sometimes it takes 500 Mb or even 1 Gb. >>>>>>> However it's highly browser-implementation specific stuff, but what can >>>>>>> explain this? >>>>>>> >>>>>>> Thanks, >>>>>>> Endre >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "emscripten-discuss" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected] >>>>>>> <javascript:>. >>>>>>> >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "emscripten-discuss" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/emscripten-discuss/CZiENma0uU0/unsubscribe >>>>>> . >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected] <javascript:>. >>>>>> >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "emscripten-discuss" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected] >>>>> <javascript:>. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "emscripten-discuss" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/emscripten-discuss/CZiENma0uU0/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected] <javascript:>. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "emscripten-discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] >>> <javascript:>. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "emscripten-discuss" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/emscripten-discuss/CZiENma0uU0/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
