All,
with lots of debug lines I am able to isolate one error in my code,
however, I have not been able yet to build a reduced test case for it.
I tested this with a single worker to avoid races.
The pointer in question got allocated in the worker first. Later I needed
to enlarge the size. Using code like the following snippet I detected that
the content of the reallocated block was different from the original one
(considering the old size only, of course):
Assume p is the block that was allocated initially with size oldSize:
#define smallChunkSize 256
uint8_T test1[smallChunkSize],test2[smallChunkSize];
uint8Ptr_T p = (uint8Ptr_T)myP;
uint8Ptr_T newP = 0L;
int i,n;
n = min(oldSize,smallChunkSize); memcpy(test1,p,n);
newP = realloc(p,newSize);
assert(newP != 0L);
memcpy(test2,newP,n);
for (i=0;i<n;i++){
assert(test1[i] == test2[i]);
}
The newP contained garbage.
I can't say when this happens, but I can reproduce it with my code. A wild
guess is that it happens during a heap resize, but I don't have tools to
verify this.
Just as a reminder, if I set the number of workers to 0 and run this code
in a single thread nothing happens.
As a workaround I tried to allocate a new block, copy everything from the
old block to the new one, and dispose of the old block. That resolves this
particular problem, but then other problems arise.
So I will keep searching.
Cheers,
Dieter
jj schrieb am Sonntag, 28. Mai 2023 um 14:51:57 UTC+2:
> Oops, that looks like an oversight with emmalloc + multithreading +
> memvalidate assertion checks implementation. Posted a fix PR to that at
> https://github.com/emscripten-core/emscripten/pull/19465
>
> On Sun, May 28, 2023 at 11:32 AM 'Dieter Weidenbrück' via
> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>
>> One more detail:
>> - use hello_wasm_worker.c from the test lib without changes
>> - use
>> emcc ./hello_wasm_worker.c ^
>> -g3 ^
>> --source-map-base ./ ^
>> -gsource-map ^
>> -s ALLOW_MEMORY_GROWTH=1 ^
>> -s ENVIRONMENT=web,worker ^
>> -s SUPPORT_ERRNO=0 ^
>> -sWASM_WORKERS ^
>> -s ASSERTIONS=2 ^
>> -sMALLOC=emmalloc-memvalidate ^
>> -o hello_wasm_worker.html
>>
>> Result:
>> [image: hello_error.png]
>>
>> Tested on Dell XPS17, 12th Gen Intel(R) Core(TM) i9-12900HK 2.50 GHz,
>> 64 GB RAM, Windows 11 64bit
>> Browsers: current Chrome and Firefox
>>
>> Hope this helps,
>> Dieter
>> Dieter Weidenbrück schrieb am Samstag, 27. Mai 2023 um 18:03:37 UTC+2:
>>
>>> JJ,
>>> thanks for your recommendations. Here are some first results (I hope it
>>> is ok to post these screenshots here, if not I can post links):
>>>
>>> SAFE_HEAP switched off.
>>>
>>> Using *emmalloc*:
>>> First try:
>>> [image: emmalloc.png]
>>>
>>> Second try:
>>> [image: emmalloc2.png]
>>> Third try (try == same binary, just restarted and performed the same
>>> steps)
>>> [image: emmalloc3.png]
>>> So the behavior is not really predictable nor reproducable.
>>>
>>> Using *emmalloc-debug*:
>>> First try:
>>> [image: emmalloc-debug.png]
>>>
>>> Second try:
>>> [image: emmalloc-debug2.png]
>>>
>>> In both cases Chrome was left in a state using 98+% of the CPU without
>>> returning. I had to kill Chrome
>>>
>>> Using *emmalloc-memvalidate*:
>>> [image: emmalloc-memvalidate.png]
>>> Right after startup without any interaction from my side, the app
>>> aborted with this message.
>>>
>>> Setting the number of wasm workers used to 0 (i.e. run as a single
>>> thread app), everything worked fine with emmalloc and emmalloc-debug. For
>>> emmalloc-memvalidate I had to #if 0 all calls to wasm workers and remove
>>> -sWASM_WORKERS, then the app would run without problems.
>>>
>>> *A comment about debugging in general:*
>>> source code debugging in a browser is a tedious task. Every line
>>> requires several clicks/keys to advance, and I have not found a way to
>>> inspect the values of variables.
>>> If I have to run through a loop with a couple of hundred or thousand
>>> iterations this doesn't help a lot to raise my enthusiasm.
>>> So what I do is, I have set up a project in Visual Studio with a
>>> slightly different main function as a cmdline-app. From there I can call
>>> specific functions and debug them with all bells and whistles.
>>> So I have some good confidence that the code works before entering the
>>> wasm/browser world.
>>>
>>> Cheers,
>>> Dieter
>>>
>>> jj schrieb am Samstag, 27. Mai 2023 um 01:25:24 UTC+2:
>>>
>>>> This is a good bug report about debuggability shortcomings of Wasm on
>>>> two accounts:
>>>> 1. when we had the old JS-based SAFE_HEAP feature, the feature said
>>>> something along the lines of "segmentation fault: attempted to write i32
>>>> value x to memory address y", so one could deduce whether the memory
>>>> address was zero, out of wasm memory bounds, or compare it against the
>>>> memory map, e.g. if it is out of the dynamic malloc region memory bounds
>>>> even if it might have been a dynamically allocated ptr. And one could
>>>> verify the alignment as well, to use that as a heuristic to figure out if
>>>> the ptr was stomped on and was garbled.
>>>>
>>>> The wasm-based SAFE_HEAP feature does not provide any of above, but
>>>> only prints that "Aborted(segmentation fault)".
>>>>
>>>> 2. Wasm browser debuggers should be able to show the line of code that
>>>> does the segfault operation. Not sure if the browser debuggers are yet
>>>> powerful enough to do that. Also currently we have no integration with
>>>> those - maybe a SAFE_HEAP should generate a wasm trap instead of going out
>>>> to JS to throw a JS exception? (although catching that exception should
>>>> allow one to still manually inspect the callstack to find the offending
>>>> line)
>>>>
>>>> Given that there exists an error print into dlmalloc, might be
>>>> interesting to see how the error message adjust if you build against
>>>> emmalloc, by using the linker flag -sMALLOC=emmalloc. There also exist
>>>> variants -sMALLOC=emmalloc-debug, -sMALLOC=emmalloc-memvalidate, and
>>>> -sMALLOC=emmalloc-memvalidate-verbose with increasing levels of dynamic
>>>> memory allocator internal consistency checks (and increasing levels of
>>>> slowness and log spamming).
>>>>
>>>> If nothing stands out, getting to a reduced repro test case would be
>>>> helpful.
>>>>
>>>> On Fri, May 26, 2023 at 11:47 PM 'Sam Clegg' via emscripten-discuss <
>>>> emscripte...@googlegroups.com> wrote:
>>>>
>>>>> Can I ask why you chose not to use pthreads to start with? I'd like
>>>>> to understand better why folks would choose wasm workers over pthreads.
>>>>>
>>>>> On Fri, May 26, 2023 at 3:25 AM 'Dieter Weidenbrück' via
>>>>> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>>>>>
>>>>>> Hi Sam,
>>>>>> IIRC, when I started with Emscripten a while ago the program would
>>>>>> abort in case of a memory error. As my app is comparable to a desktop
>>>>>> app,
>>>>>> this was not acceptable, so I set ABORTING_MALLOC to 0. I understand
>>>>>> that
>>>>>> this flag has a different meaning today. Here is how all my allocation
>>>>>> calls work:
>>>>>>
>>>>>> Error_T allocMemPtr(MemPtr_T *p,uint32_T size,boolean_T clear) {
>>>>>> _MemPtr_T mp;
>>>>>>
>>>>>> if (clear)
>>>>>> mp = (_MemPtr_T)calloc(1,size + sizeof(_Mem_T));
>>>>>> else
>>>>>> mp = (_MemPtr_T)malloc(size + sizeof(_Mem_T));
>>>>>> if (mp) {
>>>>>> mp->size = size;
>>>>>> *p = (MemPtr_T)((char_T*)mp + sizeof(_Mem_T));
>>>>>> return kErr_NoErr;
>>>>>> }
>>>>>> return kErr_MemErr;
>>>>>> }
>>>>>> Error_T setMemPtrSize(MemPtr_T *p,uint32_T size){
>>>>>> _MemPtr_T m = _MP(*p);
>>>>>> MemPtr_T newPtr;
>>>>>>
>>>>>> newPtr = realloc(m,size + sizeof(_Mem_T));
>>>>>> if (newPtr) {
>>>>>> m = (_MemPtr_T)newPtr;
>>>>>> m->size = size;
>>>>>> *p = (MemPtr_T)((char_T*)m + sizeof(_Mem_T));
>>>>>> return kErr_NoErr;
>>>>>> }
>>>>>> return kErr_MemErr;
>>>>>> }
>>>>>>
>>>>>> So I should catch all errors. However, errors (i.e. return value ==
>>>>>> 0) are not reported by malloc or calloc during the problems I am
>>>>>> experiencing. I added debug lines, but not a single failure was recorded.
>>>>>> Removing ABORTING_MALLOC did not result in any change of error
>>>>>> outcome.
>>>>>>
>>>>>> I see two different behaviors now:
>>>>>> - setting up workers and checking that they run by
>>>>>> static void startUpWorker(void) {
>>>>>> #ifdef __EMSCRIPTEN__
>>>>>> int32_T w = emscripten_wasm_worker_self_id();
>>>>>> if (! emscripten_current_thread_is_wasm_worker()){
>>>>>> EM_ASM_({
>>>>>> console.log("Error: No worker: " + $0);
>>>>>> },w);
>>>>>> }
>>>>>> #endif //__EMSCRIPTEN__
>>>>>> }
>>>>>> - then I do my stuff and receive about 10 of the "Uncaught
>>>>>> RuntimeError: memory access out of bounds" errors.
>>>>>> - no failures of malloc/calloc recognized
>>>>>>
>>>>>> The second behavior is
>>>>>> - in main() I call this routine:
>>>>>> static void memtest(void) {
>>>>>> #define NUM_CHUNKS 15
>>>>>> const int CHUNK_SIZE = 100 * 1024 * 1024;
>>>>>> int i;
>>>>>> void* p[NUM_CHUNKS];
>>>>>> Error_T err = kErr_NoErr;
>>>>>>
>>>>>> for (int i = 0; i < NUM_CHUNKS; i++) {
>>>>>> err = allocMemPtr(&p[i],CHUNK_SIZE,FALSE); //see function above
>>>>>> if (err != kErr_NoErr || p[i] == NULLPTR) {
>>>>>> printf("Error chunk %d\n",i);
>>>>>> break;
>>>>>> }
>>>>>> }
>>>>>> for (int i = 0; i < NUM_CHUNKS; i++) {
>>>>>> if (p[i] == NULLPTR)
>>>>>> break;
>>>>>> disposeMemPtr(p[i]);
>>>>>> }
>>>>>> }
>>>>>> - then I start up the workers as described above
>>>>>> - then I do my stuff
>>>>>> - sometimes this results in error free behavior, but not always. If
>>>>>> an error occurs, I only get one "Uncaught RuntimeError" message.
>>>>>>
>>>>>> I am pretty confident that I handle memory allocation correctly,
>>>>>> because my background is in development of desktop apps in C for 30+
>>>>>> years,
>>>>>> and there you better not have any leaks and keep the app running
>>>>>> whenever
>>>>>> possible. So I must be doing something wrong when dealing with multiple
>>>>>> threads.
>>>>>> I will try out pthreads next, because I have no idea anymore what the
>>>>>> cause could be here.
>>>>>>
>>>>>> Cheers,
>>>>>> Dieter
>>>>>> s...@google.com schrieb am Donnerstag, 25. Mai 2023 um 23:20:33
>>>>>> UTC+2:
>>>>>>
>>>>>>> Is there some reason you added `-sABORTING_MALLOC=0`.. that looks a
>>>>>>> little suspicious, since it means the program can continue after malloc
>>>>>>> fails.. which mean that any callsite that doesn't check the return
>>>>>>> value of
>>>>>>> malloc can lead to segfaults. If you remove that setting does the
>>>>>>> behaviour change?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, May 25, 2023 at 1:27 PM 'Dieter Weidenbrück' via
>>>>>>> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>>>>>>>
>>>>>>>> Hi Sam,
>>>>>>>>
>>>>>>>> I can run the code in a single thread without problems, and I have
>>>>>>>> done that for a while. So I assume that the code is stable.
>>>>>>>>
>>>>>>>> Here is the command line I use in a .bat file:
>>>>>>>> emcc ./src/main.c ^
>>>>>>>> ...
>>>>>>>> ./src/w_com.c ^
>>>>>>>> -I ./include/ ^
>>>>>>>> -g3 ^
>>>>>>>> --source-map-base ./ ^
>>>>>>>> -gsource-map ^
>>>>>>>> -s ALLOW_MEMORY_GROWTH=1 ^
>>>>>>>> -s ENVIRONMENT=web,worker ^
>>>>>>>> --shell-file ./index_template.html ^
>>>>>>>> -s SUPPORT_ERRNO=0 ^
>>>>>>>> -s MODULARIZE=1 ^
>>>>>>>> -s ABORTING_MALLOC=0 ^
>>>>>>>> -sWASM_WORKERS ^
>>>>>>>> -s "EXPORT_NAME='wasmMod'" ^
>>>>>>>> -s EXPORTED_FUNCTIONS="['_malloc','_free','_main']" ^
>>>>>>>> -s EXPORTED_RUNTIME_METHODS=
>>>>>>>> "['cwrap','UTF16ToString','UTF8ToString','stringToUTF8','allocateUTF8']"
>>>>>>>>
>>>>>>>> ^
>>>>>>>> -o index.html
>>>>>>>>
>>>>>>>> I will start familiarizing myself with pthreads to test whether
>>>>>>>> that would work better.
>>>>>>>>
>>>>>>>> BTW, as an old C programmer I am fascinated by emscripten and its
>>>>>>>> possibilities. Excellent job!
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Dieter
>>>>>>>>
>>>>>>>> s...@google.com schrieb am Donnerstag, 25. Mai 2023 um 20:29:58
>>>>>>>> UTC+2:
>>>>>>>>
>>>>>>>>> This looks like some kind of memory corruption, most likely due to
>>>>>>>>> the use of muiltithreading/wasm_workers Are you able to build a
>>>>>>>>> single
>>>>>>>>> threaded version of your program, or one that uses normal pthreads
>>>>>>>>> rather
>>>>>>>>> than wasm workers?
>>>>>>>>>
>>>>>>>>> Also, can you share the full link command you are using?
>>>>>>>>>
>>>>>>>>> cheers,
>>>>>>>>> sam
>>>>>>>>>
>>>>>>>>> On Thu, May 25, 2023 at 9:20 AM 'Dieter Weidenbrück' via
>>>>>>>>> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>>>>>>>>>
>>>>>>>>>> This is a memory snapshot when using SAFE_HEAP. So here I am
>>>>>>>>>> quite below the browser limits, still the segfault occurs in
>>>>>>>>>> different
>>>>>>>>>> places.
>>>>>>>>>> Ignore the first console line, it results from Norton Utilities I
>>>>>>>>>> think.
>>>>>>>>>>
>>>>>>>>>> [image: error2.png]
>>>>>>>>>>
>>>>>>>>>> Dieter Weidenbrück schrieb am Donnerstag, 25. Mai 2023 um
>>>>>>>>>> 18:06:27 UTC+2:
>>>>>>>>>>
>>>>>>>>>>> Hi Sam,
>>>>>>>>>>> I noticed already that I am bumping against browser limits,
>>>>>>>>>>> especially with sanitizer switched on, so I reduced the
>>>>>>>>>>> pre-allocation
>>>>>>>>>>> calls.
>>>>>>>>>>> It turns out that asan uses so much memory that I can't use it
>>>>>>>>>>> to analyze this case.
>>>>>>>>>>>
>>>>>>>>>>> I use
>>>>>>>>>>> -s ALLOW_MEMORY_GROWTH=1
>>>>>>>>>>> but don't specify any MAXIMUM_MEMORY.
>>>>>>>>>>>
>>>>>>>>>>> No pthreads version so far. I might try this next.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Dieter
>>>>>>>>>>>
>>>>>>>>>>> s...@google.com schrieb am Donnerstag, 25. Mai 2023 um 17:55:41
>>>>>>>>>>> UTC+2:
>>>>>>>>>>>
>>>>>>>>>>>> Firstly, if you are allocating 1.8Gb you are likely pushing up
>>>>>>>>>>>> against browser limits. Are you specifying a MAXIMUM_MEMORY of
>>>>>>>>>>>> larger than
>>>>>>>>>>>> 2GB?
>>>>>>>>>>>>
>>>>>>>>>>>> Secondly, it looks like you are using wasm workers, which are
>>>>>>>>>>>> still relatively new. Do you have a version of your code that
>>>>>>>>>>>> uses
>>>>>>>>>>>> pthreads instead? It might tell is if the issue is related to
>>>>>>>>>>>> wasm workers.
>>>>>>>>>>>>
>>>>>>>>>>>> cheers,
>>>>>>>>>>>> sam
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, May 25, 2023 at 8:06 AM 'Dieter Weidenbrück' via
>>>>>>>>>>>> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The joy was premature, even with pre-allocated heap size
>>>>>>>>>>>>> segfaults occur. :(
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dieter Weidenbrück schrieb am Donnerstag, 25. Mai 2023 um
>>>>>>>>>>>>> 16:28:37 UTC+2:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> All,
>>>>>>>>>>>>>> I am experiencing segmentation faults when using wasm workers.
>>>>>>>>>>>>>> Overview:
>>>>>>>>>>>>>> I am working on a project with considerable 3D data sets. The
>>>>>>>>>>>>>> code has been stable for a while when running in the main thread
>>>>>>>>>>>>>> alone.
>>>>>>>>>>>>>> Then I started using js workers (no shared memory), and again
>>>>>>>>>>>>>> all was well.
>>>>>>>>>>>>>> Now I've switched to SharedArrayBuffers and wasm workers, and
>>>>>>>>>>>>>> I keep running into random problems.
>>>>>>>>>>>>>> I have prepared the code such that I can run with 0 workers
>>>>>>>>>>>>>> up to hardware.concurrency workers. All is well with 0 workers,
>>>>>>>>>>>>>> but as soon
>>>>>>>>>>>>>> as I use one or more workers, I keep getting segfaults because
>>>>>>>>>>>>>> of invalid
>>>>>>>>>>>>>> pointers, access out of bounds and similar.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What happens in main thread and what in the wasm workers:
>>>>>>>>>>>>>> I allocate all objects in the main thread when importing the
>>>>>>>>>>>>>> 3D file. Then i fire off a function for each object that will do
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>> serious calculations of the data, including allocating and
>>>>>>>>>>>>>> disposing of
>>>>>>>>>>>>>> memory. The workers allocate approx. 300 to 400 MB in addition
>>>>>>>>>>>>>> to the main
>>>>>>>>>>>>>> thread. All this happens in the same sharedArrayBuffer, of
>>>>>>>>>>>>>> course.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here is what I've tried so far:
>>>>>>>>>>>>>> - compiling with SAFE_HEAP=1
>>>>>>>>>>>>>> not a lot of helpful information,
>>>>>>>>>>>>>> - compiling with -fsanitize=address
>>>>>>>>>>>>>> everything works without problems here!
>>>>>>>>>>>>>> - compiling with ASSERTIONS=2
>>>>>>>>>>>>>> gave me this information:
>>>>>>>>>>>>>> [image: error.png]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To me it looks like another resize call is executed while
>>>>>>>>>>>>>> other workers keep working on the buffer, and then something
>>>>>>>>>>>>>> gets into
>>>>>>>>>>>>>> conflict.
>>>>>>>>>>>>>> To test this, I allocated 1.8 GB right after startup in the
>>>>>>>>>>>>>> main thread and disposed the mem blocks again just to trigger
>>>>>>>>>>>>>> heap resize.
>>>>>>>>>>>>>> After that everything works like a charm.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there anything I am doing wrong?
>>>>>>>>>>>>>> Sorry for not providing a sample, but there is a lot of code
>>>>>>>>>>>>>> involved, and it is not easy to simulate this behavior. Happy to
>>>>>>>>>>>>>> answer
>>>>>>>>>>>>>> questions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> All comments are appreciated.
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Dieter
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "emscripten-discuss" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>>> it, send an email to emscripten-disc...@googlegroups.com.
>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/80d56314-59d8-4332-bb2e-ebe00fe52ea3n%40googlegroups.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/80d56314-59d8-4332-bb2e-ebe00fe52ea3n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>>>> .
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "emscripten-discuss" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to emscripten-disc...@googlegroups.com.
>>>>>>>>>>
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/cfc03512-f69f-44b0-8c14-1f1a8e4ffe9fn%40googlegroups.com
>>>>>>>>>>
>>>>>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/cfc03512-f69f-44b0-8c14-1f1a8e4ffe9fn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "emscripten-discuss" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to emscripten-disc...@googlegroups.com.
>>>>>>>>
>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/e568e189-4259-460f-9601-e7996927cdb7n%40googlegroups.com
>>>>>>>>
>>>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/e568e189-4259-460f-9601-e7996927cdb7n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "emscripten-discuss" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to emscripten-disc...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/b20d2de8-2532-4441-b8fc-3ef8f049f7f0n%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/b20d2de8-2532-4441-b8fc-3ef8f049f7f0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "emscripten-discuss" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to emscripten-disc...@googlegroups.com.
>>>>>
>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/emscripten-discuss/CAL_va28k7RyF2n-x6B8M9pbgri2bCDCQA7N%2BG7x-6GVP%2Bpqumg%40mail.gmail.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/CAL_va28k7RyF2n-x6B8M9pbgri2bCDCQA7N%2BG7x-6GVP%2Bpqumg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "emscripten-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to emscripten-disc...@googlegroups.com.
>>
> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/emscripten-discuss/7de28d6a-0cc9-4953-b9eb-bb9a2844ac9dn%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/emscripten-discuss/7de28d6a-0cc9-4953-b9eb-bb9a2844ac9dn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/emscripten-discuss/c438a02c-af55-4c44-9999-064c49a27d78n%40googlegroups.com.
