Hmm, well the stack trace is what I mentioned previously. Is there anything specific that would help you to track down the problem? I am unfortunately unable to provide source code or equivalent since the library is closed source.
Here is some additional information about my recent experience with these problems. 1. I was wrapping each exposed library function using a function pointers which would each be added to a map from string -> function pointer. When I increase the number of such functions + size of the map the memory usage of optimizer goes up exponentially. (a little over half the map took 5 GB of memory and the full map took over 30 GB of memory and failed to complete) 2. When I switch to using a map of strings -> int and make a large switch statement containing the wrapper implementations this problem no longer occurs. (although the memory usage of optimizer,exe does go up to about 11 GB and it still takes about 10 minutes to finish) Thanks, David On Monday, July 13, 2015 at 11:44:37 AM UTC-7, Alon Zakai wrote: > > When juj gets back I hope he can look at why we are building a 32-bit > executable there. > > Meanwhile, is there any chance you can find the stack trace that causes > the error? Increasing the stack size is a workaround, but it would be good > to fix the underlying issue. > > > On Thu, Jul 9, 2015 at 5:54 PM, <[email protected] <javascript:>> wrote: > >> All right, I was finally able to get around the errors with the >> optimizer. I did the following: >> >> 1. Changed Optimizer project to build 64-bit executable. (despite being >> under the folder incoming_64bit_optimizer it builds a 32-bit executable on >> windows!) >> 2. Greatly increased stack size. >> 3. Rebuilt the optimizer executable >> >> The whole emcc process still takes about 8 minutes to run on my library, >> but at least it can actually complete the process. >> >> David >> >> On Tuesday, July 7, 2015 at 6:43:22 PM UTC-7, [email protected] wrote: >>> >>> Thanks for the tips. I was able to get around the issues that I was >>> encountering when using just the latest build. >>> >>> However, I still encounter the Optimizer stack overflow crash. Here is a >>> partial call stack: >>> >>> msvcr120.dll!6544a5d6() >>> [Frames below may be incorrect and/or missing, no symbols loaded for >>> msvcr120.dll] >>> msvcr120.dll!654605db() >>> msvcr120.dll!654b9bf0() >>> msvcr120.dll!654b9c8d() >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::Frag::Frag(char * >>> src=0x0cdffdc9) Line 231 + 0xe bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40448) Line 290 C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseExpression(cashew::Parser<Ref,ValueBuilder>::ExpressionElement >>> >>> initial={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40448) Line >>> 726 C++ >>> > optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x1cf4e3dc) Line 301 + 0x28 bytes C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseExpression(cashew::Parser<Ref,ValueBuilder>::ExpressionElement >>> >>> initial={...}, char * & src=0x20202020, const char * seps=0x00b40448) Line >>> 726 C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x00000002, const char * seps=0x00b40448) Line 304 + 0x37 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterParen(char >>> * & src=0x0cdffdc9) Line 590 C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40478) Line 304 + 0x18 bytes C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseExpression(cashew::Parser<Ref,ValueBuilder>::ExpressionElement >>> >>> initial={...}, char * & src=0x20202020, const char * seps=0x00b40478) Line >>> 726 C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterIdent(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x1cf4e37c) Line >>> 539 C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40478) Line 300 + 0x18 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>> 806 + 0x33 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseIf(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 420 + 0x27 bytes C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 334 + 0x18 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>> 806 + 0x33 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseDo(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 434 + 0x26 bytes C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 335 + 0x18 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>> src=0x00000002, Ref block={...}, const char * seps=0x00b40478, >>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>> 806 + 0x33 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseIf(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 420 + 0x27 bytes C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 334 + 0x18 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>> 806 + 0x33 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseDo(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 434 + 0x26 bytes C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 335 + 0x18 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes C++ >>> >>> >>> Prior to that the following stack is repeated. (probably infinitely >>> although depending on how the parser is written it could just be a very >>> large stack) >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>> src=0x00000002, Ref block={...}, const char * seps=0x00b40478, >>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>> 806 + 0x33 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseIf(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 420 + 0x27 bytes C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 334 + 0x18 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>> 806 + 0x33 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseDo(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 434 + 0x26 bytes C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 335 + 0x18 bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes C++ >>> >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterIdent(cashew::Parser<Ref,ValueBuilder>::Frag >>> >>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>> 533 + 0xf bytes C++ >>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * & >>> src=0x0cdffdc9, const char * seps=0x00b40478) Line 300 + 0x18 bytes C++ >>> >>> I will investigate to see if I can get any other information tomorrow. >>> >>> Thanks, >>> >>> David >>> >>> On Tuesday, July 7, 2015 at 1:18:20 PM UTC-7, Alon Zakai wrote: >>>> >>>> That looks like a different error. It is caused by the repos not being >>>> in sync - you need latest incoming on all 3 repos, see >>>> >>>> >>>> http://kripken.github.io/emscripten-site/docs/building_from_source/building_fastcomp_manually_from_source.html#branches >>>> >>>> On Tue, Jul 7, 2015 at 12:55 PM, <[email protected]> wrote: >>>> >>>>> I still get the same error using Emscripten in the incoming branch and >>>>> can't even run the samples without getting errors. This is my output >>>>> trying >>>>> to run test_hello_world: >>>>> >>>>> E:\Program Files\Emscripten\emscripten\incoming>python tests/runner.py >>>>> test_hell >>>>> o_world >>>>> test_hello_world (test_core.default) ... (checking sanity from test >>>>> runner) >>>>> INFO root: (Emscripten: Running sanity checks) >>>>> FAIL >>>>> >>>>> ====================================================================== >>>>> FAIL: test_hello_world (test_core.default) >>>>> ---------------------------------------------------------------------- >>>>> Traceback (most recent call last): >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\tests\test_core.py", lin >>>>> e 18, in test_hello_world >>>>> self.do_run_from_file(src, output) >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\tests\runner.py", line 4 >>>>> 79, in do_run_from_file >>>>> includes, force_c, build_ll_hook, extra_emscripten_args) >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\tests\runner.py", line 4 >>>>> 91, in do_run >>>>> build_ll_hook=build_ll_hook, >>>>> extra_emscripten_args=extra_emscripten_args, po >>>>> st_build=post_build) >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\tests\runner.py", line 2 >>>>> 57, in build >>>>> assert os.path.exists(filename + '.o.js'), 'Source compilation >>>>> error: ' + ou >>>>> tput >>>>> AssertionError: Source compilation error: Traceback (most recent call >>>>> last): >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 106 >>>>> 7, in <module> >>>>> _main(environ=os.environ) >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 105 >>>>> 7, in _main >>>>> temp_files.run_and_clean(lambda: main( >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\tools\tempfiles.py", lin >>>>> e 39, in run_and_clean >>>>> return func() >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 106 >>>>> 3, in <lambda> >>>>> DEBUG_CACHE=DEBUG_CACHE, >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 964 >>>>> , in main >>>>> temp_files=temp_files, DEBUG=DEBUG, DEBUG_CACHE=DEBUG_CACHE) >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 183 >>>>> , in emscript >>>>> settings['MAX_GLOBAL_ALIGN'] = metadata['maxGlobalAlign'] >>>>> KeyError: 'maxGlobalAlign' >>>>> Traceback (most recent call last): >>>>> File "E:\Program Files\Emscripten\emscripten\incoming\emcc", line >>>>> 1294, in <mo >>>>> dule> >>>>> final = shared.Building.emscripten(final, append_ext=False, >>>>> extra_args=extra >>>>> _args) >>>>> File "E:\Program >>>>> Files\Emscripten\emscripten\incoming\tools\shared.py", line 1 >>>>> 535, in emscripten >>>>> assert os.path.exists(filename + '.o.js'), 'Emscripten failed to >>>>> generate .j >>>>> s' >>>>> AssertionError: Emscripten failed to generate .js >>>>> >>>>> >>>>> ---------------------------------------------------------------------- >>>>> Ran 1 test in 1.128s >>>>> >>>>> FAILED (failures=1) >>>>> >>>>> Any ideas? >>>>> >>>>> Thanks, >>>>> >>>>> David >>>>> >>>>> >>>>> On Monday, July 6, 2015 at 8:29:59 PM UTC-7, Alon Zakai wrote: >>>>>> >>>>>> Is that the latest on the incoming branch? There was a recent fix >>>>>> there for something that looks like your issue. >>>>>> >>>>>> >>>>>> On Mon, Jul 6, 2015 at 6:43 PM, <[email protected]> wrote: >>>>>> >>>>>>> When I try using the latest Emscripten version 1.34.1 I get the >>>>>>> following error: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File "E:\Program Files\Emscripten\emscripten\1.34.1\\emcc", line >>>>>>> 1260, in <mod >>>>>>> ule> >>>>>>> shared.Building.llvm_opt(final, link_opts) >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\1.34.1\tools\shared.py", line 142 >>>>>>> 9, in llvm_opt >>>>>>> assert os.path.exists(target), 'Failed to run llvm >>>>>>> optimizations: ' + output >>>>>>> >>>>>>> AssertionError: Failed to run llvm optimizations: >>>>>>> >>>>>>> Any help would be much appreciated. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> David >>>>>>> >>>>>> >>>>>> >>>>> >>>> -- >> >> > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
