Oh, sorry, I missed the stack trace before. Ok, then we must have some code generation issue where we emit a highly recursive structure, probably unnecessarily. I understand you can't share your source code, but if you can create a new standalone example showing a similar issue - wrapping library functions, as you mention, I'm just not sure how you are doing that, but if I saw it, the problem for us to fix might be obvious.
On Wed, Jul 22, 2015 at 10:57 AM, <[email protected]> wrote: > Hmm, well the stack trace is what I mentioned previously. Is there > anything specific that would help you to track down the problem? I am > unfortunately unable to provide source code or equivalent since the library > is closed source. > > Here is some additional information about my recent experience with these > problems. > > 1. I was wrapping each exposed library function using a function pointers > which would each be added to a map from string -> function pointer. When I > increase the number of such functions + size of the map the memory usage of > optimizer goes up exponentially. (a little over half the map took 5 GB of > memory and the full map took over 30 GB of memory and failed to complete) > 2. When I switch to using a map of strings -> int and make a large switch > statement containing the wrapper implementations this problem no longer > occurs. (although the memory usage of optimizer,exe does go up to about 11 > GB and it still takes about 10 minutes to finish) > > Thanks, > > David > > On Monday, July 13, 2015 at 11:44:37 AM UTC-7, Alon Zakai wrote: >> >> When juj gets back I hope he can look at why we are building a 32-bit >> executable there. >> >> Meanwhile, is there any chance you can find the stack trace that causes >> the error? Increasing the stack size is a workaround, but it would be good >> to fix the underlying issue. >> >> >> On Thu, Jul 9, 2015 at 5:54 PM, <[email protected]> wrote: >> >>> All right, I was finally able to get around the errors with the >>> optimizer. I did the following: >>> >>> 1. Changed Optimizer project to build 64-bit executable. (despite being >>> under the folder incoming_64bit_optimizer it builds a 32-bit executable on >>> windows!) >>> 2. Greatly increased stack size. >>> 3. Rebuilt the optimizer executable >>> >>> The whole emcc process still takes about 8 minutes to run on my library, >>> but at least it can actually complete the process. >>> >>> David >>> >>> On Tuesday, July 7, 2015 at 6:43:22 PM UTC-7, [email protected] wrote: >>>> >>>> Thanks for the tips. I was able to get around the issues that I was >>>> encountering when using just the latest build. >>>> >>>> However, I still encounter the Optimizer stack overflow crash. Here is >>>> a partial call stack: >>>> >>>> msvcr120.dll!6544a5d6() >>>> [Frames below may be incorrect and/or missing, no symbols loaded for >>>> msvcr120.dll] >>>> msvcr120.dll!654605db() >>>> msvcr120.dll!654b9bf0() >>>> msvcr120.dll!654b9c8d() >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::Frag::Frag(char * >>>> src=0x0cdffdc9) Line 231 + 0xe bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40448) Line 290 C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseExpression(cashew::Parser<Ref,ValueBuilder>::ExpressionElement >>>> initial={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40448) Line >>>> 726 C++ >>>> > optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x1cf4e3dc) Line 301 + 0x28 bytes >>>> C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseExpression(cashew::Parser<Ref,ValueBuilder>::ExpressionElement >>>> initial={...}, char * & src=0x20202020, const char * seps=0x00b40448) Line >>>> 726 C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x00000002, const char * seps=0x00b40448) Line 304 + 0x37 bytes >>>> C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterParen(char >>>> * & src=0x0cdffdc9) Line 590 C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 304 + 0x18 bytes >>>> C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseExpression(cashew::Parser<Ref,ValueBuilder>::ExpressionElement >>>> initial={...}, char * & src=0x20202020, const char * seps=0x00b40478) Line >>>> 726 C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterIdent(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x1cf4e37c) Line >>>> 539 C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 300 + 0x18 bytes >>>> C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>>> 806 + 0x33 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseIf(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 420 + 0x27 bytes C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 334 + 0x18 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>> C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>>> 806 + 0x33 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseDo(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 434 + 0x26 bytes C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 335 + 0x18 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>> C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>> src=0x00000002, Ref block={...}, const char * seps=0x00b40478, >>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>>> 806 + 0x33 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseIf(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 420 + 0x27 bytes C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 334 + 0x18 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>> C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>>> 806 + 0x33 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseDo(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 434 + 0x26 bytes C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 335 + 0x18 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>> C++ >>>> >>>> >>>> Prior to that the following stack is repeated. (probably infinitely >>>> although depending on how the parser is written it could just be a very >>>> large stack) >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>> src=0x00000002, Ref block={...}, const char * seps=0x00b40478, >>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>>> 806 + 0x33 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseIf(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 420 + 0x27 bytes C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 334 + 0x18 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>> C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) Line >>>> 806 + 0x33 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseDo(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 434 + 0x26 bytes C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 335 + 0x18 bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>> C++ >>>> >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterIdent(cashew::Parser<Ref,ValueBuilder>::Frag >>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) Line >>>> 533 + 0xf bytes C++ >>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 300 + 0x18 bytes >>>> C++ >>>> >>>> I will investigate to see if I can get any other information tomorrow. >>>> >>>> Thanks, >>>> >>>> David >>>> >>>> On Tuesday, July 7, 2015 at 1:18:20 PM UTC-7, Alon Zakai wrote: >>>>> >>>>> That looks like a different error. It is caused by the repos not being >>>>> in sync - you need latest incoming on all 3 repos, see >>>>> >>>>> >>>>> http://kripken.github.io/emscripten-site/docs/building_from_source/building_fastcomp_manually_from_source.html#branches >>>>> >>>>> On Tue, Jul 7, 2015 at 12:55 PM, <[email protected]> wrote: >>>>> >>>>>> I still get the same error using Emscripten in the incoming branch >>>>>> and can't even run the samples without getting errors. This is my output >>>>>> trying to run test_hello_world: >>>>>> >>>>>> E:\Program Files\Emscripten\emscripten\incoming>python >>>>>> tests/runner.py test_hell >>>>>> o_world >>>>>> test_hello_world (test_core.default) ... (checking sanity from test >>>>>> runner) >>>>>> INFO root: (Emscripten: Running sanity checks) >>>>>> FAIL >>>>>> >>>>>> ====================================================================== >>>>>> FAIL: test_hello_world (test_core.default) >>>>>> ---------------------------------------------------------------------- >>>>>> Traceback (most recent call last): >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\tests\test_core.py", lin >>>>>> e 18, in test_hello_world >>>>>> self.do_run_from_file(src, output) >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\tests\runner.py", line 4 >>>>>> 79, in do_run_from_file >>>>>> includes, force_c, build_ll_hook, extra_emscripten_args) >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\tests\runner.py", line 4 >>>>>> 91, in do_run >>>>>> build_ll_hook=build_ll_hook, >>>>>> extra_emscripten_args=extra_emscripten_args, po >>>>>> st_build=post_build) >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\tests\runner.py", line 2 >>>>>> 57, in build >>>>>> assert os.path.exists(filename + '.o.js'), 'Source compilation >>>>>> error: ' + ou >>>>>> tput >>>>>> AssertionError: Source compilation error: Traceback (most recent call >>>>>> last): >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 106 >>>>>> 7, in <module> >>>>>> _main(environ=os.environ) >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 105 >>>>>> 7, in _main >>>>>> temp_files.run_and_clean(lambda: main( >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\tools\tempfiles.py", lin >>>>>> e 39, in run_and_clean >>>>>> return func() >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 106 >>>>>> 3, in <lambda> >>>>>> DEBUG_CACHE=DEBUG_CACHE, >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 964 >>>>>> , in main >>>>>> temp_files=temp_files, DEBUG=DEBUG, DEBUG_CACHE=DEBUG_CACHE) >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 183 >>>>>> , in emscript >>>>>> settings['MAX_GLOBAL_ALIGN'] = metadata['maxGlobalAlign'] >>>>>> KeyError: 'maxGlobalAlign' >>>>>> Traceback (most recent call last): >>>>>> File "E:\Program Files\Emscripten\emscripten\incoming\emcc", line >>>>>> 1294, in <mo >>>>>> dule> >>>>>> final = shared.Building.emscripten(final, append_ext=False, >>>>>> extra_args=extra >>>>>> _args) >>>>>> File "E:\Program >>>>>> Files\Emscripten\emscripten\incoming\tools\shared.py", line 1 >>>>>> 535, in emscripten >>>>>> assert os.path.exists(filename + '.o.js'), 'Emscripten failed to >>>>>> generate .j >>>>>> s' >>>>>> AssertionError: Emscripten failed to generate .js >>>>>> >>>>>> >>>>>> ---------------------------------------------------------------------- >>>>>> Ran 1 test in 1.128s >>>>>> >>>>>> FAILED (failures=1) >>>>>> >>>>>> Any ideas? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> David >>>>>> >>>>>> >>>>>> On Monday, July 6, 2015 at 8:29:59 PM UTC-7, Alon Zakai wrote: >>>>>>> >>>>>>> Is that the latest on the incoming branch? There was a recent fix >>>>>>> there for something that looks like your issue. >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 6, 2015 at 6:43 PM, <[email protected]> wrote: >>>>>>> >>>>>>>> When I try using the latest Emscripten version 1.34.1 I get the >>>>>>>> following error: >>>>>>>> >>>>>>>> Traceback (most recent call last): >>>>>>>> File "E:\Program Files\Emscripten\emscripten\1.34.1\\emcc", line >>>>>>>> 1260, in <mod >>>>>>>> ule> >>>>>>>> shared.Building.llvm_opt(final, link_opts) >>>>>>>> File "E:\Program >>>>>>>> Files\Emscripten\emscripten\1.34.1\tools\shared.py", line 142 >>>>>>>> 9, in llvm_opt >>>>>>>> assert os.path.exists(target), 'Failed to run llvm >>>>>>>> optimizations: ' + output >>>>>>>> >>>>>>>> AssertionError: Failed to run llvm optimizations: >>>>>>>> >>>>>>>> Any help would be much appreciated. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> David >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> -- >>> >>> >> -- > You received this message because you are subscribed to the Google Groups > "emscripten-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
