Sorry for the delay, My colleague submitted a simplified source file and bug report for this issue at https://github.com/kripken/emscripten/issues/3718.
Thanks, David On Wednesday, July 22, 2015 at 11:05:37 AM UTC-7, Alon Zakai wrote: > > Oh, sorry, I missed the stack trace before. > > Ok, then we must have some code generation issue where we emit a highly > recursive structure, probably unnecessarily. I understand you can't share > your source code, but if you can create a new standalone example showing a > similar issue - wrapping library functions, as you mention, I'm just not > sure how you are doing that, but if I saw it, the problem for us to fix > might be obvious. > > On Wed, Jul 22, 2015 at 10:57 AM, <[email protected] <javascript:>> > wrote: > >> Hmm, well the stack trace is what I mentioned previously. Is there >> anything specific that would help you to track down the problem? I am >> unfortunately unable to provide source code or equivalent since the library >> is closed source. >> >> Here is some additional information about my recent experience with these >> problems. >> >> 1. I was wrapping each exposed library function using a function pointers >> which would each be added to a map from string -> function pointer. When I >> increase the number of such functions + size of the map the memory usage of >> optimizer goes up exponentially. (a little over half the map took 5 GB of >> memory and the full map took over 30 GB of memory and failed to complete) >> 2. When I switch to using a map of strings -> int and make a large switch >> statement containing the wrapper implementations this problem no longer >> occurs. (although the memory usage of optimizer,exe does go up to about 11 >> GB and it still takes about 10 minutes to finish) >> >> Thanks, >> >> David >> >> On Monday, July 13, 2015 at 11:44:37 AM UTC-7, Alon Zakai wrote: >>> >>> When juj gets back I hope he can look at why we are building a 32-bit >>> executable there. >>> >>> Meanwhile, is there any chance you can find the stack trace that causes >>> the error? Increasing the stack size is a workaround, but it would be good >>> to fix the underlying issue. >>> >>> >>> On Thu, Jul 9, 2015 at 5:54 PM, <[email protected]> wrote: >>> >>>> All right, I was finally able to get around the errors with the >>>> optimizer. I did the following: >>>> >>>> 1. Changed Optimizer project to build 64-bit executable. (despite being >>>> under the folder incoming_64bit_optimizer it builds a 32-bit executable on >>>> windows!) >>>> 2. Greatly increased stack size. >>>> 3. Rebuilt the optimizer executable >>>> >>>> The whole emcc process still takes about 8 minutes to run on my >>>> library, but at least it can actually complete the process. >>>> >>>> David >>>> >>>> On Tuesday, July 7, 2015 at 6:43:22 PM UTC-7, [email protected] wrote: >>>>> >>>>> Thanks for the tips. I was able to get around the issues that I was >>>>> encountering when using just the latest build. >>>>> >>>>> However, I still encounter the Optimizer stack overflow crash. Here is >>>>> a partial call stack: >>>>> >>>>> msvcr120.dll!6544a5d6() >>>>> [Frames below may be incorrect and/or missing, no symbols loaded >>>>> for msvcr120.dll] >>>>> msvcr120.dll!654605db() >>>>> msvcr120.dll!654b9bf0() >>>>> msvcr120.dll!654b9c8d() >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::Frag::Frag(char * >>>>> src=0x0cdffdc9) Line 231 + 0xe bytes C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40448) Line 290 C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseExpression(cashew::Parser<Ref,ValueBuilder>::ExpressionElement >>>>> >>>>> initial={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40448) >>>>> Line >>>>> 726 C++ >>>>> > optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x1cf4e3dc) Line 301 + 0x28 bytes >>>>> C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseExpression(cashew::Parser<Ref,ValueBuilder>::ExpressionElement >>>>> >>>>> initial={...}, char * & src=0x20202020, const char * seps=0x00b40448) >>>>> Line >>>>> 726 C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x00000002, const char * seps=0x00b40448) Line 304 + 0x37 bytes >>>>> C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterParen(char >>>>> * & src=0x0cdffdc9) Line 590 C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 304 + 0x18 bytes >>>>> C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseExpression(cashew::Parser<Ref,ValueBuilder>::ExpressionElement >>>>> >>>>> initial={...}, char * & src=0x20202020, const char * seps=0x00b40478) >>>>> Line >>>>> 726 C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterIdent(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x1cf4e37c) >>>>> Line >>>>> 539 C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 300 + 0x18 bytes >>>>> C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) >>>>> Line >>>>> 806 + 0x33 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseIf(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 420 + 0x27 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 334 + 0x18 bytes C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>>> C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) >>>>> Line >>>>> 806 + 0x33 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseDo(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 434 + 0x26 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 335 + 0x18 bytes C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>>> C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>>> src=0x00000002, Ref block={...}, const char * seps=0x00b40478, >>>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) >>>>> Line >>>>> 806 + 0x33 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseIf(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 420 + 0x27 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 334 + 0x18 bytes C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>>> C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) >>>>> Line >>>>> 806 + 0x33 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseDo(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 434 + 0x26 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 335 + 0x18 bytes C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>>> C++ >>>>> >>>>> >>>>> Prior to that the following stack is repeated. (probably infinitely >>>>> although depending on how the parser is written it could just be a very >>>>> large stack) >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>>> src=0x00000002, Ref block={...}, const char * seps=0x00b40478, >>>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) >>>>> Line >>>>> 806 + 0x33 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseIf(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 420 + 0x27 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 334 + 0x18 bytes C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>>> C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBlock(char * & >>>>> src=0x0cdffdc9, Ref block={...}, const char * seps=0x00b40478, >>>>> cashew::IString keywordSep1={...}, cashew::IString keywordSep2={...}) >>>>> Line >>>>> 806 + 0x33 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseBracketedBlock(char >>>>> * & src=0x0cdffdc9, Ref block={...}) Line 819 C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseDo(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 434 + 0x26 bytes C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterKeyword(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 335 + 0x18 bytes C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 293 + 0x13 bytes >>>>> C++ >>>>> >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseAfterIdent(cashew::Parser<Ref,ValueBuilder>::Frag >>>>> >>>>> & frag={...}, char * & src=0x0cdffdc9, const char * seps=0x00b40478) >>>>> Line >>>>> 533 + 0xf bytes C++ >>>>> optimizer.exe!cashew::Parser<Ref,ValueBuilder>::parseElement(char * >>>>> & src=0x0cdffdc9, const char * seps=0x00b40478) Line 300 + 0x18 bytes >>>>> C++ >>>>> >>>>> I will investigate to see if I can get any other information tomorrow. >>>>> >>>>> Thanks, >>>>> >>>>> David >>>>> >>>>> On Tuesday, July 7, 2015 at 1:18:20 PM UTC-7, Alon Zakai wrote: >>>>>> >>>>>> That looks like a different error. It is caused by the repos not >>>>>> being in sync - you need latest incoming on all 3 repos, see >>>>>> >>>>>> >>>>>> http://kripken.github.io/emscripten-site/docs/building_from_source/building_fastcomp_manually_from_source.html#branches >>>>>> >>>>>> On Tue, Jul 7, 2015 at 12:55 PM, <[email protected]> wrote: >>>>>> >>>>>>> I still get the same error using Emscripten in the incoming branch >>>>>>> and can't even run the samples without getting errors. This is my >>>>>>> output >>>>>>> trying to run test_hello_world: >>>>>>> >>>>>>> E:\Program Files\Emscripten\emscripten\incoming>python >>>>>>> tests/runner.py test_hell >>>>>>> o_world >>>>>>> test_hello_world (test_core.default) ... (checking sanity from test >>>>>>> runner) >>>>>>> INFO root: (Emscripten: Running sanity checks) >>>>>>> FAIL >>>>>>> >>>>>>> >>>>>>> ====================================================================== >>>>>>> FAIL: test_hello_world (test_core.default) >>>>>>> >>>>>>> ---------------------------------------------------------------------- >>>>>>> Traceback (most recent call last): >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\tests\test_core.py", lin >>>>>>> e 18, in test_hello_world >>>>>>> self.do_run_from_file(src, output) >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\tests\runner.py", line 4 >>>>>>> 79, in do_run_from_file >>>>>>> includes, force_c, build_ll_hook, extra_emscripten_args) >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\tests\runner.py", line 4 >>>>>>> 91, in do_run >>>>>>> build_ll_hook=build_ll_hook, >>>>>>> extra_emscripten_args=extra_emscripten_args, po >>>>>>> st_build=post_build) >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\tests\runner.py", line 2 >>>>>>> 57, in build >>>>>>> assert os.path.exists(filename + '.o.js'), 'Source compilation >>>>>>> error: ' + ou >>>>>>> tput >>>>>>> AssertionError: Source compilation error: Traceback (most recent >>>>>>> call last): >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 106 >>>>>>> 7, in <module> >>>>>>> _main(environ=os.environ) >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 105 >>>>>>> 7, in _main >>>>>>> temp_files.run_and_clean(lambda: main( >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\tools\tempfiles.py", lin >>>>>>> e 39, in run_and_clean >>>>>>> return func() >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 106 >>>>>>> 3, in <lambda> >>>>>>> DEBUG_CACHE=DEBUG_CACHE, >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 964 >>>>>>> , in main >>>>>>> temp_files=temp_files, DEBUG=DEBUG, DEBUG_CACHE=DEBUG_CACHE) >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\emscripten.py", line 183 >>>>>>> , in emscript >>>>>>> settings['MAX_GLOBAL_ALIGN'] = metadata['maxGlobalAlign'] >>>>>>> KeyError: 'maxGlobalAlign' >>>>>>> Traceback (most recent call last): >>>>>>> File "E:\Program Files\Emscripten\emscripten\incoming\emcc", line >>>>>>> 1294, in <mo >>>>>>> dule> >>>>>>> final = shared.Building.emscripten(final, append_ext=False, >>>>>>> extra_args=extra >>>>>>> _args) >>>>>>> File "E:\Program >>>>>>> Files\Emscripten\emscripten\incoming\tools\shared.py", line 1 >>>>>>> 535, in emscripten >>>>>>> assert os.path.exists(filename + '.o.js'), 'Emscripten failed to >>>>>>> generate .j >>>>>>> s' >>>>>>> AssertionError: Emscripten failed to generate .js >>>>>>> >>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------------------- >>>>>>> Ran 1 test in 1.128s >>>>>>> >>>>>>> FAILED (failures=1) >>>>>>> >>>>>>> Any ideas? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> David >>>>>>> >>>>>>> >>>>>>> On Monday, July 6, 2015 at 8:29:59 PM UTC-7, Alon Zakai wrote: >>>>>>>> >>>>>>>> Is that the latest on the incoming branch? There was a recent fix >>>>>>>> there for something that looks like your issue. >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Jul 6, 2015 at 6:43 PM, <[email protected]> wrote: >>>>>>>> >>>>>>>>> When I try using the latest Emscripten version 1.34.1 I get the >>>>>>>>> following error: >>>>>>>>> >>>>>>>>> Traceback (most recent call last): >>>>>>>>> File "E:\Program Files\Emscripten\emscripten\1.34.1\\emcc", line >>>>>>>>> 1260, in <mod >>>>>>>>> ule> >>>>>>>>> shared.Building.llvm_opt(final, link_opts) >>>>>>>>> File "E:\Program >>>>>>>>> Files\Emscripten\emscripten\1.34.1\tools\shared.py", line 142 >>>>>>>>> 9, in llvm_opt >>>>>>>>> assert os.path.exists(target), 'Failed to run llvm >>>>>>>>> optimizations: ' + output >>>>>>>>> >>>>>>>>> AssertionError: Failed to run llvm optimizations: >>>>>>>>> >>>>>>>>> Any help would be much appreciated. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> David >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> -- >>>> >>>> >>> >> > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
