Thanks, turns out this was indeed doing unnecessary quadratic work. Fix in https://github.com/WebAssembly/binaryen/pull/2614
It takes 5 seconds on my laptop now. - Alon On Wed, Jan 22, 2020 at 3:46 PM Patrik Weiskircher <[email protected]> wrote: > Just sent the file. Thank you for looking into this, Alon! > > On Jan 22, 2020, at 5:56 PM, Alon Zakai <[email protected]> wrote: > > > > > On Wed, Jan 22, 2020 at 2:00 PM Patrik Weiskircher <[email protected]> > wrote: > >> > [PassRunner] running pass: legalize-js-interface... 863.403 seconds. >> > I wonder if that's an error in the measurement, otherwise, that's very >> surprising, I don't remember any non-linear algorithms there... >> >> It is NOT an error in measurement, it really takes that long. I've been >> profiling using Instruments on my Mac and from what I see most of the time >> is spent here: >> >> >> https://github.com/WebAssembly/binaryen/blob/e8f9d207427bda2f6e22c28ff0210b294b1f70e1/src/passes/LegalizeJSInterface.cpp#L121-L133 >> >> Specifically modifying the `unordered_map`. >> >> If I understand the code correctly, this basically means we have a *ton* >> of functions that we're trying to export and therefor have to "legalize"? >> Is it possible that we just have our project a little misconfigured and >> instead of only exporting methods with Embind everything is exported? I >> couldn't explain why the unordered_map would be used for 15 minutes(!) >> otherwise. >> >> anyhow, if you share the binary I can investigate, should be easy to >>> narrow something that noticeable down ;) >>> >> >> Would you mind if I send it to you personally instead of to the mailing >> list? If you think it is even necessary anymore seeing the above statement. >> >> > Feel free to send it to me personally, yeah. I still need to figure out > why that code is taking nonlinear time - that should just be doing a linear > scan over the wasm module... > > Is this built with MAIN_MODULE perhaps? That would increase the # of > exports. Still, no number can explain that slowdown, so it's still a > mystery to me. > > >> >> On Wed, Jan 22, 2020 at 4:50 PM Alon Zakai <[email protected]> wrote: >> >>> > The code that I am compiling is proprietary but I feel like at that >>> stage it's just one big object file without any source? >>> >>> Yeah, if you look at the input to wasm-emscripten-finalize, it's very >>> close to the final wasm binary you'd ship on the web. So it's all compiled >>> into wasm binary code at that point. >>> >>> One possible difference is this binary might contain function names, >>> which might be stripped out by emcc later depending on the options. You >>> might also want to take a look at the binary yourself to see if there's >>> anything that looks like you can't share it, just to be sure. >>> >>> > [PassRunner] running pass: legalize-js-interface... 863.403 seconds. >>> >>> I wonder if that's an error in the measurement, otherwise, that's very >>> surprising, I don't remember any non-linear algorithms there... anyhow, if >>> you share the binary I can investigate, should be easy to narrow something >>> that noticeable down ;) >>> >>> On Wed, Jan 22, 2020 at 1:23 PM Patrik Weiskircher <[email protected]> >>> wrote: >>> >>>> >>>> If you built binaryen by yourself, I'd make sure that's not a debug >>>>> build. >>>>> >>>> >>>> Simply did `emsdk install latest` and nothing is built by myself. >>>> >>>> Another thing is to check you don't set BINARYEN_CORES=1 in the >>>>> environment (which would prevent an almost linear speedup with more >>>>> cores). >>>>> >>>> >>>> Don't think this is a problem here. I see `top` report 800% CPU use so >>>> it's definitely using all Cores. >>>> >>>> And in a very big project perhaps it takes enough memory to cause >>>>> swapping, so checking peak memory usage etc. might be interesting. >>>>> >>>> >>>> It's only at 15.8% memory usage. >>>> >>>> If those aren't it, I'd run it in a profiler to see what's taking so >>>>> long. If you can share the file I can take a look too. >>>>> >>>> >>>> I honestly don't know enough about what format the file is in to make a >>>> good decision here. The code that I am compiling is proprietary but I feel >>>> like at that stage it's just one big object file without any source? As >>>> long as we're not leaking source code I'm very happy to share the file :) >>>> >>>> I recently tracing channels to binaryen and some minimal tracing to >>>>> `wasm-emscripten-finalize`. You might be able to get some clues from >>>>> adding `--debug=emscripten` to the command line. >>>>> >>>> >>>> I just added this! Here is the result: >>>> >>>> [PassRunner] running passes... >>>>> [PassRunner] running pass: legalize-js-interface... 863.403 seconds. >>>>> [PassRunner] (validating) >>>>> [PassRunner] passes took 863.403 seconds. >>>>> [PassRunner] (final validation) >>>>> >>>> >>>> >>>> On Wed, Jan 22, 2020 at 4:00 PM 'Sam Clegg' via emscripten-discuss < >>>> [email protected]> wrote: >>>> >>>>> I recently tracing channels to binaryen and some minimal tracing to >>>>> `wasm-emscripten-finalize`. You might be able to get some clues from >>>>> adding `--debug=emscripten` to the command line. >>>>> >>>>> On Wed, Jan 22, 2020 at 12:52 PM Alon Zakai <[email protected]> >>>>> wrote: >>>>> >>>>>> Oh, wow, it shouldn't be anything like that... >>>>>> >>>>>> If you built binaryen by yourself, I'd make sure that's not a debug >>>>>> build. Another thing is to check you don't set BINARYEN_CORES=1 in the >>>>>> environment (which would prevent an almost linear speedup with more >>>>>> cores). >>>>>> And in a very big project perhaps it takes enough memory to cause >>>>>> swapping, >>>>>> so checking peak memory usage etc. might be interesting. >>>>>> >>>>>> If those aren't it, I'd run it in a profiler to see what's taking so >>>>>> long. If you can share the file I can take a look too. >>>>>> >>>>>> - Alon >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jan 22, 2020 at 11:31 AM Patrik Weiskircher < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> I'm working at PSPDFKit trying to upgrade our Web viewer from >>>>>>> Emscripten version 1.38.31 to 1.39.6. >>>>>>> I got everything working so far, I'm just having a little trouble >>>>>>> with the linking speed. Everything goes fine and quick >>>>>>> enough until we hit linking the library - specifically the >>>>>>> `wasm-emscripten-finalize` step. >>>>>>> >>>>>>> The exact command line is this: >>>>>>> > /opt/emsdk/upstream/bin/wasm-emscripten-finalize --detect-features >>>>>>> --global-base=1024 --check-stack-overflow \ >>>>>>> > /tmp/emscripten_temp_rjgvGG/pspdfkit.wasm.wasm -o >>>>>>> /tmp/emscripten_temp_rjgvGG/pspdfkit.wasm.wasm.o.wasm >>>>>>> >>>>>>> This takes around 15 minutes. While this isn't a deal breaker, it >>>>>>> does make our CI quite slow and keeps agents occupied >>>>>>> longer than necessary. It'll also not be great when we have to debug >>>>>>> a problem on Emscripten in the future. >>>>>>> >>>>>>> I looked around on the GitHub issue board and also on the mailing >>>>>>> list but I couldn't find any information about this. I >>>>>>> understand we have quite a big code base and things will take some >>>>>>> time - I'm just wondering if anyone has any tips >>>>>>> on how I could debug what *exactly* takes this long or how to make >>>>>>> this any quicker? >>>>>>> >>>>>>> Thanks so much! >>>>>>> Patrik >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "emscripten-discuss" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/cca83aa0-c022-4a94-8cc6-42063b5ccebc%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/cca83aa0-c022-4a94-8cc6-42063b5ccebc%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "emscripten-discuss" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpREWRPRu1M%3DEx2t%2BQXkt7pQ64sgFC%2BaGU6h_H5vbX9ocA%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpREWRPRu1M%3DEx2t%2BQXkt7pQ64sgFC%2BaGU6h_H5vbX9ocA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "emscripten-discuss" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/emscripten-discuss/Tj7C5Ij4sx8/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/emscripten-discuss/CAL_va28KeT9VfF45%3D%3DAEmVSML79uB9L%2B%2BWJV6EG6%3DL3NoGtvjg%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/emscripten-discuss/CAL_va28KeT9VfF45%3D%3DAEmVSML79uB9L%2B%2BWJV6EG6%3DL3NoGtvjg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "emscripten-discuss" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/emscripten-discuss/CANMzfzh6oQ6656xYf-C7T7eo5Es57TsnkkrNzFiJx64%3Dcx4whA%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/emscripten-discuss/CANMzfzh6oQ6656xYf-C7T7eo5Es57TsnkkrNzFiJx64%3Dcx4whA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "emscripten-discuss" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/emscripten-discuss/Tj7C5Ij4sx8/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpTqGBC4eJGJE4hrCCY_k4pA%3D4nXtBxGemSYqKQw_juGoQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpTqGBC4eJGJE4hrCCY_k4pA%3D4nXtBxGemSYqKQw_juGoQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "emscripten-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/emscripten-discuss/CANMzfzieX98ksvWvCQcD9StwOA95bjy7mkZ5kUvweZyvok71XQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/emscripten-discuss/CANMzfzieX98ksvWvCQcD9StwOA95bjy7mkZ5kUvweZyvok71XQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "emscripten-discuss" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/emscripten-discuss/Tj7C5Ij4sx8/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpTE3GfoebGiKutkQEhT5cPkFAK4ohvJLHK-HdkffnZCwA%40mail.gmail.com > <https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpTE3GfoebGiKutkQEhT5cPkFAK4ohvJLHK-HdkffnZCwA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "emscripten-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/emscripten-discuss/8109E6E4-DBDE-420E-B3A8-276FD3DE8A2A%40pspdfkit.com > <https://groups.google.com/d/msgid/emscripten-discuss/8109E6E4-DBDE-420E-B3A8-276FD3DE8A2A%40pspdfkit.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpQDbNBbatjCZbKzgSJehhppgdwRzWvWuK8JpghZkD7dvw%40mail.gmail.com.
