> * Are there tests in your codebase that we could run in upstream emscripten?
Come to think of it, there's one thorough automated test that we have to run at each upgrade: building dependencies! https://github.com/renpy/renpyweb https://github.com/renpy/renpyweb/tree/master/scripts https://www.beuc.net/python-emscripten/python/dir?ci=tip Typing a simple: $ make will download and build all dependencies, and it's not uncommon that something breaks on Emscripten upgrade. Cheers! Sylvain On 10/09/2019 12:41, Sylvain Beucler wrote: > > Hi, > > (I was hoping other people would give their PoV...) > > Generally-speaking, we may need more patches because of: > > - the combination of Emscripten features (Python + uninterruptible > main loop -- aka fpcast+async; thankfully I could get rid of dynamic > linking...) > > - centrally improving Emscripten portability rather than adapting the > codebases (e.g. spending weeks on the 1-line #8751 / LANG support, > rather than adding a quick EM_ASM in all my projects) > > - working directly with Emscripten rather than using a framework that > provides a layer of work-arounds (e.g. Unity) > > Again, at a given point of time we always have a buffer of pending > patches (I forgot to mention #6511 and #7631 that were added to our > shell.html template, btw), the ones we can upstream are replaced by > newer ones: old bugs we eventually tackle like fullscreen support or > autosave-on-quit; backports for regressions in a given Emscripten release. > > Hopefully one day the web ecosystem will be so mature that we'll be > able to take our distro's years-old compiler, `./configure > --host=browser` without any change, and publish. Meanwhile... we patch :) > > To answer your specific questions: > > - This is a game engine, much of the tests are manual, sadly (but I > contribute tests when I submit Emscripten patches); we often get > integration errors, or errors due to combining multiple Emscripten > options. > > - I upgrade Emscripten to a tagged version ~every other month; since > this is likely to introduce breaking changes, I only do it when I have > the time to deal with the consequences (e.g. when I'm not in the > middle of developing RenPyWeb itself). I avoid ToT for stability and > reproducibility reasons. > > - EMT_STACK_MAX is the exception, not the rule ;) That's about the > only thing I could have upstream but didn't due to lack of time -- and > it's now deprecated. > > - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used > for emsdk 1.38.42" from > https://chromium.googlesource.com/emscripten-releases/ > > Cheers! > Sylvain > > On 07/09/2019 00:58, Alon Zakai wrote: >> I think those are all reasonable requests, thanks for opening those >> issues and highlighting them here! I hope we can make progress on each. >> >> And hopefully other users can provide useful advice from their >> perspective and experience. >> >> About "no matching tags for llvm and bynarien, commit IDs quite hard >> to track (discussion in #9317/#9383)" - I think we have resolved >> that? Please let me know if not. >> >> And a general comment: Your overall experience does surprise me >> (which shows why it's good you wrote it up!) - that's a lot more >> local changes than I've seen elsewhere, and that includes some very >> large production user setups. So I hope we can work together to make >> your experience closer to theirs. >> >> * Are there tests in your codebase that we could run in upstream >> emscripten? >> * Do you test latest emscripten releases on your codebase as they >> are tagged? (Another option is to test the much more frequent >> tip-of-tree builds, but you may see more noise there.) >> * It would be good to discuss upstreaming long-term changes you >> mention like that EMT_STACK_MAX one - we may not be able to in all >> cases, but let's try at least (I don't think I remember a discussion >> on that one). >> >> - Alon >> >> >> >> On Fri, Sep 6, 2019 at 12:32 AM Beuc <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi, >> >> We almost never could use Emscripten without applying a few patches. >> It's become increasingly difficult, not only to patch, but to >> convince the developers that it's legitimate, so I thought I'd >> detail our use case for good here. >> >> So we've been using Emscripten with patches in emscripten, llvm >> and SDL2 port [1]. >> By the time a patch is officially included in a release, we >> usually need more patches as we face new issues -- or regressions. >> Rinse and repeat. >> >> Consequently for our production use (the RenPyWeb build that we >> publish to our users), we build an identified, versioned release >> (e.g. 1.38.42) with specific fixes. >> >> >> New difficulties: >> - scarse documentation for recompiling "upstream" (patch in >> progress #9383) >> - no matching tags for llvm and bynarien, commit IDs quite hard >> to track (discussion in #9317/#9383) >> - misleading versioning in tagged commits (#9282) >> - grab-from-git port system; EMCC_LOCAL_PORTS considered for >> "local debugging", not cached, and causing issues (e.g. #9342 #9402) >> >> >> Rebutals: >> >> - "99% users use emsdk binaries" >> Well, the remaining 1% will provide 99% of external >> contributions, take care of them :) >> Also being able to test fixes before they finish the full >> review&ci process allows spotting issues in advance. >> >> - "users will either use the unmodified binaries, or work on >> master/tip-of-tree" >> No, we use an identified, tagged Emscripten version, because it's >> more stable than a random commit. >> Also when we publish a hot fix, sometimes a couple months after >> release, we need to reproduce the same build environment >> >> - "emsdk can build from source" >> This build process doesn't allow patching, so it's only when one >> doesn't trust the binaries and wants to rebuild them as-is AFAIU. >> (To be honest, I'd like to be able to use the binaries, but >> Emscripten being bleeding-edge, I figure that'll have to wait :)) >> >> >> I hope this makes clear why we need to checkout >> emscripten/llvm/bynarien for an existing release, patch it, and >> build it. >> Preferably without reverse-engineering the build process and >> fighting all the stack. >> That's the first time I have to justify this while working on >> free/open source software. >> >> How do you people deal with this? >> >> >> [1] For instance, our current release needs these patches, some >> of them still needing work & official integration: >> >> - https://github.com/emscripten-core/emscripten/issues/9257 >> Work-around ./configure misdetection >> >> - https://github.com/emscripten-core/emscripten/issues/9097 >> Fix fullscreen exit causing wrong screen size >> >> - hard-coded change to EMT_STACK_MAX which isn't configurable >> >> - https://github.com/emscripten-core/emscripten-fastcomp/pull/195 >> Compiler limitation when combining several Emscripten features >> >> - https://github.com/emscripten-ports/SDL2/issues/88 >> Send SDL_APP_TERMINATING on page exit >> >> - https://github.com/emscripten-ports/SDL2/issues/70 >> Support Emterpreter/Asincify in SDL2 >> >> - Search "Emscripten" in https://renpy.beuc.net/#status_reports >> for all the other patches that were eventually integrated over >> the year >> -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/14c22302-1e4c-e5af-bab8-20e55e14151a%40beuc.net.
