On Mon, Sep 16, 2019 at 6:50 AM Beuc <[email protected]> wrote: > > * Are there tests in your codebase that we could run in upstream > emscripten? > > Come to think of it, there's one thorough automated test that we have to > run at each upgrade: > building dependencies! > > https://github.com/renpy/renpyweb > https://github.com/renpy/renpyweb/tree/master/scripts > https://www.beuc.net/python-emscripten/python/dir?ci=tip > > Typing a simple: > > $ make > > will download and build all dependencies, and it's not uncommon that > something breaks on Emscripten upgrade. >
It might be useful to set up CI that runs the emscripten tip-of-tree builds on that (emsdk install tot-fastcomp or tot-upstream). Those are literally the very newest code, that passed chromium CI but is not as heavily tested as the actual release tags. You may sometimes see a temporary breakage you can ignore, but it would also catch regressions. Cheers! > Sylvain > On 10/09/2019 12:41, Sylvain Beucler wrote: > > Hi, > > (I was hoping other people would give their PoV...) > > Generally-speaking, we may need more patches because of: > > - the combination of Emscripten features (Python + uninterruptible main > loop -- aka fpcast+async; thankfully I could get rid of dynamic linking...) > > - centrally improving Emscripten portability rather than adapting the > codebases (e.g. spending weeks on the 1-line #8751 / LANG support, rather > than adding a quick EM_ASM in all my projects) > > - working directly with Emscripten rather than using a framework that > provides a layer of work-arounds (e.g. Unity) > > Again, at a given point of time we always have a buffer of pending patches > (I forgot to mention #6511 and #7631 that were added to our shell.html > template, btw), the ones we can upstream are replaced by newer ones: old > bugs we eventually tackle like fullscreen support or autosave-on-quit; > backports for regressions in a given Emscripten release. > > Hopefully one day the web ecosystem will be so mature that we'll be able > to take our distro's years-old compiler, `./configure --host=browser` > without any change, and publish. Meanwhile... we patch :) > > To answer your specific questions: > - This is a game engine, much of the tests are manual, sadly (but I > contribute tests when I submit Emscripten patches); we often get > integration errors, or errors due to combining multiple Emscripten options. > > - I upgrade Emscripten to a tagged version ~every other month; since this > is likely to introduce breaking changes, I only do it when I have the time > to deal with the consequences (e.g. when I'm not in the middle of > developing RenPyWeb itself). I avoid ToT for stability and reproducibility > reasons. > > - EMT_STACK_MAX is the exception, not the rule ;) That's about the only > thing I could have upstream but didn't due to lack of time -- and it's now > deprecated. > > - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used for > emsdk 1.38.42" from https://chromium.googlesource.com/emscripten-releases/ > > Is that still an issue? Are the docs at https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten not clear enough? - Alon > Cheers! > Sylvain > On 07/09/2019 00:58, Alon Zakai wrote: > > I think those are all reasonable requests, thanks for opening those issues > and highlighting them here! I hope we can make progress on each. > > And hopefully other users can provide useful advice from their perspective > and experience. > > About "no matching tags for llvm and bynarien, commit IDs quite hard to > track (discussion in #9317/#9383)" - I think we have resolved that? Please > let me know if not. > > And a general comment: Your overall experience does surprise me (which > shows why it's good you wrote it up!) - that's a lot more local changes > than I've seen elsewhere, and that includes some very large production user > setups. So I hope we can work together to make your experience closer to > theirs. > > * Are there tests in your codebase that we could run in upstream > emscripten? > * Do you test latest emscripten releases on your codebase as they are > tagged? (Another option is to test the much more frequent tip-of-tree > builds, but you may see more noise there.) > * It would be good to discuss upstreaming long-term changes you mention > like that EMT_STACK_MAX one - we may not be able to in all cases, but let's > try at least (I don't think I remember a discussion on that one). > > - Alon > > > > On Fri, Sep 6, 2019 at 12:32 AM Beuc <[email protected]> wrote: > >> Hi, >> >> We almost never could use Emscripten without applying a few patches. >> It's become increasingly difficult, not only to patch, but to convince >> the developers that it's legitimate, so I thought I'd detail our use case >> for good here. >> >> So we've been using Emscripten with patches in emscripten, llvm and SDL2 >> port [1]. >> By the time a patch is officially included in a release, we usually need >> more patches as we face new issues -- or regressions. >> Rinse and repeat. >> >> Consequently for our production use (the RenPyWeb build that we publish >> to our users), we build an identified, versioned release (e.g. 1.38.42) >> with specific fixes. >> >> >> New difficulties: >> - scarse documentation for recompiling "upstream" (patch in progress >> #9383) >> - no matching tags for llvm and bynarien, commit IDs quite hard to track >> (discussion in #9317/#9383) >> - misleading versioning in tagged commits (#9282) >> - grab-from-git port system; EMCC_LOCAL_PORTS considered for "local >> debugging", not cached, and causing issues (e.g. #9342 #9402) >> >> >> Rebutals: >> >> - "99% users use emsdk binaries" >> Well, the remaining 1% will provide 99% of external contributions, take >> care of them :) >> Also being able to test fixes before they finish the full review&ci >> process allows spotting issues in advance. >> >> - "users will either use the unmodified binaries, or work on >> master/tip-of-tree" >> No, we use an identified, tagged Emscripten version, because it's more >> stable than a random commit. >> Also when we publish a hot fix, sometimes a couple months after release, >> we need to reproduce the same build environment >> >> - "emsdk can build from source" >> This build process doesn't allow patching, so it's only when one doesn't >> trust the binaries and wants to rebuild them as-is AFAIU. >> (To be honest, I'd like to be able to use the binaries, but Emscripten >> being bleeding-edge, I figure that'll have to wait :)) >> >> >> I hope this makes clear why we need to checkout emscripten/llvm/bynarien >> for an existing release, patch it, and build it. >> Preferably without reverse-engineering the build process and fighting all >> the stack. >> That's the first time I have to justify this while working on free/open >> source software. >> >> How do you people deal with this? >> >> >> [1] For instance, our current release needs these patches, some of them >> still needing work & official integration: >> >> - https://github.com/emscripten-core/emscripten/issues/9257 >> Work-around ./configure misdetection >> >> - https://github.com/emscripten-core/emscripten/issues/9097 >> Fix fullscreen exit causing wrong screen size >> >> - hard-coded change to EMT_STACK_MAX which isn't configurable >> >> - https://github.com/emscripten-core/emscripten-fastcomp/pull/195 >> Compiler limitation when combining several Emscripten features >> >> - https://github.com/emscripten-ports/SDL2/issues/88 >> Send SDL_APP_TERMINATING on page exit >> >> - https://github.com/emscripten-ports/SDL2/issues/70 >> Support Emterpreter/Asincify in SDL2 >> >> - Search "Emscripten" in https://renpy.beuc.net/#status_reports for all >> the other patches that were eventually integrated over the year >> > -- > You received this message because you are subscribed to the Google Groups > "emscripten-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/emscripten-discuss/14c22302-1e4c-e5af-bab8-20e55e14151a%40beuc.net > <https://groups.google.com/d/msgid/emscripten-discuss/14c22302-1e4c-e5af-bab8-20e55e14151a%40beuc.net?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpT-B-MPkVooWDi1ZOEaznvh10ZBAH0w3VJ%3D53R3q_8vTA%40mail.gmail.com.
