Re: Patching Emscripten: preserving good practices

['Sam Clegg' via emscripten-discuss]

On Wed, Oct 2, 2019 at 12:35 PM Beuc  wrote:
>
> I just upgraded 1.38.45->1.38.46 and I got a new build failure in an 
> unmodified ffmpeg build.
> Which is also what happened last time I upgraded Emscripten 1 month ago 
> (different error).
> I don't see wasm-ld-related notes in the ChangeLog, but apparently it got 
> stricter again.
> Compiling a few external libs may actually be a good release test case for 
> the Emscripten project ¯\_(ツ)_/¯
>

I think are you referring to this bug
https://github.com/emscripten-core/emscripten/issues/9565?

The good news is that we now have a unit test to cover that specific
issue.  But I agree we also need larger system tests.  Perhaps we
could add ffmpeg to our build somehow, perhaps via an improved ports
system.

> - Beuc
>
> On 20/09/2019 20:06, Beuc wrote:
>
> Hi,
>
> On 17/09/2019 00:18, Alon Zakai wrote:
>
> On Mon, Sep 16, 2019 at 6:50 AM Beuc  wrote:
>>
>> > * Are there tests in your codebase that we could run in upstream 
>> > emscripten?
>>
>> Come to think of it, there's one thorough automated test that we have to run 
>> at each upgrade:
>> building dependencies!
>> [...]
>
> It might be useful to set up CI that runs the emscripten tip-of-tree builds 
> on that (emsdk install tot-fastcomp or tot-upstream). Those are literally the 
> very newest code, that passed chromium CI but is not as heavily tested as the 
> actual release tags. You may sometimes see a temporary breakage you can 
> ignore, but it would also catch regressions.
>
> There seem to be a misunderstanding, you asked for test you could run, not 
> the other way around ;)
>
> It isn't worth it for us to constantly rebuild with ToT: most of the time 
> we're working on Ren'Py/RenPyWeb itself and can't afford varying dependencies.
> Investigating all temporary and non-temporary breakages sounds like time I 
> should reserve for properly upgrading and adapting to a newer, stable 
> Emscripten once every other month.
>
> - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used for 
> emsdk 1.38.42" from https://chromium.googlesource.com/emscripten-releases/
>
>
> Is that still an issue? Are the docs at
>
> https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten
>
> (I hadn't seen this question on first read.)
>
> My first attempts with binaryen seems to work, and for those interested Sam 
> Clegg posted a recap:
>
> https://github.com/emscripten-core/emscripten/issues/8995#issuecomment-532984238
>
> --
> You received this message because you are subscribed to the Google Groups 
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/emscripten-discuss/3f1f8c14-8d50-d66d-f6fb-31c588f42acf%40beuc.net.

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/CAL_va29dxeiMwU8mO%2BKTHbp383tCp6vzMTt%3D9oCd8H2Te%3DS7UQ%40mail.gmail.com.

Re: Patching Emscripten: preserving good practices

[Alon Zakai]

Yeah, given this has happened more than once, I tend to agree that it may
be worth us having ffmpeg tests in upstream. We'd need to measure how long
the build takes, but if it's reasonable, I'd support it.

On Wed, Oct 2, 2019 at 12:35 PM Beuc  wrote:

> I just upgraded 1.38.45->1.38.46 and I got a new build failure in an
> unmodified ffmpeg build.
> Which is also what happened last time I upgraded Emscripten 1 month ago
> (different error).
> I don't see wasm-ld-related notes in the ChangeLog, but apparently it got
> stricter again.
> Compiling a few external libs may actually be a good release test case for
> the Emscripten project ¯\_(ツ)_/¯
>
> - Beuc
> On 20/09/2019 20:06, Beuc wrote:
>
> Hi,
> On 17/09/2019 00:18, Alon Zakai wrote:
>
> On Mon, Sep 16, 2019 at 6:50 AM Beuc  wrote:
>
>> > * Are there tests in your codebase that we could run in upstream
>> emscripten?
>>
>> Come to think of it, there's one thorough automated test that we have to
>> run at each upgrade:
>> building dependencies!
>> [...]
>>
> It might be useful to set up CI that runs the emscripten tip-of-tree
> builds on that (emsdk install tot-fastcomp or tot-upstream). Those are
> literally the very newest code, that passed chromium CI but is not as
> heavily tested as the actual release tags. You may sometimes see a
> temporary breakage you can ignore, but it would also catch regressions.
>
> There seem to be a misunderstanding, you asked for test you could run, not
> the other way around ;)
>
> It isn't worth it for us to constantly rebuild with ToT: most of the time
> we're working on Ren'Py/RenPyWeb itself and can't afford varying
> dependencies.
> Investigating all temporary and non-temporary breakages sounds like time I
> should reserve for properly upgrading and adapting to a newer, stable
> Emscripten once every other month.
>
> - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used for
> emsdk 1.38.42" from https://chromium.googlesource.com/emscripten-releases/
>
>
> Is that still an issue? Are the docs at
>
>
> https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten
>
> (I hadn't seen this question on first read.)
>
> My first attempts with binaryen seems to work, and for those interested
> Sam Clegg posted a recap:
>
>
> https://github.com/emscripten-core/emscripten/issues/8995#issuecomment-532984238
>
> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/3f1f8c14-8d50-d66d-f6fb-31c588f42acf%40beuc.net
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpS1%3DJEaQA%3DVXBZOYAbRjKwtq6u9aTUwE0nVTF_L__1H5Q%40mail.gmail.com.

Re: Patching Emscripten: preserving good practices

[Mehdi Sabwat]

Hi,

I am interested by this, will you please share the logs?

It might be irrelevant but, did you clear your cache before, rebuilding?

emcc --clear-cache

On Wed, Oct 2, 2019, 21:35 Beuc  wrote:

> I just upgraded 1.38.45->1.38.46 and I got a new build failure in an
> unmodified ffmpeg build.
> Which is also what happened last time I upgraded Emscripten 1 month ago
> (different error).
> I don't see wasm-ld-related notes in the ChangeLog, but apparently it got
> stricter again.
> Compiling a few external libs may actually be a good release test case for
> the Emscripten project ¯\_(ツ)_/¯
>
> - Beuc
> On 20/09/2019 20:06, Beuc wrote:
>
> Hi,
> On 17/09/2019 00:18, Alon Zakai wrote:
>
> On Mon, Sep 16, 2019 at 6:50 AM Beuc  wrote:
>
>> > * Are there tests in your codebase that we could run in upstream
>> emscripten?
>>
>> Come to think of it, there's one thorough automated test that we have to
>> run at each upgrade:
>> building dependencies!
>> [...]
>>
> It might be useful to set up CI that runs the emscripten tip-of-tree
> builds on that (emsdk install tot-fastcomp or tot-upstream). Those are
> literally the very newest code, that passed chromium CI but is not as
> heavily tested as the actual release tags. You may sometimes see a
> temporary breakage you can ignore, but it would also catch regressions.
>
> There seem to be a misunderstanding, you asked for test you could run, not
> the other way around ;)
>
> It isn't worth it for us to constantly rebuild with ToT: most of the time
> we're working on Ren'Py/RenPyWeb itself and can't afford varying
> dependencies.
> Investigating all temporary and non-temporary breakages sounds like time I
> should reserve for properly upgrading and adapting to a newer, stable
> Emscripten once every other month.
>
> - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used for
> emsdk 1.38.42" from https://chromium.googlesource.com/emscripten-releases/
>
>
> Is that still an issue? Are the docs at
>
>
> https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten
>
> (I hadn't seen this question on first read.)
>
> My first attempts with binaryen seems to work, and for those interested
> Sam Clegg posted a recap:
>
>
> https://github.com/emscripten-core/emscripten/issues/8995#issuecomment-532984238
>
> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/3f1f8c14-8d50-d66d-f6fb-31c588f42acf%40beuc.net
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/CANLCaymuSPabZcWnfNzKEyfVkLppenMg4VShVGDqPpSsvdAK%2Bw%40mail.gmail.com.

Re: Patching Emscripten: preserving good practices

[Beuc]

I just upgraded 1.38.45->1.38.46 and I got a new build failure in an
unmodified ffmpeg build.
Which is also what happened last time I upgraded Emscripten 1 month ago
(different error).
I don't see wasm-ld-related notes in the ChangeLog, but apparently it
got stricter again.
Compiling a few external libs may actually be a good release test case
for the Emscripten project ¯\_(ツ)_/¯

- Beuc

On 20/09/2019 20:06, Beuc wrote:
>
> Hi,
>
> On 17/09/2019 00:18, Alon Zakai wrote:
>> On Mon, Sep 16, 2019 at 6:50 AM Beuc > > wrote:
>>
>> > * Are there tests in your codebase that we could run in
>> upstream emscripten?
>>
>> Come to think of it, there's one thorough automated test that we
>> have to run at each upgrade:
>> building dependencies!
>> [...]
>>
>> It might be useful to set up CI that runs the emscripten tip-of-tree
>> builds on that (emsdk install tot-fastcomp or tot-upstream). Those
>> are literally the very newest code, that passed chromium CI but is
>> not as heavily tested as the actual release tags. You may sometimes
>> see a temporary breakage you can ignore, but it would also catch
>> regressions.
>>
> There seem to be a misunderstanding, you asked for test you could run,
> not the other way around ;)
>
> It isn't worth it for us to constantly rebuild with ToT: most of the
> time we're working on Ren'Py/RenPyWeb itself and can't afford varying
> dependencies.
> Investigating all temporary and non-temporary breakages sounds like
> time I should reserve for properly upgrading and adapting to a newer,
> stable Emscripten once every other month.
>
>>> - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is
>>> used for emsdk 1.38.42" from
>>> https://chromium.googlesource.com/emscripten-releases/
>>>
>>
>> Is that still an issue? Are the docs at
>>
>> https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten
>
> (I hadn't seen this question on first read.)
>
> My first attempts with binaryen seems to work, and for those
> interested Sam Clegg posted a recap:
>
> https://github.com/emscripten-core/emscripten/issues/8995#issuecomment-532984238
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/3f1f8c14-8d50-d66d-f6fb-31c588f42acf%40beuc.net.

Re: Patching Emscripten: preserving good practices

[Beuc]

HHi,

On 17/09/2019 00:18, Alon Zakai wrote:
> On Mon, Sep 16, 2019 at 6:50 AM Beuc  > wrote:
>
> > * Are there tests in your codebase that we could run in upstream
> emscripten?
>
> Come to think of it, there's one thorough automated test that we
> have to run at each upgrade:
> building dependencies!
> [...]
>
> It might be useful to set up CI that runs the emscripten tip-of-tree
> builds on that (emsdk install tot-fastcomp or tot-upstream). Those are
> literally the very newest code, that passed chromium CI but is not as
> heavily tested as the actual release tags. You may sometimes see a
> temporary breakage you can ignore, but it would also catch regressions.
>
There seem to be a misunderstanding, you asked for test you could run,
not the other way around ;)

It isn't worth it for us to constantly rebuild with ToT: most of the
time we're working on Ren'Py/RenPyWeb itself and can't afford varying
dependencies.
Investigating all temporary and non-temporary breakages sounds like time
I should reserve for properly upgrading and adapting to a newer, stable
Emscripten once every other month.

>> - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used
>> for emsdk 1.38.42" from
>> https://chromium.googlesource.com/emscripten-releases/
>>
>
> Is that still an issue? Are the docs at
>
> https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten

(I hadn't seen this question on first read.)

My first attempts with binaryen seems to work, and for those interested
Sam Clegg posted a recap:

https://github.com/emscripten-core/emscripten/issues/8995#issuecomment-532984238

Cheers!
Beuc

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/82726f26-e8eb-5b4d-4938-c82d79fa02cf%40beuc.net.

Re: Patching Emscripten: preserving good practices

[Alon Zakai]

On Mon, Sep 16, 2019 at 6:50 AM Beuc  wrote:

> > * Are there tests in your codebase that we could run in upstream
> emscripten?
>
> Come to think of it, there's one thorough automated test that we have to
> run at each upgrade:
> building dependencies!
>
> https://github.com/renpy/renpyweb
> https://github.com/renpy/renpyweb/tree/master/scripts
> https://www.beuc.net/python-emscripten/python/dir?ci=tip
>
> Typing a simple:
>
> $ make
>
> will download and build all dependencies, and it's not uncommon that
> something breaks on Emscripten upgrade.
>

It might be useful to set up CI that runs the emscripten tip-of-tree builds
on that (emsdk install tot-fastcomp or tot-upstream). Those are literally
the very newest code, that passed chromium CI but is not as heavily tested
as the actual release tags. You may sometimes see a temporary breakage you
can ignore, but it would also catch regressions.

Cheers!
> Sylvain
> On 10/09/2019 12:41, Sylvain Beucler wrote:
>
> Hi,
>
> (I was hoping other people would give their PoV...)
>
> Generally-speaking, we may need more patches because of:
>
> - the combination of Emscripten features (Python + uninterruptible main
> loop -- aka fpcast+async; thankfully I could get rid of dynamic linking...)
>
> - centrally improving Emscripten portability rather than adapting the
> codebases (e.g. spending weeks on the 1-line #8751 / LANG support, rather
> than adding a quick EM_ASM in all my projects)
>
> - working directly with Emscripten rather than using a framework that
> provides a layer of work-arounds (e.g. Unity)
>
> Again, at a given point of time we always have a buffer of pending patches
> (I forgot to mention #6511 and #7631 that were added to our shell.html
> template, btw), the ones we can upstream are replaced by newer ones: old
> bugs we eventually tackle like fullscreen support or autosave-on-quit;
> backports for regressions in a given Emscripten release.
>
> Hopefully one day the web ecosystem will be so mature that we'll be able
> to take our distro's years-old compiler, `./configure --host=browser`
> without any change, and publish. Meanwhile... we patch :)
>
> To answer your specific questions:
> - This is a game engine, much of the tests are manual, sadly (but I
> contribute tests when I submit Emscripten patches); we often get
> integration errors, or errors due to combining multiple Emscripten options.
>
> - I upgrade Emscripten to a tagged version ~every other month; since this
> is likely to introduce breaking changes, I only do it when I have the time
> to deal with the consequences (e.g. when I'm not in the middle of
> developing RenPyWeb itself). I avoid ToT for stability and reproducibility
> reasons.
>
> - EMT_STACK_MAX is the exception, not the rule ;) That's about the only
> thing I could have upstream but didn't due to lack of time -- and it's now
> deprecated.
>
> - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used for
> emsdk 1.38.42" from https://chromium.googlesource.com/emscripten-releases/
>
>
Is that still an issue? Are the docs at

https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten

not clear enough?

- Alon


> Cheers!
> Sylvain
> On 07/09/2019 00:58, Alon Zakai wrote:
>
> I think those are all reasonable requests, thanks for opening those issues
> and highlighting them here! I hope we can make progress on each.
>
> And hopefully other users can provide useful advice from their perspective
> and experience.
>
> About "no matching tags for llvm and bynarien, commit IDs quite hard to
> track (discussion in #9317/#9383)" - I think we have resolved that? Please
> let me know if not.
>
> And a general comment: Your overall experience does surprise me (which
> shows why it's good you wrote it up!) - that's a lot more local changes
> than I've seen elsewhere, and that includes some very large production user
> setups. So I hope we can work together to make your experience closer to
> theirs.
>
>  * Are there tests in your codebase that we could run in upstream
> emscripten?
>  * Do you test latest emscripten releases on your codebase as they are
> tagged? (Another option is to test the much more frequent tip-of-tree
> builds, but you may see more noise there.)
>  * It would be good to discuss upstreaming long-term changes you mention
> like that EMT_STACK_MAX one - we may not be able to in all cases, but let's
> try at least (I don't think I remember a discussion on that one).
>
> - Alon
>
>
>
> On Fri, Sep 6, 2019 at 12:32 AM Beuc  wrote:
>
>> Hi,
>>
>> We almost never could use Emscripten without applying a few patches.
>> It's become increasingly difficult, not only to patch, but to convince
>> the developers that it's legitimate, so I thought I'd detail our use case
>> for good here.
>>
>> So we've been using Emscripten with patches in emscripten, llvm and SDL2
>> port [1].
>> By the time a patch is officially included in a release, we usually need
>> more patches as we face new 

Re: Patching Emscripten: preserving good practices

[Beuc]

> * Are there tests in your codebase that we could run in upstream
emscripten?

Come to think of it, there's one thorough automated test that we have to
run at each upgrade:
building dependencies!

https://github.com/renpy/renpyweb
https://github.com/renpy/renpyweb/tree/master/scripts
https://www.beuc.net/python-emscripten/python/dir?ci=tip

Typing a simple:

$ make

will download and build all dependencies, and it's not uncommon that
something breaks on Emscripten upgrade.

Cheers!
Sylvain

On 10/09/2019 12:41, Sylvain Beucler wrote:
>
> Hi,
>
> (I was hoping other people would give their PoV...)
>
> Generally-speaking, we may need more patches because of:
>
> - the combination of Emscripten features (Python + uninterruptible
> main loop -- aka fpcast+async; thankfully I could get rid of dynamic
> linking...)
>
> - centrally improving Emscripten portability rather than adapting the
> codebases (e.g. spending weeks on the 1-line #8751 / LANG support,
> rather than adding a quick EM_ASM in all my projects)
>
> - working directly with Emscripten rather than using a framework that
> provides a layer of work-arounds (e.g. Unity)
>
> Again, at a given point of time we always have a buffer of pending
> patches (I forgot to mention #6511 and #7631 that were added to our
> shell.html template, btw), the ones we can upstream are replaced by
> newer ones: old bugs we eventually tackle like fullscreen support or
> autosave-on-quit; backports for regressions in a given Emscripten release.
>
> Hopefully one day the web ecosystem will be so mature that we'll be
> able to take our distro's years-old compiler, `./configure
> --host=browser` without any change, and publish. Meanwhile... we patch :)
>
> To answer your specific questions:
>
> - This is a game engine, much of the tests are manual, sadly (but I
> contribute tests when I submit Emscripten patches); we often get
> integration errors, or errors due to combining multiple Emscripten
> options.
>
> - I upgrade Emscripten to a tagged version ~every other month; since
> this is likely to introduce breaking changes, I only do it when I have
> the time to deal with the consequences (e.g. when I'm not in the
> middle of developing RenPyWeb itself). I avoid ToT for stability and
> reproducibility reasons.
>
> - EMT_STACK_MAX is the exception, not the rule ;) That's about the
> only thing I could have upstream but didn't due to lack of time -- and
> it's now deprecated.
>
> - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used
> for emsdk 1.38.42" from
> https://chromium.googlesource.com/emscripten-releases/
>
> Cheers!
> Sylvain
>
> On 07/09/2019 00:58, Alon Zakai wrote:
>> I think those are all reasonable requests, thanks for opening those
>> issues and highlighting them here! I hope we can make progress on each.
>>
>> And hopefully other users can provide useful advice from their
>> perspective and experience.
>>
>> About "no matching tags for llvm and bynarien, commit IDs quite hard
>> to track (discussion in #9317/#9383)" - I think we have resolved
>> that? Please let me know if not.
>>
>> And a general comment: Your overall experience does surprise me
>> (which shows why it's good you wrote it up!) - that's a lot more
>> local changes than I've seen elsewhere, and that includes some very
>> large production user setups. So I hope we can work together to make
>> your experience closer to theirs.
>>
>>  * Are there tests in your codebase that we could run in upstream
>> emscripten?
>>  * Do you test latest emscripten releases on your codebase as they
>> are tagged? (Another option is to test the much more frequent
>> tip-of-tree builds, but you may see more noise there.)
>>  * It would be good to discuss upstreaming long-term changes you
>> mention like that EMT_STACK_MAX one - we may not be able to in all
>> cases, but let's try at least (I don't think I remember a discussion
>> on that one).
>>
>> - Alon
>>
>>
>>
>> On Fri, Sep 6, 2019 at 12:32 AM Beuc > > wrote:
>>
>> Hi,
>>
>> We almost never could use Emscripten without applying a few patches.
>> It's become increasingly difficult, not only to patch, but to
>> convince the developers that it's legitimate, so I thought I'd
>> detail our use case for good here.
>>
>> So we've been using Emscripten with patches in emscripten, llvm
>> and SDL2 port [1].
>> By the time a patch is officially included in a release, we
>> usually need more patches as we face new issues -- or regressions.
>> Rinse and repeat.
>>
>> Consequently for our production use (the RenPyWeb build that we
>> publish to our users), we build an identified, versioned release
>> (e.g. 1.38.42) with specific fixes.
>>
>>
>> New difficulties:
>> - scarse documentation for recompiling "upstream" (patch in
>> progress #9383)
>> - no matching tags for llvm and bynarien, commit IDs quite hard
>> to track (discussion in #9317/#9383)
>> - misleading 

Re: Patching Emscripten: preserving good practices

[Sylvain Beucler]

Hi,

(I was hoping other people would give their PoV...)

Generally-speaking, we may need more patches because of:

- the combination of Emscripten features (Python + uninterruptible main
loop -- aka fpcast+async; thankfully I could get rid of dynamic linking...)

- centrally improving Emscripten portability rather than adapting the
codebases (e.g. spending weeks on the 1-line #8751 / LANG support,
rather than adding a quick EM_ASM in all my projects)

- working directly with Emscripten rather than using a framework that
provides a layer of work-arounds (e.g. Unity)

Again, at a given point of time we always have a buffer of pending
patches (I forgot to mention #6511 and #7631 that were added to our
shell.html template, btw), the ones we can upstream are replaced by
newer ones: old bugs we eventually tackle like fullscreen support or
autosave-on-quit; backports for regressions in a given Emscripten release.

Hopefully one day the web ecosystem will be so mature that we'll be able
to take our distro's years-old compiler, `./configure --host=browser`
without any change, and publish. Meanwhile... we patch :)

To answer your specific questions:

- This is a game engine, much of the tests are manual, sadly (but I
contribute tests when I submit Emscripten patches); we often get
integration errors, or errors due to combining multiple Emscripten options.

- I upgrade Emscripten to a tagged version ~every other month; since
this is likely to introduce breaking changes, I only do it when I have
the time to deal with the consequences (e.g. when I'm not in the middle
of developing RenPyWeb itself). I avoid ToT for stability and
reproducibility reasons.

- EMT_STACK_MAX is the exception, not the rule ;) That's about the only
thing I could have upstream but didn't due to lack of time -- and it's
now deprecated.

- Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used
for emsdk 1.38.42" from
https://chromium.googlesource.com/emscripten-releases/

Cheers!
Sylvain

On 07/09/2019 00:58, Alon Zakai wrote:
> I think those are all reasonable requests, thanks for opening those
> issues and highlighting them here! I hope we can make progress on each.
>
> And hopefully other users can provide useful advice from their
> perspective and experience.
>
> About "no matching tags for llvm and bynarien, commit IDs quite hard
> to track (discussion in #9317/#9383)" - I think we have resolved that?
> Please let me know if not.
>
> And a general comment: Your overall experience does surprise me (which
> shows why it's good you wrote it up!) - that's a lot more local
> changes than I've seen elsewhere, and that includes some very large
> production user setups. So I hope we can work together to make your
> experience closer to theirs.
>
>  * Are there tests in your codebase that we could run in upstream
> emscripten?
>  * Do you test latest emscripten releases on your codebase as they are
> tagged? (Another option is to test the much more frequent tip-of-tree
> builds, but you may see more noise there.)
>  * It would be good to discuss upstreaming long-term changes you
> mention like that EMT_STACK_MAX one - we may not be able to in all
> cases, but let's try at least (I don't think I remember a discussion
> on that one).
>
> - Alon
>
>
>
> On Fri, Sep 6, 2019 at 12:32 AM Beuc  > wrote:
>
> Hi,
>
> We almost never could use Emscripten without applying a few patches.
> It's become increasingly difficult, not only to patch, but to
> convince the developers that it's legitimate, so I thought I'd
> detail our use case for good here.
>
> So we've been using Emscripten with patches in emscripten, llvm
> and SDL2 port [1].
> By the time a patch is officially included in a release, we
> usually need more patches as we face new issues -- or regressions.
> Rinse and repeat.
>
> Consequently for our production use (the RenPyWeb build that we
> publish to our users), we build an identified, versioned release
> (e.g. 1.38.42) with specific fixes.
>
>
> New difficulties:
> - scarse documentation for recompiling "upstream" (patch in
> progress #9383)
> - no matching tags for llvm and bynarien, commit IDs quite hard to
> track (discussion in #9317/#9383)
> - misleading versioning in tagged commits (#9282)
> - grab-from-git port system; EMCC_LOCAL_PORTS considered for
> "local debugging", not cached, and causing issues (e.g. #9342 #9402)
>
>
> Rebutals:
>
> - "99% users use emsdk binaries"
> Well, the remaining 1% will provide 99% of external contributions,
> take care of them :)
> Also being able to test fixes before they finish the full
> review process allows spotting issues in advance.
>
> - "users will either use the unmodified binaries, or work on
> master/tip-of-tree"
> No, we use an identified, tagged Emscripten version, because it's
> more stable than a random commit.
> Also when we 

Re: Patching Emscripten: preserving good practices

[Alon Zakai]

I think those are all reasonable requests, thanks for opening those issues
and highlighting them here! I hope we can make progress on each.

And hopefully other users can provide useful advice from their perspective
and experience.

About "no matching tags for llvm and bynarien, commit IDs quite hard to
track (discussion in #9317/#9383)" - I think we have resolved that? Please
let me know if not.

And a general comment: Your overall experience does surprise me (which
shows why it's good you wrote it up!) - that's a lot more local changes
than I've seen elsewhere, and that includes some very large production user
setups. So I hope we can work together to make your experience closer to
theirs.

 * Are there tests in your codebase that we could run in upstream
emscripten?
 * Do you test latest emscripten releases on your codebase as they are
tagged? (Another option is to test the much more frequent tip-of-tree
builds, but you may see more noise there.)
 * It would be good to discuss upstreaming long-term changes you mention
like that EMT_STACK_MAX one - we may not be able to in all cases, but let's
try at least (I don't think I remember a discussion on that one).

- Alon



On Fri, Sep 6, 2019 at 12:32 AM Beuc  wrote:

> Hi,
>
> We almost never could use Emscripten without applying a few patches.
> It's become increasingly difficult, not only to patch, but to convince the
> developers that it's legitimate, so I thought I'd detail our use case for
> good here.
>
> So we've been using Emscripten with patches in emscripten, llvm and SDL2
> port [1].
> By the time a patch is officially included in a release, we usually need
> more patches as we face new issues -- or regressions.
> Rinse and repeat.
>
> Consequently for our production use (the RenPyWeb build that we publish to
> our users), we build an identified, versioned release (e.g. 1.38.42) with
> specific fixes.
>
>
> New difficulties:
> - scarse documentation for recompiling "upstream" (patch in progress #9383)
> - no matching tags for llvm and bynarien, commit IDs quite hard to track
> (discussion in #9317/#9383)
> - misleading versioning in tagged commits (#9282)
> - grab-from-git port system; EMCC_LOCAL_PORTS considered for "local
> debugging", not cached, and causing issues (e.g. #9342 #9402)
>
>
> Rebutals:
>
> - "99% users use emsdk binaries"
> Well, the remaining 1% will provide 99% of external contributions, take
> care of them :)
> Also being able to test fixes before they finish the full review
> process allows spotting issues in advance.
>
> - "users will either use the unmodified binaries, or work on
> master/tip-of-tree"
> No, we use an identified, tagged Emscripten version, because it's more
> stable than a random commit.
> Also when we publish a hot fix, sometimes a couple months after release,
> we need to reproduce the same build environment
>
> - "emsdk can build from source"
> This build process doesn't allow patching, so it's only when one doesn't
> trust the binaries and wants to rebuild them as-is AFAIU.
> (To be honest, I'd like to be able to use the binaries, but Emscripten
> being bleeding-edge, I figure that'll have to wait :))
>
>
> I hope this makes clear why we need to checkout emscripten/llvm/bynarien
> for an existing release, patch it, and build it.
> Preferably without reverse-engineering the build process and fighting all
> the stack.
> That's the first time I have to justify this while working on free/open
> source software.
>
> How do you people deal with this?
>
>
> [1] For instance, our current release needs these patches, some of them
> still needing work & official integration:
>
> - https://github.com/emscripten-core/emscripten/issues/9257
> Work-around ./configure misdetection
>
> - https://github.com/emscripten-core/emscripten/issues/9097
> Fix fullscreen exit causing wrong screen size
>
> - hard-coded change to EMT_STACK_MAX which isn't configurable
>
> - https://github.com/emscripten-core/emscripten-fastcomp/pull/195
> Compiler limitation when combining several Emscripten features
>
> - https://github.com/emscripten-ports/SDL2/issues/88
> Send SDL_APP_TERMINATING on page exit
>
> - https://github.com/emscripten-ports/SDL2/issues/70
> Support Emterpreter/Asincify in SDL2
>
> - Search "Emscripten" in https://renpy.beuc.net/#status_reports for all
> the other patches that were eventually integrated over the year
>
> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/20190906073253.yh5wv677v3l4xn4k%40mail.beuc.net
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Patching Emscripten: preserving good practices

[Beuc]

Hi,

We almost never could use Emscripten without applying a few patches.
It's become increasingly difficult, not only to patch, but to convince the 
developers that it's legitimate, so I thought I'd detail our use case for good 
here.

So we've been using Emscripten with patches in emscripten, llvm and SDL2 port 
[1].
By the time a patch is officially included in a release, we usually need more 
patches as we face new issues -- or regressions.
Rinse and repeat.

Consequently for our production use (the RenPyWeb build that we publish to our 
users), we build an identified, versioned release (e.g. 1.38.42) with specific 
fixes.


New difficulties:
- scarse documentation for recompiling "upstream" (patch in progress #9383)
- no matching tags for llvm and bynarien, commit IDs quite hard to track 
(discussion in #9317/#9383)
- misleading versioning in tagged commits (#9282)
- grab-from-git port system; EMCC_LOCAL_PORTS considered for "local debugging", 
not cached, and causing issues (e.g. #9342 #9402)


Rebutals:

- "99% users use emsdk binaries"
Well, the remaining 1% will provide 99% of external contributions, take care of 
them :)
Also being able to test fixes before they finish the full review process 
allows spotting issues in advance.

- "users will either use the unmodified binaries, or work on master/tip-of-tree"
No, we use an identified, tagged Emscripten version, because it's more stable 
than a random commit.
Also when we publish a hot fix, sometimes a couple months after release, we 
need to reproduce the same build environment

- "emsdk can build from source"
This build process doesn't allow patching, so it's only when one doesn't trust 
the binaries and wants to rebuild them as-is AFAIU.
(To be honest, I'd like to be able to use the binaries, but Emscripten being 
bleeding-edge, I figure that'll have to wait :))


I hope this makes clear why we need to checkout emscripten/llvm/bynarien for an 
existing release, patch it, and build it.
Preferably without reverse-engineering the build process and fighting all the 
stack.
That's the first time I have to justify this while working on free/open source 
software.

How do you people deal with this?


[1] For instance, our current release needs these patches, some of them still 
needing work & official integration:

- https://github.com/emscripten-core/emscripten/issues/9257
Work-around ./configure misdetection

- https://github.com/emscripten-core/emscripten/issues/9097
Fix fullscreen exit causing wrong screen size

- hard-coded change to EMT_STACK_MAX which isn't configurable

- https://github.com/emscripten-core/emscripten-fastcomp/pull/195
Compiler limitation when combining several Emscripten features

- https://github.com/emscripten-ports/SDL2/issues/88
Send SDL_APP_TERMINATING on page exit

- https://github.com/emscripten-ports/SDL2/issues/70
Support Emterpreter/Asincify in SDL2

- Search "Emscripten" in https://renpy.beuc.net/#status_reports for all the 
other patches that were eventually integrated over the year

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/20190906073253.yh5wv677v3l4xn4k%40mail.beuc.net.