Re: Wine MinGW system libraries
Hello all, Since this conversation several months ago I've been working with the Wine maintainer on implementing a solution upstream that is compatible with our requirements and the pretty much universal desire by packagers to avoid system library imports. I believe I've found a solution that will work for now for everyone, although it has raised some questions about standardization of MinGW packages—see the "Open Questions" section of this mail for more details. == Summary of upstream progress == Currently Wine bundles the source of several libraries. At the same time, as was discussed we've implemented a system upstream by which external shared libraries can be linked to and loaded (while keeping them in a separate "namespace" so that they don't conflict with identically named application DLLs.) There are still some concerns on the Wine side that this will break applications in subtle ways, but unfortunately there has been almost no testing of this feature yet. The libraries which we import are as follows: * libFAudio * libgsm * libjpeg * jxrlib * liblcms2 * libmpg123 * libpng * libtiff * libvkd3d * libxml2 * libxslt * zlib This list is probably going to remain stable. Despite what was discussed earlier I do *not* forsee us importing libOSMesa (it turns out that it actually makes more sense on the Win32 "kernel" side). An import of libfreetype or libgnutls is also unlikely at this point (we've found it more worthwhile to write thunks for both). We currently do *not* support external static linking. The basic reason for this is that Wine builds against its own CRT rather than MinGW's, and we cannot safely statically link to a module built with a different CRT. == Technical details == In order to use system shared libraries, Wine needs to know where to find them at runtime. This is done with a configure argument: configure --with-system-dllpath=/first/directory/:/second/directory/ Note that this may need to include the location of libgcc_s and other dependencies, which is in a different location on Debian. Separately, for each argument, we need to know the compiler and linker flags used to find the library. This also can be done with configure arguments, much like for native libraries: configure GSM_PE_CFLAGS="-Iwhatever" GSM_PE_LIBS="-Lwhatever -lgsm" If MinGW pkg-config is present, and --with-system-dllpath is specified, Wine will automatically use it to detect the compiler and linker flags for all of the above packages supporting pkg-config, which in particular excludes libgsm and jxrlib. Thus in practice it should not be necessary to hand-code most CFLAGS and LIBS variables. == Open Question(s) == Currently it's necessary to use a configure argument to specify where to find the runtime path. This is fine for distributions, which is where most people are going to be getting Wine anyway. It's a bit unfortunate for anyone self-compiling Wine, though, and it'd be nice if there was some way to avoid it. Unfortunately, there are two things standing in our way. The first problem is that the location of runtime DLLs varies wildly between distributions, and there's no common independent way to detect it. We could potentially hardcode a few "guesses" at the runtime path into Wine's configure script, but that brings us to the second problem: there's no way to verify the presence of runtime DLLs. We *are* the loader and lower-level APIs and would have to bootstrap ourselves first, and this is pretty much infeasible. Verifying by name doesn't really work either, since the name isn't really predictable. Hence "standardize the location of runtime DLLs across distributions" isn't really a feasible solution either (or at least not a complete one), as nice as it would be—we have no way to know whether a distribution is post-standardization or not. Rather, I think the best solution is to have some tool which, if present or successful, lists one or more directories where libraries can be found. I'm not sure what the most sensible thing here is: * pkg-config would be an obvious tool to extend, but would presumably require modifying .pc files for all relevant packages upstream. The location also seems to be consistent across all packages within each distribution I was able to test, so there probably isn't much point in making it package-specific. [For the packages we care about, cmake and libtool both put things in /bin by default, and zlib apparently has no built-in cross-compilation support.] * Introducing an analogue of ld.so.conf seems like a good idea. It would allow for per-package extra directories, and is relatively easy to parse, although ideally it would be even easier than that (i.e. if someone else could deal with the "include" part, that would be great). ἔρρωσθε, Zeb Figura
Re: Wine MinGW system libraries
On Mon, Sep 13, 2021 at 08:38:52PM +, Bastien ROUCARIES wrote: > Le lun. 13 sept. 2021 à 20:24, Adrian Bunk a écrit : > > On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote: >... > > > The regressions are significant though: if packages can’t stay > > > cross-satisfiable for Debian cross-targets, there’s little hope they can > > > stay > > > cross-satisfiable for Windows! This means that separate source packages > > > are > > > probably the only viable option. > > >... > > > > Separate source packages don't bring any real benefits compared to > > just bundling and building everything in src:wine. > > No please do not do that. Vendoring is bad and I fight on the > javascript side since four years, against it. >... Separate source packages that duplicate existing sources are as much vendoring as doing the same inside src:wine. There are only two realistic options for Wine: 1. building multilib packages from the regular sources of the libraries in the archive, or 2. vendoring Noone seems to be willing to do 1., which leaves 2. as only option. cu Adrian
Re: Wine MinGW system libraries
Le lun. 13 sept. 2021 à 20:24, Adrian Bunk a écrit : > > On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote: > >... > > For Wine (and even a wider MinGW-w64 > > ecosystem) we don’t need all that many source packages to be > > cross-satisfiable for the whole endeavour to be useful... > > But you would still need to create and maintain the whole > infrastructure. > > The larger picture is that there are at least 3 big topics: > - cross-architecture dependencies > - partial release architectures > - cross-building a release architecture > > Every single of these topics would be a huge amount of work. > > If you need all 3 as basis for a solution for Wine, > then that's simply not realistic in the foreseeable future. > > > The regressions are significant though: if packages can’t stay > > cross-satisfiable for Debian cross-targets, there’s little hope they can > > stay > > cross-satisfiable for Windows! This means that separate source packages are > > probably the only viable option. > >... > > Separate source packages don't bring any real benefits compared to > just bundling and building everything in src:wine. No please do not do that. Vendoring is bad and I fight on the javascript side since four years, against it. > > Either would likely imply that Wine in Debian comes without security > support, and the Release Notes stating that Wine should only be used > for trusted local contents. > > > Regards, > > > > Stephen > > cu > Adrian >
Re: Wine MinGW system libraries
On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote: >... > For Wine (and even a wider MinGW-w64 > ecosystem) we don’t need all that many source packages to be > cross-satisfiable for the whole endeavour to be useful... But you would still need to create and maintain the whole infrastructure. The larger picture is that there are at least 3 big topics: - cross-architecture dependencies - partial release architectures - cross-building a release architecture Every single of these topics would be a huge amount of work. If you need all 3 as basis for a solution for Wine, then that's simply not realistic in the foreseeable future. > The regressions are significant though: if packages can’t stay > cross-satisfiable for Debian cross-targets, there’s little hope they can stay > cross-satisfiable for Windows! This means that separate source packages are > probably the only viable option. >... Separate source packages don't bring any real benefits compared to just bundling and building everything in src:wine. Either would likely imply that Wine in Debian comes without security support, and the Release Notes stating that Wine should only be used for trusted local contents. > Regards, > > Stephen cu Adrian
Re: Wine MinGW system libraries
On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote: > Is the process documented anywhere? Or does one simply pick a failure from > http://crossqa.debian.net and figure out what’s going wrong (and hope that > pulling the threads doesn’t reveal a monster...)? That describes the process quite well. In particular, crossqa.d.n links to existing bug reports to avoid duplication of work. A major lack (beyond people doing the work) is domain-specific knowledge of the various archive sections. So once you locate a monster, don't stop there, but contact debian-cr...@lists.debian.org or #debian-bootstrap@oftc and let us combine your domain-specific knowledge with some cross compilation experience to create an original solution. So yeah, start with those packages that you know well. Check http://crossqa.debian.net/src/ for your pet packages or follow the "cross" link in the right sidebar at tracker.d.o. Also (thanks to IOhannes zmölnig) enable cross building in your salsa-ci pipeline. In any case though, don't let wine wait for cross building to become a thing. Helmut
Re: Wine MinGW system libraries
On Sun, 12 Sep 2021 20:54:56 +0300, Adrian Bunk wrote: > On Sun, Sep 12, 2021 at 07:03:54PM +0200, Stephen Kitt wrote: > > On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk wrote: > > > On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote: > > > >... > > > > While one could imagine adding support to all the appropriate > > > > source packages to build similar “Architecture: all” packages, that > > > > would require convincing all the relevant maintainers, > > > > > > Adding a new release architecture (partial or not) requires convincing > > > the ftp team and the release team. > > > > > > It would also require many software changes. > > > > Yes, it would. However, it requires convincing a well-defined set of > > people *once*. If we handle Windows dependencies using “Architecture: > > all” packages, we would either end up with something like Fedora’s > > MinGW-w64 SIG (with a complete set of independent packages), or we would > > end up having to convince a huge set of package maintainers over and over > > again. > > Based on the package list provided in the initial email in this thread > (and guessing the amount of transitive dependencies) I would have guessed > that there are perhaps ~ 30 packages that would have to be touched once. ... which means ~30 maintainers who need convincing individually. My experience so far is that maintainers tend not to be interested (justifiably) in Windows cross-builds. > And after that things will just continue working, just like with udebs > which are in some ways similar to what is being discussed here. udebs are mostly a packaging concern. Keeping Windows cross-builds working tends to require a bit more effort unfortunately. > > > How would for example the dependencies of wine:amd64 be fulfilled? > > > > > > Just supporting that package dependencies might only be fulfilled by > > > also using packages from a different architecture would require changes > > > in many places, like for example the testing migration scripts that > > > ensure installability after migration. > > > > In the same way as the none-arch packages would be. Yes, it’s a lot of > > work, but it’s useful for more than just Wine, and it can be done once > > for all the interested architectures. > > It is also the kind of change where it might be required that it is > fully supported in one release before it can be used by packages in the > next release, if for example any of dpkg/apt/aptitude/... in the > previous stable gives an error when seeing dependencies to another > architecture. That is indeed a good point, thanks for bringing it up! Regards, Stephen pgp6oea3PS5Hm.pgp Description: OpenPGP digital signature
Re: Wine MinGW system libraries
On Mon, 13 Sep 2021 17:39:25 +0200, Helmut Grohne wrote: > I believe I can speak with my "main cross building porter" hat on. > > On Sun, Sep 12, 2021 at 01:57:44PM +, Bastien ROUCARIES wrote: > > They are a full color gradiant between: > > - freestanding arches pure cross compile without any depends except > > arch:all > > - partial cross built arch > > - partial arch > > - full arch > > > > I believe the first step to get partial cross built arch is to begin > > by freestanding arch. > > Certainly not. As much as I would love to see this happen, it is very > unrealistic. wine can quite simply use -$arch-cross:all packages like > very other bare metal target and switch to our pipe dream once it is > ready (if ever). And even that assumes that we can get agreement on what $arch should be for Windows targets :-(. > cross building is very far off becoming the thing you want it to be. It > presently has a bus factor of around 1.5. Around 59% of source packages > are cross-satisfiable. Around 78% of cross-satisfiable packages are > cross buildable. In other words: less than 50% of packages are cross > buildable. Beyond that, the cross porters cannot keep up with the > current rate of regressions. Fractions are falling. The bus factor is indeed a problem. For Wine (and even a wider MinGW-w64 ecosystem) we don’t need all that many source packages to be cross-satisfiable for the whole endeavour to be useful... The regressions are significant though: if packages can’t stay cross-satisfiable for Debian cross-targets, there’s little hope they can stay cross-satisfiable for Windows! This means that separate source packages are probably the only viable option. > To make cross building a viable thing we'd need at least three more > porters determined to make it happen. Talk doesn't make it happen. > Patches do. Let me know if you want to help. I'll get you started. Is the process documented anywhere? Or does one simply pick a failure from http://crossqa.debian.net and figure out what’s going wrong (and hope that pulling the threads doesn’t reveal a monster...)? Regards, Stephen pgpCQ0PLoIbRU.pgp Description: OpenPGP digital signature
Re: Wine MinGW system libraries
I believe I can speak with my "main cross building porter" hat on. On Sun, Sep 12, 2021 at 01:57:44PM +, Bastien ROUCARIES wrote: > They are a full color gradiant between: > - freestanding arches pure cross compile without any depends except arch:all > - partial cross built arch > - partial arch > - full arch > > I believe the first step to get partial cross built arch is to begin > by freestanding arch. Certainly not. As much as I would love to see this happen, it is very unrealistic. wine can quite simply use -$arch-cross:all packages like very other bare metal target and switch to our pipe dream once it is ready (if ever). cross building is very far off becoming the thing you want it to be. It presently has a bus factor of around 1.5. Around 59% of source packages are cross-satisfiable. Around 78% of cross-satisfiable packages are cross buildable. In other words: less than 50% of packages are cross buildable. Beyond that, the cross porters cannot keep up with the current rate of regressions. Fractions are falling. To make cross building a viable thing we'd need at least three more porters determined to make it happen. Talk doesn't make it happen. Patches do. Let me know if you want to help. I'll get you started. Helmut
Re: Wine MinGW system libraries
On Sun, Sep 12, 2021 at 07:03:54PM +0200, Stephen Kitt wrote: > On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk wrote: > > On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote: > > >... > > > While one could imagine adding support to all the appropriate > > > source packages to build similar “Architecture: all” packages, that would > > > require convincing all the relevant maintainers, > > > > Adding a new release architecture (partial or not) requires convincing > > the ftp team and the release team. > > > > It would also require many software changes. > > Yes, it would. However, it requires convincing a well-defined set of people > *once*. If we handle Windows dependencies using “Architecture: all” packages, > we would either end up with something like Fedora’s MinGW-w64 SIG (with a > complete set of independent packages), or we would end up having to convince > a huge set of package maintainers over and over again. Based on the package list provided in the initial email in this thread (and guessing the amount of transitive dependencies) I would have guessed that there are perhaps ~ 30 packages that would have to be touched once. And after that things will just continue working, just like with udebs which are in some ways similar to what is being discussed here. > > How would for example the dependencies of wine:amd64 be fulfilled? > > > > Just supporting that package dependencies might only be fulfilled by > > also using packages from a different architecture would require changes > > in many places, like for example the testing migration scripts that > > ensure installability after migration. > > In the same way as the none-arch packages would be. Yes, it’s a lot of work, > but it’s useful for more than just Wine, and it can be done once for all the > interested architectures. It is also the kind of change where it might be required that it is fully supported in one release before it can be used by packages in the next release, if for example any of dpkg/apt/aptitude/... in the previous stable gives an error when seeing dependencies to another architecture. >... > Regards, > > Stephen cu Adrian
Re: Wine MinGW system libraries
On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk wrote: > On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote: > >... > > While one could imagine adding support to all the appropriate > > source packages to build similar “Architecture: all” packages, that would > > require convincing all the relevant maintainers, > > Adding a new release architecture (partial or not) requires convincing > the ftp team and the release team. > > It would also require many software changes. Yes, it would. However, it requires convincing a well-defined set of people *once*. If we handle Windows dependencies using “Architecture: all” packages, we would either end up with something like Fedora’s MinGW-w64 SIG (with a complete set of independent packages), or we would end up having to convince a huge set of package maintainers over and over again. > How would for example the dependencies of wine:amd64 be fulfilled? > > Just supporting that package dependencies might only be fulfilled by > also using packages from a different architecture would require changes > in many places, like for example the testing migration scripts that > ensure installability after migration. In the same way as the none-arch packages would be. Yes, it’s a lot of work, but it’s useful for more than just Wine, and it can be done once for all the interested architectures. > > and it would end up tying the testing migrations to MinGW-w64... > > If this partial arch (and Wine) should be part of Debian releases, > then testing migrations would have to be tied to it in any case. True, I’d missed that. (In my mind I was comparing this with having separate source packages.) > >... > > The buildd situation isn’t necessarily that much of an obstacle: it seems > > to me we could have “Windows” buildds which are really Linux amd64 > > systems, that cross-build for Windows. > > The first obstacle is that if you want to be the first who does cross > building packages on the buildds, there is likely a lot of work and > bugfixing ahead for you for getting that working. A lot of that work has already been done for http://crossqa.debian.net/ Fixing this would reduce the burden on all the maintainers who currently handle cross-built packages in the archive, so overall I think it would be a net win. Regards, Stephen pgpnUkwnkb6Zl.pgp Description: OpenPGP digital signature
Re: Wine MinGW system libraries
On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote: >... > While one could imagine adding support to all the appropriate > source packages to build similar “Architecture: all” packages, that would > require convincing all the relevant maintainers, Adding a new release architecture (partial or not) requires convincing the ftp team and the release team. It would also require many software changes. How would for example the dependencies of wine:amd64 be fulfilled? Just supporting that package dependencies might only be fulfilled by also using packages from a different architecture would require changes in many places, like for example the testing migration scripts that ensure installability after migration. > and it would end up tying the testing migrations to MinGW-w64... If this partial arch (and Wine) should be part of Debian releases, then testing migrations would have to be tied to it in any case. >... > The buildd situation isn’t necessarily that much of an obstacle: it seems to > me we could have “Windows” buildds which are really Linux amd64 systems, that > cross-build for Windows. The first obstacle is that if you want to be the first who does cross building packages on the buildds, there is likely a lot of work and bugfixing ahead for you for getting that working. > Regards, > > Stephen cu Adrian
Re: Wine MinGW system libraries
On Sun, 12 Sep 2021 10:38:52 +0300, Adrian Bunk wrote: > On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote: > >... > > Improve dpkg to support partial arch. I volonteer to implement none arch > > but i am waiting from guillem here. > >... > > There is also plenty of infrastructure on the buildd, archive and > release team sides that would likely need changes for supporting > anything like that. > > A multilib based approach might be more realistic for bookworm. > > What benefits would a "none arch" even bring compared to building > binary-all packages? > The ability to binNMU is the only one that comes into my mind. IMO the main benefit of using multiarch would be that packages could be built for the new architectures without changes (ideally). libz is currently built for MinGW-w64 in an “Architecture: all” package, but it’s a separate source package; various other libraries are built with specific support in their source packages. While one could imagine adding support to all the appropriate source packages to build similar “Architecture: all” packages, that would require convincing all the relevant maintainers, and it would end up tying the testing migrations to MinGW-w64... In this scenario the solution wouldn’t be a “none” arch, but a Windows arch, if we can ever resolve https://bugs.debian.org/606825 The buildd situation isn’t necessarily that much of an obstacle: it seems to me we could have “Windows” buildds which are really Linux amd64 systems, that cross-build for Windows. Regards, Stephen pgpgVleJyWbI1.pgp Description: OpenPGP digital signature
Re: Wine MinGW system libraries
On Sun, Sep 12, 2021 at 02:57:20PM +, Bastien ROUCARIES wrote: > Le dim. 12 sept. 2021 à 14:16, Adrian Bunk a écrit : > > > > On Sun, Sep 12, 2021 at 01:57:44PM +, Bastien ROUCARIES wrote: > > > > > > I think you misunderstand: > > > https://wiki.debian.org/Teams/Dpkg/Spec/FreestandingArches > > > > > > They are a full color gradiant between: > > > - freestanding arches pure cross compile without any depends except > > > arch:all > > > - partial cross built arch > > > - partial arch > > > - full arch > > > > > > I believe the first step to get partial cross built arch is to begin > > > by freestanding arch. > > > > I believe this is mostly irrelevant for the discussion about Wine. > > > > Making Wine in Debian depend on such lofty ideas that might be available > > very far in the future (if ever) would imply no Wine in the next Debian > > releases. > > > > Debian 12 will be released mid-2023, which is only 2 years away. > > Debian 13 will be released mid-2025, which is only 4 years away. > > > > If your partial cross built arch is ever fully implemented and supported > > by Debian it would be a good idea to migrate Wine to using that. > > > > But a realistic solution for Wine in Debian 12 has to be multilib, > > not multiarch. > Yes but current solution used by wine should be compatible with > multiarch. And It was since 2012 that multiarch exists. Maybe time to > speed up I do consider it quite evil how you are trying to steer upstream and Debian maintainers of Wine in a direction where Wine would be taken hostage for some lofty idea you have. > bastien cu Adrian
Re: Wine MinGW system libraries
Le dim. 12 sept. 2021 à 14:16, Adrian Bunk a écrit : > > On Sun, Sep 12, 2021 at 01:57:44PM +, Bastien ROUCARIES wrote: > > > > I think you misunderstand: > > https://wiki.debian.org/Teams/Dpkg/Spec/FreestandingArches > > > > They are a full color gradiant between: > > - freestanding arches pure cross compile without any depends except arch:all > > - partial cross built arch > > - partial arch > > - full arch > > > > I believe the first step to get partial cross built arch is to begin > > by freestanding arch. > > I believe this is mostly irrelevant for the discussion about Wine. > > Making Wine in Debian depend on such lofty ideas that might be available > very far in the future (if ever) would imply no Wine in the next Debian > releases. > > Debian 12 will be released mid-2023, which is only 2 years away. > Debian 13 will be released mid-2025, which is only 4 years away. > > If your partial cross built arch is ever fully implemented and supported > by Debian it would be a good idea to migrate Wine to using that. > > But a realistic solution for Wine in Debian 12 has to be multilib, > not multiarch. Yes but current solution used by wine should be compatible with multiarch. And It was since 2012 that multiarch exists. Maybe time to speed up bastien > > > Bastien > > cu > Adrian
Re: Wine MinGW system libraries
Le sam. 11 sept. 2021 à 06:00, Paul Wise a écrit :
>
> Disclaimer: I know precisely zero of the details here nor
> if the PE loader can support any of the below features.
>
> On Fri, 2021-09-10 at 09:23 +, Bastien ROUCARIES wrote:
>
> > The problem is that windows apps particularly games try to check if
> > mapped ram exec pages are from dll from disk and not modified in
> > memory.
>
> The fake dlopen could presumably be provided by an on-disk DLL?
> The PE loader would just load the fake dlopen DLL first, like
> glibc does when you set the LD_PRELOAD environment variable.
>
> > You propose:
>
> I'm not proposing any on-disk symlinks, renaming DLLs or any other
> changes to any PE DLLs, just loading into memory like dlmopen.
>
> The solution I'm thinking of looks something like this:
>
> src:libpng -> libpng16-16:$winarch -> /usr/lib/$wintriplet/libpng16.dll
> (requires Debian to support cross-only, partial and Windows arch types)
>
> PE loader sets up in-memory Wine namespace.
>
> PE loader loads into Wine namespace:
> /usr/lib/$windowstriplet/wine/fake-dlopen.dll
>
> PE loader loads Wine dependencies into Wine namespace.
>
> Some of the Wine dependencies call the Windows equivalent of dlopen,
> which resolves to the fake dlopen equivalent within the namespace,
> which calls the Windows equivalent of dlmopen("Wine", dll), which makes
> the PE loader load those libraries into the Wine namespace.
>
> PE loader finishes loading Wine dependencies and proceeds loading the
> PE file and its dependencies into the default namespace.
Seems that python do what you describe:
https://svn.python.org/projects/python/trunk/PC/dl_nt.c
Windows already implemented the name space called activation context.
Need only windows XP or better but wine could offer the API
> Windows apps check mapped RAM pages and they show up correctly.
> The main issue would be that apps could check they run in Wine.
> OTOH the PE loader could also just hide the pages somehow?
>
> --
> bye,
> pabs
>
> https://wiki.debian.org/PaulWise
Re: Wine MinGW system libraries
On Sun, Sep 12, 2021 at 01:57:44PM +, Bastien ROUCARIES wrote: > > I think you misunderstand: > https://wiki.debian.org/Teams/Dpkg/Spec/FreestandingArches > > They are a full color gradiant between: > - freestanding arches pure cross compile without any depends except arch:all > - partial cross built arch > - partial arch > - full arch > > I believe the first step to get partial cross built arch is to begin > by freestanding arch. I believe this is mostly irrelevant for the discussion about Wine. Making Wine in Debian depend on such lofty ideas that might be available very far in the future (if ever) would imply no Wine in the next Debian releases. Debian 12 will be released mid-2023, which is only 2 years away. Debian 13 will be released mid-2025, which is only 4 years away. If your partial cross built arch is ever fully implemented and supported by Debian it would be a good idea to migrate Wine to using that. But a realistic solution for Wine in Debian 12 has to be multilib, not multiarch. > Bastien cu Adrian
Re: Wine MinGW system libraries
Le dim. 12 sept. 2021 à 13:44, Adrian Bunk a écrit : > > On Sun, Sep 12, 2021 at 01:18:11PM +, Bastien ROUCARIES wrote: > > Le dim. 12 sept. 2021 à 07:38, Adrian Bunk a écrit : > > > > > > On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote: > > > >... > > > > Improve dpkg to support partial arch. I volonteer to implement none arch > > > > but i am waiting from guillem here. > > > >... > > > > > > There is also plenty of infrastructure on the buildd, archive and > > > release team sides that would likely need changes for supporting > > > anything like that. > > > > > > A multilib based approach might be more realistic for bookworm. > > > > > > What benefits would a "none arch" even bring compared to building > > > binary-all packages? > > > The ability to binNMU is the only one that comes into my mind. > > we have at least 3 architectures that need uefi none arch... > > > > So we build three arch-all package with paterning name. > > I might be misunderstanding what you are trying to do. > > What release architecture would "none" packages be in the Debian > archive, and on what buildds would they be built? > > Debian buildds are not cross-compiling packages for a different Debian > architecture, a partial architecture would still require DSA-maintained > buildds doing native building on this partial architecture. I think you misunderstand: https://wiki.debian.org/Teams/Dpkg/Spec/FreestandingArches They are a full color gradiant between: - freestanding arches pure cross compile without any depends except arch:all - partial cross built arch - partial arch - full arch I believe the first step to get partial cross built arch is to begin by freestanding arch. Bastien > cu > Adrian
Re: Wine MinGW system libraries
On Sun, Sep 12, 2021 at 01:18:11PM +, Bastien ROUCARIES wrote: > Le dim. 12 sept. 2021 à 07:38, Adrian Bunk a écrit : > > > > On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote: > > >... > > > Improve dpkg to support partial arch. I volonteer to implement none arch > > > but i am waiting from guillem here. > > >... > > > > There is also plenty of infrastructure on the buildd, archive and > > release team sides that would likely need changes for supporting > > anything like that. > > > > A multilib based approach might be more realistic for bookworm. > > > > What benefits would a "none arch" even bring compared to building > > binary-all packages? > > The ability to binNMU is the only one that comes into my mind. > we have at least 3 architectures that need uefi none arch... > > So we build three arch-all package with paterning name. I might be misunderstanding what you are trying to do. What release architecture would "none" packages be in the Debian archive, and on what buildds would they be built? Debian buildds are not cross-compiling packages for a different Debian architecture, a partial architecture would still require DSA-maintained buildds doing native building on this partial architecture. cu Adrian
Re: Wine MinGW system libraries
Le dim. 12 sept. 2021 à 07:38, Adrian Bunk a écrit : > > On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote: > >... > > Improve dpkg to support partial arch. I volonteer to implement none arch > > but i am waiting from guillem here. > >... > > There is also plenty of infrastructure on the buildd, archive and > release team sides that would likely need changes for supporting > anything like that. > > A multilib based approach might be more realistic for bookworm. > > What benefits would a "none arch" even bring compared to building > binary-all packages? > The ability to binNMU is the only one that comes into my mind. we have at least 3 architectures that need uefi none arch... So we build three arch-all package with paterning name. > > cu > Adrian
Re: Wine MinGW system libraries
On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote: >... > Improve dpkg to support partial arch. I volonteer to implement none arch > but i am waiting from guillem here. >... There is also plenty of infrastructure on the buildd, archive and release team sides that would likely need changes for supporting anything like that. A multilib based approach might be more realistic for bookworm. What benefits would a "none arch" even bring compared to building binary-all packages? The ability to binNMU is the only one that comes into my mind. cu Adrian
Re: Wine MinGW system libraries
Le sam. 11 sept. 2021 à 17:30, Adrian Bunk a écrit : > On Sat, Sep 04, 2021 at 08:17:53PM -0500, Zebediah Figura wrote: > > Hello all, > > > > I'm a contributor to the Wine project. To summarize the following mail, > Wine > > needs special versions of some of its normal dependencies, such as > > libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > > sending out a mail to major distributions in order to get some feedback > from > > our packagers on how these should be built and packaged. > > > > For a long time Wine has built all of its Win32 libraries (DLLs and > EXEs) as > > ELF binaries. For various reasons related to application compatibility, > we > > have started building our binaries as PE instead, using the MinGW > > cross-compiler. It is our intent to expand this to some of our > dependencies > > as well. The list of dependencies that we intend to build using MinGW is > not > > quite fixed yet, but we expect it to include and be mostly limited to the > > following: > > > > * libvkd3d > > * libFAudio > > * libgnutls > > * zlib (currently included via manual source import) > > * libmpg123 > > * libgsm > > * libpng > > * libjpeg-turbo > > * libtiff > > * libfreetype > > * liblcms2 > > * jxrlib > > > > and dependencies of the above packages (not including CRT dependencies, > > which Wine provides). > >... > > Accordingly, although static linking and source imports are generally > > disprefered, it may quite likely be preferable in our case. We don't get > the > > benefits of on-disk deduplication, since Wine is essentially the only > piece > > of software which needs these libraries. > >... > > How are distributions supposed to provide security support for Wine? > > As an example, Debian 10 that is security supported by Debian > until next summer ships Wine 4.0. > > The libgnutls in Debian 10 has already been patched several times > by Debian for CVE fixes. > > Having to patch several different versions of the same library > in different packages multiplies the effort required to provide > security support for a library. > Yes what why Paul and myself ask for partial arch. No static linking no embeded lib but we need answer from upstream. Note that windows application that use old library are not a concern from the debian si de Bastien > > cu > Adrian > >
Re: Wine MinGW system libraries
On Sat, Sep 04, 2021 at 08:17:53PM -0500, Zebediah Figura wrote: > Hello all, > > I'm a contributor to the Wine project. To summarize the following mail, Wine > needs special versions of some of its normal dependencies, such as > libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > sending out a mail to major distributions in order to get some feedback from > our packagers on how these should be built and packaged. > > For a long time Wine has built all of its Win32 libraries (DLLs and EXEs) as > ELF binaries. For various reasons related to application compatibility, we > have started building our binaries as PE instead, using the MinGW > cross-compiler. It is our intent to expand this to some of our dependencies > as well. The list of dependencies that we intend to build using MinGW is not > quite fixed yet, but we expect it to include and be mostly limited to the > following: > > * libvkd3d > * libFAudio > * libgnutls > * zlib (currently included via manual source import) > * libmpg123 > * libgsm > * libpng > * libjpeg-turbo > * libtiff > * libfreetype > * liblcms2 > * jxrlib > > and dependencies of the above packages (not including CRT dependencies, > which Wine provides). >... > Accordingly, although static linking and source imports are generally > disprefered, it may quite likely be preferable in our case. We don't get the > benefits of on-disk deduplication, since Wine is essentially the only piece > of software which needs these libraries. >... How are distributions supposed to provide security support for Wine? As an example, Debian 10 that is security supported by Debian until next summer ships Wine 4.0. The libgnutls in Debian 10 has already been patched several times by Debian for CVE fixes. Having to patch several different versions of the same library in different packages multiplies the effort required to provide security support for a library. cu Adrian
Re: Wine MinGW system libraries
Disclaimer: I know precisely zero of the details here nor
if the PE loader can support any of the below features.
On Fri, 2021-09-10 at 09:23 +, Bastien ROUCARIES wrote:
> The problem is that windows apps particularly games try to check if
> mapped ram exec pages are from dll from disk and not modified in
> memory.
The fake dlopen could presumably be provided by an on-disk DLL?
The PE loader would just load the fake dlopen DLL first, like
glibc does when you set the LD_PRELOAD environment variable.
> You propose:
I'm not proposing any on-disk symlinks, renaming DLLs or any other
changes to any PE DLLs, just loading into memory like dlmopen.
The solution I'm thinking of looks something like this:
src:libpng -> libpng16-16:$winarch -> /usr/lib/$wintriplet/libpng16.dll
(requires Debian to support cross-only, partial and Windows arch types)
PE loader sets up in-memory Wine namespace.
PE loader loads into Wine namespace:
/usr/lib/$windowstriplet/wine/fake-dlopen.dll
PE loader loads Wine dependencies into Wine namespace.
Some of the Wine dependencies call the Windows equivalent of dlopen,
which resolves to the fake dlopen equivalent within the namespace,
which calls the Windows equivalent of dlmopen("Wine", dll), which makes
the PE loader load those libraries into the Wine namespace.
PE loader finishes loading Wine dependencies and proceeds loading the
PE file and its dependencies into the default namespace.
Windows apps check mapped RAM pages and they show up correctly.
The main issue would be that apps could check they run in Wine.
OTOH the PE loader could also just hide the pages somehow?
--
bye,
pabs
https://wiki.debian.org/PaulWise
signature.asc
Description: This is a digitally signed message part
Re: Wine MinGW system libraries
Le jeu. 9 sept. 2021 à 07:32, Paul Wise a écrit : > > On Thu, 2021-09-09 at 00:59 -0500, Zebediah Figura wrote: > > > Unfortunately, no. We have no way of knowing the caller. > > Can the PE loading mechanism do something like inject a fake dlopen > function available only in the Wine namespace that just passes the Wine > namespace to the dlmopen function? Or the same but for each library? The problem is that windows apps particularly games try to check if mapped ram exec pages are from dll from disk and not modified in memory. Thus if we create namespace, this crappy software will stop to work except if we create or symlink a library with an unique name that the wine applications could fetch Note that windows use for libraryname only the basename not the fullpatch (zebediah could you confirm) I am proposing (for simplicity I call the problematic lib libpng that depends on libz) 1. build like usual a libpng and libz. libpng depend on linbz 2. copy libpng to libwinepng and libz to libwinez 3 use a new patchpe command (like the patchelf command) to patch the depend of libwinepng So globally a post link time namespace (it will be better on debian side than to recompile and pass flags to library). You propose: 1. build like usual a libpng and libz. libpng 2 create symlink libwinepng -> libpng, liwinez->libz 3. instruct the loader to load libwinepng privatly using namespace approach and do at run time the patchpe work I believe that your approach is technically superior but could be done on the top of my first approach and a second step. Zebediah could you please give use feedback from wine team ? If patchpe is workable (and I firmly believe it is), you could add a task to see if paul wise solution is workable and report here. Next step, will be to refine your proposal. For instance i really dislike the libpng libwinepng mapping I will prefer a mapping given as perlre and something that encode the wine ABI like s/^lib/lib-wineinternaluseABI1.0/g It will allow to create pkgconfig file for wine version of lib automatically Bastien > > -- > bye, > pabs > > https://wiki.debian.org/PaulWise
Re: Wine MinGW system libraries
On Thu, 2021-09-09 at 00:59 -0500, Zebediah Figura wrote: > Unfortunately, no. We have no way of knowing the caller. Can the PE loading mechanism do something like inject a fake dlopen function available only in the Wine namespace that just passes the Wine namespace to the dlmopen function? Or the same but for each library? -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Wine MinGW system libraries
On 9/9/21 12:45 AM, Paul Wise wrote: On Thu, 2021-09-09 at 00:39 -0500, Zebediah Figura wrote: Right, but we don't have any guarantee that library A will load library B in its constructor routines. In fact, if it's loading library B dynamically, it's probably not doing that. Can the loader tell which library asked it to load a library? Then the namespace could be inherited from the parent library. Unfortunately, no. We have no way of knowing the caller.
Re: Wine MinGW system libraries
On Thu, 2021-09-09 at 00:39 -0500, Zebediah Figura wrote: > Right, but we don't have any guarantee that library A will load library > B in its constructor routines. In fact, if it's loading library B > dynamically, it's probably not doing that. Can the loader tell which library asked it to load a library? Then the namespace could be inherited from the parent library. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Wine MinGW system libraries
On 9/9/21 12:15 AM, Paul Wise wrote: On Wed, 2021-09-08 at 23:47 -0500, Zebediah Figura wrote: Unfortunately, while thinking about the answer to this question, I realized another snag, which I think really does make using identically-named dynamic libraries impossible: if system library A loads system library B dynamically, i.e. does the equivalent of a dlopen(), we have no way of knowing which namespace to apply to it. [AFAICT dlmopen() has this problem too, and just isn't meant for that use case.] Even rewriting the import table doesn't help here. While loading library A into a namespace Foo, the loader could set a process global variable containing the current namespace to load into, and then have the dlopen equivalent use namespace Foo in preference to the default namespace for the process. Then when library B is loaded it will go into namespace Foo. Once loading library B and A has completed the loader could revert to using the default namespace. Right, but we don't have any guarantee that library A will load library B in its constructor routines. In fact, if it's loading library B dynamically, it's probably not doing that.
Re: Wine MinGW system libraries
On Wed, 2021-09-08 at 23:47 -0500, Zebediah Figura wrote: > Unfortunately, while thinking about the answer to this question, I > realized another snag, which I think really does make using > identically-named dynamic libraries impossible: if system library A > loads system library B dynamically, i.e. does the equivalent of a > dlopen(), we have no way of knowing which namespace to apply to it. > [AFAICT dlmopen() has this problem too, and just isn't meant for that > use case.] Even rewriting the import table doesn't help here. While loading library A into a namespace Foo, the loader could set a process global variable containing the current namespace to load into, and then have the dlopen equivalent use namespace Foo in preference to the default namespace for the process. Then when library B is loaded it will go into namespace Foo. Once loading library B and A has completed the loader could revert to using the default namespace. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Wine MinGW system libraries
On 9/8/21 8:12 PM, Paul Wise wrote: On 9/8/21 3:13 AM, Simon McVittie wrote: As far as I understand it, the PE loader used for Wine is part of Wine, so it has total control over the libraries that it loads and how it loads them. This means that if Wine developers (the experts on this codebase) have decided a libcapsule-like approach isn't feasible or isn't desirable, they are probably right. Could the PE loader add support for some sort of namespaces and then be told to load Wine dependencies within a Wine namespace? Does the Microsoft Windows loader support anything like this? Recompiling every PE library with special flags or renaming every PE library with a Wine prefix seems a bit tedious if just the PE loader and anything that requests Wine deps can use a namespace/capsule/etc. The closest thing that Windows has is SxS, which is meant to solve I think some of the same problems (well, it's mainly meant to solve the versioning problem that Microsoft dug themselves into). The problem is that I believe it requires the namespace to be effectively hard-coded into the library in question, in the form of a manifest resource. Unfortunately, while thinking about the answer to this question, I realized another snag, which I think really does make using identically-named dynamic libraries impossible: if system library A loads system library B dynamically, i.e. does the equivalent of a dlopen(), we have no way of knowing which namespace to apply to it. [AFAICT dlmopen() has this problem too, and just isn't meant for that use case.] Even rewriting the import table doesn't help here.
Re: Wine MinGW system libraries
On 9/8/21 3:13 AM, Simon McVittie wrote: > As far as I understand it, the PE loader used for Wine is part of Wine, > so it has total control over the libraries that it loads and how it loads > them. This means that if Wine developers (the experts on this codebase) > have decided a libcapsule-like approach isn't feasible or isn't desirable, > they are probably right. Could the PE loader add support for some sort of namespaces and then be told to load Wine dependencies within a Wine namespace? Does the Microsoft Windows loader support anything like this? Recompiling every PE library with special flags or renaming every PE library with a Wine prefix seems a bit tedious if just the PE loader and anything that requests Wine deps can use a namespace/capsule/etc. On Wed, 2021-09-08 at 10:42 -0500, Zebediah Figura wrote: > The basic reason is that Wine is essentially trying to split itself into > "PE" and "Unix" parts. The idea is that all of the "user" code is built > in PE format, and only calls into unix code by making a sort of fake > syscall. The "kernel" code is built in ELF format. > > There are a few basic reasons for doing this: Definitely seems like the right thing to do for Wine. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Wine MinGW system libraries
On 9/8/21 3:13 AM, Simon McVittie wrote: On Wed, 08 Sep 2021 at 07:31:59 +, Bastien ROUCARIES wrote: Simon, do you think you could implement a version of libcapasule for PE object ? Given that libcapsule is very glibc- and ELF-specific, doesn't work properly without new glibc feature work that isn't in Debian yet (I've lost track of whether/when it got merged upstream, it might be in 2.34) and wasn't written by me, the answer is likely to be no. As far as I understand it, the PE loader used for Wine is part of Wine, so it has total control over the libraries that it loads and how it loads them. This means that if Wine developers (the experts on this codebase) have decided a libcapsule-like approach isn't feasible or isn't desirable, they are probably right. More generally, if Wine developers wanted Wine's dependency libraries to be in one version of reality, and Windows DLLs to be in a different version of reality, then there's a straightforward way to do that: use ELF .so dependency libraries, like older versions of Wine did, and don't give them a representation in the PE layer. Presumably they have put in the work to move from ELF to PE dependency libraries because it has some compelling advantage, rather than creating work for themselves for no reason! Yeah, basically :-) The basic reason is that Wine is essentially trying to split itself into "PE" and "Unix" parts. The idea is that all of the "user" code is built in PE format, and only calls into unix code by making a sort of fake syscall. The "kernel" code is built in ELF format. There are a few basic reasons for doing this: * Many applications, mostly anti-cheat and anti-tamper engines, depend on our libraries actually being in PE format on disk, and matching that in memory. We used to have "fake" DLLs for this, but they weren't verisimilar enough. * Some of the important hosts for Wine have dropped support for 32-bit libraries or are going to drop it. Mac is the obvious example here, but many commercial Linux distributions also want to drop support. By limiting the surface through which code can transition between PE and Unix code, it becomes feasible to do 32-to-64 translation, where previously this was quite infeasible. * There's some demand for running Win32 debuggers under wine. These have always worked more than a little tenuously, but a couple of them are causing problems when they try to break in and unwind from Unix code. By acting like we're making actual syscalls, things work much better. Of the several dozen modules we have that currently call into Unix libraries, about half of them aren't going to use PE dependencies. Most of these have already been fully split into PE and Unix parts, including things like the socket layer and winegstreamer. This raises the question: why not just call into Unix for the other half as well? There are a few reasons: * Making fake syscalls is, like real syscalls, not cheap. It involves a full context switch into and out of Unix code. * Making callbacks from Unix code is difficult, and not cheap either. For some libraries this doesn't matter, but for others it really does. * Writing the wrappers between Unix and PE code involves some nontrivial work in itself. The smaller the interface is, the easier things are for us.
Re: Wine MinGW system libraries
On Wed, 08 Sep 2021 at 07:31:59 +, Bastien ROUCARIES wrote: > Simon, do you think you could implement a version of libcapasule for PE > object ? Given that libcapsule is very glibc- and ELF-specific, doesn't work properly without new glibc feature work that isn't in Debian yet (I've lost track of whether/when it got merged upstream, it might be in 2.34) and wasn't written by me, the answer is likely to be no. As far as I understand it, the PE loader used for Wine is part of Wine, so it has total control over the libraries that it loads and how it loads them. This means that if Wine developers (the experts on this codebase) have decided a libcapsule-like approach isn't feasible or isn't desirable, they are probably right. More generally, if Wine developers wanted Wine's dependency libraries to be in one version of reality, and Windows DLLs to be in a different version of reality, then there's a straightforward way to do that: use ELF .so dependency libraries, like older versions of Wine did, and don't give them a representation in the PE layer. Presumably they have put in the work to move from ELF to PE dependency libraries because it has some compelling advantage, rather than creating work for themselves for no reason! smcv
Re: Wine MinGW system libraries
Hi, Adding smcv to the thread. Le mar. 7 sept. 2021 à 23:25, Bastien ROUCARIES a écrit : > > Le mar. 7 sept. 2021 à 21:16, Zebediah Figura > a écrit : > > > > On 9/7/21 12:05 PM, Bastien ROUCARIES wrote: > > > I disagree. > > > > > > Le mar. 7 sept. 2021 à 17:48, Zebediah Figura a > > > écrit : > > > > > >> On 9/7/21 5:16 AM, Bastien Roucariès wrote: > > >>> Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : > > On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > > > The basic problem is that applications can and often do ship with PE > > > builds of cross-platform libraries. These libraries can be ahead of > > > Wine's system libraries, behind them, or even built with custom > > >> patches. > > > Accordingly we really don't want to load "our" freetype in place of > > > "their" freetype, or "theirs" in place of "ours". But because of the > > >> way > > > the Win32 loader works, you can generally only have one DLL of a given > > > name loaded in a process, and further attempts to dlopen() [as it > > > were] > > > "libfreetype-6.dll" will return the handle to the already loaded > > > library, potentially breaking either Wine or the application. > > > > I don't know the details here, but my immediate thought when reading > > this is that you need some kind of namespace. I then found that linker > > namespaces are a thing, perhaps they would provide the solution for > > you. It sounds like the OpenGL shim loader solution listed on the > > glibc wiki might work for your use-case. Or perhaps the LD_AUDIT > > feature would also work. > > > > https://www.sourceware.org/glibc/wiki/LinkerNamespaces > > >>> I agree with pabs, implementing name space is a good solution > > >>> and will allow to be distrib friendly. > > >>> > > >>> Moreover it will be best if marking a library as wine system one could > > >> be done > > >>> post build. We will prefer to not modify upstream (like libpng) source. > > >>> > > >>> It is easier for us to apply small patch to upstream, or better in your > > >> case > > >>> to add a post link linker script or even some compiler flag that will > > >> add some > > >>> notes section to dll in order to mark it as use namespace system, or so > > >> on. > > >>> > > >>> The problem on windows side is well known as dll hell > > >>> > > >>> Renaming of the lib could be done post install. Moreover a crazy idea > > >> why not > > >>> renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll > > >> (note the > > >>> version is manadaroty) Therefore we could use upstream source. Then add > > >> a > > >>> small linker script that will rewrite the depends of libpng-2.0.0.sll > > >> to > > >>> .sll file (aka binary sed s/.dll/.sll/g). > > >>> > > >>> then add a small wraper (shim) like said pab that load the .sll library, > > >> the > > >>> best pratice will be a command tool that take the name of the sll and > > >> create > > >>> magically the dll > > >>> > > >>> Therefore from distrib point of side we only changes the way we build > > >>> the > > >>> package, it could be automated, it is scalable and it does not need to > > >> patch > > >>> package by package. Only to add some linker script magic > > >> > > >> The problem isn't particularly about detecting what is and isn't a > > >> system library; I think we can come up with a reliable heuristic for > > >> that, without even needing linker namespaces or anything. > > >> > > >> The outstanding problem seems to be more about potentially breaking > > >> applications because they see two identically named DLLs loaded in the > > >> same process. Applications can and do trawl the internal loader state, > > >> although the Win32 loader also exposes some APIs so they don't even have > > >> to. > > >> > > >> It may also be that the aforementioned heuristic is too hacky to be > > >> accepted upstream into Wine. > > >> > > > I do not propose to change the loader, i propose to change the link or > > > post > > > link stage of system library. You could rename the lib and change the > > > dépend to new name after build. So you will have an unique name per system > > > lib without changing the way you build these lib. > > > > > > It is équivalent to use a namespace resolved at late build time. > > > > > > > Ah, so what you're proposing is that we do some sort of objcopy-like > > patching at runtime, e.g. when copying the dependency into the prefix. > > > > It probably won't be easy, but it might be more feasible than modifying > > the loader. I'll look into it... > Yes sort of obcopy patching under linux it is called patchelf... > > If it work, it could be latter done in memory like paul wise suggest > implementing namespace and LT_AUDIT (but it is a long term goal) > 1. capture library loading > 2. do the patching at loading time (using the same code then patchelf > for pe replacing file read/write by mmap operation) > > And voila Simon, do you
Re: Wine MinGW system libraries
Le mar. 7 sept. 2021 à 21:16, Zebediah Figura a écrit : > > On 9/7/21 12:05 PM, Bastien ROUCARIES wrote: > > I disagree. > > > > Le mar. 7 sept. 2021 à 17:48, Zebediah Figura a > > écrit : > > > >> On 9/7/21 5:16 AM, Bastien Roucariès wrote: > >>> Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : > On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > > The basic problem is that applications can and often do ship with PE > > builds of cross-platform libraries. These libraries can be ahead of > > Wine's system libraries, behind them, or even built with custom > >> patches. > > Accordingly we really don't want to load "our" freetype in place of > > "their" freetype, or "theirs" in place of "ours". But because of the > >> way > > the Win32 loader works, you can generally only have one DLL of a given > > name loaded in a process, and further attempts to dlopen() [as it were] > > "libfreetype-6.dll" will return the handle to the already loaded > > library, potentially breaking either Wine or the application. > > I don't know the details here, but my immediate thought when reading > this is that you need some kind of namespace. I then found that linker > namespaces are a thing, perhaps they would provide the solution for > you. It sounds like the OpenGL shim loader solution listed on the > glibc wiki might work for your use-case. Or perhaps the LD_AUDIT > feature would also work. > > https://www.sourceware.org/glibc/wiki/LinkerNamespaces > >>> I agree with pabs, implementing name space is a good solution > >>> and will allow to be distrib friendly. > >>> > >>> Moreover it will be best if marking a library as wine system one could > >> be done > >>> post build. We will prefer to not modify upstream (like libpng) source. > >>> > >>> It is easier for us to apply small patch to upstream, or better in your > >> case > >>> to add a post link linker script or even some compiler flag that will > >> add some > >>> notes section to dll in order to mark it as use namespace system, or so > >> on. > >>> > >>> The problem on windows side is well known as dll hell > >>> > >>> Renaming of the lib could be done post install. Moreover a crazy idea > >> why not > >>> renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll > >> (note the > >>> version is manadaroty) Therefore we could use upstream source. Then add > >> a > >>> small linker script that will rewrite the depends of libpng-2.0.0.sll > >> to > >>> .sll file (aka binary sed s/.dll/.sll/g). > >>> > >>> then add a small wraper (shim) like said pab that load the .sll library, > >> the > >>> best pratice will be a command tool that take the name of the sll and > >> create > >>> magically the dll > >>> > >>> Therefore from distrib point of side we only changes the way we build the > >>> package, it could be automated, it is scalable and it does not need to > >> patch > >>> package by package. Only to add some linker script magic > >> > >> The problem isn't particularly about detecting what is and isn't a > >> system library; I think we can come up with a reliable heuristic for > >> that, without even needing linker namespaces or anything. > >> > >> The outstanding problem seems to be more about potentially breaking > >> applications because they see two identically named DLLs loaded in the > >> same process. Applications can and do trawl the internal loader state, > >> although the Win32 loader also exposes some APIs so they don't even have > >> to. > >> > >> It may also be that the aforementioned heuristic is too hacky to be > >> accepted upstream into Wine. > >> > > I do not propose to change the loader, i propose to change the link or post > > link stage of system library. You could rename the lib and change the > > dépend to new name after build. So you will have an unique name per system > > lib without changing the way you build these lib. > > > > It is équivalent to use a namespace resolved at late build time. > > > > Ah, so what you're proposing is that we do some sort of objcopy-like > patching at runtime, e.g. when copying the dependency into the prefix. > > It probably won't be easy, but it might be more feasible than modifying > the loader. I'll look into it... Yes sort of obcopy patching under linux it is called patchelf... If it work, it could be latter done in memory like paul wise suggest implementing namespace and LT_AUDIT (but it is a long term goal) 1. capture library loading 2. do the patching at loading time (using the same code then patchelf for pe replacing file read/write by mmap operation) And voila bastien
Re: Wine MinGW system libraries
Le mar. 7 sept. 2021 à 23:01, Paul Wise a écrit : > > On Tue, 2021-09-07 at 10:48 -0500, Zebediah Figura wrote: > > > The outstanding problem seems to be more about potentially breaking > > applications because they see two identically named DLLs loaded in the > > same process. Applications can and do trawl the internal loader state, > > although the Win32 loader also exposes some APIs so they don't even have to. > > I might be wrong but I got the impression that runtime linker > namespaces & LD_AUDIT are designed to solve this exact issue. Yes but it does not work for PE because win native application do messy stuff with lib loader... see dll hell for instance. Spec was wrong for the beginning for PE. And wine does not use the ELF loader but a homemade PE in order to be bug to bug compatible with windows. In the long term implementing a LT_AUDIT in the wine loader will be worthless, but in the short term elfpatch like approach could work. Python use it. Bastien > > -- > bye, > pabs > > https://wiki.debian.org/PaulWise
Re: Wine MinGW system libraries
On Tue, 2021-09-07 at 10:48 -0500, Zebediah Figura wrote: > The outstanding problem seems to be more about potentially breaking > applications because they see two identically named DLLs loaded in the > same process. Applications can and do trawl the internal loader state, > although the Win32 loader also exposes some APIs so they don't even have to. I might be wrong but I got the impression that runtime linker namespaces & LD_AUDIT are designed to solve this exact issue. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Wine MinGW system libraries
On 9/7/21 12:05 PM, Bastien ROUCARIES wrote: I disagree. Le mar. 7 sept. 2021 à 17:48, Zebediah Figura a écrit : On 9/7/21 5:16 AM, Bastien Roucariès wrote: Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: The basic problem is that applications can and often do ship with PE builds of cross-platform libraries. These libraries can be ahead of Wine's system libraries, behind them, or even built with custom patches. Accordingly we really don't want to load "our" freetype in place of "their" freetype, or "theirs" in place of "ours". But because of the way the Win32 loader works, you can generally only have one DLL of a given name loaded in a process, and further attempts to dlopen() [as it were] "libfreetype-6.dll" will return the handle to the already loaded library, potentially breaking either Wine or the application. I don't know the details here, but my immediate thought when reading this is that you need some kind of namespace. I then found that linker namespaces are a thing, perhaps they would provide the solution for you. It sounds like the OpenGL shim loader solution listed on the glibc wiki might work for your use-case. Or perhaps the LD_AUDIT feature would also work. https://www.sourceware.org/glibc/wiki/LinkerNamespaces I agree with pabs, implementing name space is a good solution and will allow to be distrib friendly. Moreover it will be best if marking a library as wine system one could be done post build. We will prefer to not modify upstream (like libpng) source. It is easier for us to apply small patch to upstream, or better in your case to add a post link linker script or even some compiler flag that will add some notes section to dll in order to mark it as use namespace system, or so on. The problem on windows side is well known as dll hell Renaming of the lib could be done post install. Moreover a crazy idea why not renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll (note the version is manadaroty) Therefore we could use upstream source. Then add a small linker script that will rewrite the depends of libpng-2.0.0.sll to .sll file (aka binary sed s/.dll/.sll/g). then add a small wraper (shim) like said pab that load the .sll library, the best pratice will be a command tool that take the name of the sll and create magically the dll Therefore from distrib point of side we only changes the way we build the package, it could be automated, it is scalable and it does not need to patch package by package. Only to add some linker script magic The problem isn't particularly about detecting what is and isn't a system library; I think we can come up with a reliable heuristic for that, without even needing linker namespaces or anything. The outstanding problem seems to be more about potentially breaking applications because they see two identically named DLLs loaded in the same process. Applications can and do trawl the internal loader state, although the Win32 loader also exposes some APIs so they don't even have to. It may also be that the aforementioned heuristic is too hacky to be accepted upstream into Wine. I do not propose to change the loader, i propose to change the link or post link stage of system library. You could rename the lib and change the dépend to new name after build. So you will have an unique name per system lib without changing the way you build these lib. It is équivalent to use a namespace resolved at late build time. Ah, so what you're proposing is that we do some sort of objcopy-like patching at runtime, e.g. when copying the dependency into the prefix. It probably won't be easy, but it might be more feasible than modifying the loader. I'll look into it...
Re: Wine MinGW system libraries
Le mar. 7 sept. 2021 à 19:05, Bastien ROUCARIES a écrit : > I disagree. > > Le mar. 7 sept. 2021 à 17:48, Zebediah Figura a > écrit : > >> On 9/7/21 5:16 AM, Bastien Roucariès wrote: >> > Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : >> >> On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: >> >>> The basic problem is that applications can and often do ship with PE >> >>> builds of cross-platform libraries. These libraries can be ahead of >> >>> Wine's system libraries, behind them, or even built with custom >> patches. >> >>> Accordingly we really don't want to load "our" freetype in place of >> >>> "their" freetype, or "theirs" in place of "ours". But because of the >> way >> >>> the Win32 loader works, you can generally only have one DLL of a given >> >>> name loaded in a process, and further attempts to dlopen() [as it >> were] >> >>> "libfreetype-6.dll" will return the handle to the already loaded >> >>> library, potentially breaking either Wine or the application. >> >> >> >> I don't know the details here, but my immediate thought when reading >> >> this is that you need some kind of namespace. I then found that linker >> >> namespaces are a thing, perhaps they would provide the solution for >> >> you. It sounds like the OpenGL shim loader solution listed on the >> >> glibc wiki might work for your use-case. Or perhaps the LD_AUDIT >> >> feature would also work. >> >> >> >> https://www.sourceware.org/glibc/wiki/LinkerNamespaces >> > I agree with pabs, implementing name space is a good solution >> > and will allow to be distrib friendly. >> > >> > Moreover it will be best if marking a library as wine system one could >> be done >> > post build. We will prefer to not modify upstream (like libpng) source. >> > >> > It is easier for us to apply small patch to upstream, or better in your >> case >> > to add a post link linker script or even some compiler flag that will >> add some >> > notes section to dll in order to mark it as use namespace system, or so >> on. >> > >> > The problem on windows side is well known as dll hell >> > >> > Renaming of the lib could be done post install. Moreover a crazy idea >> why not >> > renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll >> (note the >> > version is manadaroty) Therefore we could use upstream source. Then >> add a >> > small linker script that will rewrite the depends of libpng-2.0.0.sll >> to >> > .sll file (aka binary sed s/.dll/.sll/g). >> > >> > then add a small wraper (shim) like said pab that load the .sll >> library, the >> > best pratice will be a command tool that take the name of the sll and >> create >> > magically the dll >> > >> > Therefore from distrib point of side we only changes the way we build >> the >> > package, it could be automated, it is scalable and it does not need to >> patch >> > package by package. Only to add some linker script magic >> >> The problem isn't particularly about detecting what is and isn't a >> system library; I think we can come up with a reliable heuristic for >> that, without even needing linker namespaces or anything. >> >> The outstanding problem seems to be more about potentially breaking >> applications because they see two identically named DLLs loaded in the >> same process. Applications can and do trawl the internal loader state, >> although the Win32 loader also exposes some APIs so they don't even have >> to. >> >> It may also be that the aforementioned heuristic is too hacky to be >> accepted upstream into Wine. >> > I do not propose to change the loader, i propose to change the link or > post link stage of system library. You could rename the lib and change the > dépend to new name after build. So you will have an unique name per system > lib without changing the way you build these lib. > > It is équivalent to use a namespace resolved at late build time. > And it is already coded :) https://pypi.org/project/machomachomangler/ > >
Re: Wine MinGW system libraries
I disagree. Le mar. 7 sept. 2021 à 17:48, Zebediah Figura a écrit : > On 9/7/21 5:16 AM, Bastien Roucariès wrote: > > Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : > >> On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > >>> The basic problem is that applications can and often do ship with PE > >>> builds of cross-platform libraries. These libraries can be ahead of > >>> Wine's system libraries, behind them, or even built with custom > patches. > >>> Accordingly we really don't want to load "our" freetype in place of > >>> "their" freetype, or "theirs" in place of "ours". But because of the > way > >>> the Win32 loader works, you can generally only have one DLL of a given > >>> name loaded in a process, and further attempts to dlopen() [as it were] > >>> "libfreetype-6.dll" will return the handle to the already loaded > >>> library, potentially breaking either Wine or the application. > >> > >> I don't know the details here, but my immediate thought when reading > >> this is that you need some kind of namespace. I then found that linker > >> namespaces are a thing, perhaps they would provide the solution for > >> you. It sounds like the OpenGL shim loader solution listed on the > >> glibc wiki might work for your use-case. Or perhaps the LD_AUDIT > >> feature would also work. > >> > >> https://www.sourceware.org/glibc/wiki/LinkerNamespaces > > I agree with pabs, implementing name space is a good solution > > and will allow to be distrib friendly. > > > > Moreover it will be best if marking a library as wine system one could > be done > > post build. We will prefer to not modify upstream (like libpng) source. > > > > It is easier for us to apply small patch to upstream, or better in your > case > > to add a post link linker script or even some compiler flag that will > add some > > notes section to dll in order to mark it as use namespace system, or so > on. > > > > The problem on windows side is well known as dll hell > > > > Renaming of the lib could be done post install. Moreover a crazy idea > why not > > renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll > (note the > > version is manadaroty) Therefore we could use upstream source. Then add > a > > small linker script that will rewrite the depends of libpng-2.0.0.sll > to > > .sll file (aka binary sed s/.dll/.sll/g). > > > > then add a small wraper (shim) like said pab that load the .sll library, > the > > best pratice will be a command tool that take the name of the sll and > create > > magically the dll > > > > Therefore from distrib point of side we only changes the way we build the > > package, it could be automated, it is scalable and it does not need to > patch > > package by package. Only to add some linker script magic > > The problem isn't particularly about detecting what is and isn't a > system library; I think we can come up with a reliable heuristic for > that, without even needing linker namespaces or anything. > > The outstanding problem seems to be more about potentially breaking > applications because they see two identically named DLLs loaded in the > same process. Applications can and do trawl the internal loader state, > although the Win32 loader also exposes some APIs so they don't even have > to. > > It may also be that the aforementioned heuristic is too hacky to be > accepted upstream into Wine. > I do not propose to change the loader, i propose to change the link or post link stage of system library. You could rename the lib and change the dépend to new name after build. So you will have an unique name per system lib without changing the way you build these lib. It is équivalent to use a namespace resolved at late build time.
Re: Wine MinGW system libraries
On 9/7/21 5:16 AM, Bastien Roucariès wrote: Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: The basic problem is that applications can and often do ship with PE builds of cross-platform libraries. These libraries can be ahead of Wine's system libraries, behind them, or even built with custom patches. Accordingly we really don't want to load "our" freetype in place of "their" freetype, or "theirs" in place of "ours". But because of the way the Win32 loader works, you can generally only have one DLL of a given name loaded in a process, and further attempts to dlopen() [as it were] "libfreetype-6.dll" will return the handle to the already loaded library, potentially breaking either Wine or the application. I don't know the details here, but my immediate thought when reading this is that you need some kind of namespace. I then found that linker namespaces are a thing, perhaps they would provide the solution for you. It sounds like the OpenGL shim loader solution listed on the glibc wiki might work for your use-case. Or perhaps the LD_AUDIT feature would also work. https://www.sourceware.org/glibc/wiki/LinkerNamespaces I agree with pabs, implementing name space is a good solution and will allow to be distrib friendly. Moreover it will be best if marking a library as wine system one could be done post build. We will prefer to not modify upstream (like libpng) source. It is easier for us to apply small patch to upstream, or better in your case to add a post link linker script or even some compiler flag that will add some notes section to dll in order to mark it as use namespace system, or so on. The problem on windows side is well known as dll hell Renaming of the lib could be done post install. Moreover a crazy idea why not renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll (note the version is manadaroty) Therefore we could use upstream source. Then add a small linker script that will rewrite the depends of libpng-2.0.0.sll to .sll file (aka binary sed s/.dll/.sll/g). then add a small wraper (shim) like said pab that load the .sll library, the best pratice will be a command tool that take the name of the sll and create magically the dll Therefore from distrib point of side we only changes the way we build the package, it could be automated, it is scalable and it does not need to patch package by package. Only to add some linker script magic The problem isn't particularly about detecting what is and isn't a system library; I think we can come up with a reliable heuristic for that, without even needing linker namespaces or anything. The outstanding problem seems to be more about potentially breaking applications because they see two identically named DLLs loaded in the same process. Applications can and do trawl the internal loader state, although the Win32 loader also exposes some APIs so they don't even have to. It may also be that the aforementioned heuristic is too hacky to be accepted upstream into Wine.
Re: Wine MinGW system libraries
Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : > On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > > The basic problem is that applications can and often do ship with PE > > builds of cross-platform libraries. These libraries can be ahead of > > Wine's system libraries, behind them, or even built with custom patches. > > Accordingly we really don't want to load "our" freetype in place of > > "their" freetype, or "theirs" in place of "ours". But because of the way > > the Win32 loader works, you can generally only have one DLL of a given > > name loaded in a process, and further attempts to dlopen() [as it were] > > "libfreetype-6.dll" will return the handle to the already loaded > > library, potentially breaking either Wine or the application. > > I don't know the details here, but my immediate thought when reading > this is that you need some kind of namespace. I then found that linker > namespaces are a thing, perhaps they would provide the solution for > you. It sounds like the OpenGL shim loader solution listed on the > glibc wiki might work for your use-case. Or perhaps the LD_AUDIT > feature would also work. > > https://www.sourceware.org/glibc/wiki/LinkerNamespaces I agree with pabs, implementing name space is a good solution and will allow to be distrib friendly. Moreover it will be best if marking a library as wine system one could be done post build. We will prefer to not modify upstream (like libpng) source. It is easier for us to apply small patch to upstream, or better in your case to add a post link linker script or even some compiler flag that will add some notes section to dll in order to mark it as use namespace system, or so on. The problem on windows side is well known as dll hell Renaming of the lib could be done post install. Moreover a crazy idea why not renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll (note the version is manadaroty) Therefore we could use upstream source. Then add a small linker script that will rewrite the depends of libpng-2.0.0.sll to .sll file (aka binary sed s/.dll/.sll/g). then add a small wraper (shim) like said pab that load the .sll library, the best pratice will be a command tool that take the name of the sll and create magically the dll Therefore from distrib point of side we only changes the way we build the package, it could be automated, it is scalable and it does not need to patch package by package. Only to add some linker script magic Bastien
Re: Wine MinGW system libraries
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > The basic problem is that applications can and often do ship with PE > builds of cross-platform libraries. These libraries can be ahead of > Wine's system libraries, behind them, or even built with custom patches. > Accordingly we really don't want to load "our" freetype in place of > "their" freetype, or "theirs" in place of "ours". But because of the way > the Win32 loader works, you can generally only have one DLL of a given > name loaded in a process, and further attempts to dlopen() [as it were] > "libfreetype-6.dll" will return the handle to the already loaded > library, potentially breaking either Wine or the application. I don't know the details here, but my immediate thought when reading this is that you need some kind of namespace. I then found that linker namespaces are a thing, perhaps they would provide the solution for you. It sounds like the OpenGL shim loader solution listed on the glibc wiki might work for your use-case. Or perhaps the LD_AUDIT feature would also work. https://www.sourceware.org/glibc/wiki/LinkerNamespaces -- bye, pabs https://wiki.debian.org/PaulWise
Re: Wine MinGW system libraries
On 9/6/21 1:34 PM, Bastien ROUCARIES wrote: Le lun. 6 sept. 2021 à 18:36, Zebediah Figura a écrit : On 9/6/21 1:57 AM, Stephen Kitt wrote: On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura < zfig...@codeweavers.com> wrote: On 9/5/21 11:19 AM, Stephen Kitt wrote: On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura wrote: I'm a contributor to the Wine project. To summarize the following mail, Wine needs special versions of some of its normal dependencies, such as libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm sending out a mail to major distributions in order to get some feedback from our packagers on how these should be built and packaged. For a long time Wine has built all of its Win32 libraries (DLLs and EXEs) as ELF binaries. For various reasons related to application compatibility, we have started building our binaries as PE instead, using the MinGW cross-compiler. It is our intent to expand this to some of our dependencies as well. The list of dependencies that we intend to build using MinGW is not quite fixed yet, but we expect it to include and be mostly limited to the following: * libvkd3d * libFAudio * libgnutls * zlib (currently included via manual source import) * libmpg123 * libgsm * libpng * libjpeg-turbo * libtiff * libfreetype * liblcms2 * jxrlib and dependencies of the above packages (not including CRT dependencies, which Wine provides). There is currently some internal discussion about how these dependencies should be built and linked. There are essentially three questions I see that need to be resolved, and while these resolutions have a significant impact on the Wine building and development process, they also have an impact on distributions, and accordingly I'd like to get input from our packagers to ensure that their considerations are accurately taken into account. (1) Should we build via source import, or link statically, or dynamically? Source imports are dispreferred by Debian [1], on the grounds that they cause duplication of libraries on disk and in memory, and make it harder to handle security updates. They also make building and bisecting harder. Static libraries don't seem to be expressly discouraged, but share most of the same downsides (see also [2]). Note however that if they are linked dynamically, we need to make sure that we load our packages instead of MinGW builds of open-source libraries with applications ship with. There's some internal discussion about whether this is possible while using "stock" builds of MinGW libraries, but, due to the way the Win32 loader works, we will probably need to compile each library, and its dependencies, with a separate, wine-specific name, e.g. "libwinefreetype-6.dll" and "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note that all we actually need to change is the name; we don't need to patch the source. Assuming Debian provides the dependencies (which is currently true only for zlib), this could be handled in packaging by providing symlinks, couldn’t it? Not in the Wine prefixes, but elsewhere. Almost :-/ Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. The problem is that libwinefreetype-1.dll is still going to link to libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. Ah yes, I hadn’t thought it through. So really Wine needs its own parallel ecosystem of DLLs in any case, which effectively means building them along with Wine (from Wine’s perspective, ignoring distribution constraints and preferences). This also works in Debian: $ sudo apt install libz-mingw-w64-dev mingw-w64-tools $ x86_64-w64-mingw32-pkg-config --libs zlib -L/usr/x86_64-w64-mingw32/lib -lz Ah, cool, I looked for that and somehow missed it. Note that's the wrong path for dynamic libraries, though; those go in /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe hardcoding a list won't be too painful, though... In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why the .pc file points there. But as you point out, that doesn’t help at runtime. Also, having pkg-config support doesn’t really help with a parallel set of DLLs, does it? I mean... eh. In theory you could say "here's a library called libwinefreetype, and to find it you do `pkg-config --cflags --libs libwinefreetype`", but that does strike me as more than a little janky. For what it's worth, the current proposed solution (which has the support of the Wine maintainer) involves source imports and submodules. There's probably room for changing our approach even after things are committed, but I'd still like to get early feedback from distributions, and make sure that their interests are accurately represented, before we commit. Realistically, I think this is the best approach for now. As Debian adds support for PE libraries, we can replace the source imports in the Wine source code; this is done in many
Re: Wine MinGW system libraries
Le lun. 6 sept. 2021 à 18:36, Zebediah Figura a écrit : > On 9/6/21 1:57 AM, Stephen Kitt wrote: > > On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura < > zfig...@codeweavers.com> > > wrote: > >> On 9/5/21 11:19 AM, Stephen Kitt wrote: > >>> On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura > >>> wrote: > I'm a contributor to the Wine project. To summarize the following > mail, > Wine needs special versions of some of its normal dependencies, such > as > libfreetype and libgnutls, built using the MinGW cross-compiler, and > I'm > sending out a mail to major distributions in order to get some > feedback > from our packagers on how these should be built and packaged. > > For a long time Wine has built all of its Win32 libraries (DLLs and > EXEs) as ELF binaries. For various reasons related to application > compatibility, we have started building our binaries as PE instead, > using the MinGW cross-compiler. It is our intent to expand this to > some > of our dependencies as well. The list of dependencies that we intend > to > build using MinGW is not quite fixed yet, but we expect it to include > and be mostly limited to the following: > > * libvkd3d > * libFAudio > * libgnutls > * zlib (currently included via manual source import) > * libmpg123 > * libgsm > * libpng > * libjpeg-turbo > * libtiff > * libfreetype > * liblcms2 > * jxrlib > > and dependencies of the above packages (not including CRT > dependencies, > which Wine provides). > > There is currently some internal discussion about how these > dependencies > should be built and linked. There are essentially three questions I > see > that need to be resolved, and while these resolutions have a > significant > impact on the Wine building and development process, they also have an > impact on distributions, and accordingly I'd like to get input from > our > packagers to ensure that their considerations are accurately taken > into > account. > > (1) Should we build via source import, or link statically, or > dynamically? > > Source imports are dispreferred by Debian [1], on the grounds that > they > cause duplication of libraries on disk and in memory, and make it > harder > to handle security updates. They also make building and bisecting > harder. Static libraries don't seem to be expressly discouraged, but > share most of the same downsides (see also [2]). > > Note however that if they are linked dynamically, we need to make sure > that we load our packages instead of MinGW builds of open-source > libraries with applications ship with. There's some internal > discussion > about whether this is possible while using "stock" builds of MinGW > libraries, but, due to the way the Win32 loader works, we will > probably > need to compile each library, and its dependencies, with a separate, > wine-specific name, e.g. "libwinefreetype-6.dll" and > "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. > Note > that all we actually need to change is the name; we don't need to > patch > the source. > >>> > >>> Assuming Debian provides the dependencies (which is currently true only > >>> for zlib), this could be handled in packaging by providing symlinks, > >>> couldn’t it? Not in the Wine prefixes, but elsewhere. > >> > >> Almost :-/ > >> > >> Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. > >> The problem is that libwinefreetype-1.dll is still going to link to > >> libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. > > > > Ah yes, I hadn’t thought it through. So really Wine needs its own > parallel > > ecosystem of DLLs in any case, which effectively means building them > along > > with Wine (from Wine’s perspective, ignoring distribution constraints and > > preferences). > > > >>> This also works in Debian: > >>> > >>> $ sudo apt install libz-mingw-w64-dev mingw-w64-tools > >>> $ x86_64-w64-mingw32-pkg-config --libs zlib > >>> -L/usr/x86_64-w64-mingw32/lib -lz > >> > >> Ah, cool, I looked for that and somehow missed it. > >> > >> Note that's the wrong path for dynamic libraries, though; those go in > >> /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe > >> hardcoding a list won't be too painful, though... > > > > In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why > > the .pc file points there. But as you point out, that doesn’t help at > > runtime. > > > > Also, having pkg-config support doesn’t really help with a parallel set > of > > DLLs, does it? > > I mean... eh. In theory you could say "here's a library called > libwinefreetype, and to find it you do `pkg-config --cflags --libs > libwinefreetype`", but that does strike me as more than a little janky. > > > > For what it's
Re: Wine MinGW system libraries
On 9/6/21 1:57 AM, Stephen Kitt wrote: On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura wrote: On 9/5/21 11:19 AM, Stephen Kitt wrote: On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura wrote: I'm a contributor to the Wine project. To summarize the following mail, Wine needs special versions of some of its normal dependencies, such as libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm sending out a mail to major distributions in order to get some feedback from our packagers on how these should be built and packaged. For a long time Wine has built all of its Win32 libraries (DLLs and EXEs) as ELF binaries. For various reasons related to application compatibility, we have started building our binaries as PE instead, using the MinGW cross-compiler. It is our intent to expand this to some of our dependencies as well. The list of dependencies that we intend to build using MinGW is not quite fixed yet, but we expect it to include and be mostly limited to the following: * libvkd3d * libFAudio * libgnutls * zlib (currently included via manual source import) * libmpg123 * libgsm * libpng * libjpeg-turbo * libtiff * libfreetype * liblcms2 * jxrlib and dependencies of the above packages (not including CRT dependencies, which Wine provides). There is currently some internal discussion about how these dependencies should be built and linked. There are essentially three questions I see that need to be resolved, and while these resolutions have a significant impact on the Wine building and development process, they also have an impact on distributions, and accordingly I'd like to get input from our packagers to ensure that their considerations are accurately taken into account. (1) Should we build via source import, or link statically, or dynamically? Source imports are dispreferred by Debian [1], on the grounds that they cause duplication of libraries on disk and in memory, and make it harder to handle security updates. They also make building and bisecting harder. Static libraries don't seem to be expressly discouraged, but share most of the same downsides (see also [2]). Note however that if they are linked dynamically, we need to make sure that we load our packages instead of MinGW builds of open-source libraries with applications ship with. There's some internal discussion about whether this is possible while using "stock" builds of MinGW libraries, but, due to the way the Win32 loader works, we will probably need to compile each library, and its dependencies, with a separate, wine-specific name, e.g. "libwinefreetype-6.dll" and "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note that all we actually need to change is the name; we don't need to patch the source. Assuming Debian provides the dependencies (which is currently true only for zlib), this could be handled in packaging by providing symlinks, couldn’t it? Not in the Wine prefixes, but elsewhere. Almost :-/ Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. The problem is that libwinefreetype-1.dll is still going to link to libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. Ah yes, I hadn’t thought it through. So really Wine needs its own parallel ecosystem of DLLs in any case, which effectively means building them along with Wine (from Wine’s perspective, ignoring distribution constraints and preferences). This also works in Debian: $ sudo apt install libz-mingw-w64-dev mingw-w64-tools $ x86_64-w64-mingw32-pkg-config --libs zlib -L/usr/x86_64-w64-mingw32/lib -lz Ah, cool, I looked for that and somehow missed it. Note that's the wrong path for dynamic libraries, though; those go in /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe hardcoding a list won't be too painful, though... In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why the .pc file points there. But as you point out, that doesn’t help at runtime. Also, having pkg-config support doesn’t really help with a parallel set of DLLs, does it? I mean... eh. In theory you could say "here's a library called libwinefreetype, and to find it you do `pkg-config --cflags --libs libwinefreetype`", but that does strike me as more than a little janky. For what it's worth, the current proposed solution (which has the support of the Wine maintainer) involves source imports and submodules. There's probably room for changing our approach even after things are committed, but I'd still like to get early feedback from distributions, and make sure that their interests are accurately represented, before we commit. Realistically, I think this is the best approach for now. As Debian adds support for PE libraries, we can replace the source imports in the Wine source code; this is done in many other Debian packages for projects which vendor dependencies. I still think this is true. With requirement for a Wine-specific set of DLLs, improving the situation in Debian
Re: Wine MinGW system libraries
Le lun. 6 sept. 2021 à 06:57, Stephen Kitt a écrit : > > On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura > wrote: > > On 9/5/21 11:19 AM, Stephen Kitt wrote: > > > On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura > > > wrote: > > >> I'm a contributor to the Wine project. To summarize the following mail, > > >> Wine needs special versions of some of its normal dependencies, such as > > >> libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > > >> sending out a mail to major distributions in order to get some feedback > > >> from our packagers on how these should be built and packaged. > > >> > > >> For a long time Wine has built all of its Win32 libraries (DLLs and > > >> EXEs) as ELF binaries. For various reasons related to application > > >> compatibility, we have started building our binaries as PE instead, > > >> using the MinGW cross-compiler. It is our intent to expand this to some > > >> of our dependencies as well. The list of dependencies that we intend to > > >> build using MinGW is not quite fixed yet, but we expect it to include > > >> and be mostly limited to the following: > > >> > > >> * libvkd3d > > >> * libFAudio > > >> * libgnutls > > >> * zlib (currently included via manual source import) > > >> * libmpg123 > > >> * libgsm > > >> * libpng > > >> * libjpeg-turbo > > >> * libtiff > > >> * libfreetype > > >> * liblcms2 > > >> * jxrlib > > >> > > >> and dependencies of the above packages (not including CRT dependencies, > > >> which Wine provides). > > >> > > >> There is currently some internal discussion about how these dependencies > > >> should be built and linked. There are essentially three questions I see > > >> that need to be resolved, and while these resolutions have a significant > > >> impact on the Wine building and development process, they also have an > > >> impact on distributions, and accordingly I'd like to get input from our > > >> packagers to ensure that their considerations are accurately taken into > > >> account. > > >> > > >> (1) Should we build via source import, or link statically, or > > >> dynamically? > > >> > > >> Source imports are dispreferred by Debian [1], on the grounds that they > > >> cause duplication of libraries on disk and in memory, and make it harder > > >> to handle security updates. They also make building and bisecting > > >> harder. Static libraries don't seem to be expressly discouraged, but > > >> share most of the same downsides (see also [2]). > > >> > > >> Note however that if they are linked dynamically, we need to make sure > > >> that we load our packages instead of MinGW builds of open-source > > >> libraries with applications ship with. There's some internal discussion > > >> about whether this is possible while using "stock" builds of MinGW > > >> libraries, but, due to the way the Win32 loader works, we will probably > > >> need to compile each library, and its dependencies, with a separate, > > >> wine-specific name, e.g. "libwinefreetype-6.dll" and > > >> "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note > > >> that all we actually need to change is the name; we don't need to patch > > >> the source. > > > > > > Assuming Debian provides the dependencies (which is currently true only > > > for zlib), this could be handled in packaging by providing symlinks, > > > couldn’t it? Not in the Wine prefixes, but elsewhere. > > > > Almost :-/ > > > > Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. > > The problem is that libwinefreetype-1.dll is still going to link to > > libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. > > Ah yes, I hadn’t thought it through. So really Wine needs its own parallel > ecosystem of DLLs in any case, which effectively means building them along > with Wine (from Wine’s perspective, ignoring distribution constraints and > preferences). I will really prefer something less laborious. Like using flexdll for your own library. Moreover using a custom loader will decrease the patching burden for debian side, so a win win. Bastien > > > This also works in Debian: > > > > > > $ sudo apt install libz-mingw-w64-dev mingw-w64-tools > > > $ x86_64-w64-mingw32-pkg-config --libs zlib > > > -L/usr/x86_64-w64-mingw32/lib -lz > > > > Ah, cool, I looked for that and somehow missed it. > > > > Note that's the wrong path for dynamic libraries, though; those go in > > /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe > > hardcoding a list won't be too painful, though... > > In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why > the .pc file points there. But as you point out, that doesn’t help at > runtime. > > Also, having pkg-config support doesn’t really help with a parallel set of > DLLs, does it? > > > >> For what it's worth, the current proposed solution (which has the > > >> support of the Wine maintainer) involves source imports and submodules. > > >> There's probably room for changing our
Re: Wine MinGW system libraries
On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura wrote: > On 9/5/21 11:19 AM, Stephen Kitt wrote: > > On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura > > wrote: > >> I'm a contributor to the Wine project. To summarize the following mail, > >> Wine needs special versions of some of its normal dependencies, such as > >> libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > >> sending out a mail to major distributions in order to get some feedback > >> from our packagers on how these should be built and packaged. > >> > >> For a long time Wine has built all of its Win32 libraries (DLLs and > >> EXEs) as ELF binaries. For various reasons related to application > >> compatibility, we have started building our binaries as PE instead, > >> using the MinGW cross-compiler. It is our intent to expand this to some > >> of our dependencies as well. The list of dependencies that we intend to > >> build using MinGW is not quite fixed yet, but we expect it to include > >> and be mostly limited to the following: > >> > >> * libvkd3d > >> * libFAudio > >> * libgnutls > >> * zlib (currently included via manual source import) > >> * libmpg123 > >> * libgsm > >> * libpng > >> * libjpeg-turbo > >> * libtiff > >> * libfreetype > >> * liblcms2 > >> * jxrlib > >> > >> and dependencies of the above packages (not including CRT dependencies, > >> which Wine provides). > >> > >> There is currently some internal discussion about how these dependencies > >> should be built and linked. There are essentially three questions I see > >> that need to be resolved, and while these resolutions have a significant > >> impact on the Wine building and development process, they also have an > >> impact on distributions, and accordingly I'd like to get input from our > >> packagers to ensure that their considerations are accurately taken into > >> account. > >> > >> (1) Should we build via source import, or link statically, or > >> dynamically? > >> > >> Source imports are dispreferred by Debian [1], on the grounds that they > >> cause duplication of libraries on disk and in memory, and make it harder > >> to handle security updates. They also make building and bisecting > >> harder. Static libraries don't seem to be expressly discouraged, but > >> share most of the same downsides (see also [2]). > >> > >> Note however that if they are linked dynamically, we need to make sure > >> that we load our packages instead of MinGW builds of open-source > >> libraries with applications ship with. There's some internal discussion > >> about whether this is possible while using "stock" builds of MinGW > >> libraries, but, due to the way the Win32 loader works, we will probably > >> need to compile each library, and its dependencies, with a separate, > >> wine-specific name, e.g. "libwinefreetype-6.dll" and > >> "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note > >> that all we actually need to change is the name; we don't need to patch > >> the source. > > > > Assuming Debian provides the dependencies (which is currently true only > > for zlib), this could be handled in packaging by providing symlinks, > > couldn’t it? Not in the Wine prefixes, but elsewhere. > > Almost :-/ > > Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. > The problem is that libwinefreetype-1.dll is still going to link to > libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. Ah yes, I hadn’t thought it through. So really Wine needs its own parallel ecosystem of DLLs in any case, which effectively means building them along with Wine (from Wine’s perspective, ignoring distribution constraints and preferences). > > This also works in Debian: > > > > $ sudo apt install libz-mingw-w64-dev mingw-w64-tools > > $ x86_64-w64-mingw32-pkg-config --libs zlib > > -L/usr/x86_64-w64-mingw32/lib -lz > > Ah, cool, I looked for that and somehow missed it. > > Note that's the wrong path for dynamic libraries, though; those go in > /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe > hardcoding a list won't be too painful, though... In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why the .pc file points there. But as you point out, that doesn’t help at runtime. Also, having pkg-config support doesn’t really help with a parallel set of DLLs, does it? > >> For what it's worth, the current proposed solution (which has the > >> support of the Wine maintainer) involves source imports and submodules. > >> There's probably room for changing our approach even after things are > >> committed, but I'd still like to get early feedback from distributions, > >> and make sure that their interests are accurately represented, before we > >> commit. > > > > Realistically, I think this is the best approach for now. As Debian adds > > support for PE libraries, we can replace the source imports in the Wine > > source code; this is done in many other Debian packages for
Re: Wine MinGW system libraries
On 9/5/21 11:19 AM, Stephen Kitt wrote: Hi Zebediah, On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura wrote: I'm a contributor to the Wine project. To summarize the following mail, Wine needs special versions of some of its normal dependencies, such as libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm sending out a mail to major distributions in order to get some feedback from our packagers on how these should be built and packaged. For a long time Wine has built all of its Win32 libraries (DLLs and EXEs) as ELF binaries. For various reasons related to application compatibility, we have started building our binaries as PE instead, using the MinGW cross-compiler. It is our intent to expand this to some of our dependencies as well. The list of dependencies that we intend to build using MinGW is not quite fixed yet, but we expect it to include and be mostly limited to the following: * libvkd3d * libFAudio * libgnutls * zlib (currently included via manual source import) * libmpg123 * libgsm * libpng * libjpeg-turbo * libtiff * libfreetype * liblcms2 * jxrlib and dependencies of the above packages (not including CRT dependencies, which Wine provides). There is currently some internal discussion about how these dependencies should be built and linked. There are essentially three questions I see that need to be resolved, and while these resolutions have a significant impact on the Wine building and development process, they also have an impact on distributions, and accordingly I'd like to get input from our packagers to ensure that their considerations are accurately taken into account. (1) Should we build via source import, or link statically, or dynamically? Source imports are dispreferred by Debian [1], on the grounds that they cause duplication of libraries on disk and in memory, and make it harder to handle security updates. They also make building and bisecting harder. Static libraries don't seem to be expressly discouraged, but share most of the same downsides (see also [2]). Note however that if they are linked dynamically, we need to make sure that we load our packages instead of MinGW builds of open-source libraries with applications ship with. There's some internal discussion about whether this is possible while using "stock" builds of MinGW libraries, but, due to the way the Win32 loader works, we will probably need to compile each library, and its dependencies, with a separate, wine-specific name, e.g. "libwinefreetype-6.dll" and "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note that all we actually need to change is the name; we don't need to patch the source. Assuming Debian provides the dependencies (which is currently true only for zlib), this could be handled in packaging by providing symlinks, couldn’t it? Not in the Wine prefixes, but elsewhere. Almost :-/ Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. The problem is that libwinefreetype-1.dll is still going to link to libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. (The Wine team also maintains libvkd3d and libFaudio, so we can take care of those at least.) Accordingly, although static linking and source imports are generally disprefered, it may quite likely be preferable in our case. We don't get the benefits of on-disk deduplication, since Wine is essentially the only piece of software which needs these libraries. (2) If we use dynamic libraries, should dependencies be included in the main wine package, or packaged separately? This is mostly a question for packagers, although it also relates to (3). I expect that Debian will want to answer "packaged separately" here, on the grounds that this lets them update (say) Wine's libgnutls separately, and in sync with ELF libgnutls, if some security fix is needed. There is a snag, though: we need libraries to be copied into the prefix (there's some internal effort to allow using something like symlinks instead, but this hard and not done yet). Normally we perform this copy every time Wine is updated, but if Wine and its dependencies aren't updated on the same schedule, we may end up loading an old version of a dependency in the prefix. Debian packaging doesn’t touch anything in users’ home directories, so this would have to be handled in Wine itself, perhaps in a similar fashion to existing provisions for Gecko and Mono. (3) If dependencies are packaged separately, should Wine build them as part of its build tree (e.g. using submodules), or find and link (statically or dynamically) to existing binaries? Linking to existing binaries is generally preferable: it avoids duplication on disk; it reduces compile times when compiling a single package from source (especially the first time). However, we aren't going to benefit from on-disk duplication. And, most importantly, unlike with ELF dependencies, there is no standardized way to locate MinGW libraries—especially if it comes to
Re: Wine MinGW system libraries
Hi Bastien, On Sun, 5 Sep 2021 08:53:49 +0200, Bastien ROUCARIES wrote: > Le dim. 5 sept. 2021 à 03:34, Zebediah Figura a > écrit : > > I'm a contributor to the Wine project. To summarize the following mail, > > Wine needs special versions of some of its normal dependencies, such as > > libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > > sending out a mail to major distributions in order to get some feedback > > from our packagers on how these should be built and packaged. > > > > For a long time Wine has built all of its Win32 libraries (DLLs and > > EXEs) as ELF binaries. For various reasons related to application > > compatibility, we have started building our binaries as PE instead, > > using the MinGW cross-compiler. It is our intent to expand this to some > > of our dependencies as well. The list of dependencies that we intend to > > build using MinGW is not quite fixed yet, but we expect it to include > > and be mostly limited to the following: > > > > * libvkd3d > > * libFAudio > > * libgnutls > > * zlib (currently included via manual source import) > > * libmpg123 > > * libgsm > > * libpng > > * libjpeg-turbo > > * libtiff > > * libfreetype > > * liblcms2 > > * jxrlib > > > > and dependencies of the above packages (not including CRT dependencies, > > which Wine provides). > > > > There is currently some internal discussion about how these dependencies > > should be built and linked. There are essentially three questions I see > > that need to be resolved, and while these resolutions have a significant > > impact on the Wine building and development process, they also have an > > impact on distributions, and accordingly I'd like to get input from our > > packagers to ensure that their considerations are accurately taken into > > account. > > > > (1) Should we build via source import, or link statically, or dynamically? > > > > Source imports are dispreferred by Debian [1], on the grounds that they > > cause duplication of libraries on disk and in memory, and make it harder > > to handle security updates. They also make building and bisecting > > harder. Static libraries don't seem to be expressly discouraged, but > > share most of the same downsides (see also [2]). > > > > Note however that if they are linked dynamically, we need to make sure > > that we load our packages instead of MinGW builds of open-source > > libraries with applications ship with. There's some internal discussion > > about whether this is possible while using "stock" builds of MinGW > > libraries, but, due to the way the Win32 loader works, we will probably > > need to compile each library, and its dependencies, with a separate, > > wine-specific name, e.g. "libwinefreetype-6.dll" and > > "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note > > that all we actually need to change is the name; we don't need to patch > > the source. > > > > Accordingly, although static linking and source imports are generally > > disprefered, it may quite likely be preferable in our case. We don't get > > the benefits of on-disk deduplication, since Wine is essentially the > > only piece of software which needs these libraries. > > > > (2) If we use dynamic libraries, should dependencies be included in the > > main wine package, or packaged separately? > > > > > Improve dpkg to support partial arch. I volonteer to implement none arch > but i am waiting from guillem here. Thanks for stepping up! For MinGW-w64 dependencies, an additional requirement is figuring out a solution for https://bugs.debian.org/606825; https://wiki.debian.org/Teams/Dpkg/FAQ#Q._Can_we_add_support_for_new_dpkg_architectures.3F gives additional context. Regards, Stephen pgpHCEnwcfbXF.pgp Description: OpenPGP digital signature
Re: Wine MinGW system libraries
Hi Zebediah, On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura wrote: > I'm a contributor to the Wine project. To summarize the following mail, > Wine needs special versions of some of its normal dependencies, such as > libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > sending out a mail to major distributions in order to get some feedback > from our packagers on how these should be built and packaged. > > For a long time Wine has built all of its Win32 libraries (DLLs and > EXEs) as ELF binaries. For various reasons related to application > compatibility, we have started building our binaries as PE instead, > using the MinGW cross-compiler. It is our intent to expand this to some > of our dependencies as well. The list of dependencies that we intend to > build using MinGW is not quite fixed yet, but we expect it to include > and be mostly limited to the following: > > * libvkd3d > * libFAudio > * libgnutls > * zlib (currently included via manual source import) > * libmpg123 > * libgsm > * libpng > * libjpeg-turbo > * libtiff > * libfreetype > * liblcms2 > * jxrlib > > and dependencies of the above packages (not including CRT dependencies, > which Wine provides). > > There is currently some internal discussion about how these dependencies > should be built and linked. There are essentially three questions I see > that need to be resolved, and while these resolutions have a significant > impact on the Wine building and development process, they also have an > impact on distributions, and accordingly I'd like to get input from our > packagers to ensure that their considerations are accurately taken into > account. > > (1) Should we build via source import, or link statically, or dynamically? > > Source imports are dispreferred by Debian [1], on the grounds that they > cause duplication of libraries on disk and in memory, and make it harder > to handle security updates. They also make building and bisecting > harder. Static libraries don't seem to be expressly discouraged, but > share most of the same downsides (see also [2]). > > Note however that if they are linked dynamically, we need to make sure > that we load our packages instead of MinGW builds of open-source > libraries with applications ship with. There's some internal discussion > about whether this is possible while using "stock" builds of MinGW > libraries, but, due to the way the Win32 loader works, we will probably > need to compile each library, and its dependencies, with a separate, > wine-specific name, e.g. "libwinefreetype-6.dll" and > "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note > that all we actually need to change is the name; we don't need to patch > the source. Assuming Debian provides the dependencies (which is currently true only for zlib), this could be handled in packaging by providing symlinks, couldn’t it? Not in the Wine prefixes, but elsewhere. (The Wine team also maintains libvkd3d and libFaudio, so we can take care of those at least.) > Accordingly, although static linking and source imports are generally > disprefered, it may quite likely be preferable in our case. We don't get > the benefits of on-disk deduplication, since Wine is essentially the > only piece of software which needs these libraries. > > (2) If we use dynamic libraries, should dependencies be included in the > main wine package, or packaged separately? > > This is mostly a question for packagers, although it also relates to (3). > > I expect that Debian will want to answer "packaged separately" here, on > the grounds that this lets them update (say) Wine's libgnutls > separately, and in sync with ELF libgnutls, if some security fix is > needed. There is a snag, though: we need libraries to be copied into the > prefix (there's some internal effort to allow using something like > symlinks instead, but this hard and not done yet). Normally we perform > this copy every time Wine is updated, but if Wine and its dependencies > aren't updated on the same schedule, we may end up loading an old > version of a dependency in the prefix. Debian packaging doesn’t touch anything in users’ home directories, so this would have to be handled in Wine itself, perhaps in a similar fashion to existing provisions for Gecko and Mono. > (3) If dependencies are packaged separately, should Wine build them as > part of its build tree (e.g. using submodules), or find and link > (statically or dynamically) to existing binaries? > > Linking to existing binaries is generally preferable: it avoids > duplication on disk; it reduces compile times when compiling a single > package from source (especially the first time). However, we aren't > going to benefit from on-disk duplication. And, most importantly, unlike > with ELF dependencies, there is no standardized way to locate MinGW > libraries—especially if it comes to Wine-specific libraries. We would > need a way for
Re: Wine MinGW system libraries
Le dim. 5 sept. 2021 à 03:34, Zebediah Figura a écrit : > Hello all, > > I'm a contributor to the Wine project. To summarize the following mail, > Wine needs special versions of some of its normal dependencies, such as > libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > sending out a mail to major distributions in order to get some feedback > from our packagers on how these should be built and packaged. > > For a long time Wine has built all of its Win32 libraries (DLLs and > EXEs) as ELF binaries. For various reasons related to application > compatibility, we have started building our binaries as PE instead, > using the MinGW cross-compiler. It is our intent to expand this to some > of our dependencies as well. The list of dependencies that we intend to > build using MinGW is not quite fixed yet, but we expect it to include > and be mostly limited to the following: > > * libvkd3d > * libFAudio > * libgnutls > * zlib (currently included via manual source import) > * libmpg123 > * libgsm > * libpng > * libjpeg-turbo > * libtiff > * libfreetype > * liblcms2 > * jxrlib > > and dependencies of the above packages (not including CRT dependencies, > which Wine provides). > > There is currently some internal discussion about how these dependencies > should be built and linked. There are essentially three questions I see > that need to be resolved, and while these resolutions have a significant > impact on the Wine building and development process, they also have an > impact on distributions, and accordingly I'd like to get input from our > packagers to ensure that their considerations are accurately taken into > account. > > (1) Should we build via source import, or link statically, or dynamically? > > Source imports are dispreferred by Debian [1], on the grounds that they > cause duplication of libraries on disk and in memory, and make it harder > to handle security updates. They also make building and bisecting > harder. Static libraries don't seem to be expressly discouraged, but > share most of the same downsides (see also [2]). > > Note however that if they are linked dynamically, we need to make sure > that we load our packages instead of MinGW builds of open-source > libraries with applications ship with. There's some internal discussion > about whether this is possible while using "stock" builds of MinGW > libraries, but, due to the way the Win32 loader works, we will probably > need to compile each library, and its dependencies, with a separate, > wine-specific name, e.g. "libwinefreetype-6.dll" and > "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note > that all we actually need to change is the name; we don't need to patch > the source. > > Accordingly, although static linking and source imports are generally > disprefered, it may quite likely be preferable in our case. We don't get > the benefits of on-disk deduplication, since Wine is essentially the > only piece of software which needs these libraries. > > (2) If we use dynamic libraries, should dependencies be included in the > main wine package, or packaged separately? > Improve dpkg to support partial arch. I volonteer to implement none arch but i am waiting from guillem here. > This is mostly a question for packagers, although it also relates to (3). > > I expect that Debian will want to answer "packaged separately" here, on > the grounds that this lets them update (say) Wine's libgnutls > separately, and in sync with ELF libgnutls, if some security fix is > needed. There is a snag, though: we need libraries to be copied into the > prefix (there's some internal effort to allow using something like > symlinks instead, but this hard and not done yet). Normally we perform > this copy every time Wine is updated, but if Wine and its dependencies > aren't updated on the same schedule, we may end up loading an old > version of a dependency in the prefix. > > (3) If dependencies are packaged separately, should Wine build them as > part of its build tree (e.g. using submodules), or find and link > (statically or dynamically) to existing binaries? > > Linking to existing binaries is generally preferable: it avoids > duplication on disk; it reduces compile times when compiling a single > package from source (especially the first time). However, we aren't > going to benefit from on-disk duplication. And, most importantly, unlike > with ELF dependencies, there is no standardized way to locate MinGW > libraries—especially if it comes to Wine-specific libraries. We would > need a way for Wine's configure script to find these packages—and > ideally find them automatically, or else fall back to a submodule-based > approach. > > If we rely on distributions to provide our dependencies, the best idea I > have here would be something like a x86_64-w64-mingw32-pkg-config > (Fedora actually already ships this, but I think Fedora is the only > one). And if we use shared libraries rather than static, things get >
