Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On Fri, Dec 20, 2019 at 8:41 AM Gerion Entrup wrote: > > Am Donnerstag, 19. Dezember 2019, 19:43:37 CET schrieb Sebastian Pipping: > > On 19.12.19 18:37, Michał Górny wrote: > > > We have a better alternative that lets us limit the impact on the users. > > > Why not use it? > > > > Which one? The CMake bootstrap copy? The adding to stage3 one? > > If an error message can be shown, maybe this is enough as a hint: > "expat and cmake have circular dependencies. Emerge it the first time with: > USE=bundled-cmake emerge -1 cmake expat > and then just don't care anymore about this use flag." > Not sure about custom error messages but in any case when using that bundled-cmake USE flag it would probably make sense to do an ewarn that the bundling took place and is only intended for temporary use, and that the package should be re-merged without the flag once expat is working. That would also cover users who inadvertently set the flag. I think that bundled-cmake also might make more sense than system-cmake, even though the latter is more established. We have a lot of users who set USE=-* and that means they're going to end up with lots of stuff bundled that doesn't need to be. I'm no fan of USE=-* in general but it seems like we should try to avoid making not setting a USE flag the less secure option since it does happen a lot. I'm just commenting on the USE-based solution. The compat package solution obviously bypasses this particular problem entirely. -- Rich
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
Am Donnerstag, 19. Dezember 2019, 19:43:37 CET schrieb Sebastian Pipping: > On 19.12.19 18:37, Michał Górny wrote: > > We have a better alternative that lets us limit the impact on the users. > > Why not use it? > > Which one? The CMake bootstrap copy? The adding to stage3 one? Is it possible to show a specific error message when running in a circular dependency? When I get it right, the problem occurs only one time when solved with a bundled use flag. As soon as expat and/or cmake are installed, follow up versions can be merge without problems. If an error message can be shown, maybe this is enough as a hint: "expat and cmake have circular dependencies. Emerge it the first time with: USE=bundled-cmake emerge -1 cmake expat and then just don't care anymore about this use flag." Of course, the same applies for other known circular dependencies, too. At least for me as a user, this would be enough. Best, Gerion signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On 12/19/19 12:37 PM, Michał Górny wrote: > > Just because someone did something crappy, it doesn't mean it was > considered 'good enough'. It was just a cheap hack that someone once > did just to get it over with and stop caring. Not a good solution we > should keep copying. > These should all be USE=bundled-foo, and off by default at the very least.
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On Thu, 2019-12-19 at 19:43 +0100, Sebastian Pipping wrote: > On 19.12.19 18:37, Michał Górny wrote: > > We have a better alternative that lets us limit the impact on the users. > > Why not use it? > > Which one? The CMake bootstrap copy? The adding to stage3 one? > Bootstrap version. It doesn't have to be a strict copy. The whole point is that even if it's buggy, unmaintained, vulnerable, it's impact is going to be minimal. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On 19.12.19 18:37, Michał Górny wrote: > We have a better alternative that lets us limit the impact on the users. > Why not use it? Which one? The CMake bootstrap copy? The adding to stage3 one?
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On Thu, 2019-12-19 at 18:28 +0100, Sebastian Pipping wrote: > Hey! > > > On 19.12.19 17:03, Michał Górny wrote: > > > B) Introduce USE flag "system-expat" to CMake similar to existing > > >flag "system-jsoncpp", have it off by default, keep reminding > > >CMake upstream to update their bundle > > > > > > [..] > > > > It violates the policy on bundled libraries. > > Same for the dev-util/cmake-bootstrap approach, right? > > > > What's worse, the awful > > USE flags solution means that most of the Gentoo devs end up using > > bundled libraries just because people are manually required to figure > > out what to do in order to disable them. > > I didn't say that it's perfect :) It's the same approach that we have > have with the system-jsoncpp USE flag already so that was considered > good enough at some point in the past. I guess we want the same for > Expat and jsoncpp? Which alternative do you see as better than a new > flag system-expat? > Just because someone did something crappy, it doesn't mean it was considered 'good enough'. It was just a cheap hack that someone once did just to get it over with and stop caring. Not a good solution we should keep copying. We have a better alternative that lets us limit the impact on the users. Why not use it? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
Hey! On 19.12.19 17:03, Michał Górny wrote: >> B) Introduce USE flag "system-expat" to CMake similar to existing >>flag "system-jsoncpp", have it off by default, keep reminding >>CMake upstream to update their bundle >> >> [..] > > It violates the policy on bundled libraries. Same for the dev-util/cmake-bootstrap approach, right? > What's worse, the awful > USE flags solution means that most of the Gentoo devs end up using > bundled libraries just because people are manually required to figure > out what to do in order to disable them. I didn't say that it's perfect :) It's the same approach that we have have with the system-jsoncpp USE flag already so that was considered good enough at some point in the past. I guess we want the same for Expat and jsoncpp? Which alternative do you see as better than a new flag system-expat? Best Sebastian
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On Thu, 2019-12-19 at 15:39 +0100, Sebastian Pipping wrote: > Hey! > > > Thanks everyone for your thoughts so far! > > From what I heard, these two options seem realistic to me: > > A) Ask the KDE team for help with teaming up on a new package >dev-util/cmake-bootstrap, keep it in sync with dev-util/cmake, >make sure both packages co-exists with full disjoint operation, >i.e. zero file conflicts + zero cross package file usage (tricky?). > > B) Introduce USE flag "system-expat" to CMake similar to existing >flag "system-jsoncpp", have it off by default, keep reminding >CMake upstream to update their bundle > > I favor (B) by more than just a bit. Does anyone have strong concerns > against moving in the dev-util/cmake[-system-expat] (B) direction? Is > it acceptable if I make those changes to the CMake ebuild myself? > It violates the policy on bundled libraries. What's worse, the awful USE flags solution means that most of the Gentoo devs end up using bundled libraries just because people are manually required to figure out what to do in order to disable them. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
Hey! Thanks everyone for your thoughts so far! >From what I heard, these two options seem realistic to me: A) Ask the KDE team for help with teaming up on a new package dev-util/cmake-bootstrap, keep it in sync with dev-util/cmake, make sure both packages co-exists with full disjoint operation, i.e. zero file conflicts + zero cross package file usage (tricky?). B) Introduce USE flag "system-expat" to CMake similar to existing flag "system-jsoncpp", have it off by default, keep reminding CMake upstream to update their bundle I favor (B) by more than just a bit. Does anyone have strong concerns against moving in the dev-util/cmake[-system-expat] (B) direction? Is it acceptable if I make those changes to the CMake ebuild myself? Thanks again Sebastian
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On 19.12.19 14:32, Rolf Eike Beer wrote: > These things _are_ updated regularly To be fair they update because I keep opening update requests: https://gitlab.kitware.com/cmake/cmake/issues?scope=all=%E2%9C%93=closed=expat+update
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
Am 2019-12-18 22:44, schrieb Francesco Riosa: Il giorno mer 18 dic 2019 alle ore 22:03 Sebastian Pipping ha scritto: CMake bundles a (previously outdated and vulnerable) copy of expat so I'm not sure if re-activating that bundle — say with a new use flag "system-expat" — would be a good thing to resort to for breaking the cycle, with regard to security in particular. Pushing gently upstream to upgrade bundled expat copy would (at least temporarily) fix the issue and also benefit other use cases. Maybe they are Gentoo friendly they also release quite often, which would fix the problem soon This is in CMake 3.16.0: commit 50bc359184472700e9776a0a9d6f7e06ea82b9ce Author: Brad King Date: Mon Nov 11 10:44:17 2019 -0500 expat: Update CMake build for 2.2.9 commit b63a5c88a2089494e53f22f83db1925435161934 Merge: 512fabaa9d 1712885b4f Author: Brad King Date: Mon Nov 11 10:42:32 2019 -0500 Merge branch 'upstream-expat' into update-expat * upstream-expat: expat 2019-09-25 (a7bc26b6) These things _are_ updated regularly, but in case something is missed just file a bug at gitlab.kitware.com. All these bundled thing bumps are scripted as far as possible, so the actual overhead is quite small. Eike
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On Wed, 2019-12-18 at 23:58 +, Sergei Trofimovich wrote:
> On Wed, 18 Dec 2019 22:02:47 +0100
> Sebastian Pipping wrote:
>
> > Hi all,
> >
> >
> > I noticed that dev-util/cmake depends on dev-libs/expat and that
> > libexpat upstream (where I'm involved) is in the process of
> > dropping GNU Autotools altogether in favor of CMake in the near future,
> > potentially the next release (without any known target release date).
> >
> > CMake bundles a (previously outdated and vulnerable) copy of expat so
> > I'm not sure if re-activating that bundle — say with a new use flag
> > "system-expat" — would be a good thing to resort to for breaking the
> > cycle, with regard to security in particular.
> >
> > Do you have any ideas how to avoid a bad circular dependency issue for
> > our users in the future? Are you aware of similar problems and
> > solutions from the past?
>
> Some other distributions provide two packages to break the cycle.
> Example Gentoo solution would be: "cmake.ebuild" depends on "expat.ebuild",
> "expat.ebuild" depends on "cmake-with-bundled-expat.ebuild".
>
I actually think this is the cleanest solution of all. To be more
specific, create dev-util/cmake-bootstrap that either includes bundled
dependencies (let's not forget about jsoncpp here) and installs into
some dedicated prefix (e.g. /usr/lib/cmake-bootstrap).
Then you'd have expat and jsoncpp would BDEPEND:
|| ( dev-util/cmake-bootstrap dev-util/cmake )
and the ebuild would do something like, roughly:
has_version -b dev-util/cmake ||
local -x PATH=${BROOT}/usr/lib/cmake-bootstrap/bin:${PATH}
Since we don't need blockers there, Portage should be able to resolve
the depgraph peacefully and pull both packages in gracefully. You
wouldn't have to do anything else in further revdeps. The bootstrap
package would be safely isolated from the other revdeps, and it would be
depcleaned once other packages pull in regular cmake.
I can make a proof-of-concept based on jsoncpp if you like.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On Wed, 18 Dec 2019 23:58:22 + Sergei Trofimovich wrote: > [c] would be nice to be solved at portage level if portage could schedule > multiple merges for the same package with different USE flags. Related bugs: - Portage should be able to auto-flip USE="test" & FEATURES="test" https://bugs.gentoo.org/697914#c0 - Portage needs an extension to package.use specification to provide a level of discretion to portage to flip flags on its own https://bugs.gentoo.org/698920#c0 pgp4qmLQa8qhA.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On 12/18/19 4:02 PM, Sebastian Pipping wrote: > Hi all, > > > I noticed that dev-util/cmake depends on dev-libs/expat and that > libexpat upstream (where I'm involved) is in the process of > dropping GNU Autotools altogether in favor of CMake in the near future, > potentially the next release (without any known target release date). >> ... > > Do you have any ideas how to avoid a bad circular dependency issue for > our users in the future? Are you aware of similar problems and > solutions from the past? Keep autotools?
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On Wed, 18 Dec 2019 22:02:47 +0100 Sebastian Pipping wrote: > Hi all, > > > I noticed that dev-util/cmake depends on dev-libs/expat and that > libexpat upstream (where I'm involved) is in the process of > dropping GNU Autotools altogether in favor of CMake in the near future, > potentially the next release (without any known target release date). > > CMake bundles a (previously outdated and vulnerable) copy of expat so > I'm not sure if re-activating that bundle — say with a new use flag > "system-expat" — would be a good thing to resort to for breaking the > cycle, with regard to security in particular. > > Do you have any ideas how to avoid a bad circular dependency issue for > our users in the future? Are you aware of similar problems and > solutions from the past? Fun fact: cmake is not keyworded for riscv. To think of solutions enumerating the arising problems explicitly might help here. I see a few: 1. initial system bootstrap requires solving a circular dependency 2. recovery from broken state (expat soname bump without preserved libs or cmake broken by one of many depends it has) [2.] effectively forbids a dependency circle. [1.] is hard to solve without users' intervention to at least flip a default flag. Some other distributions provide two packages to break the cycle. Example Gentoo solution would be: "cmake.ebuild" depends on "expat.ebuild", "expat.ebuild" depends on "cmake-with-bundled-expat.ebuild". Some examples of circular dependencies are: a) gcc/glibc/binutils toolchain. the solution is to provide prebuilt binaries as part of stage3. b) self-hosted languages without an interpreter in C (rust, golang, ghc). The solution is to provide prebuilt binaries in ebuilds. c) circular dependencies in tests. The solution is careful user's USE flags juggling: enable USE=test only for a package being tested. d) glibc depends on libidn2 to implement modern DNS demangling. glibc fixes it by not depending on libidn at build time and does manual library loading with dlopen()/dlsym(). e) vast majority of others: dependency bundling (users of autotools, gnulib, zlib, lua). All the above are not pretty. a) is simplest to maintain by ebuild maintainer and gentoo user, but probably not by releng. c) moves the burden to user. I personally would explore [e]: unconditional bundling of expat into cmake and make sure it's kept up-to-date there. [c] would be nice to be solved at portage level if portage could schedule multiple merges for the same package with different USE flags. -- Sergei
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
Il giorno mer 18 dic 2019 alle ore 22:03 Sebastian Pipping ha scritto: > > CMake bundles a (previously outdated and vulnerable) copy of expat so > I'm not sure if re-activating that bundle — say with a new use flag > "system-expat" — would be a good thing to resort to for breaking the > cycle, with regard to security in particular. > > Pushing gently upstream to upgrade bundled expat copy would (at least temporarily) fix the issue and also benefit other use cases. Maybe they are Gentoo friendly they also release quite often, which would fix the problem soon
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On Wed, 2019-12-18 at 22:10 +0100, Piotr Karbowski wrote: > Hi, > > On 18/12/2019 22.08, Michał Górny wrote: > > I know that's an unhappy idea but maybe it's time to include CMake > > in stage3. Then it would be just a matter of temporarily enabling > > bundled libs for stage builds, I guess. > > Not sure what's unhappy about it, but I like the idea, it will be > painless for new users with rather limited cost of ownership on Gentoo side. > Increase of stage size may make people unhappy. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
Hi, On 18/12/2019 22.08, Michał Górny wrote: > I know that's an unhappy idea but maybe it's time to include CMake > in stage3. Then it would be just a matter of temporarily enabling > bundled libs for stage builds, I guess. Not sure what's unhappy about it, but I like the idea, it will be painless for new users with rather limited cost of ownership on Gentoo side. -- Piotr. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake
On Wed, 2019-12-18 at 22:02 +0100, Sebastian Pipping wrote: > Hi all, > > > I noticed that dev-util/cmake depends on dev-libs/expat and that > libexpat upstream (where I'm involved) is in the process of > dropping GNU Autotools altogether in favor of CMake in the near future, > potentially the next release (without any known target release date). > > CMake bundles a (previously outdated and vulnerable) copy of expat so > I'm not sure if re-activating that bundle — say with a new use flag > "system-expat" — would be a good thing to resort to for breaking the > cycle, with regard to security in particular. > > Do you have any ideas how to avoid a bad circular dependency issue for > our users in the future? Are you aware of similar problems and > solutions from the past? > I know that's an unhappy idea but maybe it's time to include CMake in stage3. Then it would be just a matter of temporarily enabling bundled libs for stage builds, I guess. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
