Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[William L. Thomson Jr.]

On Tuesday, November 1, 2016 1:31:55 AM EDT Michał Górny wrote:
>
> 1. How often do you find '~' useful? Do you think there should be
> additional operators that ignore revision part?

IMHO, this one should be a requirement anytime a project/package is split 
between multiple ebuilds. This is common for Java projects, where parts are 
built in pieces, and packaged on their own.

The rationale is if you are splitting up a package from a single source/
release version. All dependencies should be using the same version, beyond the 
same slot. I do not think upstreams will want to work or handle bugs that 
result from mixing versions of stuff. Nor should there even be potential for 
such issue by mixing versions.

I went so far as to make it a recommendation for Gentoo Java, but really 
should be expanded as a general rule, requirement vs recommendation.

https://wiki.gentoo.org/wiki/Java_Developer_Guide#Versions

This does come with additional overhead, as you cannot just bump a piece of a 
package without doing all. It also makes the order of bumping and removal a 
bit more complex as well. There can also be some issues when it comes to doing 
system updates, if not doing world. May have blockers with different versions 
being pulled in vs installed on system.

Regardless of the negatives, I think the positives are justified.

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[Sergei Trofimovich]

On Wed, 2 Nov 2016 07:43:04 +0100
Michał Górny  wrote:

> On Tue, 1 Nov 2016 23:09:25 +
> Sergei Trofimovich  wrote:
> 
> > On Tue, 1 Nov 2016 14:16:18 +0100
> > Michał Górny  wrote:
> >   
> > > However, while at it I should
> > > point out that the spec doesn't really cover pure-DEPEND :=, so there
> > > is no guarantee that Haskell := operators will work at all.
> > 
> > Yeah. It settled mostly because DEPEND="$RDEPEND + foo" is idiomatic
> > at least in non ':=' world and no tools complained about ':=' in DEPEND.  
> 
> No, it's allowed because of DEPEND=$RDEPEND.

I'm guessing (again) I should read it as "syntax-wise they are defined by the
same grammar".

> However, DEPEND without RDEPEND is not guaranteed to do anything useful.

I'm not aware of any haskell ebuilds that set ':=' only in DEPEND but not
propagate it to RDEPEND.

Crude hack
gentoo/metadata/md5-cache $ for f in */*; do if egrep -q '^DEPEND=.*:=' $f; 
then if ! egrep -q '^RDEPEND=.*:=' $f; then echo $f; fi; fi; done
reports a few unrelated packages.

-- 

  Sergei


pgphFCQ8W7m9L.pgp
Description: Цифровая подпись OpenPGP

Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[Daniel Campbell]

On 10/31/2016 05:31 PM, Michał Górny wrote:
> Hello, everyone.

Answering inline to help context.

> 
> I would like to work on a major version depedencny specification
> improvements as part of the next EAPI. For this reason, I'd like to
> first gather some research on how developers are using the current
> system, what they find efficient and what they find cumbersome.
> 
> Therefore, I would like to ask the following questions:
> 
> 1. How often do you find '~' useful? Do you think there should be
> additional operators that ignore revision part?

I've never really used it. After reading the devmanual on it, it seems
like it's redundant compared to *.

> 
> 2. How often do you find '=...*' wildcard syntax useful? To what
> purpose do you use it? Do you find its behavior confusing [1]?

It was confusing until I read your footnote. Obviously, I had to know it
back when I was applying, but it seems counter intuitive. Could we
instead make ~ take on the _pre _r behavior and let * be a more
intuitive option to match ranged deps without slotting?

> 
> 3. Do you sometimes find yourself using '<'/'<=' specs that
> accidentally match _pre/_rc/... versions?

Nope.

> 
> 4. What are the common tasks that you find unnecessarily complex /
> lengthy with the current version specifications?

Maybe I'm not writing enough ebuilds, but slot dependencies are a bit
complex for me. I have a hard time deciding whether to use := or :*, or
even if a slotted dependency is necessary. Every ebuild I end up having
peer-reviewed has a slot dep mistake that I end up fixing, and it's one
of the things that makes me ask for review before committing something.

> 
> 5. Do you find any other parts of the current version dependency
> specifications confusing?
> 
> Please just list the problems and your feeling about the current system
> now, not solutions. We will focus on the solutions in a next thread
> once we determine what the problems are.
> 
> Thanks in advance for your input.
> 
> 
> [1]. Few less-known facts about =...*:
> 
> a. it does NOT do string pattern matching but allows any version
> components following (i.e. 1.1* does not match 1.11),
> 
> b. it matches any version components, including _pre/_p/_rc/...
> suffixes (i.e. 1.1* matches 1.1_rc1).
> 

Maybe it could be covered in the devmanual (or elsewhere?) on
discovering "real" dependencies. Some projects just tell you what they
use without specifying version numbers or ranges. ldd may be useful
there, but that will only show you what it linked to when you built it.
If it's a "smart" build system it detects dependencies anyway. We don't
want that behavior, though.

I think, personally, if I knew more about nailing down dep versions and
handling slot dependencies, I'd feel a bit more confident writing new
ebuilds or taking on maintenance of others. Most of the syntax is
legible and understandable. Superficially, I understand slot
dependencies, but manipulating them in an older ebuild is different than
a fresh ebuild you're looking to bring to the main tree.

Thanks for bringing up the discussion.

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[Michał Górny]

On Tue, 1 Nov 2016 23:09:25 +
Sergei Trofimovich  wrote:

> On Tue, 1 Nov 2016 14:16:18 +0100
> Michał Górny  wrote:
> 
> > However, while at it I should
> > point out that the spec doesn't really cover pure-DEPEND :=, so there
> > is no guarantee that Haskell := operators will work at all.  
> 
> Yeah. It settled mostly because DEPEND="$RDEPEND + foo" is idiomatic
> at least in non ':=' world and no tools complained about ':=' in DEPEND.

No, it's allowed because of DEPEND=$RDEPEND. However, DEPEND without
RDEPEND is not guaranteed to do anything useful.

-- 
Best regards,
Michał Górny



pgpMTNBN34ush.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[Sergei Trofimovich]

On Tue, 1 Nov 2016 14:16:18 +0100
Michał Górny  wrote:

> On Tue, 1 Nov 2016 09:29:09 +
> Sergei Trofimovich  wrote:
> 
> > > 4. What are the common tasks that you find unnecessarily complex /
> > > lengthy with the current version specifications?  
> > 
> > Problem 1:
> > 
> > Sometimes upstream decides that one of their dependencies
> > is broken on a version range (package compiles but does
> > horrible things due to accidental semantic change in a dependency)
> > 
> > Common example is aeson. Upstream dependency looks like:
> > || ( ( >=aeson-0.5  >   ( >=aeson-0.11  
> So version ranges with exclusions. How many versions usually need to be
> excluded? Is this usually limited to a few versions? Are the versions
> very specific or would you need to restrict them e.g. 0.10*?

Usually the exclusions is a single range of 2 unpredictables versions:
[when_broke .. when_fixed)
and usually they can't be easily covered by '*' wildcard.
https://hackage.haskell.org/package/Agda-2.5.1.1 is a good example:
hashtables (>=1.0.1.8 && <1.2 || >=1.2.0.2 && <1.3)
# removes >=1.2 ... <1.2.0.2, today expands as: 1.2.0.0, 1.2.0.1
mtl (>=2.1.1 && <=2.1.3.1 || >=2.2.1 && <2.3)
# removes >2.1.3.1 ... <2.2.1, today expands as: 2.2, 2.2.0.1
transformers (==0.3.* || >=0.4.1.0 && <0.6)
# removes >=0.4.0.0 <0.4.1.0, today expands as: 0.4.0.0

> > Problem 2:
> > 
> > Haskell ABI it a SUM of:
> > - ABIs of its dependency libraries
> > - own exports
> > Library version is a part of that library ABI.
> > Thus library ABI is guaranteed to change every library release.
> > 
> > It means that once end user merges a library version bump
> > user needs to rebuild all library's transitive reverse dependencies.
> > Sometimes library is deep in the dependency chain (5-10 levels deep).
> > 
> > There is no way to express that style of dependency in portage:
> > https://bugs.gentoo.org/show_bug.cgi?id=449094
> > and app-admin/haskell-updater is still a needed thing to restore
> > world sanity after a library upgrade.  
> 
> Just to be clear, I was asking purely about versions and not
> slots/subslots and other magic like that.

Fair.

> However, while at it I should
> point out that the spec doesn't really cover pure-DEPEND :=, so there
> is no guarantee that Haskell := operators will work at all.

Yeah. It settled mostly because DEPEND="$RDEPEND + foo" is idiomatic
at least in non ':=' world and no tools complained about ':=' in DEPEND.

This slight deviation in DEPEND/RDEPEND representation of the same
intent, say:
aeson (>= 0.10 && < 1.2)
would have to be solved with something from the list:

1. use some cheap sed-like bash hack to strip away ':=' from RDEPEND
  when assigning it to DEPEND:
RDEPEND=">=aeson-0.10:= 

Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[Andreas K. Huettel]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512


Hi Michał, 

> 
> Therefore, I would like to ask the following questions:
> 
> 1. How often do you find '~' useful? Do you think there should be
> additional operators that ignore revision part?
>

Very useful for virtuals (especially perl virtuals :), since you can pin one 
virtual to one upstream version and allow Gentoo patch revisions.

> 
> 2. How often do you find '=...*' wildcard syntax useful? To what
> purpose do you use it? Do you find its behavior confusing [1]?
>

Mostly not so useful. However, ...

* Libreoffice versions 5.x tend to depend on one specific version of support 
libraries, e.g. libixion-0.y ... We take care to have equal visibility and 
synchronous stabilization there, otherwise dependency resolution becomes a 
mess.

* I'm using it in libreoffice-bin to pin dependency versions when I need a 
specific ABI of a library and I can't do that via a fixed :x/y subslot.

> 
> 4. What are the common tasks that you find unnecessarily complex /
> lengthy with the current version specifications?
> 

Version ranges, obviously.

> 5. Do you find any other parts of the current version dependency
> specifications confusing?

Not really.

If we do make a bigger overhaul we might revisit the DEPENDENCIES variant 
though. Better than HDEPEND, BDEPEND, ØDEPEND and €DEPEND...

Also I would really really like to see IUSE_RUNTIME implemented so we can 
finally get away cleanly from the unrealistic "no useflags for optional 
runtime deps" policy.

Cheers, Andreas

- -- 

Andreas K. Huettel
Gentoo Linux developer 
dilfri...@gentoo.org
http://www.akhuettel.de/

-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQJ8BAEBCgBmBQJYGPe4XxSAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRDMjhGQ0IwRjdCRUQxMzdBQUNCMDJEODlB
NDRDRjM3M0U3RUU5OUU0AAoJEKRM83Pn7pnkr4kQAN/GkCm6fepgPEd70O6wwyys
+aMKVZK7oCrAaKCJ+zYM5qJTmlAYVJhArgq8qgBVHeaEv9XE7zIxXdJ5NGDurow5
Dnu9tPS+lQDjyDgu05apkEK+vTzpfTQhRA7i5qbDQia6d1oJT7HKRZ0/Dcy0FUVA
fOAwc3bLnoQ+Sp9N4HZpc/bu5FIYcLL6rhq1TcJMqCiWZdCktX1TRWjbafW6ujtN
G4Yhzc3iW45615M7ioXVHIel6O5oB5RaASdOcjj9W9gQ+3PwdomU+W3nB21vxCeU
baG6kGx28XZMTYdiUxg281aF2SbqgIdT45FwR8jlN9tlMHVj9NU7z6nqY1fg9nlR
YRQ9Ee1gpvIim78qTwUzrs0kQujdUO3ZwDSt7RfEX1TJn8/Y6gGE6HHdver/o3vD
b/8b5zThSqmRTs2p/xBHTsUkDX22DOlJgNwUxuF7wtn1pegumQWhyyT2DuSSnU4I
ZJ8gr9+xoc8D7n19/6oq1N4mTaYL+INN1PRfMWvR0PzvZp5BoDvhCkAYNX8/w3RI
/AbTFPkhUO+MR3I1X3+bmoQkUSV/4LJ7Dz4b8HJ8pzoeKb64T3f3+/IPVaAjksF2
qriCecmEiagtZv2YB1peHSMm9JsOhho8k2v/oDU32teR3dWqWpW+Xj+QCdOGS4KA
fVQfnVdNmjutR9HxIIK7
=73pl
-END PGP SIGNATURE-

Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[Michał Górny]

On Tue, 1 Nov 2016 09:29:09 +
Sergei Trofimovich  wrote:

> > 4. What are the common tasks that you find unnecessarily complex /
> > lengthy with the current version specifications?
> 
> Problem 1:
> 
> Sometimes upstream decides that one of their dependencies
> is broken on a version range (package compiles but does
> horrible things due to accidental semantic change in a dependency)
> 
> Common example is aeson. Upstream dependency looks like:
> || ( ( >=aeson-0.5( >=aeson-0.11  Problem 2:
> 
> Haskell ABI it a SUM of:
> - ABIs of its dependency libraries
> - own exports
> Library version is a part of that library ABI.
> Thus library ABI is guaranteed to change every library release.
> 
> It means that once end user merges a library version bump
> user needs to rebuild all library's transitive reverse dependencies.
> Sometimes library is deep in the dependency chain (5-10 levels deep).
> 
> There is no way to express that style of dependency in portage:
> https://bugs.gentoo.org/show_bug.cgi?id=449094
> and app-admin/haskell-updater is still a needed thing to restore
> world sanity after a library upgrade.

Just to be clear, I was asking purely about versions and not
slots/subslots and other magic like that. However, while at it I should
point out that the spec doesn't really cover pure-DEPEND :=, so there
is no guarantee that Haskell := operators will work at all.

-- 
Best regards,
Michał Górny



pgp75IiURfNCW.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[Michael Orlitzky]

On 10/31/2016 08:31 PM, Michał Górny wrote:
> 
> 4. What are the common tasks that you find unnecessarily complex /
> lengthy with the current version specifications?

Slotted version ranges, for example:

  berkdb? ( || ( sys-libs/db:5.3
 sys-libs/db:5.1
 sys-libs/db:4.8
 sys-libs/db:4.7
 sys-libs/db:4.6
 sys-libs/db:4.5 ) )

The verbosity is to avoid something like

  ( >sys-libs/db-4.4 

Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[Sergei Trofimovich]

On Tue, 1 Nov 2016 01:31:55 +0100
Michał Górny  wrote:

Haskell package ecosystem (hackage) encourages
upper version bounds and strict version bounds on
dependencies: http://pvp.haskell.org/

Most of dependencies look like:
>=foo-1.3 =bar-2.1 =baz-2.1 =baz-2.9  Therefore, I would like to ask the following questions:
> 
> 1. How often do you find '~' useful? Do you think there should be
> additional operators that ignore revision part?

Haskell packages occasionally bind to very specific library version.
The set of packages is usually released together because
package author tests against that single set of versions.

 ::gentoo examples:

dev-haskell/haddock/haddock-2.16.1.ebuild:  ~dev-haskell/haddock-api-2.16.1
dev-haskell/hspec/hspec-2.2.1.ebuild:RDEPEND="~dev-haskell/hspec-core-2.2.1:=[profile?]
dev-haskell/hspec/hspec-2.2.1.ebuild:   
~dev-haskell/hspec-discover-2.2.1:=[profile?]

::haskell examples:

dev-haskell/edisoncore/edisoncore-1.2.2.1.ebuild:RDEPEND="~dev-haskell/edisonapi-1.2.2.1:=[profile?]
dev-haskell/ghc-api/ghc-api-7.10.3.ebuild:  ~dev-haskell/binary-0.7.6.1:=
dev-haskell/ghc-api/ghc-api-7.10.3.ebuild:  
~dev-haskell/transformers-0.4.3.0:=
dev-haskell/hledger-web/hledger-web-0.27.ebuild:
~dev-haskell/hledger-0.27:=[profile?]
dev-haskell/hledger-web/hledger-web-0.27.ebuild:
~dev-haskell/hledger-lib-0.27:=[profile?]
dev-haskell/hledger/hledger-0.27.ebuild:
~dev-haskell/hledger-lib-0.27:=[profile?]
dev-haskell/hprotoc/hprotoc-2.4.0.ebuild:   
~dev-haskell/protocol-buffers-2.4.0:=[profile?]
dev-haskell/hprotoc/hprotoc-2.4.0.ebuild:   
~dev-haskell/protocol-buffers-descriptor-2.4.0:=[profile?]
dev-haskell/llvm-general/llvm-general-3.5.1.2.ebuild:RDEPEND="~dev-haskell/llvm-general-pure-3.5.1.0:=[profile?]
dev-haskell/protocol-buffers-descriptor/protocol-buffers-descriptor-2.4.0.ebuild:RDEPEND="~dev-haskell/protocol-buffers-2.4.0:=[profile?]
dev-haskell/yarr/yarr-1.4.0.2.ebuild:   
~dev-haskell/missing-foreign-0.1.1:=[profile?]
dev-lang/elm-make/elm-make-0.17.1-r1.ebuild:~dev-lang/elm-compiler-0.17.1:=
dev-lang/elm-meta/elm-meta-0.17.1.ebuild:   ~dev-lang/elm-compiler-${PV}
dev-lang/elm-meta/elm-meta-0.17.1.ebuild:   ~dev-lang/elm-make-${PV}
dev-lang/elm-meta/elm-meta-0.17.1.ebuild:   ~dev-lang/elm-package-${PV}
dev-lang/elm-meta/elm-meta-0.17.1.ebuild:   ~dev-lang/elm-reactor-${PV}
dev-lang/elm-meta/elm-meta-0.17.1.ebuild:   ~dev-lang/elm-repl-${PV}
dev-lang/elm-package/elm-package-0.17.1-r2.ebuild:  
~dev-lang/elm-compiler-0.17.1:=[profile?]
dev-lang/elm-reactor/elm-reactor-0.17.1.ebuild: ~dev-lang/elm-compiler-0.17.1:=
dev-lang/elm-reactor/elm-reactor-0.17.1.ebuild: ~dev-lang/elm-make-${PV}
dev-lang/elm-repl/elm-repl-0.17.1.ebuild:   ~dev-lang/elm-compiler-0.17.1:=
sci-mathematics/agda/agda-.ebuild:  ~dev-haskell/edisonapi-1.3:=[profile?]

> 2. How often do you find '=...*' wildcard syntax useful? To what
> purpose do you use it? Do you find its behavior confusing [1]?

haskell used to optimise ranged dependencies of style
>=foo-2 && =foo-2 &&  4. What are the common tasks that you find unnecessarily complex /
> lengthy with the current version specifications?

Problem 1:

Sometimes upstream decides that one of their dependencies
is broken on a version range (package compiles but does
horrible things due to accidental semantic change in a dependency)

Common example is aeson. Upstream dependency looks like:
|| ( ( >=aeson-0.5 =aeson-0.11 =aeson-0.5:= 

Re: [gentoo-dev] [RFC] Problems and limitations of the current version dependency specs

[Ian Stakenvicius]

On 31/10/16 08:31 PM, Michał Górny wrote:
> Hello, everyone.
> 
> I would like to work on a major version depedencny specification
> improvements as part of the next EAPI. For this reason, I'd like to
> first gather some research on how developers are using the current
> system, what they find efficient and what they find cumbersome.
> 
> Therefore, I would like to ask the following questions:
> 
> 1. How often do you find '~' useful? Do you think there should be
> additional operators that ignore revision part?

The only time I have used this is when I've needed to synchronize the
version of two inter-dependent packages, while wanting to allow the
revision portion to vary.

I have generally found it a lot more useful to use this in my various
/etc/portage/ files than in ebuilds, though, especially
package.accept_keywords


> 
> 2. How often do you find '=...*' wildcard syntax useful? To what
> purpose do you use it? Do you find its behavior confusing [1]?

I find that this syntax is rather useful and often is necessary, due
to it being difficult to properly define both minimum and maximun
versions otherwise at times -- especially when I want to use the := or
:0= slot operator.  I don't find its behavior confusing once I became
educated on how it actually works; I do recognize that it *is not*
intuitive to the layman.


> 3. Do you sometimes find yourself using '<'/'<=' specs that
> accidentally match _pre/_rc/... versions?

No, but that is likely just because I tend not to deal with
dependencies that have _pre and _rc suffixes.


> 
> 4. What are the common tasks that you find unnecessarily complex /
> lengthy with the current version specifications?

It is cumbersome to deal with limiting slots to a specific subset,
when wanting to leverage slot-operator rebuilds right now --
effectively you need to have two atoms, one with :=, and a second set
||()'d with each of the slots that are compatible.


> 5. Do you find any other parts of the current version dependency
> specifications confusing?

The as yet still impossible means of dealing properly with multiple
packages that should either independently or when switching from one
to another trigger a (sub)slot rebuild is quite vexing at times.





signature.asc
Description: OpenPGP digital signature