Re: cmdliner 1.0.0 uses topkg for building

[Stéphane Glondu]

On 18/07/2017 15:11, Daniel Bünzli wrote:

As for automation, it remains to be seen whether we want to map 1 opam package 
to 1 Debian
package, or to map 1 git repo containing many opam packages to 1 Debian 
package. The downside
of the first approach is similar to the above points. Also the overhead of 
Debian packaging
is more than opam packaging (it takes care of more things) so if people want to 
do left-pad
in ocaml it might be better to group those together in Debian. The second 
approach would
be easy to automate in the case that whole-git-repos do not form cyclic 
dependencies
with other whole-git-repos.


I don't understand why you are talking about git repos. Tarballs are the source 
of releases. I'm not a debian packager but I also see a lot of packages in 
debian itself that I presume are based on the same tarballs (e.g. dev and 
non-dev) packages.


In Debian, we have the notion of "source packages" (upstream tarball + 
debian directory) and "binary packages" (.deb files). The compilation of 
one source package produces one or several binary packages. What you see 
with dev and non-dev packages is two binary packages generated by the 
same source package and indeed it is very common.


For topkg and topkg-care, we would need two source packages with the 
same tarball, which is not common in Debian (I don't know if it even 
exists) and was what was bothering Hendrik. In my own opinion, it's fine.



- If you release {tokpg,topkg-care} v0.8 then v0.9, but in v0.9 topkg-care 
actually
did not change at all and was exactly the same as in v0.8, this makes it harder 
to resolve
dependencies. Something that uses topkg-care >= v0.9 can actually be satisified 
by
v0.8, but opam wouldn't know this.


You are trying to make things more complex than they are here. topkg and 
topkg-care must always be used with identical versions. This is enforced in the 
opam files which are machine readable and from which you can infer your 
constraints.


Lockstep upgrades are frowned upon in general in Debian, but of course 
they are supported. In the specific case of topkg and topkg-care, I 
don't think there would be any practical problem.



Cheers,

--
Stéphane

Re: cmdliner 1.0.0 uses topkg for building

[Hendrik Tews]

Daniel Bünzli  writes:

>> I thought topkg-care is needed to run the tests, for instance,
>> for cmdliner? 
>
> In fact no, the `topkg` tool is really just a convenience as far as

OK, I'll keep this in mind and will have a closer look when I
look at the tests.

Hendrik

Re: cmdliner 1.0.0 uses topkg for building

[Daniel Bünzli]

On 18 July 2017 at 23:16:03, Hendrik Tews (hend...@askra.de) wrote:
> Everybody,
>  
> I am a bit shocked that my simple question about how to best
> package topkg and topkg-care lead to such a heated and quite
> personal discussion. I would rather prefer to have a friendly and
> constructive relation to our upstream providers and to base
> discussions on technical arguments.

I'm sorry about this. I think we need to get over this now.

> More often than not upstream
> providers will -often for good reason- not be convinced by our
> arguments.

Maybe, but I think it still good that expectations and their reasons are 
actually explained and that more communication occurs in general. If the point 
of view of system packagers is never represented in the platform level 
discussions it's unlikely we'll do something upstream that naturally satisfies 
you (or at least makes your life easier). The goals of a developer oriented 
package system and a system one are certainly different but they do share quite 
a bit of their means and procedures and we should make sure that at least part 
of the work done upstream can be reused downstream in an efficient manner.


> I thought topkg-care is needed to run the tests, for instance,
> for cmdliner? 

In fact no, the `topkg` tool is really just a convenience as far as building 
and testing is concerned. It simply forwards to the `pkg/pkg.ml` file of the 
package which only needs the `topkg` package to function. This section [0] of 
the documentation has the details for building and running the tests of a 
`topkg` package.

Best, 

Daniel

[0] http://erratique.ch/software/topkg/doc/Topkg#build

Re: cmdliner 1.0.0 uses topkg for building

[Hendrik Tews]

Everybody,

I am a bit shocked that my simple question about how to best
package topkg and topkg-care lead to such a heated and quite
personal discussion. I would rather prefer to have a friendly and
constructive relation to our upstream providers and to base
discussions on technical arguments. More often than not upstream
providers will -often for good reason- not be convinced by our
arguments.

Daniel Bünzli  writes:

> On 18 July 2017 at 10:42:51, Hendrik Tews (hend...@askra.de) wrote:
>
>> Preparing Debian packages would of course be much more
>> straightforward, if topkg and
>> topkg-care would not share the same source tarball ;-)
>
> I am a bit curious on why this is the case exactly. What kind of
> difficulties does it specifically bring ?

The "much more straightforward" was certainly an exaggeration. I
should have written "easier for me" instead.

The difficulty is just that I have not yet encountered a
situation that we have with topkg and topkg-care and that I don't
know yet what would be the preferred way to package both.

> In any case I suspect packaging topkg-care is not going to be useful

I thought topkg-care is needed to run the tests, for instance,
for cmdliner? If I am not mixing things up here, then it would
certainly be interesting to package topkg-care eventually to
enable Debian continuous integration to run these tests too.

But I agree that it is much more urgent to package topkg.

> More generally. Always feel free to ping me in when you encounter

Thanks a lot, this is certainly appreciated!

> It would somehow be sad if all this work was not able to
> simplify the work of system packagers downstream by being able to
> semi-automate the creation of your packages via cuts in the package
> universe of the OCaml opam repository.

I also expect that packaging opam packages for Debian should be
simpler, but I have no experience yet. I also thought about
automation already, but I first need some manual experience.

Best regards,

Hendrik

Re: cmdliner 1.0.0 uses topkg for building

[Daniel Bünzli]

On 18 July 2017 at 14:09:36, Ximin Luo (infini...@debian.org) wrote:

> In Debian they have to go in two separate source packages, because the 
> distributed nature  
> of package building means (if it were in a single source package) we can't 
> run the build  
> for topkg, run the build for cmdliner (another package), then run the build 
> for topkg-care.

I don't think anyone suggested that. If you do not want to follow opam's exact 
packaging structure it is your duty to figure out how you want to handle that 
rather than come on the upstream issue tracker shouting that the build 
structure is circular.


> Then, maintenance of two identical source packages requires more work than 
> maintaining  
> two different source packages:

They should of course not be treated as *identical*. It seems that most of the 
problem you mention come from 
that idea.


> As for automation, it remains to be seen whether we want to map 1 opam 
> package to 1 Debian 
> package, or to map 1 git repo containing many opam packages to 1 Debian 
> package. The downside  
> of the first approach is similar to the above points. Also the overhead of 
> Debian packaging  
> is more than opam packaging (it takes care of more things) so if people want 
> to do left-pad  
> in ocaml it might be better to group those together in Debian. The second 
> approach would  
> be easy to automate in the case that whole-git-repos do not form cyclic 
> dependencies  
> with other whole-git-repos. 

I don't understand why you are talking about git repos. Tarballs are the source 
of releases. I'm not a debian packager but I also see a lot of packages in 
debian itself that I presume are based on the same tarballs (e.g. dev and 
non-dev) packages.


> - If you release {tokpg,topkg-care} v0.8 then v0.9, but in v0.9 topkg-care 
> actually  
> did not change at all and was exactly the same as in v0.8, this makes it 
> harder to resolve  
> dependencies. Something that uses topkg-care >= v0.9 can actually be 
> satisified by  
> v0.8, but opam wouldn't know this.

You are trying to make things more complex than they are here. topkg and 
topkg-care must always be used with identical versions. This is enforced in the 
opam files which are machine readable and from which you can infer your 
constraints.


> I could construct more examples by spending more time, but they all point to 
> the general  
> principle that this sort of arrangement is really not worth it. On your side, 
> what do you  
> think you gain from it?

This is explained here: 

https://github.com/dbuenzli/topkg/blob/master/DEVEL.md

It is not my intention to multiply packages that behave like this and I 
actually remain largely unconvinced using this solution for replacing depopts, 
a move that is being promoted now in opam but that I don't see as a viable long 
term solution. However this is sometimes needed for being able to bootstrap 
OCaml infrastructure tools that are not part of OCaml but very early in the 
whole dependency cone.


> Would you like to provide us with a list of your "trigger phrases" such as 
> "circular dependency"  
> and "intuition" that cause you to completely shut down, assume the person 
> writing it  
> is "out to get you", and go into panic-defense-mode?

It should be entirely sufficient for you to try to improve on your 
interpersonal communication skills. As far as I'm concerned, given your style 
and behaviour yesterday on topkg's issue tracker, I'm no longer interested in 
interacting with you for the time being.

Best, 

Daniel

Re: cmdliner 1.0.0 uses topkg for building

[Ximin Luo]

Daniel Bünzli:
> On 18 July 2017 at 10:42:51, Hendrik Tews (hend...@askra.de) wrote:
> 
>> Preparing Debian packages would of course be much more straightforward, if 
>> topkg and
>> topkg-care would not share the same source tarball ;-)
> 
> I am a bit curious on why this is the case exactly. What kind of difficulties 
> does it specifically bring ?
> 

In Debian they have to go in two separate source packages, because the 
distributed nature of package building means (if it were in a single source 
package) we can't run the build for topkg, run the build for cmdliner (another 
package), then run the build for topkg-care. Then, maintenance of two identical 
source packages requires more work than maintaining two different source 
packages:

- for developers, to figure out which files "actually belong" to a particular 
package, in the cases where you'd like to install them to places that opam 
doesn't yet cover.

- for users and reviewers, to figure out why certain files are being ignored as 
part of the build

- bug reports that affect the source package (such as metadata), may or may not 
have to be duplicated to both packages.

- if the package has to be patched for urgent bugfixes or security fixes, the 
person handling this has to figure out whether they have to patch both packages 
or just one. (assuming the patch is in the form of commit(s) to the whole git 
repo.) patching both packages and re-uploading them and rebuilding its 
dependent packages involves more work, so it's not just a "might as well patch 
both" situation.

As for automation, it remains to be seen whether we want to map 1 opam package 
to 1 Debian package, or to map 1 git repo containing many opam packages to 1 
Debian package. The downside of the first approach is similar to the above 
points. Also the overhead of Debian packaging is more than opam packaging (it 
takes care of more things) so if people want to do left-pad in ocaml it might 
be better to group those together in Debian. The second approach would be easy 
to automate in the case that whole-git-repos do not form cyclic dependencies 
with other whole-git-repos. It would not be easy to automate in the case of 
topkg. In general, ignoring opam and Debian packages altogether, one does not 
usually see build instructions like "run ./configure, run make partA, git clone 
project Y, cd Y && make && cd .., run make partB, etc etc". 

The shared version history can make things harder to upgrade.

- If you release {tokpg,topkg-care} v0.8 then v0.9, but in v0.9 topkg-care 
actually did not change at all and was exactly the same as in v0.8, this makes 
it harder to resolve dependencies. Something that uses topkg-care >= v0.9 can 
actually be satisified by v0.8, but opam wouldn't know this.

- Depending on the situation we might be forced to upgrade cmdliner at the same 
time, e.g. if topkg-care 0.9 depends on cmdliner >= 1.2, topkg = 0.9, then we 
would be forced to upgrade cmdliner when upgrading topkg to v0.9, and 
everything else that can't use an older cmdliner. If the versions were split, 
we could upgrade topkg 0.9 independently of topkg-care. Of course in many cases 
there is no problem, but in specific cases it causes pain. If you like, you can 
make topkg-care 0.9 depend on >= topkg 0.9 instead of = 0.9, then you can take 
on the burden of tracking the compatibility yourself as the version numbers 
increase. I suggested that it would be easier to do this, if the version number 
of topkg-care was a totally different number from topkg, but you seem not to 
have read that part of what I wrote.

I could construct more examples by spending more time, but they all point to 
the general principle that this sort of arrangement is really not worth it. On 
your side, what do you think you gain from it?

> In any case I suspect packaging topkg-care is not going to be useful as it is 
> a tool to help with the bureaucracy of making and distributing releases and 
> submitting them to the OCaml opam respository (see e.g. here [1] for a 
> high-level description). [..]

I also recommended (in a previous email) ignoring topkg-care for the time being 
in Debian. I'm glad you agree with this approach.

> More generally. Always feel free to ping me in when you encounter 
> difficulties packaging my software. Besides we are keen in general in hearing 
> more about system packagers and their needs on the ocaml platform mailing 
> list [2] or whichever medium you may find fitting, opam or OCaml opam 
> repository issue tracker or discuss.opam.org or the ocaml mailing list. 
> 
> [..]

Would you like to provide us with a list of your "trigger phrases" such as 
"circular dependency" and "intuition" that cause you to completely shut down, 
assume the person writing it is "out to get you", and go into 
panic-defense-mode?

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Re: cmdliner 1.0.0 uses topkg for building

[Daniel Bünzli]

On 18 July 2017 at 10:42:51, Hendrik Tews (hend...@askra.de) wrote:

> Preparing Debian packages would of course be much more straightforward, if 
> topkg and
> topkg-care would not share the same source tarball ;-)

I am a bit curious on why this is the case exactly. What kind of difficulties 
does it specifically bring ?

In any case I suspect packaging topkg-care is not going to be useful as it is a 
tool to help with the bureaucracy of making and distributing releases and 
submitting them to the OCaml opam respository (see e.g. here [1] for a 
high-level description). Anyone who needs this will certainly be using opam 
where this can be easily installed. Packaging `topkg` itself should be easy as 
it has no dependency beyond ocaml, ocamlbuild and ocamlfind and should be 
generally useful as it became the dependency of quite a number of packages 
(~200 according to `opam list --depends-on topkg | wc -l`).

More generally. Always feel free to ping me in when you encounter difficulties 
packaging my software. Besides we are keen in general in hearing more about 
system packagers and their needs on the ocaml platform mailing list [2] or 
whichever medium you may find fitting, opam or OCaml opam repository issue 
tracker or discuss.opam.org or the ocaml mailing list. 

A great deal of work and effort is being poured into opam and the OCaml opam 
repository. This includes careful metadata linting, continuous integration with 
revdep checking and specifying external dependencies on system packages via the 
depext field of opam files (see [3]). It would somehow be sad if all this work 
was not able to simplify the work of system packagers downstream by being able 
to semi-automate the creation of your packages via cuts in the package universe 
of the OCaml opam repository.

Best, 

Daniel

[1] http://ocamllabs.io/projects/2017/02/23/topkg.html
[2] http://lists.ocaml.org/listinfo/platform
[3] https://github.com/ocaml/opam-depext

Re: cmdliner 1.0.0 uses topkg for building

[Hendrik Tews]

Daniel Bünzli  writes:

> Please note that there is no circular dependency. There are two
> different opam packages: topkg and topkg-care with different build
> instructions and each package should be built on its own, as done in
> opam. See the topkg.opam and topkg-care.opam files for a precise
> definition of the dependencies and the build instructions.

Yes, I saw that there is not a real circular build dependency and
I saw how you packaged it for opam. Preparing Debian packages
would of course be much more straightforward, if topkg and
topkg-care would not share the same source tarball ;-) But I
assumed from the beginning that you might have good reasons for
not separating the two things.

Making it two packages -as in opam- is the obvious option. I only
brought up the topic here to find out if there are any other
sensible options. For instance, tagging the build dependency with
.

Hendrik

Re: Re: cmdliner 1.0.0 uses topkg for building

[Daniel Bünzli]

Hello Hendrik, 

Please note that there is no circular dependency. There are two different opam 
packages: topkg and topkg-care with different build instructions and each 
package should be built on its own, as done in opam. See the topkg.opam and 
topkg-care.opam files for a precise definition of the dependencies and the 
build instructions.

Best, 

Daniel

Re: cmdliner 1.0.0 uses topkg for building

[Ximin Luo]

Hendrik Tews:
> 
>>> I just see:  topkg depends on Fmt, Logs, Bos, Cmdliner and
>>> opam-format. Out of this only cmlliner is currently in Debian,
>>> right?
> 
> There is some kind of circular build dependency:
> 
> - topkg contains the topkg library and the topkg-care tool
> - building the topkg library requires nothing
> - building topkg-care requires cmdliner
> - building cmdliner requires the topkg library
> 
> How should I break this cycle? My idea is to duplicate topkg into
> topkg-lib and topkg-care, where topkg-lib builds the library only
> and topkg-care builds the topkg-care tool only. Both packages
> have the same upstream sources but different build dependencies.
> 
> Is there a better way to deal with this kind of circular build
> dependencies?
> 

I've filed a bug here: https://github.com/dbuenzli/topkg/issues/115

For the time being, it might be possible to ignore topkg-care, it sounds like a 
developer "maintenance" tool that shouldn't be needed to build other things 
with. I don't have it installed in any of my opam switches, but plenty of 
things I use, do depend on topkg.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Re: cmdliner 1.0.0 uses topkg for building

[Hendrik Tews]

>> I just see:  topkg depends on Fmt, Logs, Bos, Cmdliner and
>> opam-format. Out of this only cmlliner is currently in Debian,
>> right?

There is some kind of circular build dependency:

- topkg contains the topkg library and the topkg-care tool
- building the topkg library requires nothing
- building topkg-care requires cmdliner
- building cmdliner requires the topkg library

How should I break this cycle? My idea is to duplicate topkg into
topkg-lib and topkg-care, where topkg-lib builds the library only
and topkg-care builds the topkg-care tool only. Both packages
have the same upstream sources but different build dependencies.

Is there a better way to deal with this kind of circular build
dependencies?

Hendrik

Re: cmdliner 1.0.0 uses topkg for building

[Ximin Luo]

Stéphane Glondu:
> On 16/07/2017 22:15, Hendrik Tews wrote:
 IMO, a proper solution would require packaging topkg and, in the
 long run, also odig. What do you think?
>>>
>>> I agree. Feel free to do it! I'm not planning to do it right now, unless
>>> it is absolutely necessary for the transition.
>>
>> I just see:  topkg depends on Fmt, Logs, Bos, Cmdliner and
>> opam-format. Out of this only cmlliner is currently in Debian,
>> right?
> 
> Right.
> 
> This balkanization of OCaml packages is horrifying.
> 

It might be worth writing an automated tool to convert a github repo containing 
an opam package, into a Debian package. We're doing that for rust with the 
debcargo(1) tool.

licensecheck can also help to generate d/copyright files by scanning file 
headers, even if it's not in the opam metadata.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Re: cmdliner 1.0.0 uses topkg for building

[Stéphane Glondu]

On 16/07/2017 22:15, Hendrik Tews wrote:

IMO, a proper solution would require packaging topkg and, in the
long run, also odig. What do you think?


I agree. Feel free to do it! I'm not planning to do it right now, unless
it is absolutely necessary for the transition.


I just see:  topkg depends on Fmt, Logs, Bos, Cmdliner and
opam-format. Out of this only cmlliner is currently in Debian,
right?


Right.

This balkanization of OCaml packages is horrifying.


Cheers,

--
Stéphane

Re: cmdliner 1.0.0 uses topkg for building

[Hendrik Tews]

Stéphane Glondu  writes:

>> IMO, a proper solution would require packaging topkg and, in the
>> long run, also odig. What do you think?
>
> I agree. Feel free to do it! I'm not planning to do it right now, unless
> it is absolutely necessary for the transition.

I just see:  topkg depends on Fmt, Logs, Bos, Cmdliner and
opam-format. Out of this only cmlliner is currently in Debian,
right?

Hendrik

Re: cmdliner 1.0.0 uses topkg for building

[Hendrik Tews]

>> IMO, a proper solution would require packaging topkg and, in the
>> long run, also odig. What do you think?
>
> I agree. Feel free to do it! I'm not planning to do it right now, unless
> it is absolutely necessary for the transition.

OK, unless you have another suggestion how I could contribute, I
will look into topkg.

Hendrik

Re: cmdliner 1.0.0 uses topkg for building

[Stéphane Glondu]

On 15/07/2017 22:17, Hendrik Tews wrote:
> I looked into updating the cmdliner package for the new upstream
> version 1.0.0. This new version uses topkg
> (http://erratique.ch/software/topkg) for building and it seems
> that the documentation is generated with odig
> (http://erratique.ch/software/odig). I can neither find topkg nor
> odig in Debian.

The new upstream of ocaml-base64 build depends on topkg, too. I guess
other Daniel Bünzli's packages (jsonm, xmlm, uutf...) too. That's why I
delayed their update in my uploads of today.

> One could probably hack around the missing topkg with a
> customized ocamlbuild command in debian/rules and drop the
> documentation (which is available online).
> 
> IMO, a proper solution would require packaging topkg and, in the
> long run, also odig. What do you think?

I agree. Feel free to do it! I'm not planning to do it right now, unless
it is absolutely necessary for the transition.


BTW, I've updated my repo:

  http://ocaml.debian.net/debian/ocaml-4.05.0%2Brc1/pool/

with logs/packages following what I've uploaded today to unstable
(mostly adding ocamlbuild to bdeps). The most unpleasant "case" I've
seen so far is labltk (because it requires many fixes and has many rdeps).


Cheers,

-- 
Stéphane