Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Enrico Weigelt, metux IT consult]

On 11.04.19 09:44, Mo Zhou wrote:

> Different from that, duprkit's design don't hope to limit> the user with any 
> pre-defined "sequence", but enable the users to>
selectively call the functions they need. In other words, the> user can
define how to deal with the prepared source+debian directories,>
afterall the .durpkg header is a shell script. That said, I think> some
more helper functions would be nice: [1].
I'm still struggling to understand why simply using old-fashioned
./debian/rules file (possibly w/o using debhelper+friends) isn't
sufficient here.

AFAIK, all that tools like buildd do for actual build is calling that
rule file w/ some specific target names (eg. 'binary'). You can put in
anything you like here - it's just a makefile. Theoretically, you could
also use shellscript instead.

If you drop the idea of having everything in a single file in favour
of debian trees (= something that has the 'debian' subdirectory with
a 'rules' file in it), the existing debian toolchains could be used.

> My blueprint includes a very light-weight/simple dependency tracking> 
> mechanism. And I assume the project don't need to handle complex dep>
trees like apt. Because:> > 1. If a package is common and useful enough,
such that it has been>adopted by many other projects, why don't I
package it for the>official Debian release? So, I expect that most
packages that DUPR>will deal with, are actually leaf or near-leaf
packages on the>dependency tree.
Okay, that's a different topic. We have three options here:

a) put it into official debian repo. that would go the usual ways, but
   takes pretty long, until the next release is out and the desired
   audience actually uses it.

b) add it to backports repos. i'm not sure how the actual workflows
   and release timelines look here.

c) go the PPA route. here we'd need some repo-level dependency handling
   (not sure what tools exist here), and we'd have to coordinate between
   several PPAs

> 2. Some of my targeted upstreams do sourceless binary tarball release.>
> They seldom get into the dependency trouble...

When I have to touch those stuff, I basically always run into trouble.
Many subtle breaks, that are extremly hard to resolve (even to track
down). Those stuff I'm only doing in containers. Binary-only stuff is
not trustworthy at all, so it really should be strictly confined.

Those vendors (eg. Microsoft/Skype) also like to mess w/ package manager
configuration, have implicit dependencies like silly Lennartware, etc.
I never ever run such crap outside a strictly confined container.

One of the worst things I've ever seen is coming from National
Instruments (which don't support Debian anyways, just ancient RHEL)
Traditionally they only provided ridiculous installer programs
(just like they're used to from the dillettantic Windows world)
that do lots of really weird things, even messing w/ the kernel
(yeah, they still insist on binary-only kernel modules, that's
always broken-by-design).

Somewhere in last summer they learned what package repos are for,
well, just partially learned. They now messed w/ the repo configs and
installed a globally trusted package source with explicitly disabled
authentication and plain http. Boom - 0day !

Due to their long history of hostility, total bullshit and censorship in
their own "community", I've posted that @full-disclosure (even goverment
institutions like BSI called by for interviews on that matter - their
products also run in critical infrastructure like power plants). Again
it took several month for the issue to be migitated by NI.

> 3. Inserting a DUPR package into the near-root part of the Debian>
> dependency tree is, generally speaking, a terrible idea.>Only
those who aware of what they are doing will do this.
ACK. Those stuff belongs into throwaway-containers.

> The `bin/duprCollector` will collect meta information from a collection> (and 
> will form a dependency tree in the future). I have no plan to>
rethink about the "get-orig-source" target since there are ... lots> of
weird upstreams in my list...
Maybe we should talk about some of these cases, to get a better idea.

In general, we IMHO should rethink the workflows for creating the actual
buildable debian tree from upstream releases (in many packages that's
still pretty manual and hackish)


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Enrico Weigelt, metux IT consult]

On 10.04.19 16:56, Helmut Grohne wrote:

Hi,

> I looked into this. Your reasons are sound and you are scratching your> itch. 
> This is great.
ACK. It's always good when people make their hands dirty and work on
solving actual problems. Even if the actual output (=code, etc) finally
doesn't get wide use or even thrown away completely, we still a lot
that way.

When I look back into my earlier years, I've written lots of things that
never have been actually used, but I've learned a lot that way.

> Your implementation goes straight from .durpkg -> .deb. I question this> 
> decision: We already have lots of tools to go from .dsc to .deb. Your>
implementation replicates part of that and I think, this is bad as it>
makes it harder to collaborate.
I did a similar decision w/ dck-buildpackage, because I came to the
conclusion that intermediate steps via dsc are just unncessary
complexity and slowdown. But the motivation of dck-buildpackage was
getting rid of complicated and cumbersome things like pbuilder.

So, I can understand to his decision - he probably doesn't need anything
from the dsc-based tools, as he's operating in a completely different
scope.

> Let me propose a rather intrusive interface change to duprkit. What if> the 
> result of building a .durpkg was a .dsc rather than a .deb? Then
you> could split duprkit into two tools:> >  * One tool to build source
packages from .durpkg files on demand.>  * One tool to build a specific
binary package from a given deb-src>repository.
Let me propose an even more consequent approach: let it operate even one
step earlier in the pipeline by just generating a debianized source
tree. You could then use the tool of your choice to create dsc from that
and put in whatever kind of pipeline you prefer. My personal choice here
would be dck-buildpackage, and my infrastructure ontop of that.

By the way, this opens up another common topic: how do we get from an
upstream tree (in git repo) to a debianzed source tree w/ minimal manual
efforts ?

> Now in principle, the latter is simply sbuild or pbuilder, but there is> more 
> to it:>  * Given the binary package name, figure out which source
package must>be built.

Yet another tricky issue. The primary data source for that usually are
the control files. But they also somethimes are autogenerated.

Could we invent some metadata language for this, that also can handle
tricky cases like the kernel ?

>  * Given that source package's Build-Depends, figure out what other>
> binary packages need to be built.>  * Recurse.>  * Build them in a
suitable order.

You're talking about building whole overlay repos ?
Then I might have something for you:

https://github.com/metux/deb-pkg

Note: it's still pretty hackish and needs some local per-project
customizations. Haven't had the time to make some general purpose
standalone package of it.

I'm just using it for building per private extra repos for my customers.

If anybody likes to join in and turn it into some general purpose
package, let's talk about that in a different thread. The first step
would be creating a decent command line interface (for now, the run-*
scripts are just project-specific dirty hacks to save me from typing
too much ;-)).

> (Some people will observe that this is the "bootstrap" problem. ;)

Not really bootstrap problem, but depenency problem. Easier to solve :p

> There is one major difficulty here (and duprkit doesn't presently solve> that 
> either): If you figure that some binary package is missing, you>
have no way of knowing which .durpkg file to build to get the relevant>
source package.

Yes, he'd have to reinvent the dependency handling. This is one of the
points that let me question the whole approach and favour completely
different approaches like classic containers.

> Now let's assume that you do want to allow complex dependencies in this
> user repository. In this case, it would make sense to trade .durpkg
> files for plain "debian" directories with an additional debian/rules
> target to obtain the source. (We removed "get-orig-source" from policy a
> while ago, but maybe this is what you want here.)

Sounds a good idea.

Maybe we should put this to a broader discussion, along w/ the control
file generation problem. My desired outcome of that would be a generic
way for fully automatically building everything from a debianzed source
tree (eg. git repo) within a minimal container/jail, w/o any other extra
configuration outside that source tree - even for those cases where the
control file needs to be generated, which again needs some deps.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Shengjing Zhu]

On Sun, Apr 14, 2019 at 4:02 AM Thomas Goirand  wrote:
>
> On 4/8/19 7:16 PM, Shengjing Zhu wrote:
> >> from PPA (source+binary-based).
> >
> > If people just want a PPA which supports Debian, please just take a
> > look at OBS[1].
> >
> > I've seen many upstreams provide packages with OBS, and most
> > distributions are supported.
> > Not only deb, but also rpm, from Debian/Ubuntu to OpenSuse/Fedora, and
> > even Archlinux, in a uniform way.
> >
> > [1] https://build.opensuse.org/
>
> I don't see how OBS may help us. The proposal from Ganneff about
> bikesheds was much nicer (ie: using our already existing buildd and FTP
> infrastructure, and keeping our quality standard), than then path this
> thread is taking.
>
> If we are to propose sub-standard PPA like repositories, I would myself
> try to avoid them.

Who is "us"? If they are Debian {Developer, Maintainer, Contributor},
why they ever want to use PPA and its alternatives?

And I don't think OBS is much different from PPA, except one supports
build against Debian archive, another just Ubuntu.

PS, OBS here refers to the free service, not the OBS software.

-- 
Shengjing Zhu

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Thomas Goirand]

On 4/8/19 7:16 PM, Shengjing Zhu wrote:
>> from PPA (source+binary-based).
> 
> If people just want a PPA which supports Debian, please just take a
> look at OBS[1].
> 
> I've seen many upstreams provide packages with OBS, and most
> distributions are supported.
> Not only deb, but also rpm, from Debian/Ubuntu to OpenSuse/Fedora, and
> even Archlinux, in a uniform way.
> 
> [1] https://build.opensuse.org/

I don't see how OBS may help us. The proposal from Ganneff about
bikesheds was much nicer (ie: using our already existing buildd and FTP
infrastructure, and keeping our quality standard), than then path this
thread is taking.

If we are to propose sub-standard PPA like repositories, I would myself
try to avoid them.

Cheers,

Thomas Goirand (zigo)

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Paul Wise]

On Fri, Apr 12, 2019 at 3:27 PM Simon McVittie wrote:

> Flatpak treats /usr as immutable (with the exception of mounting
> "extensions" on pre-prepared empty directories) and mounts it read-only in
> the container. If it didn't, it wouldn't be able to use content-addressed
> storage (the storage can't be content-addressed if the content changes).
> In the usual way to use Flatpak, with a shared runtime for a family of
> related packages like "the GNOME apps", apps would end up accidentally
> or deliberately modifying each other's runtimes if they weren't read-only.

I expected the Flatpak /app directory to also be entirely read-only,
are there parts of /app that are not read-only?

> If the user-facing leaf package is really installed in the runtime, with
> a different runtime for each user-facing leaf package, you're right that
> the Flatpak app could mostly contain symlinks into /usr. A few metadata
> and integration files that get "exported" to the host system (such as
> .desktop files and icons) would have to be copied instead of symlinked,
> because the host system needs to be able to read those without entering
> the container.

It sounds like my idea might be a viable way to generate automatically
Flatpaks directly from leaf Debian packages then, thanks for the info.
I might at some point take a look at adding such a mode to flatdeb,
would you accept having that as an option?

> flatdeb installs apt/dpkg/essential, but then deletes apt, dpkg and ...

Yeah, I noticed that so I was glad when mmdebstrap's sub-essential
stuff came along.

> The matching SDK runtime that is used to compile apps

I don't know enough about SDKs in the Flatpak world, would they
contain something like the Build-Depends and their recursive deps?

> Instead of using sudo or ssh to a remote (usually virtual) machine to
> get root so that it can run debootstrap and dpkg, flatdeb now runs debos
> on the local machine.  This means it's root in debos' temporary qemu VM,
> and doesn't need privileges other than /dev/kvm access in the host system.

That sounds like a much better design.

> In principle there'd be nothing to stop debos from using something
> like user-mode-linux as an alternative to qemu/KVM.

I guess you could also use container tools like systemd-run with user
namespaces (once those are enabled by default)?

> One per desktop task, plus one for the Priority >= standard default
> CLI installation, would be quite a lot of data but could be good for
> bisecting, yes. I'd suggest doing this with debos, which has built-in
> support for committing trees to libostree and doesn't need root on the
> host system.

Cool, thanks for the information :)

--
bye,
pabs

https://wiki.debian.org/PaulWise

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Simon McVittie]

On Fri, 12 Apr 2019 at 10:49:57 +0800, Paul Wise wrote:
> Is there any reason that making /app a
> symlink to /usr (or a directory containing only links to /usr)
> wouldn't work inside Flatpak packages?

Flatpak treats /usr as immutable (with the exception of mounting
"extensions" on pre-prepared empty directories) and mounts it read-only in
the container. If it didn't, it wouldn't be able to use content-addressed
storage (the storage can't be content-addressed if the content changes).
In the usual way to use Flatpak, with a shared runtime for a family of
related packages like "the GNOME apps", apps would end up accidentally
or deliberately modifying each other's runtimes if they weren't read-only.

If the user-facing leaf package is really installed in the runtime, with
a different runtime for each user-facing leaf package, you're right that
the Flatpak app could mostly contain symlinks into /usr. A few metadata
and integration files that get "exported" to the host system (such as
.desktop files and icons) would have to be copied instead of symlinked,
because the host system needs to be able to read those without entering
the container.

> I had planned to do this using
> mmdebstrap, which now offers installation of Debian systems that don't
> have apt/dpkg/essential, by using the new dpkg root stuff.

flatdeb installs apt/dpkg/essential, but then deletes apt, dpkg and
a few other parts of essential/important that make little sense in a
single-user container (such as mount, adduser, passwd) with dpkg --purge
--force-depends --force-remove-essential before preparing the Platform
runtime that is used to run apps. The matching SDK runtime that is used
to compile apps still has the full set of packages.

It also deletes perl-base if perl isn't installed, and python-minimal
if python isn't installed.

> Hmm, last time you presented flatdeb, it wasn't using debos, how does
> debos get used in flatdeb now?

Instead of using sudo or ssh to a remote (usually virtual) machine to
get root so that it can run debootstrap and dpkg, flatdeb now runs debos
on the local machine.  This means it's root in debos' temporary qemu VM,
and doesn't need privileges other than /dev/kvm access in the host system.

In principle there'd be nothing to stop debos from using something
like user-mode-linux as an alternative to qemu/KVM.

> It might also be interesting to start importing the standard Debian
> installations into an OSTree archive

One per desktop task, plus one for the Priority >= standard default
CLI installation, would be quite a lot of data but could be good for
bisecting, yes. I'd suggest doing this with debos, which has built-in
support for committing trees to libostree and doesn't need root on the
host system.

smcv

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Paul Wise]

On Thu, Apr 11, 2019 at 7:01 PM Simon McVittie wrote:

> The "app" (directly-user-facing part) in a Flatpak package will be mounted
> on /app and so is expected to be built with --prefix=/app, so you can't
> reuse a compiled binary .deb unless it's for something that happens to be
> relocatable already. The runtime (chroot-like library stack to run apps)
> is built with --prefix=/usr, so it's usually easy to reuse binary .debs
> for that.

Has anyone done any rebuild testing to see how many packages hard-code
the prefix in their binaries?

Having to recompile everything once more for each "app" package seems
like a lot of extra CPU time. Is there any reason that making /app a
symlink to /usr (or a directory containing only links to /usr)
wouldn't work inside Flatpak packages? I had planned to do this using
mmdebstrap, which now offers installation of Debian systems that don't
have apt/dpkg/essential, by using the new dpkg root stuff. I was
blocked by the lack of a DPKG_CONFIG variable though, otherwise the
system dpkg config interferes with the process if you have certain
packages installed.

> https://gitlab.collabora.com/smcv/flatdeb builds Flatpak runtimes from
> an apt repository using debos and debootstrap.

Hmm, last time you presented flatdeb, it wasn't using debos, how does
debos get used in flatdeb now?

> The example runtime recipes provided with
> flatdeb are "Base" ... and "Games"

I still think one runtime per "app" package is the way to go.

It might also be interesting to start importing the standard Debian
installations into an OSTree archive, so we could do things like
bisect bugs between stable and current sid. As well as share files
between all the runtimes and apps.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Simon McVittie]

On Thu, 11 Apr 2019 at 09:54:47 +, Mo Zhou wrote:
> On Thu, Apr 11, 2019 at 09:26:15AM +0100, Simon McVittie wrote:
> > On Thu, 11 Apr 2019 at 07:44:30 +, Mo Zhou wrote:
> > It might be interesting to look at game-data-packager, which is another
> > tool that builds and optionally installs .deb files for data that is
> > not suitable for the Debian archive (in most cases not even for non-free),
> > without going via a .dsc or source package.
> 
> Any link please? Both apt-file-search and google found nothing.

https://tracker.debian.org/pkg/game-data-packager

It's in contrib.

> > I can't help thinking that a sandboxed app system like Flatpak or Snap
> > would be a better fit than .deb for leaf packages that have had minimal
> > or no review and don't really need to be part of the operating system.
> > For various reasons my preference is Flatpak (obviously, I wouldn't
> > maintain it otherwise) but I'm sure that proponents of Snap would tell
> > you that it should also be a candidate.
> 
> I don't have experience with Flatpak/Snap. However sandboxing sounds
> great.  Is there any existing work that helps one convert existing .deb
> into flatpack/snap format? I'd be glad to enable DUPR build Flatpack/Snap
> packages in the future, if possible.

If your goal is to turn something that is not in Debian into a Flatpak
app, going via a .deb seems an unnecessarily indirect route. Flatpak
already has the flatpak-builder package, which takes a JSON or YAML
manifest and uses it to build software from source, typically by cloning
from a git URL and compiling it in a special "SDK" container that has all
the necessary development tools. As with dpkg, the "source" can be a set
of binary blobs if that's all you have. The software to be downloaded
can be checked against hashes supplied by the maintainer of the Flatpak
app or runtime.

Flathub, the reference Flatpak repository, works by accepting
flatpak-builder manifests and building them. Flathub is analogous to the
role of Dockerhub in Docker, Github in the Go ecosystem, or Launchpad PPAs
in third-party addons for Ubuntu: Flatpak is a federated/decentralized
system in which anyone can run their own repository, a lot like apt,
but Flathub is the large "hub" repository used by anyone who doesn't
have a reason to host their project elsewhere.

Flatpak also has an "extra data" mechanism which downloads and manipulates
binary blobs directly from upstream on the end-user system without hosting
a copy on the Flatpak equivalent of an apt repository, which can be used
for binary blobs that the maintainer of the Flatpak app or runtime does
not have permission to redistribute. The main use case for this is (the
user-space part of) the proprietary Nvidia driver. Again, the binary
blobs are authenticated against hashes supplied by the maintainer of
the Flatpak app or runtime, so that only known-good versions are used.

I assume Snap has something similar to some or all of these but I don't
use it myself and don't know the details. My understanding is that Snap
is less federated/more centralized than Flatpak, but that might be
outdated information.

The "app" (directly-user-facing part) in a Flatpak package will be mounted
on /app and so is expected to be built with --prefix=/app, so you can't
reuse a compiled binary .deb unless it's for something that happens to be
relocatable already. The runtime (chroot-like library stack to run apps)
is built with --prefix=/usr, so it's usually easy to reuse binary .debs
for that.

https://gitlab.collabora.com/smcv/flatdeb builds Flatpak runtimes from
an apt repository using debos and debootstrap. The resulting runtimes
are a Debian-based alternative to the reference "freedesktop" runtimes
published on Flathub, which are either used directly by apps with lighter
dependencies, or used as the basis for the GNOME and KDE runtimes used by
apps with heavier dependencies. The example runtime recipes provided with
flatdeb are "Base", which is a minimal CLI system, and "Games", which can
run OpenArena (and probably nothing else yet).

flatdeb also supports building Flatpak apps from a binary .deb (assuming
its contents are relocatable) or an unpacked source package, but that part
isn't my focus right now and might have stopped working. The example app
recipes provided with flatdeb are hello, bash-static and mesa-utils (built
by unpacking a binary .deb that happens to be sufficiently relocatable),
and OpenArena (a hybrid approach, with ioquake3 and openarena rebuilt from
source, and the openarena-data family of packages unpacked and relocated).

For more information please see my Debconf 17 presentation, linked from
.

At the moment my focus in my work project is using flatdeb-built
runtimes for ad-hoc containers that share bubblewrap, Flatpak's
container-runner/sandbox helper, but do not actually use Flatpak. I'll try
to publish more information about this when it goes into production 

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Andrey Rahmatullin]

On Thu, Apr 11, 2019 at 09:54:47AM +, Mo Zhou wrote:
> Any link please? Both apt-file-search and google found nothing.
It's in contrib.
https://tracker.debian.org/pkg/game-data-packager

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Mo Zhou]

Hi,

On Thu, Apr 11, 2019 at 09:26:15AM +0100, Simon McVittie wrote:
> On Thu, 11 Apr 2019 at 07:44:30 +, Mo Zhou wrote:
> It might be interesting to look at game-data-packager, which is another
> tool that builds and optionally installs .deb files for data that is
> not suitable for the Debian archive (in most cases not even for non-free),
> without going via a .dsc or source package.

Any link please? Both apt-file-search and google found nothing.

> If I had enough free time, my long-term plan for g-d-p is to make
> it optionally produce Flatpak apps or extensions (based on either
> the reference "freedesktop" runtimes available on Flathub, or a new
> Debian-provided runtime) instead of .deb files, to make it easier to run
> its supported games in a sandbox to mitigate any security vulnerabilities
> that they might have.
>
> I can't help thinking that a sandboxed app system like Flatpak or Snap
> would be a better fit than .deb for leaf packages that have had minimal
> or no review and don't really need to be part of the operating system.
> For various reasons my preference is Flatpak (obviously, I wouldn't
> maintain it otherwise) but I'm sure that proponents of Snap would tell
> you that it should also be a candidate.

I don't have experience with Flatpak/Snap. However sandboxing sounds
great.  Is there any existing work that helps one convert existing .deb
into flatpack/snap format? I'd be glad to enable DUPR build Flatpack/Snap
packages in the future, if possible.

> > afterall the .durpkg header is a shell script
> 
> I'm sure the costs and benefits are different for your use case, but as
> a data point: game-data-packager used to have an ad-hoc shell script per
> supported game. It has been *much* nicer to maintain since its redesign
> as a Python program that reads declarative recipes.

I've ever thought about providing a Python interface in duprkit, such
that a .durpkg can be either
  (shell script) + (HFT part)
or
  (python script) + (HFT part)

I hesitate to move forward on that direction because not everybody
can write python; but everyone who types commands in the terminal
can write basic shell.
 
> What proportion of AUR users do you think genuinely do this?
> 
> What proportion of AUR users do you think are sufficiently experienced
> to be able to recognise malicious code in a packaging recipe or an
> upstream release?

TBH IDK. Umm... At least I'm not blindly accepting PRs to the
DefaultCollection.

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Simon McVittie]

On Thu, 11 Apr 2019 at 07:44:30 +, Mo Zhou wrote:
> On Wed, Apr 10, 2019 at 04:56:51PM +0200, Helmut Grohne wrote:
> > It seems that a key aspect of this thing is avoiding to (re)distribute
> > sources.

It might be interesting to look at game-data-packager, which is another
tool that builds and optionally installs .deb files for data that is
not suitable for the Debian archive (in most cases not even for non-free),
without going via a .dsc or source package.

Some key differences:

* game-data-packager recipes are part of its source code. Each version
  uploaded to Debian contrib supports a fixed set of games.

* game-data-packager normally only has a trivial "compilation" step,
  although for Quake II expansions (what we'd now call DLC) it does need to
  compile C code.

* game-data-packager doesn't expect users to review the recipes that build
  its packages. They're reviewed by a DD (in practice, me) before entering
  a release. For games that contain executable code (Quake II expansions),
  I also review the changes to that executable code whenever I update to a
  new version.

* game-data-packager has this design principle: if a component can have
  bugs, and they are bugs that Debian can (technically and legally)
  fix, then we try to put that component in the Debian contrib apt
  archive, not in the game-data-packager-produced .deb. For example,
  quake3.desktop and unreal.desktop can easily have bugs (missing
  translations or Keywords or similar), and they were written by Debian
  contributors under a Free Software license, so we put them in quake3.deb
  and game-data-packager-runtime.deb (respectively) in the Debian contrib
  apt archive, not in the quake3-data.deb and unreal-gold.deb produced
  by g-d-p.

* game-data-packager recipes are declarative (YAML) and don't attempt to
  be compatible with the full generality of Debian packaging. Individual
  games can have "plugins" to provide imperative build/packaging steps,
  but those are part of game-data-packager's source code.

If I had enough free time, my long-term plan for g-d-p is to make
it optionally produce Flatpak apps or extensions (based on either
the reference "freedesktop" runtimes available on Flathub, or a new
Debian-provided runtime) instead of .deb files, to make it easier to run
its supported games in a sandbox to mitigate any security vulnerabilities
that they might have.

I can't help thinking that a sandboxed app system like Flatpak or Snap
would be a better fit than .deb for leaf packages that have had minimal
or no review and don't really need to be part of the operating system.
For various reasons my preference is Flatpak (obviously, I wouldn't
maintain it otherwise) but I'm sure that proponents of Snap would tell
you that it should also be a candidate.

> afterall the .durpkg header is a shell script

I'm sure the costs and benefits are different for your use case, but as
a data point: game-data-packager used to have an ad-hoc shell script per
supported game. It has been *much* nicer to maintain since its redesign
as a Python program that reads declarative recipes.

> > I don't think I would start using such a user repository. I'm much too
> > scared about it. 
> 
> Understand. The nature of AUR is similar: every user is recommened to
> review the code before execution to confirm safety.

What proportion of AUR users do you think genuinely do this?

What proportion of AUR users do you think are sufficiently experienced
to be able to recognise malicious code in a packaging recipe or an
upstream release?

smcv

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Mo Zhou]

Hi Helmut,

Thank you very much for the detailed review! :-)

On Wed, Apr 10, 2019 at 04:56:51PM +0200, Helmut Grohne wrote:
> It seems that a key aspect of this thing is avoiding to (re)distribute
> sources. You give good reasons for why this is needed and I see no need
> to reiterate or discuss them.

Indeed.

> Your implementation goes straight from .durpkg -> .deb. I question this
> decision: We already have lots of tools to go from .dsc to .deb. Your
> implementation replicates part of that and I think, this is bad as it
> makes it harder to collaborate.
> 
> Let me propose a rather intrusive interface change to duprkit. What if
> the result of building a .durpkg was a .dsc rather than a .deb? Then you
> could split duprkit into two tools:
> 
>  * One tool to build source packages from .durpkg files on demand.
>  * One tool to build a specific binary package from a given deb-src
>repository.
> 
> Now in principle, the latter is simply sbuild or pbuilder, but there is
> more to it:
>  * Given the binary package name, figure out which source package must
>be built.
>  * Given that source package's Build-Depends, figure out what other
>binary packages need to be built.
>  * Recurse.
>  * Build them in a suitable order.
> 
> (Some people will observe that this is the "bootstrap" problem. ;)

Good point! This is easy to improve. And this suggestion is actually
not intrusive at all. AUR's PKGBUILD builder sources the PKGBUILD
file for variables and some functions, then execute a pre-defined
sequence. Different from that, duprkit's design don't hope to limit
the user with any pre-defined "sequence", but enable the users to
selectively call the functions they need. In other words, the
user can define how to deal with the prepared source+debian directories,
afterall the .durpkg header is a shell script. That said, I think
some more helper functions would be nice: [1].
 
> There is one major difficulty here (and duprkit doesn't presently solve
> that either): If you figure that some binary package is missing, you
> have no way of knowing which .durpkg file to build to get the relevant
> source package.

My blueprint includes a very light-weight/simple dependency tracking
mechanism. And I assume the project don't need to handle complex dep
trees like apt. Because:

1. If a package is common and useful enough, such that it has been
   adopted by many other projects, why don't I package it for the
   official Debian release? So, I expect that most packages that DUPR
   will deal with, are actually leaf or near-leaf packages on the
   dependency tree.

2. Some of my targeted upstreams do sourceless binary tarball release.
   They seldom get into the dependency trouble...

3. Inserting a DUPR package into the near-root part of the Debian
   dependency tree is, generally speaking, a terrible idea.
   Only those who aware of what they are doing will do this.

The simple dep tracking mechanism will be implemented following
the Collection-Manifest collection functionality. Everything
looks fine so far, and I'll package more stuff to see whether
the assumption/blueprint is correct.

> Before tackling that problem, the question arises of whether that
> problem is in scope. Does duprkit really want to handle complex
> dependencies or is the idea really to just get the stuff into users
> hands? 

No. As explained above.

> In the latter case, vendoring likely is a simple way to avoid
> this problem entirely.

Agreed.
 
> Now let's assume that you do want to allow complex dependencies in this
> user repository. In this case, it would make sense to trade .durpkg
> files for plain "debian" directories with an additional debian/rules
> target to obtain the source. (We removed "get-orig-source" from policy a
> while ago, but maybe this is what you want here.)

The `bin/duprCollector` will collect meta information from a collection
(and will form a dependency tree in the future). I have no plan to
rethink about the "get-orig-source" target since there are ... lots
of weird upstreams in my list...

> If you go this route, you can just scrape those debian/control files to
> figure out which .durpkg files to convert into .dsc files.

It will be easy to unfold all .durpkg files into prepared debian/
directories, which makes meta info collection straightforward.
To be implemented: [2]
 
> I don't think I would start using such a user repository. I'm much too
> scared about it. 

Understand. The nature of AUR is similar: every user is recommened to
review the code before execution to confirm safety.

> Yet, the second part of the problem seems interesting
> to me: Taking a (possibly local) repository of source packages and
> building a specific binary package (plus everything that's needed to get
> there). I think that you can increase collaboration by changing your
> interface in a way that makes it easier to reuse in other settings.

This is also a good point. At this point I think taking advantages
from sbuild would ne 

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Helmut Grohne]

Hi Mo,

On Mon, Apr 08, 2019 at 11:18:14AM +, Mo Zhou wrote:
> The proposed idea is to take some advantages from source-based
> software distribution tools. Examples are available here:
> https://github.com/dupr/duprkit
> https://github.com/dupr/DefaultCollection

I looked into this. Your reasons are sound and you are scratching your
itch. This is great.

It seems that a key aspect of this thing is avoiding to (re)distribute
sources. You give good reasons for why this is needed and I see no need
to reiterate or discuss them.

Your implementation goes straight from .durpkg -> .deb. I question this
decision: We already have lots of tools to go from .dsc to .deb. Your
implementation replicates part of that and I think, this is bad as it
makes it harder to collaborate.

Let me propose a rather intrusive interface change to duprkit. What if
the result of building a .durpkg was a .dsc rather than a .deb? Then you
could split duprkit into two tools:

 * One tool to build source packages from .durpkg files on demand.
 * One tool to build a specific binary package from a given deb-src
   repository.

Now in principle, the latter is simply sbuild or pbuilder, but there is
more to it:
 * Given the binary package name, figure out which source package must
   be built.
 * Given that source package's Build-Depends, figure out what other
   binary packages need to be built.
 * Recurse.
 * Build them in a suitable order.

(Some people will observe that this is the "bootstrap" problem. ;)

There is one major difficulty here (and duprkit doesn't presently solve
that either): If you figure that some binary package is missing, you
have no way of knowing which .durpkg file to build to get the relevant
source package.

Before tackling that problem, the question arises of whether that
problem is in scope. Does duprkit really want to handle complex
dependencies or is the idea really to just get the stuff into users
hands? In the latter case, vendoring likely is a simple way to avoid
this problem entirely.

Now let's assume that you do want to allow complex dependencies in this
user repository. In this case, it would make sense to trade .durpkg
files for plain "debian" directories with an additional debian/rules
target to obtain the source. (We removed "get-orig-source" from policy a
while ago, but maybe this is what you want here.)

If you go this route, you can just scrape those debian/control files to
figure out which .durpkg files to convert into .dsc files.

I don't think I would start using such a user repository. I'm much too
scared about it. Yet, the second part of the problem seems interesting
to me: Taking a (possibly local) repository of source packages and
building a specific binary package (plus everything that's needed to get
there). I think that you can increase collaboration by changing your
interface in a way that makes it easier to reuse in other settings.

Helmut

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Shengjing Zhu]

> from PPA (source+binary-based).

If people just want a PPA which supports Debian, please just take a
look at OBS[1].

I've seen many upstreams provide packages with OBS, and most
distributions are supported.
Not only deb, but also rpm, from Debian/Ubuntu to OpenSuse/Fedora, and
even Archlinux, in a uniform way.

[1] https://build.opensuse.org/


--
Shengjing Zhu

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Alf Gaida]

DUR is fine, DPA is fine PPA is not - as it is used before in a totally 
different context.


The idea just to require git is really nice, putting all the things into 
a single file is not. Not even Arch does it. (patches, install, config 
...) - so the default debian dir should be enough.


Please think a bit further - which files are really needed?
* control
* rules
* watch

There was discussions to make rules obsolete in standard packages - if 
there is no rules file, just assume/use the default. compat is obsoleted 
if one use debhelper-compat, copyright would be nice to have, but might 
not be needed for a possible dur. Same for all other things.


Things become a bit different if one want to use patches. So why don't 
use the debian standards. Same for packages that result in more than one 
binary package.


If anything else fails just add a file with a special set of things for 
the DUR/DPA - should be dead easy, i use something similar for years now 
to build things from several git sources.


Cheers Alf

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Jonathan Dowland]

On Mon, Apr 08, 2019 at 11:18:14AM +, Mo Zhou wrote:

The proposed idea is source-only-based, and is totally different
from PPA (source+binary-based). I'm a PPA user and I don't have
any reason to re-invent yet another PPA.


Sorry I appreciate that *your* idea is different, and effectively
my sub-thread is a hijack, but to be clear what I was referring to
was the naming of the pre-existing "Bikeshed" proposal for Debian,
which is equivalent to PPAs.

--

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Mo Zhou]

On Mon, Apr 08, 2019 at 03:50:19PM +0300, Tzafrir Cohen wrote:
> Hi,
> 
> The README states a directory structure with a top-level collection
> directory, but the repository currently does not include one.

The github.com:dupr/DefaultCollection.git repo is indeed a specification
compliant if you mangle the first layer of directory structure.

```
--- a/README
+++ b/README
@@ -17,7 +17,7 @@ name. Plus, no one prevents you from submitting a debian 
directory.

 For example:

-A-Certain-Collection/
+A-Certain-Collection/  # For example, this git repository.
 library-foo/
 library-foo.durpkg
 library-foo-ubuntu.durpkg
```

> Looking at:
> 
> 
> https://github.com/dupr/DefaultCollection/blob/master/rover/rover.durpkg
> 
> I see that the name of the directory repeats quite a number of times.
> The clean function needs to state 'rm -rf' explicitly, which I hope
> won't lead to typos.

The repetition of "pushd" and "popd" is something that could be improved
later. So the same as the 'rm -rf' hook. The two problems have been
tracked here:
https://github.com/dupr/duprkit/issues/3
https://github.com/dupr/duprkit/issues/4

Please don't expect too much from a prototype rushed within several
hours. That's used for presenting the idea instead of presenting
a graceful implementation.

> The changelog is buried deep inside the package an needs to be updated
> whenever a version number is bumped.

AUR's PKGBUILD doesn't record changes. If you feel like that the dch
has been deeply buried, the usage of traditional debian/ directory
layout is still supported:
https://github.com/dupr/DefaultCollection/tree/master/rover-traditional
https://github.com/dupr/DefaultCollection/blob/master/rover-traditional/debian/changelog

One could easity convert the packaging between single-file format and
debian/ directory with a single-command-cost (virtually zero-cost).

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Tzafrir Cohen]

Hi,


On 08/04/2019 14:18, Mo Zhou wrote:
> Hi,
>
> The proposed idea is source-only-based, and is totally different
> from PPA (source+binary-based). I'm a PPA user and I don't have
> any reason to re-invent yet another PPA.
>
> The proposed idea is to take some advantages from source-based
> software distribution tools. Examples are available here:
> https://github.com/dupr/duprkit
> https://github.com/dupr/DefaultCollection

The README states a directory structure with a top-level collection
directory, but the repository currently does not include one.


Looking at:


https://github.com/dupr/DefaultCollection/blob/master/rover/rover.durpkg


I see that the name of the directory repeats quite a number of times.
The clean function needs to state 'rm -rf' explicitly, which I hope
won't lead to typos.


The changelog is buried deep inside the package an needs to be updated
whenever a version number is bumped.


-- Tzafrir

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Mo Zhou]

Hi,

The proposed idea is source-only-based, and is totally different
from PPA (source+binary-based). I'm a PPA user and I don't have
any reason to re-invent yet another PPA.

The proposed idea is to take some advantages from source-based
software distribution tools. Examples are available here:
https://github.com/dupr/duprkit
https://github.com/dupr/DefaultCollection

Only "packaging scripts" are provided. Upstream source are
downloaded by the scritps locally before the build. The proposed
idea will never involve "distributing resulting binary .deb packages".

Again, the idea is totally different from PPA.

On Mon, Apr 08, 2019 at 10:32:46AM +, Holger Levsen wrote:
> On Mon, Apr 08, 2019 at 10:18:39AM +0100, Jonathan Dowland wrote:
> > > At the first glance I interpreted the sentence as
> > >  "This will only lead to flamewars"
> > > due to the meaning of bikeshed[1].
> > > 
> > > However, I got a hint from a fellow developer and learned that
> > > "Bikeshed" has its own meaning under Debian's context, according
> > > to some old mailing list fragments[2][3] -- which refers to a
> > > dak feature (This is the first time I heard of such thing).
> > This supports my (thus far) private feeling that naming this initiative
> > "Bikeshed" is a bad idea.
> 
> +1
> 
> I also think that using the name "D**ian" is a bad idea.
> 
> Just call it PPAs... millions of people know that term.

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Ondřej Surý]

Or DPA (Debian Personal Archive)...

Ondrej
--
Ondřej Surý 

> On 8 Apr 2019, at 12:32, Holger Levsen  wrote:
> 
> On Mon, Apr 08, 2019 at 10:18:39AM +0100, Jonathan Dowland wrote:
>>> At the first glance I interpreted the sentence as
>>> "This will only lead to flamewars"
>>> due to the meaning of bikeshed[1].
>>> 
>>> However, I got a hint from a fellow developer and learned that
>>> "Bikeshed" has its own meaning under Debian's context, according
>>> to some old mailing list fragments[2][3] -- which refers to a
>>> dak feature (This is the first time I heard of such thing).
>> This supports my (thus far) private feeling that naming this initiative
>> "Bikeshed" is a bad idea.
> 
> +1
> 
> I also think that using the name "D**ian" is a bad idea.
> 
> Just call it PPAs... millions of people know that term.
> 
> 
> -- 
> tschau,
>Holger
> 
> ---
>   holger@(debian|reproducible-builds|layer-acht).org
>   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

[Holger Levsen]

On Mon, Apr 08, 2019 at 10:18:39AM +0100, Jonathan Dowland wrote:
> > At the first glance I interpreted the sentence as
> >  "This will only lead to flamewars"
> > due to the meaning of bikeshed[1].
> > 
> > However, I got a hint from a fellow developer and learned that
> > "Bikeshed" has its own meaning under Debian's context, according
> > to some old mailing list fragments[2][3] -- which refers to a
> > dak feature (This is the first time I heard of such thing).
> This supports my (thus far) private feeling that naming this initiative
> "Bikeshed" is a bad idea.

+1

I also think that using the name "D**ian" is a bad idea.

Just call it PPAs... millions of people know that term.


-- 
tschau,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature