Bug#933312: ITP: golang-github-cloudflare-circl -- Cloudflare Interoperable Reusable Cryptographic Library
Package: wnpp Severity: wishlist Owner: Eric Dorland * Package name: golang-github-cloudflare-circl Version : 1.0.0 Upstream Author : Cloudflare * URL : https://github.com/cloudflare/circl * License : BSD Programming Lang: Go Description : Cloudflare Interoperable Reusable Cryptographic Library CIRCL is a collection of cryptographic primitives written in Go. The goal of this library is to be used as a tool for experimental deployment of cryptographic algorithms targeting Post-Quantum (PQ) and Elliptic Curve Cryptography (ECC).
Re: Salsa CI - DebConf2019 Curitiba
On 7/28/19 10:59 PM, Joaquin de Andres wrote: [..] > * Active helping developers with the utilization of Salsa CI pipeline. At > least 4 project were initialized. Actually there were more than 10! 😄 -- TiN signature.asc Description: OpenPGP digital signature
Salsa CI - DebConf2019 Curitiba
Hi! Part of the Salsa CI Team met at Debconf 2019 in Curitiba, and during the past two weeks we managed to work on continuing to improve the project on different topics: * Diffusion on the usage and the importance of the project have been made. We presented two Long talks, one by Agustin Henze titled "Salsa CI - Debian Pipeline for Developers" [1], and the other by Otto Kekäläinen titled "How MariaDB packaging uses Salsa-CI to ensure smooth upgrades and avoid regressions" [2]; a BoF titled "Salsa and Salsa-CI BoF – how the two teams could collaborate and provide a perfect platform for Debian Developers" [3]; and two Lightning talks by Santiago Ruano Rincón and Joaquin de Andres, on the second block [4]. * New tools for better use of the Salsa CI flow: we worked in two tools: the first one, tuco [5], aimed to help set up Salsa CI on your package automatically; the second, atomic-reprotest [6], a tool for helping with debugging on errors reported by reprotest. * Arm64 gitlab runner implementation on Packet.net platform. Docker-machine-kvm is beeing deployed natively for better isolation and plarform flexibility. * Active helping developers with the utilization of Salsa CI pipeline. At least 4 project were initialized. Great DebConf, thanks to the orga team! Joa Salsa CI Team [1] https://meetings-archive.debian.net/pub/debian-meetings/2019/DebConf19/salsa-ci-debian-pipeline-for-developers.webm [2] https://meetings-archive.debian.net/pub/debian-meetings/2019/DebConf19/how-mariadb-packaging-uses-salsa-ci-to-e.webm [3] https://meetings-archive.debian.net/pub/debian-meetings/2019/DebConf19/salsa-and-salsa-ci-bof-how-the-two-teams.webm [4] https://gemmei.ftp.acc.umu.se/pub/debian-meetings/2019/DebConf19/lightning-talks-2.webm [5] https://salsa.debian.org/salsa-ci-team/tuco/ [6] https://salsa.debian.org/salsa-ci-team/atomic-reprotest/ signature.asc Description: PGP signature
Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi Marco, After the merger of (64bit-indexing) * (multi-thread flavor) feature enhancement, an libopenblas-julia package will look abrupt, and break the symmetry of package layout (libopenblas-julia doesn't need development files) : ~/D/o/o/debian ❯❯❯ rg \^Package control 19:Package: libopenblas0 41:Package: libopenblas0-pthread 64:Package: libopenblas0-openmp 87:Package: libopenblas0-serial 134:Package: libopenblas-dev 156:Package: libopenblas-pthread-dev 179:Package: libopenblas-openmp-dev 202:Package: libopenblas-serial-dev 227:Package: libopenblas64-0 245:Package: libopenblas64-0-pthread 264:Package: libopenblas64-0-openmp 283:Package: libopenblas64-0-serial 302:Package: libopenblas64-dev 321:Package: libopenblas64-pthread-dev 341:Package: libopenblas64-openmp-dev 361:Package: libopenblas64-serial-dev Instead, if we embed the openblas source to src:julia, no extra binary package is needed. So I prefer (3). It's not the maintenance burden that matters most at this point. It's simply that a maintainer doesn't want his/her package to look weird. On 2019-07-29 00:12, Marco d'Itri wrote: > On Jul 28, Mo Zhou wrote: > >> 2. Specifically compile a libopenblas64_.so >>from src:openblas for julia's use. >>This looks very bad for src:openblas's >>package layout. > Why would this look bad? We have plenty of source packages which > generate multiple variations of binary packages. > udebs, for a start. Many libraries which generate both a static and > dynamic package. The older inn2 releases if you want to see something > really ugly. > While this solution may require some packaging work it is obviously both > the technically correct one and the one which will not harm users.
Re: And in 2019? Re: -flto to become more of a routine - any change in opinion since 2011?
I wrote: >in GCC 6 IIRC and *still* pertinent in GCC 8 and I believe 9, Still present in (sid/amd64): • gcc-8 (= 8.3.0-19) • gcc-9 (= 9.1.0-10) • gcc-snapshot (= 1:20190719-1) >>I'm currently compiling e2fsprogs with LTO for Debian --- and I'm >>seriously considering ditching that change. The reason why is because >Ted, I agree, please drop LTO from e2fsprogs as well. Given the wrong-code bug in mksh produced by GCC’s LTO, I shudder to think what it might do to my filesystem. bye, //mirabilos -- „Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund, mksh auf jedem System zu installieren.“ -- XTaran auf der OpenRheinRuhr, ganz begeistert (EN: “[…]uhr.gz is a reason to install mksh on every system.”)
Re: And in 2019? Re: -flto to become more of a routine - any change in opinion since 2011?
>We just had SuSE embracing LTO
Not entirely.
With my DD hat doffed and wearing the mksh upstream developer hat,
I’ve been asking the package maintainers of mksh in all distributions
to remove the LTO flags, and will remove the built-in support for
using LTO in the next release.
Why?
mksh’s testsuite has proven to be pretty good in catching kernel,
libc, compiler and toolchain bugs. (I have commits in all libcs
for Linux except notably musl, fixing tons of bugs found this
way.)
However, the GCC developers have, time and time again, ignored
all but (I think) one of my bugreports regarding LTO, first
-fwhole-program --combine, then -flto with -fno-use-linker-plugin,
with -fuse-linker-plugin, or without, with or without -fwhole-program,
and lately -flto=jobserver with or without -fuse-linker-plugin.
Some of the code generation bugs are hard to find. mksh itself
is an interpreter language and the most recent bug, appearing
in GCC 6 IIRC and *still* pertinent in GCC 8 and I believe 9,
only shows up in one single testsuite failure of a program that
is written in the mksh Korn Shell scripting language, so, very
hard to reduce down (impossible for myself).
Nevertheless, all (except IIRC one) of those bugs got ignored
by the GCC developers, and while mksh’s testsuite caught some
of the codegen bugs I’m afraid of more LTO bugs lurking and
thus recommend people to NOT enable LTO for GCC in the general
case, only if they test the exact output with the exact com‐
piler version throughoutly before even attempting to use it.
I’ve removed -flto from the packages I maintain where I had
previously added it, happy for the benefits.
CI and tests do not help if nobody fixes the bugs. Worse, the
bugs will make it into production undetected.
Oh, and that’s not on an esoteric architecture. It’s on plain amd64.
mksh itself is not very large…
tg@blau:/usr/src/bin/mksh $ cat *.c *.h | wc
34477 119600 798703
and very portable code (it works on DEC ULTRIX with the
vendor C compiler, has an active OS/2 port, etc.) and I
regularily build it with whatever modern GCC is available
in Debian (even gcc-snapshot) with as many warnings en‐
abled as I can find, have had other people looking it
over and am pretty sure it does not fall into the category
of “design[ed] with knowledge of how compilers translate[…]”
that Ian writes about. (I do agree with his “The C standard
rules are obtuse, insanely complicated, and generally hard
to satisfy or even analyse in nontrivial cases.“ I rewrote
the arithmetic code to emulate signed arithmetic using only
unsigned C types, to avoid UB/IB.) mksh works with ANSI C89,
C99, C11 compilers and even some pre-ANSI ones. I literally
revived an entire Debian architecture in order to be able
to build and test mksh on another CPU architecture because
every port has let me find new bugs (or not, in the happy
case).
Therefore I state that the tooling is nowhere near working
enough for LTO, and note that the upstream developers of
said tooling are not interested in fixing it either, while
the developers of the to-be-compiled software don’t have
skills and tooling available in aiding them (e.g. to reduce
testcases which, in the LTO case, is extremely hard to im‐
possible anyway).
Ted, I agree, please drop LTO from e2fsprogs as well.
Thanks,
//mirabilos
--
11:56⎜«liwakura:#!/bin/mksh» also, i wanted to add mksh to my own distro │
i was disappointed that there is no makefile │ but somehow the Build.sh is
the least painful built system i've ever seen │ honours CC, {CPP,C,LD}FLAGS
properly │ looks cleary like done by someone who knows what they are doing
Re: unsigned repositories
I actually have a use case: injecting packages from
/var/cache/pbuilder/base.cow-somedist/repo/ into APT
from an optional cowbuilder hook script that I can
enable (chmod +x) when needed.
The hook script has the interesting constraint that
it adds a repository to APT with*out* running apt-get
update, i.e. ceteris paribus, keeping the *other*
repositories’ state as-is and NOT updating them from
the network. I sometimes rely on this.
-BEGIN cutting here may damage your screen surface-
#!/bin/bash
# $MirOS: contrib/hosted/tg/deb/hookdir/D01slashrepo,v 1.3 2019/02/24 03:34:00
tg Exp $
#-
# Copyright © 2014, 2018, 2019
# mirabilos
#
# Provided that these terms and disclaimer and all copyright notices
# are retained or reproduced in an accompanying document, permission
# is granted to deal in this work without restriction, including un‐
# limited rights to use, publicly perform, distribute, sell, modify,
# merge, give away, or sublicence.
#
# This work is provided “AS IS” and WITHOUT WARRANTY of any kind, to
# the utmost extent permitted by applicable law, neither express nor
# implied; without malicious intent or gross negligence. In no event
# may a licensor, author or contributor be held liable for indirect,
# direct, other damage, loss, or other issues arising in any way out
# of dealing in the work, even if advised of the possibility of such
# damage or existence of a defect, except proven that it results out
# of said person’s immediate fault when using the work as intended.
#-
# Configure $base and $this at the beginning of the file. Do ensure:
# • base must be URI safe since we do not encode it for sources.list
# • this must be a valid basename for sources.list.d: [A-Za-z0-9._-]
base=/repo
this=D01slashrepo
unset LANGUAGE
LC_ALL=C; export LC_ALL
test -d "$base/." || {
echo >&2 "E: D01slashrepo: base '$base' does not exist"
exit 1
}
shopt -s extglob
base=${base%%*(/)}
pstr=${base//\//_}_._Packages
echo >&2 "I: creating Packages file for local APT cache in $base"
rm -f "$base/Packages"
(cd "$base"
#dpkg-scanpackages -h md5 -m . >Packages 2>/dev/null || \
dpkg-scanpackages -m . >Packages 2>/dev/null || \
dpkg-scanpackages . /dev/null >Packages)
paste -d_ <(sed -n '/^Package: /s///p' "$base/Packages") \
<(sed -n '/^Version: /s///p' "$base/Packages") \
<(sed -n '/^Architecture: /s///p' "$base/Packages") | \
sed 's/^/N: /' >&2
echo >&2 "I: updating APT repository information"
cp "$base/Packages" "/var/lib/apt/lists/$pstr"
if test -d /etc/apt/sources.list.d/.; then
echo "deb [trusted=yes] file://$base ./"
>"/etc/apt/sources.list.d/$this.list"
else
echo "deb file://$base ./" >>"/etc/apt/sources.list"
fi
apt-cache gencaches
echo >&2 "I: made $(grep -c '^Package: ' "$base/Packages") packages available
from $base"
exit 0
-END cutting here may damage your screen surface-
It’s also much more convenient for quickly testing the
built packages: having the following snippet…
-BEGIN cutting here may damage your screen surface-
#!/bin/sh
cd "$(dirname "$0")"
rm -f Packages* en.gz
#dpkg-scanpackages -h md5 -m . >Packages 2>/dev/null || \
dpkg-scanpackages -m . >Packages
gzip -n9 Packages.gz
(: | gzip -n9 >en.gz)
-END cutting here may damage your screen surface-
… in /var/cache/pbuilder/result/0 (for quickly running
it from mc) and the sources.list snippet…
deb [trusted=yes] file:///var/cache/pbuilder/result ./
… I can test-install stuff just built with an easy
sudo apt-get update
sudo apt-get install foo
or even
sudo apt-get update
sudo apt-get upgrade
instead of having to do
sudo apt install /var/cache/pbuilder/result/{pkg1,pkg2,pkg-common}_*.deb
sudo apt-mark auto pkg2 pkg-common
because installing .deb via apt doesn’t search the rest of
the directory for dependencies and disturbs the auto/manually
installed flag.
Please keep the code and support in APT.
Thanks,
//mirabilos
--
22:20⎜ The crazy that persists in his craziness becomes a master
22:21⎜ And the distance between the craziness and geniality is
only measured by the success 18:35⎜ "Psychotics are consistently
inconsistent. The essence of sanity is to be inconsistently inconsistent
Re: tag2upload (git-debpush) service architecture - draft
On Jul 28, Bernd Zeimetz wrote: > So I think the best thing to do is to get sha256 working in git and > force the usage of sha256 if you want to sign a tag for upload. This cannot be a goal for this project since git upstream will need apparently a few more years for the transition to sha-256 to happen. -- ciao, Marco signature.asc Description: PGP signature
Re: Challenge from Julia's non-standard vendored openblas"64_"
On Jul 28, Mo Zhou wrote: > 2. Specifically compile a libopenblas64_.so >from src:openblas for julia's use. >This looks very bad for src:openblas's >package layout. Why would this look bad? We have plenty of source packages which generate multiple variations of binary packages. udebs, for a start. Many libraries which generate both a static and dynamic package. The older inn2 releases if you want to see something really ugly. While this solution may require some packaging work it is obviously both the technically correct one and the one which will not harm users. -- ciao, Marco signature.asc Description: PGP signature
Re: And in 2019? Re: -flto to become more of a routine - any change in opinion since 2011?
On Wed, Jul 24, 2019 at 06:03:21PM +0200, Steffen Möller wrote: > Hello, > > We just had SuSE embracing LTO > (https://www.linuxtoday.com/infrastructure/opensuse-enables-lto-by-default-for-tumbleweed-smaller-faster-binaries.html). > I am not sure about the progress on issues summarised in > http://blog.regehr.org/archives/1180 that Ian pointed to. But since I > last asked in 2016 we have more pedantic compiler settings and more CI - > and LTO, as much as compilers have improved on that, does not need to be > applied everywhere. Any change in opinion? I'm currently compiling e2fsprogs with LTO for Debian --- and I'm seriously considering ditching that change. The reason why is because LTO breaks reproducible builds, and so it makes it harder when I'm verifying whether a particular packaging change (say, moving to a new debhelper compat level) is going to make any changes to the binary --- because using LTO pretty much guarantees that it will. Yeah, the binaries are a little bit smaller, and presumably a little bit more CPU efficient, but 99% of the time, e2fsprogs binary are I/O bound, not CPU bound, and the fact that my package builds aren't reproducible is !@#?! annoying. - Ted
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 10:33 PM Johannes Schauer wrote: > If we choose the "src:" prefix to pick source instead of binary packages, then > it's important that we don't have any binary packages called "src" and no > source packages with a name equal to a valid debian architecture. Could we have a double suffix instead to avoid these issues? Build-Depends: foo:src:src -> src:foo unpacked in /usr/src/foo Build-Depends: foo:src -> b-d of src:foo for the current host architecture Build-Depends: foo:src:amd64-> b-d of src:foo for amd64 Build-Depends: foo:src:native -> b-d of src:foo for the current build architecture -- bye, pabs https://wiki.debian.org/PaulWise
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 10:04 PM Mo Zhou wrote: > Is such demand common enough among developers? There are several ways this would be useful: To replace all librust-*-dev and golang-*-dev packages (they contain source code). To replace all toolchain -source packages, so that cross-compiling toolchains can be built from separate source packages. To replace all out-of-tree Linux kernel module -source packages, so that dkms/etc doesn't need a binary package duplicating the source. Anywhere you want to build multiple independent build configurations of the same source code in different ways. -- bye, pabs https://wiki.debian.org/PaulWise
Re: tag2upload (git-debpush) service architecture - draft
On 7/27/19 8:16 PM, Rebecca N. Palmer wrote:> As a way to avoid relying on SHA-1, would it work to have git-debpush > include a longer hash in the tag message, and tag2upload also verify > that hash? > The other idea would be to convince git upstream to use something better than sha1 - and after a bit of searching, I found https://github.com/git/git/blob/master/Documentation/technical/hash-function-transition.txt - Git v2.13.0 and later use a hardened sha-1 implementation by default, which isn't vulnerable to the SHAttered attack. Still sha-1, though. - there is a plan to support sha256. Googling a bit more found https://stackoverflow.com/questions/28159071/why-doesnt-git-use-more-modern-sha which gives some insight on the (plans for) implementation. So I think the best thing to do is to get sha256 working in git and force the usage of sha256 if you want to sign a tag for upload. -- Bernd ZeimetzDebian GNU/Linux Developer http://bzed.dehttp://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
Re: tag2upload (git-debpush) service architecture - draft
On 28/07/2019 20:01, Sean Whitton wrote: When I read your first e-mail what I thought you had in mind was just this -- having git-debpush compute a stronger hash of the tree object and add that to the tag metadata, ignoring commit objects. Of the files in the signer's repository, not of an actual tree object (since the second is a list of file/subtree SHA-1 hashes). But now I'm struggling to understand the relevance of your discussion of having git-debpush create a .dsc in your second e-mail, if what you're actually talking about is hashing a git tree object. "Tag with sha256" and "hidden .dsc" are two alternative options: the first is a narrowly targeted fix for the SHA-1 issue, the second a bigger redesign. (As an aside, if what you want is to hide .dsc creation from the user but still do it on their machine and upload it, `dgit push-source` is already available.) That doesn't push to salsa [0]. However, I agree that it otherwise does solve the problem of "not making the uploader think about how Debian source packages work", without requiring a server-side component. This does still "waste" the uploader's bandwidth on tarballs, but I don't know if that's an issue in practice. For most packages [1] it is a much smaller data volume than the downloads needed to keep an up-to-date sid for building/testing the package. [0] https://sources.debian.org/src/dgit/9.6/dgit-maint-gbp.7.pod/#L117 [1] Rough numbers: ~80% of .orig.tar.*z are <1MB, ~97% <10MB; a gcc update is a ~30MB download.
Re: tag2upload (git-debpush) service architecture - draft
Hello, On Sun 28 Jul 2019 at 07:05pm +01, Rebecca N. Palmer wrote: > On 28/07/2019 16:18, Ian Jackson wrote: >> What it amounts to is a parallel Merkle tree to the >> git one, just with a different data format and a better hash. > > Not really: it wouldn't need the history tree structure (in Git terms > [0], it would be a tree object not a commit object), and if we use > tar+sha256 [1], it wouldn't need the hash-per-file directory tree > structure either. When I read your first e-mail what I thought you had in mind was just this -- having git-debpush compute a stronger hash of the tree object and add that to the tag metadata, ignoring commit objects. But now I'm struggling to understand the relevance of your discussion of having git-debpush create a .dsc in your second e-mail, if what you're actually talking about is hashing a git tree object. (As an aside, if what you want is to hide .dsc creation from the user but still do it on their machine and upload it, `dgit push-source` is already available.) On Sun 28 Jul 2019 at 04:18pm +01, Ian Jackson wrote: > The downside is that the tag is no longer just a normal signed git tag > with some easy to construct and easy to understand metadata. It will > in practice then not be practical to make this tag other than with > git-debpush (or some other special utility with the same code). This is a downside, but it's not a permanent one -- it goes away if git switches away from SHA-1, which perhaps it is reasonable to expect eventually. It would be good to hear responses to Rebecca's suggestion from those who disagree that it is okay to rely on SHA-1 in the particular way that git-debpush/tag2upload does. -- Sean Whitton
Re: does libmyodbc was removed? ther's will no more mysql odbc packages?
Am 26.07.19 um 20:02 schrieb Andrey Rahmatullin: > On Fri, Jul 26, 2019 at 01:26:52PM -0400, PICCORO McKAY Lenz wrote: >> the https://packages.debian.org/search?keywords=libmyodbc show only >> for sid and oldstable.. so what will happened.. users now must >> compiled own mysql odbc? > It's in sid, so no? > Also, if you use https://tracker.debian.org/pkg/libmyodbc instead of > packages.debian.org, you can see the reasons why it's not in testing. It should probably be RMed at this point, I don't think it will work at all these days. I think even though it was still included in Stretch it was segfaulting in most cases there. mariadb-connector-odbc is waiting in NEW for quite a while, that should be a sane replacement. I plan to add it to buster-backports (and maybe stretch-backports-sloppy) as soon as it has been accepted and migrated to testing. https://ftp-master.debian.org/new/mariadb-connector-odbc_3.1.1-1.html Bernhard
Re: tag2upload (git-debpush) service architecture - draft
On 28/07/2019 16:18, Ian Jackson wrote: What it amounts to is a parallel Merkle tree to the git one, just with a different data format and a better hash. Not really: it wouldn't need the history tree structure (in Git terms [0], it would be a tree object not a commit object), and if we use tar+sha256 [1], it wouldn't need the hash-per-file directory tree structure either. The upside is the better hash, but I think our overall risk from the git SHA-1 problem is (i) still in practice quite low For attacks happening now, I agree (but am not an expert): my intent in suggesting this was "this is an easy way to have a better hash if we want it", not to take a side on the question of whether we need it. This may change, but we have the option of implementing this fix then (and if it happens suddenly, temporarily disabling tag2upload to give us time to do so). (ii) exists in all the other places we rely on git already. That suggests that working towards requiring the SHA-256 mode of git (which at least sort of exists since 2.21 [2], but I don't know if it's usable yet) might be a better use of effort. [0] https://git-scm.com/book/en/v2/Git-Internals-Git-Objects [1] needs reproducibility, but simpler than pristine-tar in that we're only trying to create _a_ reproducible tarball (not match one created by upstream) and don't need to compress it (as it can be deleted after hashing - unfortunately tar doesn't obviously have a write-to-stdout option to allow tar | sha256). Reproducible builds suggests tar --sort=name --owner=0 --group=0 --numeric-owner. [2] https://github.com/git/git/blob/master/Documentation/technical/hash-function-transition.txt
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Mo Zhou 于 2019年7月28日周日 16:03写道： > On 2019-07-28 13:13, Ian Jackson wrote: > > This is maybe not directly helpful to you right now, but: > > > > If we could build-depend on source packages, you could combine these > > two ideas into something that might be less awful. > > More than once have I thought of "what if I can B-D on a source > package". > Is such demand common enough among developers? > Yes, for those static linked languages, at least for Go. // send from my mobile device
Re: file(1) now with seccomp support enabled
Philipp Kern wrote... > On 2019-07-27 10:01, Vincent Bernat wrote: > > I am upstream for a project using seccomp since a long time and I have > > never been comfortable to enable it in Debian for this reason. However, > > they enable it in Gentoo and I get the occasional patches to update the > > whitelist (I am not doing anything fancy). > > But technically it should be possible to test this in an autopkgtest, no? I > don't think perfect has to be the enemy of good here, as long as we can > detect breakage and remediate it afterwards? Ayup, already working on this, for precisely that reason. There a question releated I haven't worded yet, stay tuned. Christoph signature.asc Description: PGP signature
Re: tag2upload (git-debpush) service architecture - draft
On 28/07/2019 10:58, Bernd Zeimetz wrote: On 7/27/19 8:16 PM, Rebecca N. Palmer wrote: As a way to avoid relying on SHA-1, would it work to have git-debpush include a longer hash in the tag message, and tag2upload also verify that hash? what exactly would you create that long hash of? The signer's local files when they run git-debpush. (To be decided: how to define the hash of a directory tree (as opposed to a single file), i.e. "tar | sha256 like a .dsc" or "what git uses but sha256".) The hash security is for ensuring that tag2upload is seeing the same content as the signer did, and not something different an attacker placed on Salsa. (If the attacker can get their changes into the signer's local copy without the signer noticing, we'd have a problem whatever method the signer uses to upload it.) This does sort of raise the question of why not prefer "keep .dscs, but hide them from the user and regenerate tarballs", but this might be inappropriately reopening an already decided issue. (I remember it being suggested before, but not what (if any) response this got.) (+/=/- are relative to the existing proposal) + Security: dak doesn't have to trust dgit-repos-server (avoids both weak hashes and potential bugs) + Compatibility: finding the signer's name from the .dsc still works = Uploader only needs to do 'git debpush' = Doesn't spend uploader's (possibly low/expensive) bandwidth on uploading what Salsa already has - Someone would have to implement it (if that's me - not in Perl and I'm not a DD or a security specialist) git-debpush: create .dsc # as normal create tag # as normal, only needs version number sign tag # not strictly required, but since the next step # needs a key anyway, good to automate best practice sign .dsc push tag to Salsa upload .dsc to dgit-repos-server # but not its tarballs dgit-repos-server --tag2upload: receive .dsc check .dsc signature # do this first to prevent DoS # maybe also check the version number to prevent DoS by # re-submitting old/non-Debian .dscs fetch source from Salsa create source package tarballs check if these match the .dsc hashes # not strictly required as # dak will do it again anyway, but easy dput the .dsc+tarballs # as normal # not sure where .changes fits into this: # replace ".dsc" by ".dsc+.changes" throughout? # or have dgit-repos-server create .changes as if it were a buildd?
Re: tag2upload (git-debpush) service architecture - draft
Rebecca N. Palmer writes ("Re: tag2upload (git-debpush) service architecture -
draft"):
> The signer's local files when they run git-debpush. (To be decided: how
> to define the hash of a directory tree (as opposed to a single file),
> i.e. "tar | sha256 like a .dsc" or "what git uses but sha256".)
This would of course be possible. I don't think it's a particularly
good idea though. What it amounts to is a parallel Merkle tree to the
git one, just with a different data format and a better hash.
The upside is the better hash, but I think our overall risk from the
git SHA-1 problem is (i) still in practice quite low (ii) exists in
all the other places we rely on git already.
The downside is that the tag is no longer just a normal signed git tag
with some easy to construct and easy to understand metadata. It will
in practice then not be practical to make this tag other than with
git-debpush (or some other special utility with the same code).
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi,
Quoting Mo Zhou (2019-07-28 16:03:43)
> On 2019-07-28 13:13, Ian Jackson wrote:
> > This is maybe not directly helpful to you right now, but:
> >
> > If we could build-depend on source packages, you could combine these
> > two ideas into something that might be less awful.
> More than once have I thought of "what if I can B-D on a source package". Is
> such demand common enough among developers?
recently there was some discussion about this in #debian-apt. I don't have the
logs of that discussion so others might want to expand on what I remember from
back then. In no particular order plus my own thoughts.
apt developers are in favour of such a feature but it would need implementation
in dpkg first. This means that dpkg would have to also track "installed"
(unpacked) source packages in /usr/src in a similar way of how it currently
tracks installed binary packages.
Tons of software that parses the Build-Depends field has to be patched. At
least: dpkg, sbuild, apt, debhelper, cdbs, pbuilder, lintian, dose3,
wanna-build, dak, devscripts, python-debian, libconfig-model-perl, augeas,
haskell-debian, dh-exec, autopkgtest...
We have to think about a good syntax for the Build-Depends field which is able
to express a build dependency on source packages unpacked to /usr/src as well
as a build dependency on the build dependencies of a certain source package for
a given host architecture. Some initial thoughts:
Build-Depends: src:foo:src -> src:foo unpacked in /usr/src/foo
Build-Depends: src:foo -> b-d of src:foo for the current host
architecture
Build-Depends: src:foo:amd64-> b-d of src:foo for amd64
Build-Depends: src:foo:native -> b-d of src:foo for the current build
architecture
Here, ":src" is a new suffix next to :native, :any or :${arch} indicating the
"source architecture", meaning an unpacked source package. It is also open what
a B-D on foo:src (without the src: prefix) would mean. Maybe there is a more
elegant solution.
If we choose the "src:" prefix to pick source instead of binary packages, then
it's important that we don't have any binary packages called "src" and no
source packages with a name equal to a valid debian architecture.
I think it's important to separate a dependency on the source code of src:foo
and a dependency on the build dependencies of src:foo. Sometimes one needs only
parts of src:foo and if unpacked source and its build dependencies are always
installed together, then there is no way to get one without the other.
And there has to be a syntax for how to "install" these from the command line
using apt-get.
And there is the question of whether source packages of different versions
should be installable at the same time, maybe into /usr/src/foo-${version}.
Currently we have foo-source binary packages which install their content into
/usr/src but this requires such a package to be created and there is currently
no way to get the precise build dependencies on that package.
There is also the question about build profiles. Should it be possible to only
request the build dependencies of a source package with a certain set of build
profiles active?
Are other distributions doing something similar already?
Thanks!
cheers, josch
signature.asc
Description: signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 07:03:43AM -0700, Mo Zhou wrote: > > This is maybe not directly helpful to you right now, but: > > > > If we could build-depend on source packages, you could combine these > > two ideas into something that might be less awful. > > More than once have I thought of "what if I can B-D on a source > package". > Is such demand common enough among developers? `apt-file search -l /usr/src` may be of interest (after you skip all kernel-related packages). -- WBR, wRAR signature.asc Description: PGP signature
B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On 2019-07-28 13:13, Ian Jackson wrote: > This is maybe not directly helpful to you right now, but: > > If we could build-depend on source packages, you could combine these > two ideas into something that might be less awful. More than once have I thought of "what if I can B-D on a source package". Is such demand common enough among developers? > Anyway, good luck.
Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi Phil, On 2019-07-28 10:14, Phil Morrell wrote: > Without knowing anything more than what you've written here (which I > didn't find too long), it sounds like maintenance burden is the concern. The situation forces we Julia package maintainers to decide whether to use such a non-standard openblas variant. If we did't link against it, the built-in package manager (similar to python's pip) will be hard for Debian users to use. If we linked against it, we get a weird and duplicated binary in the archive but the usefulness wouldn't be harmed. > Am I right in thinking that there still exists non-Julia software for > which your solution is cleaner than symbol mangling? Is that LAPACK? The 64-bit indexing variant is common in the scientific computing area. Providing the BLAS64/LAPACK64 alternative group without mangling symbol names makes sense, because the other BLAS64/LAPACK64 alternatives such as BLIS and intel's MKL don't mangle symbol. > What you would do today if you were packaging it from scratch? If you > would pick the Julia approach for ease of maintenance then a SONAME > transition seems "simple" enough. If you would implement the cleaner > no-mangling approach, then a libopenblas-julia compatibility dependency > (option 2) would isolate the problem to the julia ecosystem. I tend to choose option 3. Expecially if I'm packaging it from scratch.
And in 2019? Re: -flto to become more of a routine - any change in opinion since 2011?
Steffen Möller writes ("And in 2019? Re: -flto to become more of a routine -
any change in opinion since 2011?"):
> We just had SuSE embracing LTO
> (https://www.linuxtoday.com/infrastructure/opensuse-enables-lto-by-default-for-tumbleweed-smaller-faster-binaries.html).
> I am not sure about the progress on issues summarised in
> http://blog.regehr.org/archives/1180 that Ian pointed to. But since I
> last asked in 2016 we have more pedantic compiler settings and more CI -
> and LTO, as much as compilers have improved on that, does not need to be
> applied everywhere. Any change in opinion?
Thanks for asking. I think this needs to be dealt with by way of risk
assessment.
Within a particular source package or maybe within a particular
ecosystem, it may be possible to make an argument that there are no
(or few) new misoptimisation opportunities.
Across the distribution as a whole I think it is far too risky unless
we get some buy-in from compiler folks and a way to limit the kind of
things the compiler might do.
The biggest problem with LTO is this:
We design our whole system, and particularly C code, with knowledge of
how compilers translate source code for externally-visible objects
into a concrete ABI. (This is even documented, of course, in ABI
specifications.)
With knowledge of that ABI, we often make changes where we link
together code which has a compatible ABI but where the source code
expression of the interface is not identical.
But C's rules for what functions and structs and so on are equivalent
enough to call across translation unit boundaries are much much
stricter than the rules implied by the ABI specifications.
Now so far this seems OK because dynamic linking means that we aren't
actually doing LTO across these ABI boundaries (yet). However, of
course, we sometimes link things statically.
Without LTO the header files can describe the ABI and don't need to
exactly match, according to the stricter C rules, the definitions of
the functions. With LTO the header files must match exactly.
Making the header files match exactly sounds easy because it seems
we're talking now about things in the same source package. But:
* Sometimes different source packages intend to provide
the same ABI and then the headers might be in a different
source package.
* Authors of libraries often want to write forward- or backward-
compatibility arrangements in the headers. When you do this
it is often convenient for build system reasons to build
some parts of the system with some set of compatibility #defines
or whatever and some parts with a different set. This is
OK so long as the ABI is right - until LTO.
* Other scenarios I haven't thought of right now.
Basically, before LTO we could all assume that cross-translation-unit
compatibility was determined entirely by ABI rules. These ABI rules
are comprehensible, relatively flexible, roughly equivalent in
implications across different architectures (at least in the commonly
used subset), and embedded both in our body of existing software and
in our programming habits.
With LTO, the ABI rules go out of the window whenever LTO applies and
instead we must satisfy the C standard rules. The C standard rules
are obtuse, insanely complicated, and generally hard to satisfy or
even analyse in nontrivial cases. Textually-(nearly)-identical
declarations suffice of course, but it is often necessary or desirable
to effectively use the declaration to specify the ABI and then rely on
ABI compatibility.
I hope this explanation is helpful.
Ian.
Re: Proposal: Repository for fast-paced package backports
On 2019, ജൂലൈ 28 6:36:21 PM IST, Phil Morrell wrote: >On Tue, Jan 01, 2019 at 05:49:37PM +0530, Pirate Praveen wrote: >> I think STS (Short term support) will fit nicely with LTS. If there >is >> no serious objections, I'd go with this. > >As debconf is finishing, though I don't know if either of you attended >this year, has there been any progress on this idea? Is there an >evergreen/sts/fasttrack destination I can put in my dput.cf to support >normally unsuitable packages like jenkins/virtualbox/firefox/gitlab? Hi Phil, It stalled for a long time and we restarted work on it recently. We are in the process of getting server space to setup dak. Praveen -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Challenge from Julia's non-standard vendored openblas"64_"
Mo Zhou writes ("Challenge from Julia's non-standard vendored openblas"64_""):
> [stuff]
How awkward.
> 2. Specifically compile a libopenblas64_.so
>from src:openblas for julia's use.
...
> 3. Embed openblas source and let Julia's
This is maybe not directly helpful to you right now, but:
If we could build-depend on source packages, you could combine these
two ideas into something that might be less awful.
Anyway, good luck.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?
Marc Haber writes ("Re: do packages depend on lexical order or
{daily,weekly,monthly} cron jobs?"):
> On Sat, 27 Jul 2019 19:02:16 +0100, Ian Jackson
> >I worry about additional concurrency. Unlike ordering bugs,
> >concurrency bugs are hard to find by testing. So running these
> >scripts in parallel is risky.
> >
> >And, I think running cron.fooly scripts in parallel is a bad idea.
> >The objective is to run them "at some point", not to get it done as
> >soon as possible. Running them in sequence will save electricity,
> >may save wear on components, and will reduce overall impact on other
> >uses of the same system.
>
> I fully agree with that. However, moving away from "in sequence" thing
> would greatly ease the migration to systemd timers, making it easier
> to get away without crond on many systems.
Why can't systemd run cron.fooly as one big timer job rather than one
timer job for each script ?
That leaves cron.fooly running in parallel with cron.barly but that is
something these jobs have to cope with anyway ?
> I am wondering whether this is something we should think about giving
> up in future. We have given up so many things since we moved to
> systemd, would it be worth throwing this out of the window as well?
Obviously, I don't think it is a good idea to break this for
non-systemd users because of difficulties making it work properly
with systemd. Perhaps I have misunderstood you ?
Thanks,
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: Proposal: Repository for fast-paced package backports
On Tue, Jan 01, 2019 at 05:49:37PM +0530, Pirate Praveen wrote: > I think STS (Short term support) will fit nicely with LTS. If there is > no serious objections, I'd go with this. As debconf is finishing, though I don't know if either of you attended this year, has there been any progress on this idea? Is there an evergreen/sts/fasttrack destination I can put in my dput.cf to support normally unsuitable packages like jenkins/virtualbox/firefox/gitlab? signature.asc Description: PGP signature
Re: file(1) now with seccomp support enabled
❦ 28 juillet 2019 12:11 +02, Philipp Kern : >> Just a quick note: seccomp filters may need adaptations from one libc >> to another (and from one kernel to another as the libc may adapt to >> the current kernel). For example, with the introduction of "openat" >> syscall, the libc has started to use it for "open()" and the new >> syscall has to be whitelisted. On the other hand, if you start >> implementing seccomp filters late, you may have whitelisted only the >> "openat" syscall while older libc (or current libc running on older >> kernels) will invoke the "open" syscall. >> >> I am upstream for a project using seccomp since a long time and I have >> never been comfortable to enable it in Debian for this reason. However, >> they enable it in Gentoo and I get the occasional patches to update the >> whitelist (I am not doing anything fancy). > > But technically it should be possible to test this in an autopkgtest, > no? I don't think perfect has to be the enemy of good here, as long as > we can detect breakage and remediate it afterwards? Yes, but I was thinking to cases where you run kernel from oldstable with stable for example. This is something currently allowed that may not work anymore. I am just providing the information, I don't have a strong opinion if seccomp should or shouldn't be enabled. -- Terminate input by end-of-file or marker, not by count. - The Elements of Programming Style (Kernighan & Plauger) signature.asc Description: PGP signature
Re: anti-tarball clause and GPL
(debian-devel following Holger's advice, guessing all authors are subscribed) On Thu, Jul 25, 2019 at 10:43:12PM -0300, Yao Wei (魏銘廷) wrote: > What if, one of the upstream authors consider it violating GPL _without_ the > clause? I mean, it could happen. Indeed, and I'd argue this is already the case (implicitly). Consider a security bug reported in Buster, missing from Stretch, that when a fix is found needs to be backported to all upstream supported releases (say, for example, going all the way back to one released just after Stretch). Other than skimming release notes for the affecting change, what would a diligent upstream do to determine which release streams need patching? Would they do a git checkout of each tag, manually testing, but otherwise merely using git as the tarball distribution mechanism? Or would they do a git log on the fixed file to determine when the change was first introduced, then simply git tag --contains? Even better, if the fix is not even known yet, could they `git bisect run foo` to find the breaking commit, then use that knowledge to work out the fix? On Thu, 25 Jul 2019 at 17:40, Adam Borowski wrote: > On Wed, Jul 24, 2019 at 05:18:28AM +, Scott Kitterman wrote: > > On July 24, 2019 12:34:13 AM UTC, Adam Borowski wrote: > > >By this logic, a pile of .c files with comments removed or preprocessed > > >with cpp would be allowed as well. The VCS is also a means to store > > >human-readable comments. > > I infer from this you think projects without a public VCS (postfix is an > > example) belong in non-free? > > At this moment, not yet. Obviously, old projects didn't even _have_ a VCS, > and I'm not proposing imposing a VCS workflow on the upstream. I'd like to > consider, at some point in the future, hidden private VCSes where the upstream > occassionally releases a tarball of to be non-free, just like the same PNG Yes, yes, yes - if upstream would prefer to modify their source with the support that their private git repo provides, then publishing just the make dist tarball does not make sense. The repo doesn't have to publicly-writable or accept pull requests, perhaps even doesn't need to have the entire project history (shallow clone since last release?). On Fri, Jul 26, 2019 at 04:00:04AM +0900, Norbert Preining wrote: > On Wed, 24 Jul 2019, Yao Wei wrote: > > I believe that "flat" tarball in Adam's question means tarball stripping > > out VCS information, not tarball as a format. > > Just one hint, if this comes in I will upload texlive with about a > 70Gb tarball as source ... we have 15 years of history of "flat tarball > of 4Gb". > > I don't think that *this* is the preferred form of changes. Then why do *you* use it to make changes? This would also be a good opportunity to improve Debian's 9G+ support (also for game assets). For the record, the texlive .git is 28G and as mentioned, we could consider the replacement for .orig. to be a shallow bare clone since last upload (git-debpush wishlist?). signature.asc Description: PGP signature
Re: file(1) now with seccomp support enabled
On 2019-07-27 10:01, Vincent Bernat wrote: Just a quick note: seccomp filters may need adaptations from one libc to another (and from one kernel to another as the libc may adapt to the current kernel). For example, with the introduction of "openat" syscall, the libc has started to use it for "open()" and the new syscall has to be whitelisted. On the other hand, if you start implementing seccomp filters late, you may have whitelisted only the "openat" syscall while older libc (or current libc running on older kernels) will invoke the "open" syscall. I am upstream for a project using seccomp since a long time and I have never been comfortable to enable it in Debian for this reason. However, they enable it in Gentoo and I get the occasional patches to update the whitelist (I am not doing anything fancy). But technically it should be possible to test this in an autopkgtest, no? I don't think perfect has to be the enemy of good here, as long as we can detect breakage and remediate it afterwards? Technically you cannot use any non-vendored libraries when enabling seccomp if you reason about it this way. Practically it mostly works except sometimes when the filters need to be adjusted. And as you can see Gentoo deals with that just fine and we could accept some breakage in unstable too, as long as the migration of the breaking library is stopped until the fix for the dependencies is in. Kind regards Philipp Kern
Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 02:30:15AM -0700, Mo Zhou wrote: > problem is that the 64-bit indexing variant doesn't > have a standard SONAME. Without knowing anything more than what you've written here (which I didn't find too long), it sounds like maintenance burden is the concern. Am I right in thinking that there still exists non-Julia software for which your solution is cleaner than symbol mangling? Is that LAPACK? > long time ago, we (mainly BLAS/LAPACK maintainers) > decided to use the "SONAME=libXXX64.so" convention > without mangling symbol names. Mangling is not > considered because only openblas supports it. What you would do today if you were packaging it from scratch? If you would pick the Julia approach for ease of maintenance then a SONAME transition seems "simple" enough. If you would implement the cleaner no-mangling approach, then a libopenblas-julia compatibility dependency (option 2) would isolate the problem to the julia ecosystem. > their vendored OpenBLAS to "libopenblas64_.so.*", Ugh, vendoring "compiles for me, so it's your problem". > I have no confidence at all to convince > upstream to change their widely-spread practice. Even when that's the case, it's usually still worth reporting the issue upstream, so they know the pain they're introducing to potential users. All the best from an outsider, and thank you for tackling difficult interoperability decisions in Debian. -- Phil Morrell (emorrp1) signature.asc Description: PGP signature
Re: tag2upload (git-debpush) service architecture - draft
On 7/27/19 8:16 PM, Rebecca N. Palmer wrote: > As a way to avoid relying on SHA-1, would it work to have git-debpush > include a longer hash in the tag message, and tag2upload also verify > that hash? what exactly would you create that long hash of? If we don't trust sha-1, then we might also not be able to trust the linked list of commits a git tag is pointing to. -- Bernd ZeimetzDebian GNU/Linux Developer http://bzed.dehttp://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?
On 7/28/19 8:47 AM, Marc Haber wrote: > Setting RandomizeDelaySecs sufficiently high on our daily jobs would > probably help to chop off the load pike that especially virtualization > setups running many Debian instances suffer from at 06:25 or 07:35. I > think this could be a net gain worth pursuing. definitely. we are able to see the daily logrotate/ run in our power usage statistics... -- Bernd ZeimetzDebian GNU/Linux Developer http://bzed.dehttp://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
Challenge from Julia's non-standard vendored openblas"64_"
Hi developers, A long existing historical problem that stems from Fortran finally started to bite people. --- Background BLAS/LAPACK is a set of very stable API/ABIs for dense numerical linear algebra, of "libc-level" importance to scientific computing users. The standard/reference/low-performance implementation was written in fortran. Fortran compilers support a weird feature to change the default INTEGER length. Which means if you write a fortran subroutine (in C-pseudo) sasum(INT N, float* X, INT INCX) You'll get the sasum(int64_t, float*, int64_t) function with a "-fdefault-integer-8" option. The "INT" arguments in the above example are used for 1-d array indexing. The scientific computing community has got used to this fortran feature, and many compatible BLAS/LAPACK implementations support build flags to select the "default integer size" even if the implementation is written in C. The historical problem is that the 64-bit indexing variant doesn't have a standard SONAME. --- Debian's WIP BLAS64/LAPACK64 libraries I'm working on the 64bit-indexing support for Debian's BLAS/LAPACK libraries. As discussed very long time ago, we (mainly BLAS/LAPACK maintainers) decided to use the "SONAME=libXXX64.so" convention without mangling symbol names. Mangling is not considered because only openblas supports it. Actually, isolating the 32-bit and 64-bit indexing variants with different SONAMES is expected to be enough for avoiding confliction. --- Julia's practice To avoid symbol confliction with the 32bit-indexing BLAS/LAPACK, Julia upstream changed the SONAME of their vendored OpenBLAS to "libopenblas64_.so.*", and **MANGLED** all the BLAS/LAPACK symbols by appending the "64_" suffix. As a result, a portion of the pre-built binaries they release are linked against this libopenblas64_.so, and requiring the mangled symbols (readelf -sW xxx). At the same time the linux distribution Julia packages built against 32bit version of 64-bit version without symbol mangling will run into problem as long as the user tries to install some fundamental .jl packages with the Julia's built-in package manager. I have no confidence at all to convince upstream to change their widely-spread practice. If I 1. Patch Julia code and link it against the WIP BLAS64/LAPACK64 libraries (where there is no symbol mangling). Users will always have problem installing any external .jl packages. 2. Specifically compile a libopenblas64_.so from src:openblas for julia's use. This looks very bad for src:openblas's package layout. 3. Embed openblas source and let Julia's build system compile whatever it wants. In this case everything will be fine, except for the technical gracefulness. I hesitate to choose a solution from them. Advise? --- P.S. This mail is again very long. But I'm afraid that most people won't be able to understand the problem if I only describe the core problem with several simple words.
