Re: cryptic pkgin SSL cert error
On Tue, 23 Apr 2024 at 15:24, Martin Husemann wrote: > > On Tue, Apr 23, 2024 at 03:17:14PM +0100, David Brownlee wrote: > > However, while better checking of trust anchors is a better end state > > - assuming I am understanding the situation correctly: in an > > effectively unannounced change, pkgin on a -9 system without either > > security/mozilla-rootcerts-openssl installed or /etc/openssl will now > > just fail, including any attempt to install mozilla-rootcerts-openssl > > to resolve. > > Only if the binary pkgs repository URL was using https. > Default setup used to be http: Aha, thanks! - that would be the item of information I lacked :) > > This requires manual intervention to set an environment variable to > > allow mozilla-rootcerts-openssl to be installed, or otherwise setup > > /etc/openssl. That would appear to be an unhelpful change, to the > > extent that I would propose pkgin on netbsd < 10 might be better to > > default to disabling checking trust anchors (with a warning). > > Edit the URL, install mozilla-rootcerts-openssl, change the URL back. I would still classify it as unhelpful, but if it is only affecting users who have changed their setup from the recommended, then it is more of a "it would be good to see if there is a was to help them" rather than an "oops!!" :-p I also appreciate the amount of bikeshedding and general pulling at different angles it took to get to where we are with it working well on -10... so as long as the default & recommended pkgin install on < netbsd-10 is for http rather than https, I'm inclined to leave well enough alone Thanks David
Re: cryptic pkgin SSL cert error
On Tue, Apr 23, 2024 at 03:17:14PM +0100, David Brownlee wrote: > However, while better checking of trust anchors is a better end state > - assuming I am understanding the situation correctly: in an > effectively unannounced change, pkgin on a -9 system without either > security/mozilla-rootcerts-openssl installed or /etc/openssl will now > just fail, including any attempt to install mozilla-rootcerts-openssl > to resolve. Only if the binary pkgs repository URL was using https. Default setup used to be http: > This requires manual intervention to set an environment variable to > allow mozilla-rootcerts-openssl to be installed, or otherwise setup > /etc/openssl. That would appear to be an unhelpful change, to the > extent that I would propose pkgin on netbsd < 10 might be better to > default to disabling checking trust anchors (with a warning). Edit the URL, install mozilla-rootcerts-openssl, change the URL back. Martin
Re: cryptic pkgin SSL cert error
On Tue, 23 Apr 2024 at 12:45, Greg Troxel wrote: > > David Brownlee writes: > > > Do you have security/mozilla-rootcerts-openssl installed? (which > > should provide a full set of certs in /etc/openssl). Alternatively > > what do you have in /etc/openssl > > > > For netbsd-10 /etc/openssl is populated by the OS, but doing that > > would be a breaking change on netbsd-9, however it may be that the > > latest pkgin is enforcing SSL certificates by default on netbsd-9 > > which would be... unhelpful in this case > > I don't see it as uhelpful -- doctrine has always been that the sysadmin > should choose which CAs to configure as trust anchors. In 10, that's > still more or less doctrine, except the default set is mozilla (or ish) > rather than the empty set. If you haven't set up trust anchors, lots of > things are troubled. For -10, or systems which ship with trust anchors in /etc/openssl or equivalent I would agree the changed behaviour is an absolute improvement. However, while better checking of trust anchors is a better end state - assuming I am understanding the situation correctly: in an effectively unannounced change, pkgin on a -9 system without either security/mozilla-rootcerts-openssl installed or /etc/openssl will now just fail, including any attempt to install mozilla-rootcerts-openssl to resolve. This requires manual intervention to set an environment variable to allow mozilla-rootcerts-openssl to be installed, or otherwise setup /etc/openssl. That would appear to be an unhelpful change, to the extent that I would propose pkgin on netbsd < 10 might be better to default to disabling checking trust anchors (with a warning). If I have misunderstood the situation - my apologies. David
Re: cryptic pkgin SSL cert error
David Brownlee wrote: > On Tue, 23 Apr 2024 at 02:27, beaker wrote: > > I have a 9.3/i386 VM on which I recently ran > > $ sudo pkgin update ; sudo pkgin upgrade ;sudo pkgin autoremove > > > > which worked but subsequent attempts to use pkgin report the following > > error: > > > > -- > > $ sudo pkgin update > > cleaning database from > > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/9.3/All entries... > > reading local summary... > > processing local summary... > > processing remote summary > > (https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/9.3/All)... > > 3061459968:error:1416F086:SSL > > routines:tls_process_server_certificate:certificate verify > > failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1921: > > 3061459968:error:1416F086:SSL > > routines:tls_process_server_certificate:certificate verify > > failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1921: > > 3061459968:error:1416F086:SSL > > routines:tls_process_server_certificate:certificate verify > > failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1921: > > pkgin: Could not fetch > > https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/9.3/All/pkg_summary.xz: > > Authentication error > > -- > > > > A work-around is to edit /usr/pkg/etc/pkgin/repositories.conf so > > it only uses http not https but I'd really rather not do that going > > forward so I'm looking for some guidance on how to fix wahatever > > is causing this SSL certificate verification error. > > > > System info: > > $ pkgin -v > > pkgin 23.8.1 (using SQLite 3.26.0) > > $ uname -a |cut -d' ' -f4-12 > > NetBSD 9.3_STABLE (GENERIC) #0: Mon Mar 25 15:54:20 UTC > > $ uname -m > > i386 > > Do you have security/mozilla-rootcerts-openssl installed? (which > should provide a full set of certs in /etc/openssl). Alternatively > what do you have in /etc/openssl > > For netbsd-10 /etc/openssl is populated by the OS, but doing that > would be a breaking change on netbsd-9, however it may be that the > latest pkgin is enforcing SSL certificates by default on netbsd-9 > which would be... unhelpful in this case Thanks, installing the mozilla-rootcerts-openssl pkg then re-editing ../pkgin/repositories.conf to use "https" worked. You're probably right about this being sort of a transitory issue mostly affecting 9.x, I just hadn't encountered it before and I've a handful of 9.x systems. Probably the forementioned rootcert pkg is already present on those. -B
Re: cryptic pkgin SSL cert error
David Brownlee writes: > Do you have security/mozilla-rootcerts-openssl installed? (which > should provide a full set of certs in /etc/openssl). Alternatively > what do you have in /etc/openssl > > For netbsd-10 /etc/openssl is populated by the OS, but doing that > would be a breaking change on netbsd-9, however it may be that the > latest pkgin is enforcing SSL certificates by default on netbsd-9 > which would be... unhelpful in this case I don't see it as uhelpful -- doctrine has always been that the sysadmin should choose which CAs to configure as trust anchors. In 10, that's still more or less doctrine, except the default set is mozilla (or ish) rather than the empty set. If you haven't set up trust anchors, lots of things are troubled.
Re: cryptic pkgin SSL cert error
On Tue, 23 Apr 2024 at 02:27, beaker wrote: > > Hello, > > I have a 9.3/i386 VM on which I recently ran > $ sudo pkgin update ; sudo pkgin upgrade ;sudo pkgin autoremove > > which worked but subsequent attempts to use pkgin report the following error: > > -- > $ sudo pkgin update > cleaning database from > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/9.3/All entries... > reading local summary... > processing local summary... > processing remote summary > (https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/9.3/All)... > 3061459968:error:1416F086:SSL > routines:tls_process_server_certificate:certificate verify > failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1921: > 3061459968:error:1416F086:SSL > routines:tls_process_server_certificate:certificate verify > failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1921: > 3061459968:error:1416F086:SSL > routines:tls_process_server_certificate:certificate verify > failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1921: > pkgin: Could not fetch > https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/9.3/All/pkg_summary.xz: > Authentication error > -- > > A work-around is to edit /usr/pkg/etc/pkgin/repositories.conf so > it only uses http not https but I'd really rather not do that going > forward so I'm looking for some guidance on how to fix wahatever > is causing this SSL certificate verification error. > > System info: > $ pkgin -v > pkgin 23.8.1 (using SQLite 3.26.0) > $ uname -a |cut -d' ' -f4-12 > NetBSD 9.3_STABLE (GENERIC) #0: Mon Mar 25 15:54:20 UTC > $ uname -m > i386 Do you have security/mozilla-rootcerts-openssl installed? (which should provide a full set of certs in /etc/openssl). Alternatively what do you have in /etc/openssl For netbsd-10 /etc/openssl is populated by the OS, but doing that would be a breaking change on netbsd-9, however it may be that the latest pkgin is enforcing SSL certificates by default on netbsd-9 which would be... unhelpful in this case David
cryptic pkgin SSL cert error
Hello, I have a 9.3/i386 VM on which I recently ran $ sudo pkgin update ; sudo pkgin upgrade ;sudo pkgin autoremove which worked but subsequent attempts to use pkgin report the following error: -- $ sudo pkgin update cleaning database from http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/9.3/All entries... reading local summary... processing local summary... processing remote summary (https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/9.3/All)... 3061459968:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1921: 3061459968:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1921: 3061459968:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1921: pkgin: Could not fetch https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/9.3/All/pkg_summary.xz: Authentication error -- A work-around is to edit /usr/pkg/etc/pkgin/repositories.conf so it only uses http not https but I'd really rather not do that going forward so I'm looking for some guidance on how to fix wahatever is causing this SSL certificate verification error. System info: $ pkgin -v pkgin 23.8.1 (using SQLite 3.26.0) $ uname -a |cut -d' ' -f4-12 NetBSD 9.3_STABLE (GENERIC) #0: Mon Mar 25 15:54:20 UTC $ uname -m i386
Re: pkg_add and pkgin install taking extremely long
On Mon, Mar 25, 2024 at 05:40:44PM +0100, Enrico Weigelt, metux IT consult wrote: > A simple `pkg_add pkgin` runs for over a quarter hour, and pkgin install > call took another half an hour, until it recognized a wrong parameter: echo 'ip6addrctl=YES' >> /etc/rc.conf echo 'ip6addrctl_policy="ipv4_prefer"' >> /etc/rc.conf
Re: pkg_add and pkgin install taking extremely long
On 25.03.24 20:40, Justin Parrott wrote: This is not an issue with the local system. maybe a combination of both the guest and the host (maybe host offering IPv6 address but no actual routing). But fortunately fixed it with some tweaks now :) --mtx -- --- Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren GPG/PGP-Schlüssel zu. --- Enrico Weigelt, metux IT consult Free software and Linux embedded engineering i...@metux.net -- +49-151-27565287
Re: pkg_add and pkgin install taking extremely long
This is not an issue with the local system. On Mon, Mar 25, 2024 at 3:34 PM Enrico Weigelt, metux IT consult < l...@metux.net> wrote: > Hi @all, > > > Your timing is similar to what I had in some early tests. That said, > > have you measured what is the slow part? I bet it's the network, not > > specifically pkgin. > > meanwhile turned out it seems to be ipv6 related (somebody in irc gave > me a hint on that). calling pkgin with -4 makes it *a lot* faster > (pkg_add doesnt seem to have that switch). > > Also explicitly dropping ipv6 default route - now the job finishes > in under 5mins. > > > --mtx > > -- > --- > Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert > werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren > GPG/PGP-Schlüssel zu. > --- > Enrico Weigelt, metux IT consult > Free software and Linux embedded engineering > i...@metux.net -- +49-151-27565287 > -- Justin Allen Parrott
Re: pkg_add and pkgin install taking extremely long
Hi @all, Your timing is similar to what I had in some early tests. That said, have you measured what is the slow part? I bet it's the network, not specifically pkgin. meanwhile turned out it seems to be ipv6 related (somebody in irc gave me a hint on that). calling pkgin with -4 makes it *a lot* faster (pkg_add doesnt seem to have that switch). Also explicitly dropping ipv6 default route - now the job finishes in under 5mins. --mtx -- --- Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren GPG/PGP-Schlüssel zu. --- Enrico Weigelt, metux IT consult Free software and Linux embedded engineering i...@metux.net -- +49-151-27565287
Re: pkg_add and pkgin install taking extremely long
Probably in the queue. On Mon, Mar 25, 2024 at 3:10 PM Benny Siegert wrote: > Am 25.03.24 um 17:40 schrieb Enrico Weigelt, metux IT consult: > > I'm currently in process of setting up an CI build process for Xorg > > on NetBSD (inside Qemu), but encountering really long delays in package > > installations. > > > > A simple `pkg_add pkgin` runs for over a quarter hour, and pkgin install > > call took another half an hour, until it recognized a wrong parameter: > > I have experience with a similar setup, from setting up the NetBSD CI > images for the Go project. > > Your timing is similar to what I had in some early tests. That said, > have you measured what is the slow part? I bet it's the network, not > specifically pkgin. > > I don't see timing information in the CI log -- you could wrap the pkgin > calls with "time", or print timestamps before each command. > > Maybe you need to change something in the networking setup on the qemu > side to get more throughput? There is no dmesg output in the log, so I > would check if the network uses vioif, or an emulated interface. > > -- > Benny > -- Justin Allen Parrott
Re: pkg_add and pkgin install taking extremely long
Am 25.03.24 um 17:40 schrieb Enrico Weigelt, metux IT consult: I'm currently in process of setting up an CI build process for Xorg on NetBSD (inside Qemu), but encountering really long delays in package installations. A simple `pkg_add pkgin` runs for over a quarter hour, and pkgin install call took another half an hour, until it recognized a wrong parameter: I have experience with a similar setup, from setting up the NetBSD CI images for the Go project. Your timing is similar to what I had in some early tests. That said, have you measured what is the slow part? I bet it's the network, not specifically pkgin. I don't see timing information in the CI log -- you could wrap the pkgin calls with "time", or print timestamps before each command. Maybe you need to change something in the networking setup on the qemu side to get more throughput? There is no dmesg output in the log, so I would check if the network uses vioif, or an emulated interface. -- Benny
Re: pkg_add and pkgin install taking extremely long
All of That Sounds to be Correct to Me. On Mon, Mar 25, 2024 at 12:43 PM Enrico Weigelt, metux IT consult < l...@metux.net> wrote: > Hello folks, > > I'm currently in process of setting up an CI build process for Xorg > on NetBSD (inside Qemu), but encountering really long delays in package > installations. > > A simple `pkg_add pkgin` runs for over a quarter hour, and pkgin install > call took another half an hour, until it recognized a wrong parameter: > > https://gitlab.freedesktop.org/metux/ci-templates/-/jobs/56754224 > > Am I doing anything wrong ? > > > thx > --mtx > > -- > --- > Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert > werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren > GPG/PGP-Schlüssel zu. > --- > Enrico Weigelt, metux IT consult > Free software and Linux embedded engineering > i...@metux.net -- +49-151-27565287 > -- Justin Allen Parrott
pkg_add and pkgin install taking extremely long
Hello folks, I'm currently in process of setting up an CI build process for Xorg on NetBSD (inside Qemu), but encountering really long delays in package installations. A simple `pkg_add pkgin` runs for over a quarter hour, and pkgin install call took another half an hour, until it recognized a wrong parameter: https://gitlab.freedesktop.org/metux/ci-templates/-/jobs/56754224 Am I doing anything wrong ? thx --mtx -- --- Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren GPG/PGP-Schlüssel zu. --- Enrico Weigelt, metux IT consult Free software and Linux embedded engineering i...@metux.net -- +49-151-27565287
Re: holding back pkgs from pkgin
pin wrote:
> Apparently nobody has answered your question in the mailing-list.
>
> > Is there a way to hold back packages from pkgin similar to 'apt-mark '
> > ?
>
> The short answer is no, there isn't.
>
> > The pkg in question is ../wm/dwm which I customize in pkgscr and install.
> > pkgin sees it's installed and always over-writes my custom built binary
>
> For simple things like this, there's a way to fool pkgin that I've been using
> for years.
>
> But, first one warning. Do not change the contents of pkgsrc/wm/dwm directly.
> Use wip to do this things.
>
> Copy wm/dwm to wip, do your customization and bump the version (currently
> 6.4) to a higher number.
> Build and install the package.
> Now next time you upgrade pkgin will see that the installed version of dwm is
> than the available in the binary repository and won't touch it.
>
> Hope this solves your issue.
Sorry for late reply.. Ya this is an intersting approach. I ended
up just building dwm under /usr/local since it's not too complicated
to do.
FYI: For anyone considering building dwm under /usr/local/..
In addition to correcting the X11LIB, X11INC, and FREETYPEINC paths
one needs to do at least ONE of the following for the Xlib linking:
# option 1: tweak ../dwm-4.6/config.mk
#../dwm-4.6/config.mk
..
LIBS += -Wl,-rpath=${X11LIB}
# option 2: set LD_LIBRARY_RUN
$ export LD_LIBRARY_RUN=/usr/X11R7/lib
# option 3: create /etc/ls.so.conf
#/etc/ls.so.conf
/usr/X11R7/lib
$ sudo ldconfig
Probably many already know this. I was scratching my head for a bit
until I found the reason for the lack of linking, probably because
many projects use autoconfig to probe system and set things.
holding back pkgs from pkgin
Is there a way to hold back packages from pkgin similar to 'apt-mark ' ? Didn't see anything in the manpage and when I try to use 'pkgin import pkg_list.txt' where the not-to-be-updated pkg has been removed pkgin STILL tries to update the pkg. The pkg in question is ../wm/dwm which I customize in pkgscr and install. pkgin sees it's installed and always over-writes my custom built binary...
Re: the vine package in pkgin is wine64?
Hi Nia,Thanks for the link! However, there're plenty of win stuff that requires multiarch wine setup so neither 32 bit wine nor 64 bit wine is enough alone. E.g. even if the program is fully 64 bit but its installer is 32 bit, or if only one 32 bit lib is used then multiarch is required. I guess you know better than me but in case not: the wine64 package in pkgsrc/wip is multiarch although it's version is only 4.4. I hope it was documented during the google summer of code when it was developed how the package can be upgraded to newer versions, otherwise it'll get bitrotten :( Quite some time back I sent an email to the author about such docs to check if I was able to upgrade it but never got an answer.Best regards,r0ller Eredeti levél Feladó: nia Dátum: 2022 november 15 08:23:58Tárgy: Re: the vine package in pkgin is wine64?Címzett: r0ller On Fri, Nov 11, 2022 at 10:41:15PM +, r0ller wrote: > Running 32 bit win stuff does not work with that. I'm running 32-bit only WINE programs in a 32-bit NetBSD chroot (using sandboxctl). It works okay: https://washbear.neocities.org/wine-sandbox.html
Re: the vine package in pkgin is wine64?
On Fri, Nov 11, 2022 at 10:41:15PM +, r0ller wrote: > Running 32 bit win stuff does not work with that. I'm running 32-bit only WINE programs in a 32-bit NetBSD chroot (using sandboxctl). It works okay: https://washbear.neocities.org/wine-sandbox.html
Re: the vine package in pkgin is wine64?
Got the answer: that package in pkgin is just the wine64 package and not the WoW64. So running 32 bit win stuff does not work with that. That's a pity as the WoW64 (called wine64 in pkgsrc/wip) is only the 4.4 version. Eredeti levél Feladó: r0ller Dátum: 2022 november 11 11:14:40Tárgy: the vine package in pkgin is wine64?Címzett: netbsd-users@netbsd.org Hi All,I've just discovered that now there's a wine package in pkgin :) As I'm on amd64 it must be the wine64 which previously resided in pkgsrc/wip only, right?Therefore, I tried to enable the USER_LDT option and disable SVS in a custom kernel config but I couldn't build it (see my previous email: building new kernel on upgraded system).I know that SVS can be disabled via sysctl but is it possible to enable user_ldt on the fly as well?Best regards,r0ller
the vine package in pkgin is wine64?
Hi All,I've just discovered that now there's a wine package in pkgin :) As I'm on amd64 it must be the wine64 which previously resided in pkgsrc/wip only, right?Therefore, I tried to enable the USER_LDT option and disable SVS in a custom kernel config but I couldn't build it (see my previous email: building new kernel on upgraded system).I know that SVS can be disabled via sysctl but is it possible to enable user_ldt on the fly as well?Best regards,r0ller
Re: 9.2: pkgin packages do not register pkg_info
Riccardo Mottola writes: > Hi Greg, > > Greg Troxel wrote: >> I don't either, but my advice is to *always* set PKG_DBDIR explicitly in >> btoh mk.conf and pkg_install.conf. Check if you have the split brain >> situation, or something else: >> >> https://pkgsrc.org/pkgdb-change/ > > oh, I fear I have split. But, permit me, this is bad: I have a > super-fresh install; I used pkg_add to instappl pkgin and nothing more! > So what went wrong? > > localhost$ /usr/sbin/pkg_add -V > 20201218 > localhost$ /usr/pkg/sbin/pkg_add -V > 20210410 > > The versions are exactly the two which are cited as "good"! A good question. I'm not really sure, but would be good to check the config files and the compiled in default (by checking behavior and 'strings'). > > I will follow the script... but it is bad to land this way without > actually "doing" an upgrade... should we migrate essentially wth the > script a clean 9.2 install? I would expect 9.2 to be ok. Where did you get pkgin from? > in > /usr/pkg/pkgdb > > I essentially only have pkgin -> the one which I added with pkg_add So the pkg_add you used was ok. > in /var/db/pkg I have all the packages installed with the said pkgin! That surprises me. signature.asc Description: PGP signature
Re: 9.2: pkgin packages do not register pkg_info
Hi. replying to myself Riccardo Mottola wrote: > I fear this "dirs" come from this step: > > In /var/db/pkg.refcount > tar cf - . | (cd /usr/pkg/pkgdb && tar xfv -) > > since pkg.refcount contained "dirs" > > ls /usr/pkg/pkgdb/dirs/ > etc usr var > > is it out of place? the proper command should have been: In /var/db/pkg.refcount tar cf - . | (cd /usr/pkg/pkgdb.refcount && tar xfv -) otherwise one mixes up with th other files. Riccard
Re: 9.2: pkgin packages do not register pkg_info
Hi Greg, Greg Troxel wrote: > I don't either, but my advice is to *always* set PKG_DBDIR explicitly in > btoh mk.conf and pkg_install.conf. Check if you have the split brain > situation, or something else: > > https://pkgsrc.org/pkgdb-change/ oh, I fear I have split. But, permit me, this is bad: I have a super-fresh install; I used pkg_add to instappl pkgin and nothing more! So what went wrong? localhost$ /usr/sbin/pkg_add -V 20201218 localhost$ /usr/pkg/sbin/pkg_add -V 20210410 The versions are exactly the two which are cited as "good"! I will follow the script... but it is bad to land this way without actually "doing" an upgrade... should we migrate essentially wth the script a clean 9.2 install? in /usr/pkg/pkgdb I essentially only have pkgin -> the one which I added with pkg_add in /var/db/pkg I have all the packages installed with the said pkgin! Riccardo
Re: 9.2: pkgin packages do not register pkg_info
Hi Greg Greg Troxel wrote: > I don't either, but my advice is to *always* set PKG_DBDIR explicitly in > btoh mk.conf and pkg_install.conf. Check if you have the split brain > situation, or something else: > > https://pkgsrc.org/pkgdb-change/ I tried to perform the split migration according to the cited mail http://mail-index.netbsd.org/pkgsrc-users/2020/12/30/msg032987.html I got some warnings and I don't think it went totally well: localhost# pkg_admin check .pkg_admin: can't open /usr/pkg/pkgdb/dirs/+CONTENTS: No such file or directory localhost# pkg_admin rebuild-tree pkg_admin: Cannot read +CONTENTS of package dirs I fear this "dirs" come from this step: In /var/db/pkg.refcount tar cf - . | (cd /usr/pkg/pkgdb && tar xfv -) since pkg.refcount contained "dirs" ls /usr/pkg/pkgdb/dirs/ etc usr var is it out of place? Riccardo
Re: 9.2: pkgin packages do not register pkg_info
Riccardo Mottola writes: > I installed using pkg_add pkgin. Then I installed a myriad of packages > using "pkgin install", so to be quick and not compile stuff myself. > > I wonder however that "pkg_info" then does not list these packages, why? I don't either, but my advice is to *always* set PKG_DBDIR explicitly in btoh mk.conf and pkg_install.conf. Check if you have the split brain situation, or something else: https://pkgsrc.org/pkgdb-change/ signature.asc Description: PGP signature
9.2: pkgin packages do not register pkg_info
Hello, I just installed a fresh 9.2 on an HP ProBook amd64. It went quite smooth! (FreeBSD didn't even boot with SATA in AHCI mode...) Yay. Wireless worked out of the box, X11 too (well Intel graphics is a good bet there.) I installed using pkg_add pkgin. Then I installed a myriad of packages using "pkgin install", so to be quick and not compile stuff myself. I wonder however that "pkg_info" then does not list these packages, why? $ pkg_info pkg_install-20210410 Package management and administration tools for pkgsrc pkgin-20.12.1nb1 Apt / yum like tool for managing pkgsrc binary packages Yet I do know that I installed things like windowmaker, autoconf, etc etc and /usr/pkg/bin is full Where are they? Riccardo
Re: Regular NetBSD packaging and pkgin
On 12 Jul 2021, Greg Troxel wrote: > Benny Siegert writes: (snip) >> It sounds like pkgin and your pkg_* tools disagree on what the correct >> PKG_DBDIR is. Probably one of them is using /var/db/pkg and the other >> /usr/pkg/pkgdb. Check if you have both, set the location in the >> respective config files explicitly, and try merging the two db dirs. > > Agreed, and details at: > > https://pkgsrc.org/pkgdb-change/ Thank you both: thanks to the hints and those details, I have now restored the odd system to sanity, basically by using "pkg_delete -f", removing /usr/pkg/pkgdb/, then "pkgin install -f" and "pkgin unkeep" on what I deleted. It helped that the stray packages were simply dependencies that had no interesting associated configuration that I wanted to preserve. -- Mark
Re: pkgin does not install anything after upgrade to 9.2
Hi Greg, Thanks for the suggestion. I'm still doing the backup but will come back with the outcome. Best regards, r0ller Eredeti levél Feladó: Greg Troxel < g...@lexort.com (Link -> mailto:g...@lexort.com) > Dátum: 2021 július 9 13:06:24 Tárgy: Re: pkgin does not install anything after upgrade to 9.2 Címzett: r0ller < netbsd-users@netbsd.org (Link -> mailto:netbsd-users@netbsd.org) Boot to single user and full fsck. Also make the backups you have been meaning to get around to right away
Re: Regular NetBSD packaging and pkgin
Benny Siegert writes: > On Mon, Jul 12, 2021 at 2:46 AM Mark Carroll wrote: >> Does anybody have an idea what I messed up on the >> latter? I first noticed when "pkg_admin audit" was telling me less than >> I expected on that system. > > It sounds like pkgin and your pkg_* tools disagree on what the correct > PKG_DBDIR is. Probably one of them is using /var/db/pkg and the other > /usr/pkg/pkgdb. Check if you have both, set the location in the > respective config files explicitly, and try merging the two db dirs. Agreed, and details at: https://pkgsrc.org/pkgdb-change/ signature.asc Description: PGP signature
Re: Regular NetBSD packaging and pkgin
On Mon, Jul 12, 2021 at 2:46 AM Mark Carroll wrote: > Does anybody have an idea what I messed up on the > latter? I first noticed when "pkg_admin audit" was telling me less than > I expected on that system. It sounds like pkgin and your pkg_* tools disagree on what the correct PKG_DBDIR is. Probably one of them is using /var/db/pkg and the other /usr/pkg/pkgdb. Check if you have both, set the location in the respective config files explicitly, and try merging the two db dirs. -- Benny
Regular NetBSD packaging and pkgin
I've somehow misconfigured something and can't work out what. On one NetBSD 9.2 system, if I pkgin install something, then "pkg_info -u" reports it as installed. On the other, pkg_info doesn't seem to know anything about what I installed with pkgin, though "pkgin show-keep" still shows it fine. Does anybody have an idea what I messed up on the latter? I first noticed when "pkg_admin audit" was telling me less than I expected on that system. -- Mark
Re: pkgin does not install anything after upgrade to 9.2
Boot to single user and full fsck. Also make the backups you have been meaning to get around to right away
Re: pkgin does not install anything after upgrade to 9.2
Hi Greg, Thanks for your suggestions! It indeed seems to be corrupted :( I tried to nuke (rm -rf) the directory /usr/pkg/pkgdb/p5-Net-SSLeay-1.88nb1 but I got back that the directory is not empty which is pretty unusual for rm -rf. Now, when I try to list its contents, three files are shown: +COMMENT, +CONTENTS, +DESCR However, when listing with ls -al, I get the error: ls: +COMMENT: No such file or directory ls: +CONTENTS: No such file or directory ls: +DESCR: No such file or directory I tried to execute fsck for the node number (fsck 35752122) of the directory but I got back: fsck: cannot open '/dev/35752122': No such file or directory Is there any solution for this? Best regards, r0ller On 7/8/21 7:26 PM, Greg Troxel wrote: r0ller writes: The only error I see in pkg_install-err.log (which is not shown as error) which seems to block each pkg install is: pkg_admin: Cannot read +CONTENTS of package p5-Net-SSLeay-1.88nb1 Does anyone have any hint? pkg_admin rebuild-tree pkg_admin check make sure your variables in mk.conf are set to point to your pkgdb. See https://www.pkgsrc.org/pkgdb-change/ and set the variables even if you think it's not necessary. look in /usr/pkg/pkgdb/p5-Net-SSLeay-1.88nb1 and see if there is indeed no +CONTENTS. If true, you have had some corruption and I would rm -rf the p5-Net-SSLeay-1.88nb1 directory, and then you'll have to reinstall it.
Re: pkgin does not install anything after upgrade to 9.2
r0ller writes: > The only error I see in pkg_install-err.log (which is not shown as > error) which seems to block each pkg install is: > > pkg_admin: Cannot read +CONTENTS of package p5-Net-SSLeay-1.88nb1 > > Does anyone have any hint? pkg_admin rebuild-tree pkg_admin check make sure your variables in mk.conf are set to point to your pkgdb. See https://www.pkgsrc.org/pkgdb-change/ and set the variables even if you think it's not necessary. look in /usr/pkg/pkgdb/p5-Net-SSLeay-1.88nb1 and see if there is indeed no +CONTENTS. If true, you have had some corruption and I would rm -rf the p5-Net-SSLeay-1.88nb1 directory, and then you'll have to reinstall it. signature.asc Description: PGP signature
Re: pkgin does not install anything after upgrade to 9.2
On Thu, Jul 08, 2021 at 04:14:08PM +0200, Martin Husemann wrote: > On Thu, Jul 08, 2021 at 02:36:17PM +0200, r0ller wrote: > > is quite noticeable when starting firefox. However, after changing > > repositories.conf to point to amd64/9.2/All and upgrading all packages, > > nothing seems to have been upgraded and pkgin does not seem to install > > anything. > > The repositories are identical (9.2 is actually a symlink on the server > pointing at the 9.0 repo). > > Martin to expand: pkgsrc is developed separately and has a separate release schedule (approx. every 3 months, though packages can take a while to build). I switch the 9.0 link to the latest pkgsrc branch when I think it's ready.
Re: pkgin does not install anything after upgrade to 9.2
On Thu, Jul 08, 2021 at 02:36:17PM +0200, r0ller wrote: > is quite noticeable when starting firefox. However, after changing > repositories.conf to point to amd64/9.2/All and upgrading all packages, > nothing seems to have been upgraded and pkgin does not seem to install > anything. The repositories are identical (9.2 is actually a symlink on the server pointing at the 9.0 repo). Martin
pkgin does not install anything after upgrade to 9.2
Hi All, I've just upgraded from 9.0 (amd64) to 9.2 via sysinst successfully from ftp. Everything seems to work fine as earlier and surprisingly faster what is quite noticeable when starting firefox. However, after changing repositories.conf to point to amd64/9.2/All and upgrading all packages, nothing seems to have been upgraded and pkgin does not seem to install anything. There are many warnings about the platform difference (built for 9.0 vs 9.2) but no errors (except one maybe, see later). However, when e.g. installing firefox-86.0.1 pkgin acts as if it was doing its job, displays in the end the number of warnings (48 platform diff warns) and errors (0) but when I check the version with 'firefox --version' I still get 84.0. The only error I see in pkg_install-err.log (which is not shown as error) which seems to block each pkg install is: pkg_admin: Cannot read +CONTENTS of package p5-Net-SSLeay-1.88nb1 Does anyone have any hint? Thanks®ards, r0ller
Re: pkgin repo
On Wed, 7 Apr 2021, Dan Cîrnaț wrote: Telling pkgin: 'pkgin install gdm' led to 59 packages being "refreshed", and 4 more installed. Please be aware that those pkgin repos only have packages for the main pkgsrc tree, excluding pkgsrc-wip. The gdm version there is 2.x., an old one. I did not suffer a version downgrade; still at gdm-40.0. Thank you. -- RSB
Re: pkgin repo
On 06/04/2021 22:54, Bob Bernstein wrote: I am taking a first run at pkgin. It fails trying to get a file from: https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/9.99.81/All ...which can be found in my /usr/pkg/etc/pkgin/repositories.conf. I have here: $ uname -a NetBSD nebbytwo 9.99.81 NetBSD 9.99.81 (GENERIC) #0: Sat Mar 20 14:30:50 UTC 2021 mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64 Given the version of -current I have, what pkgin repository URI should I use? I'd use https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/9.0_current/ which should be a symlink to the latest current builds, if cdn.* follows the same layout as ftp.* -- Ottavio Caruso
Re: pkgin repo
On 07.04.21 03:44, Bob Bernstein wrote: Telling pkgin: 'pkgin install gdm' led to 59 packages being "refreshed", and 4 more installed. Please be aware that those pkgin repos only have packages for the main pkgsrc tree, excluding pkgsrc-wip. The gdm version there is 2.x., an old one. Dan
Re: pkgin repo
On Wed, 7 Apr 2021, Chavdar Ivanov wrote: I don't think there are "official" pkgin repos for -current. However, check https://pkgsrc.joyent.com/install-on-netbsd/ . What I put in my /usr/pkg/etc/pkgin/reposiories.conf was: https://pkgsrc.joyent.com/packages/NetBSD/trunk/x86_64/All/ Telling pkgin: 'pkgin install gdm' led to 59 packages being "refreshed", and 4 more installed. And I appear to have managed not to break anything that was already working. 9-) Thank you. -- RSB
Re: pkgin repo
On Tue, 6 Apr 2021 at 22:54, Bob Bernstein wrote: > > I am taking a first run at pkgin. It fails trying to get a file > from: > > https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/9.99.81/All > > ...which can be found in my > /usr/pkg/etc/pkgin/repositories.conf. > > I have here: > $ uname -a > NetBSD nebbytwo 9.99.81 NetBSD 9.99.81 (GENERIC) #0: Sat Mar 20 > 14:30:50 UTC 2021 > mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC > amd64 I don't think there are "official" pkgin repos for -current. However, check https://pkgsrc.joyent.com/install-on-netbsd/ . . I haven't used this repo yet, as I maintain a subset at home for the packages I use. > > Given the version of -current I have, what pkgin repository URI > should I use? > > Thank You! > > -- > RSB > --
pkgin repo
I am taking a first run at pkgin. It fails trying to get a file from: https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/9.99.81/All ...which can be found in my /usr/pkg/etc/pkgin/repositories.conf. I have here: $ uname -a NetBSD nebbytwo 9.99.81 NetBSD 9.99.81 (GENERIC) #0: Sat Mar 20 14:30:50 UTC 2021 mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64 Given the version of -current I have, what pkgin repository URI should I use? Thank You! -- RSB
Re: pkgin refresh vs. upgrade?
Mike, I neglected to thank you for this answer! Thanks, that actually makes sense. Cheers, /Liman On 17/06/2020 07:39, Lars-Johan Liman wrote: >> Hi! mpumf...@mudcovered.org.uk 2020-06-17 09:37 [+0100]: > This is all based on observation rather than documentation. >> When running pkgin to upgrade one or more packages, it often says "X >> packages to refresh, Y packages to upgrade". > Refresh means the version hasn't changed but the package has been > rebuilt since you last installed it from the repository. >> What's the difference between a refresh and an upgrade of a package? > Upgrade indicates a version change. >> And where is it documented? ;-) > That I don't know. Reason I know the answer is that I build my own > binary packages and because my build method builds everything every > time I see a lot more refreshes that I suspect would happen with > packages coming from a bulk build. > Mike
Re: pkgin refresh vs. upgrade?
On 17/06/2020 07:39, Lars-Johan Liman wrote: Hi! This is all based on observation rather than documentation. When running pkgin to upgrade one or more packages, it often says "X packages to refresh, Y packages to upgrade". Refresh means the version hasn't changed but the package has been rebuilt since you last installed it from the repository. What's the difference between a refresh and an upgrade of a package? Upgrade indicates a version change. And where is it documented? ;-) That I don't know. Reason I know the answer is that I build my own binary packages and because my build method builds everything every time I see a lot more refreshes that I suspect would happen with packages coming from a bulk build. Mike
pkgin refresh vs. upgrade?
Hi! When running pkgin to upgrade one or more packages, it often says "X packages to refresh, Y packages to upgrade". What's the difference between a refresh and an upgrade of a package? And where is it documented? ;-) Thanks! Cheers, /Liman
Re: pkgin error (possible workaround)
Thanks for the report. On Thu, May 21, 2020 at 12:38 PM Greg Troxel wrote: > > Lars-Johan Liman writes: > > > I've found that the x86_64 and amd64 directories seem to be out of sync > > on cdn.netbsd.org. In short, pkg_summary seems to be updated in x86_64 > > but not in amd64. For me the following simple workaround worked: > > Thanks for the specific report. > > > I modified the repository pointer in > > > > /usr/pkg/etc/pkgin/repositories.conf > > > > from: > > > > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/9.0/All > > > > to: > > > > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/x86_64/9.0/All > > amd64 is the name of the port, which is about specific hardware aas well > as the instruction set, and packages are generally shared among all > ports with the same instruction set and ABI. So arguably x86_64 is > preferred anyway -- even if x86_64 appears only on the amd64 port! > > For your info, on ftp.netbsd.org, in /pub/pkgsrc/packages/NetBSD, doing > ls -l (and skipping the other lines) one gets: > > lrwxrwxr-x 1 joerg netbsd 6 May 28 2008 amd64 -> x86_64 > drwxrwxr-x 16 pkgmastr netbsd 1024 May 15 02:57 x86_64 > > I can confirm the bad behavior. Fetching > >wget > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/9.0/All/pkg_summary.bz2 >wget > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/x86_64/9.0/All/pkg_summary.bz2 > > I get the following: (.1 is 2nd fetch): > > -rw-r--r-- 1 gdt users3270343 May 17 16:03 pkg_summary.bz2 > -rw-r--r-- 1 gdt users3258465 May 21 10:54 pkg_summary.bz2.1
Re: pkgin error (possible workaround)
Lars-Johan Liman writes: > I've found that the x86_64 and amd64 directories seem to be out of sync > on cdn.netbsd.org. In short, pkg_summary seems to be updated in x86_64 > but not in amd64. For me the following simple workaround worked: Thanks for the specific report. > I modified the repository pointer in > > /usr/pkg/etc/pkgin/repositories.conf > > from: > > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/9.0/All > > to: > > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/x86_64/9.0/All amd64 is the name of the port, which is about specific hardware aas well as the instruction set, and packages are generally shared among all ports with the same instruction set and ABI. So arguably x86_64 is preferred anyway -- even if x86_64 appears only on the amd64 port! For your info, on ftp.netbsd.org, in /pub/pkgsrc/packages/NetBSD, doing ls -l (and skipping the other lines) one gets: lrwxrwxr-x 1 joerg netbsd 6 May 28 2008 amd64 -> x86_64 drwxrwxr-x 16 pkgmastr netbsd 1024 May 15 02:57 x86_64 I can confirm the bad behavior. Fetching wget http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/9.0/All/pkg_summary.bz2 wget http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/x86_64/9.0/All/pkg_summary.bz2 I get the following: (.1 is 2nd fetch): -rw-r--r-- 1 gdt users3270343 May 17 16:03 pkg_summary.bz2 -rw-r--r-- 1 gdt users3258465 May 21 10:54 pkg_summary.bz2.1
Re: pkgin error (possible workaround)
Hi Dima (all)! I've found that the x86_64 and amd64 directories seem to be out of sync on cdn.netbsd.org. In short, pkg_summary seems to be updated in x86_64 but not in amd64. For me the following simple workaround worked: I modified the repository pointer in /usr/pkg/etc/pkgin/repositories.conf from: http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/9.0/All to: http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/x86_64/9.0/All then ran # pkgin update ... and it now works just fine. Why they are out of sync is "beyond my paygrade", and I leave it up to pkgsrc powers that be to figure that one out. ;-) Cheers, /Liman kab...@lich.phys.spbu.ru 2020-05-15 19:39 [+0300]: > I had same problems and I've found that CDN directory contents may be > not in sync with pkg_summary. I tried to work it out (thinking > there might be one server not in sync) but had short time and no > success so that time I just switched to legacy source. > How cdn is organized in short? It would be nice to know that to > catch the one next time we face the problem. > On Fri, May 15, 2020 at 12:16:18PM -0400, nottobay wrote: >> It is saying no such file or directory for /var/db/cache but /var/db/ at >> 454982 out of 38176044 so space isn't the problem, >> >> On Fri, May 15, 2020, 11:38 wrote: >> >> > Hello, >> > >> > On Fri, May 15, 2020 at 11:22:59AM -0400, nottobay wrote: >> > >I keep getting a bunch of errors saying "download error >> > >size does not match pkg_summary" I try just telling it to proceed but >> > the >> > >package still doesn't install. I have already tried forcing a pgkin >> > update >> > >and it didn't fix it, and I'm using the default repo the NetBSD 9 >> > >installer gave me. How would I fix this? >> > >> > Do you have enough disk space in the directry pkin is trying to cache >> > the downloads? That is, what does >> > >> > df /var/db/cache/ >> > >> > say? >> > >> > -is >> > > -- > Sincerely yours, > Dima Veselov > Physics R&D Establishment of Saint-Petersburg University -- #- # Lars-Johan Liman, M.Sc.! E-mail: li...@cafax.se # Cafax AB ! HTTP : //www.cafax.se/ # Computer Consultants, Sweden ! Voice : +46 8 - 564 702 30 #-
Re: pkgin error
Hi Matthew, MN> http://cdn.Netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/... MN> http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/... MN> yielded different data. Both hostnames resolved to the same IP addresses MS> I fixed the host header thing when that was pointed out. Thanks for fixing it (back when), and yes, the URL case didn't make a difference yesterday. Great to know that shouldn't be any worry anymore. MS> Anyway try it now. Much better -- pefect! Yesterday's different pkgs per 8.0 8.2 8.2 version MN> % echo 0 1 2 | xargs -n1 -I XX lynx -head -dump http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.XX/All/p5-Authen-SASL-2.16nb7.tgz | grep Length MN> Content-Length: 24900 MN> Content-Length: 24892 MN> Content-Length: 24900 changed to 24892 for all three 8.x directories, now matching what's advertised in the pkg_summary.bz2 (which was "update"able today, too). All outstanding pkgs downloaded fine today (07:23 UTC), all 60 pkgs were refreshed/upgraded/installed without a hitch. Thanks! Martin Neitzel
Re: pkgin error
On Sat, May 16, 2020 at 8:42 AM Martin Neitzel wrote: > > ill> Same here. > ill> > ill> $ echo "select file_size from remote_pkg where pkgname like > ill> 'xmlcatmgr%'" | sqlite3 pkgin.db > ill> 25004 > ill> > ill> $ ftp > ill> > https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.0/All/xmlcatmgr-2.2nb1.tgz > ill> 24864 bytes retrieved in 00:00 (16.63 MiB/s) > > Some observations on this: > > % echo 0 1 2 | xargs -n1 -I XX lynx -head -dump > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.XX/All/xmlcatmgr-2.2nb1.tgz > | grep Length > Content-Length: 24864 > Content-Length: 24864 > Content-Length: 25004 > > This actually figures with my(!, see below) long "select" info: > > sqlite> select * from remote_pkg where pkgname like 'xmlcat%' ; > PKG_ID = 21533 > FULLPKGNAME = xmlcatmgr-2.2nb1 > PKGNAME = xmlcatmgr > PKGVERS = 2.2nb1 > BUILD_DATE = 2020-03-28 20:22:48 + > COMMENT = XML and SGML catalog manager > LICENSE = modified-bsd > PKGTOOLS_VERSION = 20091115 > HOMEPAGE = http://xmlcatmgr.sourceforge.net/ > OS_VERSION = 8.0 > DESCRIPTION = > PKGPATH = textproc/xmlcatmgr > PKG_OPTIONS = > CATEGORIES = textproc > SIZE_PKG = 50583 >FILE_SIZE = 25004 >OPSYS = NetBSD > REPOSITORY = > http://cdn.Netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.2/All > > > Looks like Roland is rather using the 8.0 repo? > > I essentially noticed the same problem here, too, after... > > - an update on the netbsd-8 branch on May 2nd and > - moving my /usr/pkg/etc/pkgin/repositories.conf > from http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/8.1/All > to http://cdn.Netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.2/All > - having nothing at all happen on "pkgin update" until > May 11 00:52 /var/db/pkgin/pkgin.db > > For "pkgin upgrade", this resulted in: > > 31 packages to refresh: (xmlcatmgr-2.2nb1 ... ... ...) > 19 packages to upgrade: > 2 packages to install: heimdal-1.5.3nb24 openssl-1.1.1e > > (I was mostly surprised about the "refresh" section. Where does this > come from, what is this supposed to mean?) > > Since I'm referring to 8.2 pkg repository, details differ for me, > I guess I am seeing the same problem but maybe from the other side. > > I can still confirm the problem / the error message seen from > my side, albeit with other packages. > > I get the error message with: > download error: p5-Authen-SASL-2.16nb7 size does not match pkg_summary > > and [abridged]: > > sqlite> select * from remote_pkg where pkgname like 'p5-Authen-SASL' ; > PKG_ID = 6173 > FULLPKGNAME = p5-Authen-SASL-2.16nb7 > BUILD_DATE = 2020-04-01 03:57:23 + > OS_VERSION = 8.0 > SIZE_PKG = 119267 >FILE_SIZE = 24892 > REPOSITORY = > http://cdn.Netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.2/All > > % echo 0 1 2 | xargs -n1 -I XX lynx -head -dump > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.XX/All/p5-Authen-SASL-2.16nb7.tgz > | grep Length > Content-Length: 24900 > Content-Length: 24892 > Content-Length: 24900 > > D'oh! > > > This is not the first time this madness happens. The last time was around > last summer and some kind fellow on the ircnet #netbsd pointed out to me > that the URLs > > http://cdn.Netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/... > http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/... >^ >| > > yielded different data. Both hostnames resolved to the same IP addresses > (as they should) but the Fastly CDN servers were apparently treating the > requests in different ways depending on the case in the Host: headers. > > Martin Neitzel I fixed the host header thing when that was pointed out. Anyway try it now.
Re: pkgin error
ill> Same here. ill> ill> $ echo "select file_size from remote_pkg where pkgname like ill> 'xmlcatmgr%'" | sqlite3 pkgin.db ill> 25004 ill> ill> $ ftp ill> https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.0/All/xmlcatmgr-2.2nb1.tgz ill> 24864 bytes retrieved in 00:00 (16.63 MiB/s) Some observations on this: % echo 0 1 2 | xargs -n1 -I XX lynx -head -dump http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.XX/All/xmlcatmgr-2.2nb1.tgz | grep Length Content-Length: 24864 Content-Length: 24864 Content-Length: 25004 This actually figures with my(!, see below) long "select" info: sqlite> select * from remote_pkg where pkgname like 'xmlcat%' ; PKG_ID = 21533 FULLPKGNAME = xmlcatmgr-2.2nb1 PKGNAME = xmlcatmgr PKGVERS = 2.2nb1 BUILD_DATE = 2020-03-28 20:22:48 + COMMENT = XML and SGML catalog manager LICENSE = modified-bsd PKGTOOLS_VERSION = 20091115 HOMEPAGE = http://xmlcatmgr.sourceforge.net/ OS_VERSION = 8.0 DESCRIPTION = PKGPATH = textproc/xmlcatmgr PKG_OPTIONS = CATEGORIES = textproc SIZE_PKG = 50583 FILE_SIZE = 25004 OPSYS = NetBSD REPOSITORY = http://cdn.Netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.2/All Looks like Roland is rather using the 8.0 repo? I essentially noticed the same problem here, too, after... - an update on the netbsd-8 branch on May 2nd and - moving my /usr/pkg/etc/pkgin/repositories.conf from http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/8.1/All to http://cdn.Netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.2/All - having nothing at all happen on "pkgin update" until May 11 00:52 /var/db/pkgin/pkgin.db For "pkgin upgrade", this resulted in: 31 packages to refresh: (xmlcatmgr-2.2nb1 ... ... ...) 19 packages to upgrade: 2 packages to install: heimdal-1.5.3nb24 openssl-1.1.1e (I was mostly surprised about the "refresh" section. Where does this come from, what is this supposed to mean?) Since I'm referring to 8.2 pkg repository, details differ for me, I guess I am seeing the same problem but maybe from the other side. I can still confirm the problem / the error message seen from my side, albeit with other packages. I get the error message with: download error: p5-Authen-SASL-2.16nb7 size does not match pkg_summary and [abridged]: sqlite> select * from remote_pkg where pkgname like 'p5-Authen-SASL' ; PKG_ID = 6173 FULLPKGNAME = p5-Authen-SASL-2.16nb7 BUILD_DATE = 2020-04-01 03:57:23 + OS_VERSION = 8.0 SIZE_PKG = 119267 FILE_SIZE = 24892 REPOSITORY = http://cdn.Netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.2/All % echo 0 1 2 | xargs -n1 -I XX lynx -head -dump http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.XX/All/p5-Authen-SASL-2.16nb7.tgz | grep Length Content-Length: 24900 Content-Length: 24892 Content-Length: 24900 D'oh! This is not the first time this madness happens. The last time was around last summer and some kind fellow on the ircnet #netbsd pointed out to me that the URLs http://cdn.Netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/... http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/... ^ | yielded different data. Both hostnames resolved to the same IP addresses (as they should) but the Fastly CDN servers were apparently treating the requests in different ways depending on the case in the Host: headers. Martin Neitzel
Re: pkgin error
On 15.05.2020 17:22, nottobay wrote: I keep getting a bunch of errors saying "download error size does not match pkg_summary" I try just telling it to proceed but the package still doesn't install. I have already tried forcing a pgkin update and it didn't fix it, and I'm using the default repo the NetBSD 9 installer gave me. How would I fix this? Same here. $ echo "select file_size from remote_pkg where pkgname like 'xmlcatmgr%'" | sqlite3 pkgin.db 25004 $ ftp https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/8.0/All/xmlcatmgr-2.2nb1.tgz 24864 bytes retrieved in 00:00 (16.63 MiB/s) The strange thing is that the +BUILD_INFO inside that package says: BUILD_DATE=2019-12-21 05:36:25 + Yet the mtime on cdn.NetBSD.org is 28-Mar-2020 20:22. What's going on here? In this situation I'm missing cryptographically signed packages since the default transport protocol is "http" without any "s".
Re: pkgin error
I had same problems and I've found that CDN directory contents may be not in sync with pkg_summary. I tried to work it out (thinking there might be one server not in sync) but had short time and no success so that time I just switched to legacy source. How cdn is organized in short? It would be nice to know that to catch the one next time we face the problem. On Fri, May 15, 2020 at 12:16:18PM -0400, nottobay wrote: > It is saying no such file or directory for /var/db/cache but /var/db/ at > 454982 out of 38176044 so space isn't the problem, > > On Fri, May 15, 2020, 11:38 wrote: > > > Hello, > > > > On Fri, May 15, 2020 at 11:22:59AM -0400, nottobay wrote: > > >I keep getting a bunch of errors saying "download error > > >size does not match pkg_summary" I try just telling it to proceed but > > the > > >package still doesn't install. I have already tried forcing a pgkin > > update > > >and it didn't fix it, and I'm using the default repo the NetBSD 9 > > >installer gave me. How would I fix this? > > > > Do you have enough disk space in the directry pkin is trying to cache > > the downloads? That is, what does > > > > df /var/db/cache/ > > > > say? > > > > -is > > -- Sincerely yours, Dima Veselov Physics R&D Establishment of Saint-Petersburg University
Re: pkgin error
It is saying no such file or directory for /var/db/cache but /var/db/ at 454982 out of 38176044 so space isn't the problem, On Fri, May 15, 2020, 11:38 wrote: > Hello, > > On Fri, May 15, 2020 at 11:22:59AM -0400, nottobay wrote: > >I keep getting a bunch of errors saying "download error > >size does not match pkg_summary" I try just telling it to proceed but > the > >package still doesn't install. I have already tried forcing a pgkin > update > >and it didn't fix it, and I'm using the default repo the NetBSD 9 > >installer gave me. How would I fix this? > > Do you have enough disk space in the directry pkin is trying to cache > the downloads? That is, what does > > df /var/db/cache/ > > say? > > -is >
Re: pkgin error
Hello, On Fri, May 15, 2020 at 11:22:59AM -0400, nottobay wrote: >I keep getting a bunch of errors saying "download error >size does not match pkg_summary" I try just telling it to proceed but the >package still doesn't install. I have already tried forcing a pgkin update >and it didn't fix it, and I'm using the default repo the NetBSD 9 >installer gave me. How would I fix this? Do you have enough disk space in the directry pkin is trying to cache the downloads? That is, what does df /var/db/cache/ say? -is
pkgin error
I keep getting a bunch of errors saying "download error size does not match pkg_summary" I try just telling it to proceed but the package still doesn't install. I have already tried forcing a pgkin update and it didn't fix it, and I'm using the default repo the NetBSD 9 installer gave me. How would I fix this?
Re: pkgsrc binary packages security with pkgin
On 2020-02-01 01:38, Greg Troxel wrote: [---] > If you can't trust your local storage, you have no basis for getting > anything at all right. Your local storage is where the public keys are > stored that you use to validate, where you store files in installed > packages, and where you store /usr//bin/login. Seriously - if you can't > trust your local computer, it's all over. Sure, but I meant explicitly local storage with regards to the packages only -- they could be stored in a directory which is shared among other users, for instance. I.e. the packages could in theory be manipulated, but the tools to validate them can't. -- Kind Regards, Jan
Re: pkgsrc binary packages security with pkgin
For a quick summary from all your answers since martin's, if I may. His answer is still perfectly valid to me. Assuming you trust everything before, because not assuming that is confusing and counterproductive in this particular discussion, I wanted to focus, while there is probably work there too (there is always): - upstream softwares in pkgsrc. pkgsrc-vulnerabilities is a partial answer, right? - packagers - builders So, assuming that above: - https more trivial to use - https only protects the link between you and binary packages server - sigs protects from the builder to you, this adds a lot Also, while it is relevant to compare https and signatures, security is about resilience. There must be a valid trust chain to which add strata, if possible. Obviously, deciding to sign packages involves asking questions about key management. Be that as it may, the decision to believe is already made, the objective is to formalize it. Yours sincerely
Re: pkgsrc binary packages security with pkgin
Jan Danielsson writes: >- If you don't know if: > o the server storage can be trusted > o you can fully trust the link > o you can trust your local storage up until the point at which you > install the package > .. then you need the binary package to be signed. If you can't trust your local storage, you have no basis for getting anything at all right. Your local storage is where the public keys are stored that you use to validate, where you store files in installed packages, and where you store /usr//bin/login. Seriously - if you can't trust your local computer, it's all over.
Re: pkgsrc binary packages security with pkgin
Johnny Billquist writes: > On 2020-01-31 15:02, Greg Troxel wrote: >> The other thing https gives you is hiding the names of the packages you >> download from passive eavesdroppers on the network bewteen your computer >> and the TNF server. One such possible eavesdropper is your ISP. This >> is part of the "https everyhwere" push; there is no reason to expose the >> list of requested resources to passive eavesdroppers. > > At which point you probably should be loosing sleep because the ISP > can still see where you connect to. This is getting off topic, but exposing the set of IP addresses to which you make requests is much less than also exposing the URLs and their content. It seems you don't care, but that doesn't mean it doesn't matter.
Re: pkgsrc binary packages security with pkgin
On 2020-01-31 20:36, Manuel Bouyer wrote: [---] >>*Assuming you can trust the build environment (which includes the >> signing process)*, and assuming that you can trust the underlying crypto: >> >>- HTTPS protects the connection between you and the server (assuming >> server authentication, and not just encryption). So if you trust the >> remote server, your client, and the HTTPS implementation, then HTTPS is >> sufficient for the entire chain. > > Not really; for this to be true you have to trust the build process, the way > the binary package is uploaded to the http server and the http server itself. I should have make clear that in that particular context "trust the remote server" meant to encompass the entire build process and storage. > With signed binary pkg you only need to trust the build process. (And the tools you use to verify and install the package, but this is implied). Yes, this was the core point I was trying to convey. -- Kind Regards, Jan
Re: pkgsrc binary packages security with pkgin
On 2020-01-31 15:02, Greg Troxel wrote: The other thing https gives you is hiding the names of the packages you download from passive eavesdroppers on the network bewteen your computer and the TNF server. One such possible eavesdropper is your ISP. This is part of the "https everyhwere" push; there is no reason to expose the list of requested resources to passive eavesdroppers. At which point you probably should be loosing sleep because the ISP can still see where you connect to. Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: pkgsrc binary packages security with pkgin
On Fri, Jan 31, 2020 at 07:21:40PM +0100, Jan Danielsson wrote: > On 2020-01-31 08:49, yarl-bau...@mailoo.org wrote: > > Please Maya and Mr Billquist, can you be more specific about how it is > > insecure? > >There are different domains to consider. > >*Assuming you can trust the build environment (which includes the > signing process)*, and assuming that you can trust the underlying crypto: > >- HTTPS protects the connection between you and the server (assuming > server authentication, and not just encryption). So if you trust the > remote server, your client, and the HTTPS implementation, then HTTPS is > sufficient for the entire chain. Not really; for this to be true you have to trust the build process, the way the binary package is uploaded to the http server and the http server itself. With signed binary pkg you only need to trust the build process. In a world where there are multiple sources under different administrative domains for the same file, this is important. -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: pkgsrc binary packages security with pkgin
On 2020-01-31 08:49, yarl-bau...@mailoo.org wrote: > Please Maya and Mr Billquist, can you be more specific about how it is > insecure? There are different domains to consider. *Assuming you can trust the build environment (which includes the signing process)*, and assuming that you can trust the underlying crypto: - HTTPS protects the connection between you and the server (assuming server authentication, and not just encryption). So if you trust the remote server, your client, and the HTTPS implementation, then HTTPS is sufficient for the entire chain. - If you don't know if: o the server storage can be trusted o you can fully trust the link o you can trust your local storage up until the point at which you install the package .. then you need the binary package to be signed. Note that as long as the "trust the build environment" holds, the signed packages cover both these cases, while "use HTTPS" doesn't cover most of the second case. Depending on what kind of security work one does, relying on HTTPS may not be sufficient, as it covers less of the chain. Or to put it another way: If you're screwed in a way that your HTTPS can't be trusted, you can still trust the signed packages. If you're screwed in a way that you can't trust the signed packages, then HTTPS won't help you. -- Kind Regards, Jan
Re: pkgsrc binary packages security with pkgin
Ottavio Caruso writes: > I have interpreted "binary packages safety" as something intrinsic to > potential vulnerability of the 3rd party software itself, as opposed > to package integrity checking with digital signatures, checksums, etc, > at least related to questions 1 and 3. In my view, the biggest risks are unintentional bugs and then backdoors in the upstream packages. > It seems to me that one can sign a package all they want; if there is > a vulnerability in the code itself, this won't go away by having it > digitally signed. Absolutely true. You can an attested build of buggy sources. The point about signed packages is that it's fairly easy to do, for some definition of fairly easy. Certainly it's easier than ensuring there are no bugs in 15000 upstream packages.
Re: pkgsrc binary packages security with pkgin
Johnny Billquist writes: > (Which is why I objected to the implication that https is important, > and somehow adds some security here in the first place.) I think you are incorrect to dismiss https. In a world without signed packages, the flow of built binary packages from an official build server is surely via scp or similar to the ftp server. With https (and validation of the certificate relative to the name), you have some degree of assurance that your request is being fulfilled by the right server and that the contents are not modified. I agree that there are multiple steps that one has to trust: upload, storage, download, and that signed packages could replace that set of steps with one step (or really augment; an attacked would have to forge a signature and compromise one of those three steps). So I am not arguing that signed packages are unimportant. But "https adds nothing" is wrong. The other thing https gives you is hiding the names of the packages you download from passive eavesdroppers on the network bewteen your computer and the TNF server. One such possible eavesdropper is your ISP. This is part of the "https everyhwere" push; there is no reason to expose the list of requested resources to passive eavesdroppers. There is a further wrinkle, which is the use of a CDN, but CDNs are set up to share https certificates and public keys to make this work.
Re: pkgsrc binary packages security with pkgin
On 31/01/2020 12:05, Leonardo Taccari wrote: Ottavio Caruso writes: [...] I believe there's an internal pkgsrc security mailing list to which users have no access (I could be wrong), so I don't really know how this auditing really works. One can always "pkg_admin fetch-pkg-vulnerabilities && pkg_admin audit". [...] pkgsrc-security@ is a team, usually there isn't much traffic on it and the most possible private information that happens is on an internal RT ticket system to track tickets that then ends up in pkg-vulnerabilities file. However, this is mostly unrelated to signing binary packages (we manually sign the pkg-vulnerabilities file but that's unrelated). Leo & al., The original questions were [sic]: 1) "is safe the use pkgsrc binary packages. For example using pkgin?" 2) "Is the authenticity and integrity of packages installed this way is guaranteed assuming no bugs in software involved?" 3) "Is it safer to compile by yourself?" I have interpreted "binary packages safety" as something intrinsic to potential vulnerability of the 3rd party software itself, as opposed to package integrity checking with digital signatures, checksums, etc, at least related to questions 1 and 3. It seems to me that one can sign a package all they want; if there is a vulnerability in the code itself, this won't go away by having it digitally signed. (I'm not having a go at anybody. I'm just trying to understand what it is all about) -- Ottavio Caruso
Re: pkgsrc binary packages security with pkgin
Ottavio Caruso writes: > [...] > I believe there's an internal pkgsrc security mailing list to which > users have no access (I could be wrong), so I don't really know how this > auditing really works. > > One can always "pkg_admin fetch-pkg-vulnerabilities && pkg_admin audit". > [...] pkgsrc-security@ is a team, usually there isn't much traffic on it and the most possible private information that happens is on an internal RT ticket system to track tickets that then ends up in pkg-vulnerabilities file. However, this is mostly unrelated to signing binary packages (we manually sign the pkg-vulnerabilities file but that's unrelated).
Re: pkgsrc binary packages security with pkgin
Or putting it another way...
Martin did an excellent summary of potential risks.
You seem to be all focused on point 5 of that list, which is, I think
the least likely to be a problem or a risk. That someone would tamper
with the data en route to you is the trickiest, and least likely to
succeed in the first place.
Attacking at points 1-4 are all easier and more rewarding, and they are
all left unsolved in your world.
And any attack at points 1-4 will go undetected by a check at point 5.
Johnny
On 2020-01-31 11:08, Johnny Billquist wrote:
On 2020-01-31 10:25, yarl-bau...@mailoo.org wrote:
That's exactly the answer I was waiting and hoping for. Thank you.
I'll follow tech-pkg from now on. Packages must be signed.
And with that signature, you know that what you got from the server was
not tampered with during transport to you, which is the same thing https
would give you. And which still means you have no idea if the software
is sane, proper, does what you think, or hasn't been tampered with.
Johnny
De : Martin Husemann
À : Ottavio Caruso
Sujet : Re: pkgsrc binary packages security with pkgin
Date : 31/01/2020 09:51:53 Europe/Paris
Copie à : netbsd-users@netbsd.org
Let me (as someone not heavily involved into pkgsrc and binary pkg
building)
try to unriddle a few bits that I think get easily confused in this
context.
When it comes to 3rd party packages, you have to trust:
(1) the original source of the package ("upstream") and its release
policies.
Assuming that the released source has no bad things hidden, you then have
to trust:
(2) pkgsrc (or the commiters of the pkg and all its dependencies and all
patches involved) to not do anything bad
From that point on we can help with various checks. When building a pkg
(locally or in a bulk build environment) pkgsrc verifies the distribution
file it downloaded does match the hashes recorded at (2). The result of
that build is a binary pkg, and if you did build localy, you are done
here
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
Re: pkgsrc binary packages security with pkgin
On 2020-01-31 12:39, Johnny Billquist wrote: On 2020-01-31 12:37, Manuel Bouyer wrote: On Fri, Jan 31, 2020 at 12:32:06PM +0100, Johnny Billquist wrote: Of course you can. But then you need to have a whole list of trusted public keys that needs to be managed, which again leads to the question of which keys are now the acceptable ones. And how to you trust new builders? Can anyone then be added to the list of official builders of packages, or how to you manage that side? Key management is not trivial. Of course it's not. But this is not really a technical issue. Security is never a technical issue, more than at the surface... (Which is why I objected to the implication that https is important, and somehow adds some security here in the first place.) Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: pkgsrc binary packages security with pkgin
On 2020-01-31 12:37, Manuel Bouyer wrote: On Fri, Jan 31, 2020 at 12:32:06PM +0100, Johnny Billquist wrote: Of course you can. But then you need to have a whole list of trusted public keys that needs to be managed, which again leads to the question of which keys are now the acceptable ones. And how to you trust new builders? Can anyone then be added to the list of official builders of packages, or how to you manage that side? Key management is not trivial. Of course it's not. But this is not really a technical issue. Security is never a technical issue, more than at the surface... Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: pkgsrc binary packages security with pkgin
On Fri, Jan 31, 2020 at 12:32:06PM +0100, Johnny Billquist wrote: > Of course you can. But then you need to have a whole list of trusted public > keys that needs to be managed, which again leads to the question of which > keys are now the acceptable ones. And how to you trust new builders? Can > anyone then be added to the list of official builders of packages, or how to > you manage that side? > Key management is not trivial. Of course it's not. But this is not really a technical issue. -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: pkgsrc binary packages security with pkgin
On 2020-01-31 12:32, Johnny Billquist wrote: On 2020-01-31 12:07, Manuel Bouyer wrote: On Fri, Jan 31, 2020 at 11:39:32AM +0100, Johnny Billquist wrote: On 2020-01-31 11:34, Manuel Bouyer wrote: On Fri, Jan 31, 2020 at 11:08:05AM +0100, Johnny Billquist wrote: On 2020-01-31 10:25, yarl-bau...@mailoo.org wrote: That's exactly the answer I was waiting and hoping for. Thank you. I'll follow tech-pkg from now on. Packages must be signed. And with that signature, you know that what you got from the server was not tampered with during transport to you, which is the same thing https would give you. And which still means you have no idea if the software is sane, proper, does what you think, or hasn't been tampered with. No it's not the same thing. package signature guarantees that the data have not been modified since it has been built. https guarantees that the data have not been modified between the http server and client. It doesn't tell anything about what happened to the binary pkg between the build server and the http server at the time you download it. Right you are. I was too fast and loose on that one. Signatures are better in that sense. However, you then also have to trust that the signature have not been altered along with a alteration of the package... So does a signature really tell you much at all? I guess if you then had signatures with public/private keys. But then again, that don't really work if you have multiple places doing builds, unless they then share the private key, but that in turn leads to the question about how private do you then think that key is? Why would they have to share the same private key ? You can trust multiple public keys, this is how other binary package managers works. Of course you can. But then you need to have a whole list of trusted public keys that needs to be managed, which again leads to the question of which keys are now the acceptable ones. And how to you trust new builders? Can anyone then be added to the list of official builders of packages, or how to you manage that side? Key management is not trivial. And this of course also leads to the question, can you trust all of them to properly maintain security of their systems? Too many accepted keys eventually leads to them not giving any security at all. And as soon as a system is compromised, all the packages built there is potentially compromised. Not to mention that the key should be considered compromised, and needs to be revoked. And all systems needs to no longer accept that key, which then is also a piece of information that needs to be distributed and enforced. And how do you push such information without that also becoming an attack vector, meaning keys could be added/revoked by the wrong people, or just with the wrong data. Same thing as with anything sensitive. Too many have access and it's most likely compromised somewhere. Me paranoid? Not really... :-) Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: pkgsrc binary packages security with pkgin
On 2020-01-31 12:07, Manuel Bouyer wrote: On Fri, Jan 31, 2020 at 11:39:32AM +0100, Johnny Billquist wrote: On 2020-01-31 11:34, Manuel Bouyer wrote: On Fri, Jan 31, 2020 at 11:08:05AM +0100, Johnny Billquist wrote: On 2020-01-31 10:25, yarl-bau...@mailoo.org wrote: That's exactly the answer I was waiting and hoping for. Thank you. I'll follow tech-pkg from now on. Packages must be signed. And with that signature, you know that what you got from the server was not tampered with during transport to you, which is the same thing https would give you. And which still means you have no idea if the software is sane, proper, does what you think, or hasn't been tampered with. No it's not the same thing. package signature guarantees that the data have not been modified since it has been built. https guarantees that the data have not been modified between the http server and client. It doesn't tell anything about what happened to the binary pkg between the build server and the http server at the time you download it. Right you are. I was too fast and loose on that one. Signatures are better in that sense. However, you then also have to trust that the signature have not been altered along with a alteration of the package... So does a signature really tell you much at all? I guess if you then had signatures with public/private keys. But then again, that don't really work if you have multiple places doing builds, unless they then share the private key, but that in turn leads to the question about how private do you then think that key is? Why would they have to share the same private key ? You can trust multiple public keys, this is how other binary package managers works. Of course you can. But then you need to have a whole list of trusted public keys that needs to be managed, which again leads to the question of which keys are now the acceptable ones. And how to you trust new builders? Can anyone then be added to the list of official builders of packages, or how to you manage that side? Key management is not trivial. Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: pkgsrc binary packages security with pkgin
On 2020-01-31 11:34, Manuel Bouyer wrote: On Fri, Jan 31, 2020 at 11:08:05AM +0100, Johnny Billquist wrote: On 2020-01-31 10:25, yarl-bau...@mailoo.org wrote: That's exactly the answer I was waiting and hoping for. Thank you. I'll follow tech-pkg from now on. Packages must be signed. And with that signature, you know that what you got from the server was not tampered with during transport to you, which is the same thing https would give you. And which still means you have no idea if the software is sane, proper, does what you think, or hasn't been tampered with. No it's not the same thing. package signature guarantees that the data have not been modified since it has been built. https guarantees that the data have not been modified between the http server and client. It doesn't tell anything about what happened to the binary pkg between the build server and the http server at the time you download it. Right you are. I was too fast and loose on that one. Signatures are better in that sense. However, you then also have to trust that the signature have not been altered along with a alteration of the package... So does a signature really tell you much at all? I guess if you then had signatures with public/private keys. But then again, that don't really work if you have multiple places doing builds, unless they then share the private key, but that in turn leads to the question about how private do you then think that key is? Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: pkgsrc binary packages security with pkgin
On Fri, Jan 31, 2020 at 11:39:32AM +0100, Johnny Billquist wrote: > On 2020-01-31 11:34, Manuel Bouyer wrote: > > On Fri, Jan 31, 2020 at 11:08:05AM +0100, Johnny Billquist wrote: > > > On 2020-01-31 10:25, yarl-bau...@mailoo.org wrote: > > > > That's exactly the answer I was waiting and hoping for. Thank you. > > > > > > > > I'll follow tech-pkg from now on. Packages must be signed. > > > > > > And with that signature, you know that what you got from the server was > > > not > > > tampered with during transport to you, which is the same thing https would > > > give you. And which still means you have no idea if the software is sane, > > > proper, does what you think, or hasn't been tampered with. > > > > No it's not the same thing. > > package signature guarantees that the data have not been modified since it > > has been built. > > https guarantees that the data have not been modified between the http > > server > > and client. It doesn't tell anything about what happened to the binary pkg > > between the build server and the http server at the time you download it. > > Right you are. I was too fast and loose on that one. > > Signatures are better in that sense. However, you then also have to trust > that the signature have not been altered along with a alteration of the > package... So does a signature really tell you much at all? I guess if you > then had signatures with public/private keys. But then again, that don't > really work if you have multiple places doing builds, unless they then share > the private key, but that in turn leads to the question about how private do > you then think that key is? Why would they have to share the same private key ? You can trust multiple public keys, this is how other binary package managers works. -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: pkgsrc binary packages security with pkgin
On Fri, Jan 31, 2020 at 11:08:05AM +0100, Johnny Billquist wrote: > On 2020-01-31 10:25, yarl-bau...@mailoo.org wrote: > > That's exactly the answer I was waiting and hoping for. Thank you. > > > > I'll follow tech-pkg from now on. Packages must be signed. > > And with that signature, you know that what you got from the server was not > tampered with during transport to you, which is the same thing https would > give you. And which still means you have no idea if the software is sane, > proper, does what you think, or hasn't been tampered with. No it's not the same thing. package signature guarantees that the data have not been modified since it has been built. https guarantees that the data have not been modified between the http server and client. It doesn't tell anything about what happened to the binary pkg between the build server and the http server at the time you download it. -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: pkgsrc binary packages security with pkgin
On 2020-01-31 10:25, yarl-bau...@mailoo.org wrote:
That's exactly the answer I was waiting and hoping for. Thank you.
I'll follow tech-pkg from now on. Packages must be signed.
And with that signature, you know that what you got from the server was
not tampered with during transport to you, which is the same thing https
would give you. And which still means you have no idea if the software
is sane, proper, does what you think, or hasn't been tampered with.
Johnny
De : Martin Husemann
À : Ottavio Caruso
Sujet : Re: pkgsrc binary packages security with pkgin
Date : 31/01/2020 09:51:53 Europe/Paris
Copie à : netbsd-users@netbsd.org
Let me (as someone not heavily involved into pkgsrc and binary pkg building)
try to unriddle a few bits that I think get easily confused in this context.
When it comes to 3rd party packages, you have to trust:
(1) the original source of the package ("upstream") and its release policies.
Assuming that the released source has no bad things hidden, you then have
to trust:
(2) pkgsrc (or the commiters of the pkg and all its dependencies and all
patches involved) to not do anything bad
From that point on we can help with various checks. When building a pkg
(locally or in a bulk build environment) pkgsrc verifies the distribution
file it downloaded does match the hashes recorded at (2). The result of
that build is a binary pkg, and if you did build localy, you are done here
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
Re: pkgsrc binary packages security with pkgin
That's exactly the answer I was waiting and hoping for. Thank you.
I'll follow tech-pkg from now on. Packages must be signed.
De : Martin Husemann
À : Ottavio Caruso
Sujet : Re: pkgsrc binary packages security with pkgin
Date : 31/01/2020 09:51:53 Europe/Paris
Copie à : netbsd-users@netbsd.org
Let me (as someone not heavily involved into pkgsrc and binary pkg building)
try to unriddle a few bits that I think get easily confused in this context.
When it comes to 3rd party packages, you have to trust:
(1) the original source of the package ("upstream") and its release policies.
Assuming that the released source has no bad things hidden, you then have
to trust:
(2) pkgsrc (or the commiters of the pkg and all its dependencies and all
patches involved) to not do anything bad
>From that point on we can help with various checks. When building a pkg
(locally or in a bulk build environment) pkgsrc verifies the distribution
file it downloaded does match the hashes recorded at (2). The result of
that build is a binary pkg, and if you did build localy, you are done here
Tr: Re: pkgsrc binary packages security with pkgin
De : yarl-bau...@mailoo.org À : Ottavio Caruso Sujet : Re: pkgsrc binary packages security with pkgin Date : 31/01/2020 10:15:00 Europe/Paris De : Ottavio Caruso À : netbsd-users@netbsd.org Sujet : Re: pkgsrc binary packages security with pkgin Date : 31/01/2020 09:26:06 Europe/Paris One can always "pkg_admin fetch-pkg-vulnerabilities && pkg_admin audit". -- Ottavio Caruso I'm think this is irrelevant. The pkg-vulnerabilities is only "a list of known security vulnerabilities to packages which are (or have been) included in pkgsrc", to quote the pkgsrc guide.
Re: pkgsrc binary packages security with pkgin
Let me (as someone not heavily involved into pkgsrc and binary pkg building)
try to unriddle a few bits that I think get easily confused in this context.
When it comes to 3rd party packages, you have to trust:
(1) the original source of the package ("upstream") and its release policies.
Assuming that the released source has no bad things hidden, you then have
to trust:
(2) pkgsrc (or the commiters of the pkg and all its dependencies and all
patches involved) to not do anything bad
>From that point on we can help with various checks. When building a pkg
(locally or in a bulk build environment) pkgsrc verifies the distribution
file it downloaded does match the hashes recorded at (2). The result of
that build is a binary pkg, and if you did build localy, you are done here.
If this is a bulk build environment and the binary pkgs will be uploaded
to some server, it is good to be able to verify the pkg has not been altered.
For this there are mechanisms ("signed pkgs"), but currently they are not
widely deployed (see below).
The next steps are (3) upload to the server, (4) trusting the server
and its admins, and (5) your download.
Whether that download happens via http or https and whether the https
certificate is validated does not really matter - as long as the binary
pkg you downloaded still is untampered (matches its signature).
IIUC the original question was about trust in step 5, and the responses
tried to hint at 1-4 being insecure anyway, so 5 would not really matter.
So far the theory.
Unfortunately, as of now, there is no signing happening for most (all?) pkgs
created under TNF controll. I personally had hoped this would change for the
pkgs created for NetBSD 9.0, but right now it does not look like it.
I'll take this as a reminder and will start a thread on tech-pkg to see how
we can solve this issue ASAP.
Martin
Re: pkgsrc binary packages security with pkgin
On 31/01/2020 07:49, yarl-bau...@mailoo.org wrote: Please Maya and Mr Billquist, can you be more specific about how it is insecure? To all: Is someone working on it and what is ongoing to improve this? I feel this thread belongs to pkgsrc-users@ or even better tech-pkg@ and I'm not the OP, but here's my thoughts: binary packages are bulk-built from pkgsrc. pkgsrc is not strictly part of the operating system base but are external applications. Making a rough and totally uneducated comparison between NetBSD and, say, OpenBSD, the former is more focused on usability and making sure the system doesn't break, while the latter is totally focused on security, to the point of mutilating crucial parts of the OS, if that doesn't fit its self-imposed standards (I'm over simplifying). I believe there's an internal pkgsrc security mailing list to which users have no access (I could be wrong), so I don't really know how this auditing really works. One can always "pkg_admin fetch-pkg-vulnerabilities && pkg_admin audit". -- Ottavio Caruso
Re: pkgsrc binary packages security with pkgin
Please Maya and Mr Billquist, can you be more specific about how it is insecure? To all: Is someone working on it and what is ongoing to improve this? Thank you very much. De : J. Lewis Muir À : Johnny Billquist Sujet : Re: pkgsrc binary packages security with pkgin Date : 27/01/2020 12:08:07 Europe/Paris Copie à : m...@netbsd.org; yarl-bau...@mailoo.org; netbsd-users@netbsd.org On 01/26, Johnny Billquist wrote: > The code is not audited anyway, but just downloaded from various places, and > then built. I don't follow. What code are you saying is not audited? The source code of each package? If so, I think that's mostly true (of course there are exceptions where the source code has been audited), but that's no different than other package management systems such as RHEL's or Ubuntu's. But this thread is about pkgsrc *binary* packages. Are you instead talking about the distfiles downloaded in order to build the binary packages from source? Each pkgsrc package contains a distinfo file which contains a checksum for each distfile (or other) downloaded from the Internet, so those can all be downloaded from anywhere without HTTPS and still be trusted as long as the checksum matches. > If you really want to have some actual security, and not just a false sense > of it, https or so on is not really the answer. Anyone who thinks that just > because https is involved, it is somehow more secure, is really fooling > themselves. > > https is most properly something to use when submitting sensitive data to a > web server, which you do not want someone to pick up along the way. > > The total craziness of moving the whole internet to https is not really > improving any security in most situations. It protects against man-in-the-middle attacks, so I think for downloading binary packages it does add something significant. > Not to mention the question of how you would solve the replication of > repositories. All needs their own signatures. So there are a whole bunch of > places where to get packages from. How do you know that they are all legit, > and have the "right" binary packages even? You cannot have a single > signature to ensure they are legit, since https ties certificates to the > specific host. I'm sorry, but I also don't follow this. By "repository replication" do you mean mirroring repositories? I would say that this can be done in a number of ways including over HTTPS or SSH. And for binary packages, each package could be digitally signed by whoever built it. You can trust more than one person or organization to build packages, and if you trust whoever built it, and you can validate the signature, then you can trust the package. Regards, Lewis
Re: pkgsrc binary packages security with pkgin
On 01/26, Johnny Billquist wrote: > The code is not audited anyway, but just downloaded from various places, and > then built. I don't follow. What code are you saying is not audited? The source code of each package? If so, I think that's mostly true (of course there are exceptions where the source code has been audited), but that's no different than other package management systems such as RHEL's or Ubuntu's. But this thread is about pkgsrc *binary* packages. Are you instead talking about the distfiles downloaded in order to build the binary packages from source? Each pkgsrc package contains a distinfo file which contains a checksum for each distfile (or other) downloaded from the Internet, so those can all be downloaded from anywhere without HTTPS and still be trusted as long as the checksum matches. > If you really want to have some actual security, and not just a false sense > of it, https or so on is not really the answer. Anyone who thinks that just > because https is involved, it is somehow more secure, is really fooling > themselves. > > https is most properly something to use when submitting sensitive data to a > web server, which you do not want someone to pick up along the way. > > The total craziness of moving the whole internet to https is not really > improving any security in most situations. It protects against man-in-the-middle attacks, so I think for downloading binary packages it does add something significant. > Not to mention the question of how you would solve the replication of > repositories. All needs their own signatures. So there are a whole bunch of > places where to get packages from. How do you know that they are all legit, > and have the "right" binary packages even? You cannot have a single > signature to ensure they are legit, since https ties certificates to the > specific host. I'm sorry, but I also don't follow this. By "repository replication" do you mean mirroring repositories? I would say that this can be done in a number of ways including over HTTPS or SSH. And for binary packages, each package could be digitally signed by whoever built it. You can trust more than one person or organization to build packages, and if you trust whoever built it, and you can validate the signature, then you can trust the package. Regards, Lewis
Re: pkgsrc binary packages security with pkgin
Op 26/01/2020 om 02:55 schreef Johnny Billquist: On 2020-01-26 03:43, J. Lewis Muir wrote: On 01/25, m...@netbsd.org wrote: On Sat, Jan 25, 2020 at 01:34:34AM +0100, yarl-bau...@mailoo.org wrote: May I ask how is safe the use pkgsrc binary packages. For example using pkgin. Does libfetch is doing fine with https? Any thoughts? Is the authenticity and integrity of packages installed this way is guaranteed assuming no bugs in software involved? No. Wow! That's surprising. Can you explain why? I understand that the binary packages are not digitally signed, but if the binary repo is served over HTTPS, as long as the repo has not been compromised, the integrity and authenticity is guaranteed, no? Or as the OP asked, is pkgin not actually validating the server's SSL certificate? That would be terrible if it's silently behaving that way. If it can't handle HTTPS properly, it should refuse to use the URL. Anyway, I would be very surprised if this is what's going on, so I'm just asking to understand better. Thank you! The code is not audited anyway, but just downloaded from various places, and then built. If you really want to have some actual security, and not just a false sense of it, https or so on is not really the answer. Anyone who thinks that just because https is involved, it is somehow more secure, is really fooling themselves. https is most properly something to use when submitting sensitive data to a web server, which you do not want someone to pick up along the way. The total craziness of moving the whole internet to https is not really improving any security in most situations. Not to mention the question of how you would solve the replication of repositories. All needs their own signatures. So there are a whole bunch of places where to get packages from. How do you know that they are all legit, and have the "right" binary packages even? You cannot have a single signature to ensure they are legit, since https ties certificates to the specific host. /Me feeling very tired of the https hysteria... Johnny Incidentally, I wonder if OpenBSD's privsep [1] [2] could be a possible welcome addition to pkgsrc. [1] https://man.openbsd.org/bsd.port.mk#PORTS_PRIVSEP [2] https://dataswamp.org/~solene/2020-01-11-privsep.html -- Ottavio Caruso
Re: pkgsrc binary packages security with pkgin
Is there projects to improve this? De : m...@netbsd.org À : yarl-bau...@mailoo.org Sujet : Re: pkgsrc binary packages security with pkgin Date : 25/01/2020 23:11:25 Europe/Paris Copie à : netbsd-users@netbsd.org On Sat, Jan 25, 2020 at 01:34:34AM +0100, yarl-bau...@mailoo.org wrote: > Hello, > > May I ask how is safe the use pkgsrc binary packages. For example using > pkgin. Does libfetch is doing fine with https? Any thoughts? > > Is the authenticity and integrity of packages installed this way is > guaranteed assuming no bugs in software involved? No. > > Is it safer to compile by yourself? Yes. This is a very unfortunate case.
Re: pkgsrc binary packages security with pkgin
On 2020-01-26 03:43, J. Lewis Muir wrote: On 01/25, m...@netbsd.org wrote: On Sat, Jan 25, 2020 at 01:34:34AM +0100, yarl-bau...@mailoo.org wrote: May I ask how is safe the use pkgsrc binary packages. For example using pkgin. Does libfetch is doing fine with https? Any thoughts? Is the authenticity and integrity of packages installed this way is guaranteed assuming no bugs in software involved? No. Wow! That's surprising. Can you explain why? I understand that the binary packages are not digitally signed, but if the binary repo is served over HTTPS, as long as the repo has not been compromised, the integrity and authenticity is guaranteed, no? Or as the OP asked, is pkgin not actually validating the server's SSL certificate? That would be terrible if it's silently behaving that way. If it can't handle HTTPS properly, it should refuse to use the URL. Anyway, I would be very surprised if this is what's going on, so I'm just asking to understand better. Thank you! The code is not audited anyway, but just downloaded from various places, and then built. If you really want to have some actual security, and not just a false sense of it, https or so on is not really the answer. Anyone who thinks that just because https is involved, it is somehow more secure, is really fooling themselves. https is most properly something to use when submitting sensitive data to a web server, which you do not want someone to pick up along the way. The total craziness of moving the whole internet to https is not really improving any security in most situations. Not to mention the question of how you would solve the replication of repositories. All needs their own signatures. So there are a whole bunch of places where to get packages from. How do you know that they are all legit, and have the "right" binary packages even? You cannot have a single signature to ensure they are legit, since https ties certificates to the specific host. /Me feeling very tired of the https hysteria... Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: b...@softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol
Re: pkgsrc binary packages security with pkgin
On 01/25, m...@netbsd.org wrote: > On Sat, Jan 25, 2020 at 01:34:34AM +0100, yarl-bau...@mailoo.org wrote: > > May I ask how is safe the use pkgsrc binary packages. For example using > > pkgin. Does libfetch is doing fine with https? Any thoughts? > > > > Is the authenticity and integrity of packages installed this way is > > guaranteed assuming no bugs in software involved? > > No. Wow! That's surprising. Can you explain why? I understand that the binary packages are not digitally signed, but if the binary repo is served over HTTPS, as long as the repo has not been compromised, the integrity and authenticity is guaranteed, no? Or as the OP asked, is pkgin not actually validating the server's SSL certificate? That would be terrible if it's silently behaving that way. If it can't handle HTTPS properly, it should refuse to use the URL. Anyway, I would be very surprised if this is what's going on, so I'm just asking to understand better. Thank you! Lewis
Re: pkgsrc binary packages security with pkgin
On Sat, Jan 25, 2020 at 01:34:34AM +0100, yarl-bau...@mailoo.org wrote: > Hello, > > May I ask how is safe the use pkgsrc binary packages. For example using > pkgin. Does libfetch is doing fine with https? Any thoughts? > > Is the authenticity and integrity of packages installed this way is > guaranteed assuming no bugs in software involved? No. > > Is it safer to compile by yourself? Yes. This is a very unfortunate case.
pkgsrc binary packages security with pkgin
Hello, May I ask how is safe the use pkgsrc binary packages. For example using pkgin. Does libfetch is doing fine with https? Any thoughts? Is the authenticity and integrity of packages installed this way is guaranteed assuming no bugs in software involved? Is it safer to compile by yourself? Thank you.
Re: pkg_add vs pkgin
Hi, Mayuresh! Pkg_add/pkg_delete are reliable. Their job, however, is quite simple: - pkg_add is to install packages and their dependencies (if any); - pkg_delete is to remove packages and (if requested) those that depend on them. Pkg_add can perform a very simple upgrade, but if you request to install/upgrade a package that depends on another package older version of which is already installed on your system you will get an error as pkg_add will blindly try to install the newer version on top of the old (and get a conflict). Pkgin is more advanced: it builds a whole dependency tree and upgrade all dependencies. -- Aleksej Lebedev On Fri, May 10, 2019, at 19:10, Mayuresh wrote: > http://pkgin.net/ says: > > NetBSD, and more widely, all operating systems relying on pkgsrc have > tools like pkg_add and pkg_delete, but those are unable to correctly > handle binary upgrades, and sometimes even installation itself. > > Could someone please clarify what it means? Are pkg_add / del not > reliable? > > Mayuresh >
Re: pkg_add vs pkgin
On Tue, May 14, 2019 at 07:39:54PM +0530, Mayuresh wrote: > I often try to switch to binary mode. Then one odd time you'll come across > something that's not present in the binary repository and then you'd want > to turn to pkgsrc. Often by that time some things would have evolved and > pkgsrc would simply disregard your binary installation of some packages Just missed 1 point. binary looks a better option for forzen/stable branches of pkgsrc. Mayuresh
Re: pkg_add vs pkgin
On Tue, May 14, 2019 at 02:22:50PM +0200, Aleksej Lebedev wrote: > Pkg_add/pkg_delete are reliable. Their job, however, is quite > simple: > > - pkg_add is to install packages and their dependencies (if any); > - pkg_delete is to remove packages and (if requested) those that depend > on them. > > Pkg_add can perform a very simple upgrade, but if you request to > install/upgrade a package that > depends on another package older version of which is already installed on > your system > you will get an error as pkg_add will blindly try to install the newer > version on top of the old > (and get a conflict). Thanks Aleksej. I think the writeup can then be made a little more objective, in that case. `upgrades, and sometimes even installation itself' implies that, pkg_add sometimes may not handle a basic installation scenario (other than you mentioned) well, as well. > Pkgin is more advanced: it builds a whole dependency tree and upgrade all > dependencies. I do not know what others' experience with binary installation in general is. And I am not even talking about build options mismatch here. I often try to switch to binary mode. Then one odd time you'll come across something that's not present in the binary repository and then you'd want to turn to pkgsrc. Often by that time some things would have evolved and pkgsrc would simply disregard your binary installation of some packages (and for a reason) and you'll end up building it all over again. Further you have to take pains of finding all deps and installing manually. Recently I switched to binary mode and realized firefox wasn't present in the repo. I installed all deps manually and found that when I tried building firefox it went down to very low level like perl, python etc. to build them on its own. [ Incidentally I was able to build rust and firefox after a long gap! ] What I have learned over time is to forget trying to avoid compilation and just stick to pkgsrc. But I may be missing some tricks. I do not know if there are smarter ways to combine pkgsrc and binary repo. A pkgsrc level tool / script that installs deps transitively iff the dependency version check is also met, else switches to building by itself would be immensely useful. Besides, there may also be a check on whether the binary package (or its mk.conf) matches with options set in local mk.conf. If the options don't match it will fall back to pkgsrc. Further, mk.conf may have a directive to always force source build for some packages. etc. Mayuresh
pkg_add vs pkgin
http://pkgin.net/ says: NetBSD, and more widely, all operating systems relying on pkgsrc have tools like pkg_add and pkg_delete, but those are unable to correctly handle binary upgrades, and sometimes even installation itself. Could someone please clarify what it means? Are pkg_add / del not reliable? Mayuresh
Re: pkg/54123 (crash trying 'pkgin upgrade' with locally built pkg_summary)
At Sat, 20 Apr 2019 13:20:03 +0100, Chavdar Ivanov wrote: Subject: Re: pkg/54123 (crash trying 'pkgin upgrade' with locally built pkg_summary) > > I have always used > > cd /usr/pkgsrc/packages/All ; ( for i in *.tgz; pkg_info -X $i ) | > bzip2 > pkg_summary.bz2 Ah, that is interesting. I don't see how or why "pkg_info -X" should produce different results when looking at the .tgz vs. what is in /var/db/pkg. (without looking at the code, that is) I would call that a bug -- especially since it isn't documented as a feature! OK, I'll give that a try # pkgin upgrade calculating dependencies...done. 6 packages to refresh: perl-5.28.0nb2 yajl-2.1.0nb1 python27-2.7.15nb1 glib2-2.56.2nb3 m4-1.4.18nb1 autoconf-2.69nb8 4 packages to upgrade: py27-curses-2.7.15nb5 osabi-NetBSD-8.99.32 libarchive-3.3.3 automake-1.15.1nb1 6 to refresh, 4 to upgrade, 0 to install 0B to download, 85M to install proceed ? [Y/n] refreshing perl-5.28.0nb2... refreshing yajl-2.1.0nb1... refreshing python27-2.7.15nb1... refreshing glib2-2.56.2nb3... No schema files found: doing nothing. refreshing m4-1.4.18nb1... refreshing autoconf-2.69nb8... pkg_install warnings: 0, errors: 2 pkg_install error log can be found in /var/db/pkgin/pkg_install-err.log upgrading py27-curses-2.7.15nb5... upgrading osabi-NetBSD-8.99.32... upgrading libarchive-3.3.3... upgrading automake-1.15.1nb1... pkg_install warnings: 0, errors: 0 reading local summary... processing local summary... Well I'll be... It worked! Thanks! OK, that's definitely a bug in 'pkg_info -X', IMNSHO. Now I'll have to sort the chunks in each differently produced pkg_summary file to try doing a diff between them > This gives me the older versions of packages as well, should I need > them (and I do - e.g. qemu-3.1.0nb4.tgz was produced from > wip/qemu-nvmm, whereas qemu-3.1.0nb5.tgz from emulators/qemu, a rather > different one, ten times the size). However that is an important distinction to be sure! I don't expect I would need the older packages quite so often, given I static-link as many packages as possible (and I've tweaked pkgsrc where necessary to not record the library-only, i.e. now build-only, dependencies as runtime dependencies), though I've been unable to eliminate quite as much interdependency as I would have hoped. On the other hand part of the remaining problem is in how dependencies are specified for packages providing shared libraries and other less explicit A"B"Is. (The "B" in quotes because I'm referring more to the more dynamic interfaces outside of binary executables, such as file formats, command-line options, etc.) -- Greg A. Woods +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp3CppfPVrSo.pgp Description: OpenPGP Digital Signature
Re: pkg/54123 (crash trying 'pkgin upgrade' with locally built pkg_summary)
On Sat, 20 Apr 2019 at 07:16, Greg A. Woods wrote: > > [[ I tried sending this to gnats-admin, but it hasn't appeared yet, and > in any case folks here might have answers or suggestions too. ]] > > At Wed, 17 Apr 2019 23:32:40 +0100, Jonathan Perkin > wrote: > Subject: Re: pkg/54123 (crash trying 'pkgin upgrade' with locally built > pkg_summary) > > > > While pkgin shouldn't crash and should be able to handle bad input, it > > should be pointed out that this use-case is not expected to work at all, > > and any fix will simply enforce that. Your pkg_summary files should be > > generated from binary package files, not installed packages. > > Hmmm... OK, so how should I generate the pkg_summary file for my limited > archive of locally built binary packages? > > I couldn't find much info anywhere about handling the server-side of > things for pkgin, so I RTFM'ed and just did what it says in > pkg_summary(5): > > The pkg_summary file can be generated using the pkg_info(1) －X option. > For example, the following will list this data for all installed pack‐ > ages: > >pkg_info ‐X ‐a I have always used cd /usr/pkgsrc/packages/All ; ( for i in *.tgz; pkg_info -X $i ) | bzip2 > pkg_summary.bz2 This gives me the older versions of packages as well, should I need them (and I do - e.g. qemu-3.1.0nb4.tgz was produced from wip/qemu-nvmm, whereas qemu-3.1.0nb5.tgz from emulators/qemu, a rather different one, ten times the size). I still get errors using this repo on occasion, usually some dependencies not caught for upgrade, so I have to examine the log file and first go through their update - again with pkgin - repeating the initial installation at the end. It is not ideal, but usable. I normally setup the repo to be accessed via anonymous ftp, but also via nfs. > > And I hoped that a file of the same name was of the kind that pkgin > would be happy to use. > > Pkgin did seem to happily suck up the file, and "pkgin avail" gives me a > nice list corresponding to all the binary packages I should have > available. It's just the attempt to install one of them that failed. > > I.e. there are binaries for all the installed packages in PKG_PATH -- > that's the point, after all, as I am trying to use pkgin to install > those binaries on other local systems. (Indeed I now rely on the way > pkgsrc uses DESTDIR to create a binary package that's then installed as > the last step, even on the build machine.) > > (and yes, PKG_PATH is set when I run "pkg_info -X -a", if that matters) > > One caveat I have locally is that these binary packages may not all for > the same OS version and/or they may not all be in the same PKG_PATH > location, since it is -current, and I've built different packages at > different times while upgrading the OS from time to time (and though I > often use pgk_rolling-replace with its '-B' option on the build machine, > that doesn't seem to find absolutely everything that's not for the same > OS version and rebuild it). > > So, I'm not sure if I should be simply linking together all the > compatible OS version binary package directories, or not. With pkg_add > I can't put multiple repositories in PKG_PATH, and presumably not for > "pkg_info -X" either, though it is hinted that repositories.conf can > contain a list of locations, though it's not clear if there can only be > one per $arch and/or $osrelease, nor is it clear what happens if > different installed packages were built for different (but nominally > compatible) $osrelease values. (The issue for me is that I'll likely > never manage to build everything I want all together at once with the > exact same OS release -- and I don't want to care about this as long as > the installed binaries run, and after all that's part of the point of > using NetBSD is that the ABI is stable for the most part, and even if > I've built packages on an old release that needs a COMPAT option, I > might want to include them in the stable of binary packages that I make > available for pkgin. I only really want to use pkgin for its ease of > managing upgrades, since for the initial installs it is not much > different for me to just use pkg_add directly, provided I really can > start with an empty /var/db/pkgin database and have it rebuilt to > account for such manually installed packages.) > > -- > Greg A. Woods > > +1 250 762-7675 RoboHack > Planix, Inc. Avoncote Farms --
Re: pkg/54123 (crash trying 'pkgin upgrade' with locally built pkg_summary)
[[ I tried sending this to gnats-admin, but it hasn't appeared yet, and in any case folks here might have answers or suggestions too. ]] At Wed, 17 Apr 2019 23:32:40 +0100, Jonathan Perkin wrote: Subject: Re: pkg/54123 (crash trying 'pkgin upgrade' with locally built pkg_summary) > > While pkgin shouldn't crash and should be able to handle bad input, it > should be pointed out that this use-case is not expected to work at all, > and any fix will simply enforce that. Your pkg_summary files should be > generated from binary package files, not installed packages. Hmmm... OK, so how should I generate the pkg_summary file for my limited archive of locally built binary packages? I couldn't find much info anywhere about handling the server-side of things for pkgin, so I RTFM'ed and just did what it says in pkg_summary(5): The pkg_summary file can be generated using the pkg_info(1) −X option. For example, the following will list this data for all installed pack‐ ages: pkg_info ‐X ‐a And I hoped that a file of the same name was of the kind that pkgin would be happy to use. Pkgin did seem to happily suck up the file, and "pkgin avail" gives me a nice list corresponding to all the binary packages I should have available. It's just the attempt to install one of them that failed. I.e. there are binaries for all the installed packages in PKG_PATH -- that's the point, after all, as I am trying to use pkgin to install those binaries on other local systems. (Indeed I now rely on the way pkgsrc uses DESTDIR to create a binary package that's then installed as the last step, even on the build machine.) (and yes, PKG_PATH is set when I run "pkg_info -X -a", if that matters) One caveat I have locally is that these binary packages may not all for the same OS version and/or they may not all be in the same PKG_PATH location, since it is -current, and I've built different packages at different times while upgrading the OS from time to time (and though I often use pgk_rolling-replace with its '-B' option on the build machine, that doesn't seem to find absolutely everything that's not for the same OS version and rebuild it). So, I'm not sure if I should be simply linking together all the compatible OS version binary package directories, or not. With pkg_add I can't put multiple repositories in PKG_PATH, and presumably not for "pkg_info -X" either, though it is hinted that repositories.conf can contain a list of locations, though it's not clear if there can only be one per $arch and/or $osrelease, nor is it clear what happens if different installed packages were built for different (but nominally compatible) $osrelease values. (The issue for me is that I'll likely never manage to build everything I want all together at once with the exact same OS release -- and I don't want to care about this as long as the installed binaries run, and after all that's part of the point of using NetBSD is that the ABI is stable for the most part, and even if I've built packages on an old release that needs a COMPAT option, I might want to include them in the stable of binary packages that I make available for pkgin. I only really want to use pkgin for its ease of managing upgrades, since for the initial installs it is not much different for me to just use pkg_add directly, provided I really can start with an empty /var/db/pkgin database and have it rebuilt to account for such manually installed packages.) -- Greg A. Woods +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpcc7ip_vR0U.pgp Description: OpenPGP Digital Signature
Re: pkgin
Thx! Yes, that's understandable. Still, https://gitlab.com/iMil/pkgin looks to be outdated as well :( The best would be if http://pkgin.net/ pointed to https://github.com/joyent/pkgin which seems to be the master branch. Den fre 19 okt. 2018 15:45matthew sporleder skrev: > On Thu, Oct 18, 2018 at 3:24 AM Pedro Pinho wrote: > > > > Hi all, > > I'm rather new to NetBSD, been using it on a laptop for about 3 months > now, amd64 8.0. I'm really enjoying this OS and everything is working fine. > So, sorry if this has already been answered... > > > > Why does http://pkgin.net/ points to https://github.com/NetBSDfr/pkgin ? > > This github repo has not seen any commit for the last two years. > > Wouldn't be better if it would point to https://github.com/joyent/pkgin > , just like pkgsrc.se does? > > Is there a reason for it? > > > > Thanks! > > I'm guessing it has something to do with the text you can find here: > https://github.com/iMilnb :) > > Anyway good suggestion. >
