Re: Disable ftp in pkg_add syspatch sysupgrade
For completeness, I discovered I was having issues with downloading the sources for the sysupgrade command on my edge firewall also! So it was not limited to internet servers as first thought. Since upgrading the 6.6 (had to run sysupgrade 4 times to get it to complete the downloads), the issues seems to have been resolved and now all packages are installing first time every time.. So I am not sure if this is just me or a 6.5 issue. Most likely the former ;) Kind regards, Andy. Sent from a teeny tiny keyboard, so please excuse typos > On 31 Oct 2019, at 01:47, Stuart Henderson wrote: > >> On 2019-10-30, Andrew Lemin wrote: >> - But throws errors when I try and use flavours which is critical for >> installing python for example (NB; This is a different error to before, >> where I was getting 'timeout' instead of 'Invalid argument'); >> [HOME]root@testbsd1:/local#pkg_add python%2 py-pip python%3 py3-pip >> py3-setuptools >> quirks-3.124 signed on 2019-10-16T20:27:45Z >> http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz: >> ftp: Receiving HTTP reply: Invalid argument >> signify: gzheader truncated >> Couldn't install py3-setuptools-40.0.0v0 > > Odd. Can you try replicating on 6.6? > > Does "pkg_add py-pip py3-pip py3-setuptools" (i.e. allow pkg_add to > find the dependencies by itself) work? > >
Re: Disable ftp in pkg_add syspatch sysupgrade
Hahaha Thanks Theo, that made me smile. But you have answered my question perfectly, albeit in a round about way. Indeed it doesn’t matter what it is called, and would be clearer with a generic name, as we got caught out by a program calling another program with colliding name. For example, Having ‘pkg_add’ call a program named ‘ftp’ to perform http and https downloads. But where errors in the ftp subprocess are printed by the pkg_add process, making it seem like pkg_add was failing on an ftp protocol request, rather than the ‘ftp’ client process failing (while doing an http call).. So I think it was pretty fair for us to end up scratching our heads ;) Thanks, Andy. Sent from a teeny tiny keyboard, so please excuse typos > On 30 Oct 2019, at 15:54, Theo de Raadt wrote: > > Andrew Lemin wrote: > >> To me this seems unusual (was expecting 'curl' or 'wget' etc to avoid code >> duplication) and confusing? What do you think? > > curl is not in openbsd > > wget is not in openbsd > > Maybe we should rename our downloading software to lemin, which is > obviously a randomly chosen name with some obscure acronym we'll invent > to back the name, being a name noone recognizes we can probably avoid > assumptions as to what it does, whether it does ftp, or http, or https, > or who knows what. Of course such a strange name would also lead people > to not discovering it, and make them install some monster software > package off the internet with another strange name. > > In summary I think it's turning into a shitty world with selection by > meme. > >
Re: Disable ftp in pkg_add syspatch sysupgrade
On 2019-10-30, Andrew Lemin wrote: > - But throws errors when I try and use flavours which is critical for > installing python for example (NB; This is a different error to before, > where I was getting 'timeout' instead of 'Invalid argument'); > [HOME]root@testbsd1:/local#pkg_add python%2 py-pip python%3 py3-pip > py3-setuptools > quirks-3.124 signed on 2019-10-16T20:27:45Z > http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz: > ftp: Receiving HTTP reply: Invalid argument > signify: gzheader truncated > Couldn't install py3-setuptools-40.0.0v0 Odd. Can you try replicating on 6.6? Does "pkg_add py-pip py3-pip py3-setuptools" (i.e. allow pkg_add to find the dependencies by itself) work?
Re: Disable ftp in pkg_add syspatch sysupgrade
Andrew Lemin wrote: > To me this seems unusual (was expecting 'curl' or 'wget' etc to avoid code > duplication) and confusing? What do you think? curl is not in openbsd wget is not in openbsd Maybe we should rename our downloading software to lemin, which is obviously a randomly chosen name with some obscure acronym we'll invent to back the name, being a name noone recognizes we can probably avoid assumptions as to what it does, whether it does ftp, or http, or https, or who knows what. Of course such a strange name would also lead people to not discovering it, and make them install some monster software package off the internet with another strange name. In summary I think it's turning into a shitty world with selection by meme.
Re: Disable ftp in pkg_add syspatch sysupgrade
Hi gents, Sorry for the slow reply, and thank you for all your responses! :D Raf, you are correct. It seems that the ftp client is performing an http(s) downloads. To me this seems unusual (was expecting 'curl' or 'wget' etc to avoid code duplication) and confusing? What do you think? Stuart, thanks for your suggestion. This confirmed the ftp client is using http(s); [HOME]root@testbsd1:/local#pgrep -lf ftp 40379 /usr/bin/ftp -o - http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages-stable/amd64/quirks-3.124.tgz Tom/PJ, Understood. Was just very confused why ftp was getting involved.. Anyway, I have tested this some more, and it looks like the issue is related to when using "flavors", and looks like maybe their is some sort of timeout occuring maybe. - We can see that pkg_add is working fine when specifying packages explicitly; [HOME]root@testbsd1:/local#pkg_add sudo--gettext bash htop quirks-3.124 signed on 2019-10-16T20:27:45Z [HOME]root@testbsd1:/local#pkg_add vim--no_x11-perl-python3-ruby unzip--iconv bzip2 git fzf quirks-3.124 signed on 2019-10-16T20:27:45Z - But throws errors when I try and use flavours which is critical for installing python for example (NB; This is a different error to before, where I was getting 'timeout' instead of 'Invalid argument'); [HOME]root@testbsd1:/local#pkg_add python%2 py-pip python%3 py3-pip py3-setuptools quirks-3.124 signed on 2019-10-16T20:27:45Z http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz: ftp: Receiving HTTP reply: Invalid argument signify: gzheader truncated Couldn't install py3-setuptools-40.0.0v0 - This package is accessible as seen here; [HOME]root@testbsd1:/local#wget http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz /tmp/ --2019-10-30 14:29:28-- http://mirror.bytemark.co.uk/pub/OpenBSD/6.5/packages/amd64/py3-setuptools-40.0.0v0.tgz Resolving mirror.bytemark.co.uk (mirror.bytemark.co.uk)... 80.68.83.150, 212.110.163.12, 2001:41c8:20:5e6::150, ... Connecting to mirror.bytemark.co.uk (mirror.bytemark.co.uk)|80.68.83.150|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 731604 (714K) [application/x-gzip] Saving to: ‘py3-setuptools-40.0.0v0.tgz’ py3-setuptools-40.0.0v0.tgz 100%[===>] 714.46K 270KB/sin 2.6s - And works if specified on its own; [HOME]root@testbsd1:/local#pkg_add py3-setuptools quirks-3.124 signed on 2019-10-16T20:27:45Z If I try the line with flavors again "pkg_add python%2 py-pip python%3 py3-pip py3-setuptools" it works.. As others would be crying about this too if it were a wide issue, I thought this was maybe a bad mirror... So I have now tried every mirror in the UK, and they all do the same thing - intermittent issues accessing packages when using flavours.. I am not running squid or any kind of web proxy, http and https are being passed out with nothing more than standard NAT and a pass rule. I will try and figure out what is going on. Leave this with me. If I find anything meanful and useful I will let you know. For now, consider this an issue with my setup.. PS; has anyone managed to get ftpproxy working in an rdomain? Thanks for your time and responses.. :) Andy. On Wed, Oct 30, 2019 at 9:17 AM PJ wrote: > Am 30.10.19 um 07:32 schrieb tom ryan: > > On 2019-10-29 20:19, PJ wrote: > >> Am 28.10.19 um 23:52 schrieb Stuart Henderson: > >>> On 2019-10-28, Andy Lemin wrote: > Hi guys, > > Does anyone know if it is possible to completely disable ftp in the > package management utilities; pkg_add, syspatch, sysupgrade etc? > > My PKG_PATH references http:// urls, as does /etc/install. But I > cannot stop these tools trying to use ftp which does not work! :( > >>> Can you show some example URLs, for example from "pgrep -lf ftp" while > >>> trying to use one of these utilities? > >>> > >>> The only place I would expect to see ftp:// URLs used > >> grep ftp /usr/sbin/sysupgrade > > $ grep -ne ftp -e URL -e MIRROR /usr/sbin/sysupgrade > > 102:0) MIRROR=$(sed 's/#.*//;/^$/d' /etc/installurl) 2>/dev/null || > > 103:MIRROR=https://cdn.openbsd.org/pub/OpenBSD > > 105:1) MIRROR=$1 > > 117:URL=${MIRROR}/snapshots/${ARCH}/ > > 119:URL=${MIRROR}/${NEXT_VERSION}/${ARCH}/ > > 136:unpriv -f SHA256.sig ftp -Vmo SHA256.sig ${URL}SHA256.sig > > 176:unpriv -f $f ftp -Vmo ${f} ${URL}${f} > > > > Your point? > > I understand that I misread the question, sorry. > > > >>> is when fetching > >>> certain distfiles while building some things from ports (and they would > >>> usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if > >>> the ftp fetch failed).. > >
Re: Disable ftp in pkg_add syspatch sysupgrade
Am 30.10.19 um 07:32 schrieb tom ryan: > On 2019-10-29 20:19, PJ wrote: >> Am 28.10.19 um 23:52 schrieb Stuart Henderson: >>> On 2019-10-28, Andy Lemin wrote: Hi guys, Does anyone know if it is possible to completely disable ftp in the package management utilities; pkg_add, syspatch, sysupgrade etc? My PKG_PATH references http:// urls, as does /etc/install. But I cannot stop these tools trying to use ftp which does not work! :( >>> Can you show some example URLs, for example from "pgrep -lf ftp" while >>> trying to use one of these utilities? >>> >>> The only place I would expect to see ftp:// URLs used >> grep ftp /usr/sbin/sysupgrade > $ grep -ne ftp -e URL -e MIRROR /usr/sbin/sysupgrade > 102:0) MIRROR=$(sed 's/#.*//;/^$/d' /etc/installurl) 2>/dev/null || > 103:MIRROR=https://cdn.openbsd.org/pub/OpenBSD > 105:1) MIRROR=$1 > 117:URL=${MIRROR}/snapshots/${ARCH}/ > 119:URL=${MIRROR}/${NEXT_VERSION}/${ARCH}/ > 136:unpriv -f SHA256.sig ftp -Vmo SHA256.sig ${URL}SHA256.sig > 176:unpriv -f $f ftp -Vmo ${f} ${URL}${f} > > Your point? I understand that I misread the question, sorry. >>> is when fetching >>> certain distfiles while building some things from ports (and they would >>> usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if >>> the ftp fetch failed)..
Re: Disable ftp in pkg_add syspatch sysupgrade
On 2019-10-29 20:19, PJ wrote: > Am 28.10.19 um 23:52 schrieb Stuart Henderson: >> On 2019-10-28, Andy Lemin wrote: >>> Hi guys, >>> >>> Does anyone know if it is possible to completely disable ftp in the package >>> management utilities; pkg_add, syspatch, sysupgrade etc? >>> >>> My PKG_PATH references http:// urls, as does /etc/install. But I cannot >>> stop these tools trying to use ftp which does not work! :( >> Can you show some example URLs, for example from "pgrep -lf ftp" while >> trying to use one of these utilities? >> >> The only place I would expect to see ftp:// URLs used > > > grep ftp /usr/sbin/sysupgrade $ grep -ne ftp -e URL -e MIRROR /usr/sbin/sysupgrade 102:0) MIRROR=$(sed 's/#.*//;/^$/d' /etc/installurl) 2>/dev/null || 103:MIRROR=https://cdn.openbsd.org/pub/OpenBSD 105:1) MIRROR=$1 117:URL=${MIRROR}/snapshots/${ARCH}/ 119:URL=${MIRROR}/${NEXT_VERSION}/${ARCH}/ 136:unpriv -f SHA256.sig ftp -Vmo SHA256.sig ${URL}SHA256.sig 176:unpriv -f $f ftp -Vmo ${f} ${URL}${f} Your point? > >> is when fetching >> certain distfiles while building some things from ports (and they would >> usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if >> the ftp fetch failed).. >> >> >> >
Re: Disable ftp in pkg_add syspatch sysupgrade
On 2019-10-29, PJ wrote: > Am 28.10.19 um 23:52 schrieb Stuart Henderson: >> On 2019-10-28, Andy Lemin wrote: >>> Hi guys, >>> >>> Does anyone know if it is possible to completely disable ftp in the package >>> management utilities; pkg_add, syspatch, sysupgrade etc? >>> >>> My PKG_PATH references http:// urls, as does /etc/install. But I cannot >>> stop these tools trying to use ftp which does not work! :( >> Can you show some example URLs, for example from "pgrep -lf ftp" while >> trying to use one of these utilities? >> >> The only place I would expect to see ftp:// URLs used > > > grep ftp /usr/sbin/sysupgrade Yes I know how sysupgrade works. There is no reason why it would try to connect via the FTP protocol unless either an ftp:// url is set in /etc/installurl or an ftp:// url is specified on the command line. ftp(1) is not restricted to just the FTP protocol, it is most often used as an http/https fetcher these days.
Re: Disable ftp in pkg_add syspatch sysupgrade
Am 28.10.19 um 23:52 schrieb Stuart Henderson: > On 2019-10-28, Andy Lemin wrote: >> Hi guys, >> >> Does anyone know if it is possible to completely disable ftp in the package >> management utilities; pkg_add, syspatch, sysupgrade etc? >> >> My PKG_PATH references http:// urls, as does /etc/install. But I cannot stop >> these tools trying to use ftp which does not work! :( > Can you show some example URLs, for example from "pgrep -lf ftp" while > trying to use one of these utilities? > > The only place I would expect to see ftp:// URLs used grep ftp /usr/sbin/sysupgrade > is when fetching > certain distfiles while building some things from ports (and they would > usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if > the ftp fetch failed).. > > >
Re: Disable ftp in pkg_add syspatch sysupgrade
On 2019-10-28, Andy Lemin wrote: > Hi guys, > > Does anyone know if it is possible to completely disable ftp in the package > management utilities; pkg_add, syspatch, sysupgrade etc? > > My PKG_PATH references http:// urls, as does /etc/install. But I cannot stop > these tools trying to use ftp which does not work! :( Can you show some example URLs, for example from "pgrep -lf ftp" while trying to use one of these utilities? The only place I would expect to see ftp:// URLs used is when fetching certain distfiles while building some things from ports (and they would usually fallback to http://ftp.openbsd.org/pub/OpenBSD/distfiles if the ftp fetch failed)..
Re: Disable ftp in pkg_add syspatch sysupgrade
On Mon, Oct 28, 2019 at 06:43:03PM GMT, Andy Lemin wrote: > Hi guys, > > Does anyone know if it is possible to completely disable ftp in the package > management utilities; pkg_add, syspatch, sysupgrade etc? > > My PKG_PATH references http:// urls, as does /etc/install. But I cannot stop > these tools trying to use ftp which does not work! :( > > Every time I try and install a bunch of packages, some work fine via http, > and then some will fall back to ftp, which of course fails. > > However when I test the url for the package it reports it could not download, > due to the ftp connect timeout, the download works fine via http etc. So it > is very frustrating. > > We cannot use ftpproxy as we use rdomains, and running the ftpproxy daemon > within the rdomain that has the default route to the internet doesn’t seem to > work. And generally speaking no one in my last two companies has ever asked > me to make ftp work. > > As ftp is very much dead in most cases, could we please stop using it for > package management tools? :( I cannot update any of my OpenBSD servers which > reside inside the firewall. Only the firewall itself get et updates because > of the ftp dependence :( > > Thanks for your time. > Andy. > > > Sent from a teeny tiny keyboard, so please excuse typos > Hi Andy, If, like you say, PKG_PATH and installurl(5) both reference HTTP(S), then you're getting confused - warning/error messages come from ftp(1). It has nothing to do with FTP (the protocol). Regards, Raf
Disable ftp in pkg_add syspatch sysupgrade
Hi guys, Does anyone know if it is possible to completely disable ftp in the package management utilities; pkg_add, syspatch, sysupgrade etc? My PKG_PATH references http:// urls, as does /etc/install. But I cannot stop these tools trying to use ftp which does not work! :( Every time I try and install a bunch of packages, some work fine via http, and then some will fall back to ftp, which of course fails. However when I test the url for the package it reports it could not download, due to the ftp connect timeout, the download works fine via http etc. So it is very frustrating. We cannot use ftpproxy as we use rdomains, and running the ftpproxy daemon within the rdomain that has the default route to the internet doesn’t seem to work. And generally speaking no one in my last two companies has ever asked me to make ftp work. As ftp is very much dead in most cases, could we please stop using it for package management tools? :( I cannot update any of my OpenBSD servers which reside inside the firewall. Only the firewall itself get et updates because of the ftp dependence :( Thanks for your time. Andy. Sent from a teeny tiny keyboard, so please excuse typos