Re: Robustness in ports fetch program?
Oh, I apparently can't do FTP, but that's a recent thing so I'm not sure. I'm using a cell phone data connection. On 5/20/15, Alan Corey wrote: > I didn't override it because I didn't know how. I've defined > FETCH_CMD in the environment before but I've never messed with > mk.conf. I put in literally what you said, but _PROGRESS and > FTP_KEEPALIVE seem to be undefined. > > I'm experimenting now using wget but here's a run from trying to > install qiv. I was able to get firefox installed without much > problem. > > ===> Checking files for qiv-2.3.1 >>> Fetch http://spiegl.de/qiv//download/qiv-2.3.1.tgz > Trying 2a03:2500:1:7:5054:ff:fe95:55b1... > ftp: connect: No route to host >>> Fetch http://ftp.openbsd.org/pub/OpenBSD/distfiles//qiv-2.3.1.tgz > Trying 129.128.5.191... > Requesting http://ftp.openbsd.org/pub/OpenBSD/distfiles//qiv-2.3.1.tgz > > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% | | 0 --:-- > ETA > 0% |
Re: Robustness in ports fetch program?
I didn't override it because I didn't know how. I've defined FETCH_CMD in the environment before but I've never messed with mk.conf. I put in literally what you said, but _PROGRESS and FTP_KEEPALIVE seem to be undefined. I'm experimenting now using wget but here's a run from trying to install qiv. I was able to get firefox installed without much problem. ===> Checking files for qiv-2.3.1 >> Fetch http://spiegl.de/qiv//download/qiv-2.3.1.tgz Trying 2a03:2500:1:7:5054:ff:fe95:55b1... ftp: connect: No route to host >> Fetch http://ftp.openbsd.org/pub/OpenBSD/distfiles//qiv-2.3.1.tgz Trying 129.128.5.191... Requesting http://ftp.openbsd.org/pub/OpenBSD/distfiles//qiv-2.3.1.tgz 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0 --:-- ETA 0% | | 0
Re: Robustness in ports fetch program?
On 2015-05-17, Alan Corey wrote: > I didn't look at what FETCH_CMD was defined as by default, I just assumed > defining something non-null changed it. I did notice that when it retries > it's wrongly assumed there's a problem with the first source and gone to > another. That's a reasonable supposition because in the majority of cases, it's correct and you do want to move on to another host. It's common to see files which are no longer present return an "error" page with a 200 OK response; when that happens you *have* to move on. > Does every developer have perfect internet? That's very frustrating, maybe > counterproductive in testing. Try a modem, you can probably find a free > one. Connection interruptions and resets happen many times a day. Modem, yeah easy. Dialup ISP not so much. Per-minute connection charges in many cases too. But then back when I was on dialup I didn't get that many interruptions/resets, and when they did happen TCP sessions generally recovered. People tended to use specialist download programs for anything large though, maybe you should too?
Re: Robustness in ports fetch program?
On Sun, May 17, 2015 at 08:18:06AM -0400, Alan Corey wrote:
> I don't think it did this back in 5.0 days or maybe earlier. I started
> with OpenBSD 2.7, I just usually attributed problems to being my fault.
> And I've always used the ports tree, not packages. Distfiles are often
> useful across OpenBSD versions, sometimes in FreeBSD, I've even built some
> under Linux.
>
> I didn't look at what FETCH_CMD was defined as by default, I just assumed
> defining something non-null changed it. I did notice that when it retries
> it's wrongly assumed there's a problem with the first source and gone to
> another.
>
> Does every developer have perfect internet? That's very frustrating, maybe
> counterproductive in testing. Try a modem, you can probably find a free
> one. Connection interruptions and resets happen many times a day.
> On May 17, 2015 1:22 AM, "Marc Espie" wrote:
Why are you ranting instead of providing the info I'm asking for ?!!!
JUST OVERRIDE THE DAMN FETCH_CMD!!!
put
FETCH_CMD = /usr/bin/ftp -v ${_PROGRESS} -k ${FTP_KEEPALIVE} -C
in /etc/mk.conf
so that *at least* we can see verbose output from your fetches.
Like I said, *the error comes from ftp*.
More accurately, fetch itself has the following logic:
for site in list
do
if FETCH_CMD -o file.part ${site}url
then
ck=`check_size file.part.part`
-> leading to "size does not match, hence rm file.part, hence retry"
fi
done
this is where your problem lies: ftp returns "everything okay", so the logic
assumes the file retrieved correctly, and when it finds out the size does not
match, it assumes a corrupted mirror, and hence deletes the partial file.
ftp(1)'s code is awful. I'm not wading thru those waters without more info.
GIVE ME WHATEVER FTP IS SAYING WHEN THINGS BREAK, when you tell it to be
verbose.
Re: Robustness in ports fetch program?
Try some of these ideas.
Change the config of pf to "conservative" or "high-latency" (man
pf.conf).
Use dpb to download the distfiles:
/usr/ports/infrastructure/bin/dpb -F 2 lang/tcl/8.5
(man -m /usr/ports/infrastructure/man dpb)
Change the ports framework to download the distfiles first from the
OpenBSD mastersites, and adjust the FTP keepalive:
export PORTS_INFRA="/usr/ports/infrastructure/db/network.conf"
echo '.include "../templates/network.conf.template"' > $PORTS_INFRA
echo 'MASTER_SITE_OPENBSD=ftp://ftp.fr.openbsd.org/pub/OpenBSD/distfiles/' >>
$PORTS_INFRA
echo 'MASTER_SITE_OPENBSD=Yes' >> $PORTS_INFRA
echo 'FTP_KEEPALIVE=15' >> /etc/mk.conf
Create a shell/perl script which retries the download if the file
doesn't exists in the distfiles directory, and change FETCH_CMD to run
it. FETCH_CMD uses this by default: /usr/bin/ftp -V ${_PROGRESS}
-k ${FTP_KEEPALIVE} -C
On Sun, May 17, 2015 at 08:18:06AM -0400, Alan Corey wrote:
> I don't think it did this back in 5.0 days or maybe earlier. I started
> with OpenBSD 2.7, I just usually attributed problems to being my fault.
> And I've always used the ports tree, not packages. Distfiles are often
> useful across OpenBSD versions, sometimes in FreeBSD, I've even built some
> under Linux.
>
> I didn't look at what FETCH_CMD was defined as by default, I just assumed
> defining something non-null changed it. I did notice that when it retries
> it's wrongly assumed there's a problem with the first source and gone to
> another.
>
> Does every developer have perfect internet? That's very frustrating, maybe
> counterproductive in testing. Try a modem, you can probably find a free
> one. Connection interruptions and resets happen many times a day.
> On May 17, 2015 1:22 AM, "Marc Espie" wrote:
>
> > On Sat, May 16, 2015 at 10:31:24PM -0400, Alan Corey wrote:
> > > I'd seen this happen in 5.6 too, but I just caught an example of it in
> > > 5.7. My connection leaves a lot to be desired, but there's nothing I
> > > can do about that. I normally have FETCH_CMD set to use wget once I
> > > get it installed but this was in doing a standard make install of a
> > > port.
> > >
> > > The first time the connection gets interrupted, but something thinks
> > > it should be done and checks the size. That's wrong so it downloads
> > > it over again instead of just resuming the download. It should only
> > > download it over again if the size matches but the CRC is wrong.
> > > Seems like anyway.
> > >
> > > ===> Verifying install for tcl-8.5.16 in lang/tcl/8.5
> > > ===> Checking files for tcl-8.5.16p0
> > > >> Fetch
> > > >> http://downloads.sourceforge.net/sourceforge/tcl/tcl8.5.16-src.tar.gz
> > > tcl8.5.16-src.tar.gz 60% |*| 2696 KB
> > 00:00
> > > >> Size does not match for tcl8.5.16-src.tar.gz
> > > >> Fetch
> > http://ftp.openbsd.org/pub/OpenBSD/distfiles//tcl8.5.16-src.tar.gz
> > > tcl8.5.16-src.tar.gz 23% |** | 1024 KB
> > 00:03 ETA
> >
> > The problem lies in ftp(1).
> >
> > Logic in the ports tree is fine. But there's nothing it can do there:
> > somehow
> > your ftp returns 0 (e.g., success), so the partial file gets removed.
> >
> > If you want to get it fixed, you may have to provide more input, as we
> > obviously do not see that problem... First thing would be to override
> > FETCH_CMD to remove the -V, so that you can show us what ftp says about
> > things. Tracing the code thru the program would help.
>
--
Juan Francisco Cantero Hurtado http://juanfra.info
Re: Robustness in ports fetch program?
2015-05-17 14:18 GMT+02:00 Alan Corey : > I don't think it did this back in 5.0 days or maybe earlier. I started > with OpenBSD 2.7, I just usually attributed problems to being my fault. > And I've always used the ports tree, not packages. Distfiles are often > useful across OpenBSD versions, sometimes in FreeBSD, I've even built some > under Linux. > > I didn't look at what FETCH_CMD was defined as by default, I just assumed > defining something non-null changed it. I did notice that when it retries > it's wrongly assumed there's a problem with the first source and gone to > another. > > Does every developer have perfect internet? That's very frustrating, maybe > counterproductive in testing. Try a modem, you can probably find a free > one. Connection interruptions and resets happen many times a day. > On May 17, 2015 1:22 AM, "Marc Espie" wrote: > >> On Sat, May 16, 2015 at 10:31:24PM -0400, Alan Corey wrote: >> > I'd seen this happen in 5.6 too, but I just caught an example of it in >> > 5.7. My connection leaves a lot to be desired, but there's nothing I >> > can do about that. I normally have FETCH_CMD set to use wget once I >> > get it installed but this was in doing a standard make install of a >> > port. >> > >> > The first time the connection gets interrupted, but something thinks >> > it should be done and checks the size. That's wrong so it downloads >> > it over again instead of just resuming the download. It should only >> > download it over again if the size matches but the CRC is wrong. >> > Seems like anyway. >> > >> > ===> Verifying install for tcl-8.5.16 in lang/tcl/8.5 >> > ===> Checking files for tcl-8.5.16p0 >> > >> Fetch >> > >> http://downloads.sourceforge.net/sourceforge/tcl/tcl8.5.16-src.tar.gz >> > tcl8.5.16-src.tar.gz 60% |*| 2696 KB >> 00:00 >> > >> Size does not match for tcl8.5.16-src.tar.gz >> > >> Fetch >> http://ftp.openbsd.org/pub/OpenBSD/distfiles//tcl8.5.16-src.tar.gz >> > tcl8.5.16-src.tar.gz 23% |** | 1024 KB >> 00:03 ETA >> >> The problem lies in ftp(1). >> >> Logic in the ports tree is fine. But there's nothing it can do there: >> somehow >> your ftp returns 0 (e.g., success), so the partial file gets removed. >> >> If you want to get it fixed, you may have to provide more input, as we >> obviously do not see that problem... First thing would be to override >> FETCH_CMD to remove the -V, so that you can show us what ftp says about >> things. Tracing the code thru the program would help.
Re: Robustness in ports fetch program?
I don't think it did this back in 5.0 days or maybe earlier. I started with OpenBSD 2.7, I just usually attributed problems to being my fault. And I've always used the ports tree, not packages. Distfiles are often useful across OpenBSD versions, sometimes in FreeBSD, I've even built some under Linux. I didn't look at what FETCH_CMD was defined as by default, I just assumed defining something non-null changed it. I did notice that when it retries it's wrongly assumed there's a problem with the first source and gone to another. Does every developer have perfect internet? That's very frustrating, maybe counterproductive in testing. Try a modem, you can probably find a free one. Connection interruptions and resets happen many times a day. On May 17, 2015 1:22 AM, "Marc Espie" wrote: > On Sat, May 16, 2015 at 10:31:24PM -0400, Alan Corey wrote: > > I'd seen this happen in 5.6 too, but I just caught an example of it in > > 5.7. My connection leaves a lot to be desired, but there's nothing I > > can do about that. I normally have FETCH_CMD set to use wget once I > > get it installed but this was in doing a standard make install of a > > port. > > > > The first time the connection gets interrupted, but something thinks > > it should be done and checks the size. That's wrong so it downloads > > it over again instead of just resuming the download. It should only > > download it over again if the size matches but the CRC is wrong. > > Seems like anyway. > > > > ===> Verifying install for tcl-8.5.16 in lang/tcl/8.5 > > ===> Checking files for tcl-8.5.16p0 > > >> Fetch > > >> http://downloads.sourceforge.net/sourceforge/tcl/tcl8.5.16-src.tar.gz > > tcl8.5.16-src.tar.gz 60% |*| 2696 KB > 00:00 > > >> Size does not match for tcl8.5.16-src.tar.gz > > >> Fetch > http://ftp.openbsd.org/pub/OpenBSD/distfiles//tcl8.5.16-src.tar.gz > > tcl8.5.16-src.tar.gz 23% |** | 1024 KB > 00:03 ETA > > The problem lies in ftp(1). > > Logic in the ports tree is fine. But there's nothing it can do there: > somehow > your ftp returns 0 (e.g., success), so the partial file gets removed. > > If you want to get it fixed, you may have to provide more input, as we > obviously do not see that problem... First thing would be to override > FETCH_CMD to remove the -V, so that you can show us what ftp says about > things. Tracing the code thru the program would help.
Re: Robustness in ports fetch program?
On Sat, May 16, 2015 at 10:31:24PM -0400, Alan Corey wrote: > I'd seen this happen in 5.6 too, but I just caught an example of it in > 5.7. My connection leaves a lot to be desired, but there's nothing I > can do about that. I normally have FETCH_CMD set to use wget once I > get it installed but this was in doing a standard make install of a > port. > > The first time the connection gets interrupted, but something thinks > it should be done and checks the size. That's wrong so it downloads > it over again instead of just resuming the download. It should only > download it over again if the size matches but the CRC is wrong. > Seems like anyway. > > ===> Verifying install for tcl-8.5.16 in lang/tcl/8.5 > ===> Checking files for tcl-8.5.16p0 > >> Fetch > >> http://downloads.sourceforge.net/sourceforge/tcl/tcl8.5.16-src.tar.gz > tcl8.5.16-src.tar.gz 60% |*| 2696 KB00:00 > >> Size does not match for tcl8.5.16-src.tar.gz > >> Fetch http://ftp.openbsd.org/pub/OpenBSD/distfiles//tcl8.5.16-src.tar.gz > tcl8.5.16-src.tar.gz 23% |** | 1024 KB00:03 > ETA The problem lies in ftp(1). Logic in the ports tree is fine. But there's nothing it can do there: somehow your ftp returns 0 (e.g., success), so the partial file gets removed. If you want to get it fixed, you may have to provide more input, as we obviously do not see that problem... First thing would be to override FETCH_CMD to remove the -V, so that you can show us what ftp says about things. Tracing the code thru the program would help.
Robustness in ports fetch program?
I'd seen this happen in 5.6 too, but I just caught an example of it in 5.7. My connection leaves a lot to be desired, but there's nothing I can do about that. I normally have FETCH_CMD set to use wget once I get it installed but this was in doing a standard make install of a port. The first time the connection gets interrupted, but something thinks it should be done and checks the size. That's wrong so it downloads it over again instead of just resuming the download. It should only download it over again if the size matches but the CRC is wrong. Seems like anyway. ===> Verifying install for tcl-8.5.16 in lang/tcl/8.5 ===> Checking files for tcl-8.5.16p0 >> Fetch >> http://downloads.sourceforge.net/sourceforge/tcl/tcl8.5.16-src.tar.gz tcl8.5.16-src.tar.gz 60% |*| 2696 KB00:00 >> Size does not match for tcl8.5.16-src.tar.gz >> Fetch http://ftp.openbsd.org/pub/OpenBSD/distfiles//tcl8.5.16-src.tar.gz tcl8.5.16-src.tar.gz 23% |** | 1024 KB00:03 ETA -- Credit is the root of all evil. - AB1JX
