Re: format of i386/index.txt
http://marc.info/?l=openbsd-miscm=126678113118214w=2 Has the format of ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/index.txt changed again? It seems to be 'ls -l' now.
Re: format of i386/index.txt
On Wed, 17 Mar 2010 11:34:16 +0100 Jan Stary h...@stare.cz wrote: http://marc.info/?l=openbsd-miscm=126678113118214w=2 Has the format of ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/index.txt changed again? It seems to be 'ls -l' now. Hi Jan, I think this is the second time I've seen you mention the format of the index.txt file... so it seems you're mistakenly trying to parse index.txt to get the file/path names of the stuff you need to download. (pardon me if my mind reading skills are slightly off) --- #!/bin/ksh ftp_host=ftp.openbsd.org basepath=pub/OpenBSD/snapshots arch=`uname -m` ftp -i -n EOF ftp.log open $ftp_host user anonymous none@ nlist $basepath/$arch ftp.files EOF # exclude floppy*.fs and *.iso for FNAME in `grep -v -e \.iso -e \.fs ftp.files | sed -e s:$basepath/$arch/::`; do if [[ -z $FLIST ]]; then FLIST=$FNAME; else FLIST=$FLIST FNAME fi done --- NOTE the for ... is wrapped, The result is $FLIST contains just a list of all the file names of the stuff on the ftp server in the given directory excluding the *.fs and *.iso files. The real magic in the above is in the nlist command of ftp, and you don't necessarily need to do it in a shell script. perl would work equally well. If my mind reading skills are off, and you're trying to check the time stamp of files on the ftp server, then check out the ftp modtime command, and point it at the SHA256 file (that should be there). In other words, the format of index.txt should not matter since there are better ways to get the information you want. jcr
Re: format of i386/index.txt
On Mar 17 05:16:32, J.C. Roberts wrote:
On Wed, 17 Mar 2010 11:34:16 +0100 Jan Stary h...@stare.cz wrote:
http://marc.info/?l=openbsd-miscm=126678113118214w=2
Has the format of
ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/index.txt
changed again? It seems to be 'ls -l' now.
Hi Jan,
I think this is the second time I've seen you mention the format of the
index.txt file... so it seems you're mistakenly trying to parse
index.txt to get the file/path names of the stuff you need to download.
(pardon me if my mind reading skills are slightly off)
Indeed, that's what I've been using index.txt for.
(Sorry for not enough mind writing.)
Isn't that why index.txt is there?
#!/bin/ksh
ftp_host=ftp.openbsd.org
basepath=pub/OpenBSD/snapshots
arch=`uname -m`
ftp -i -n EOF ftp.log
open $ftp_host
user anonymous none@
nlist $basepath/$arch ftp.files
EOF
# exclude floppy*.fs and *.iso
for FNAME in `grep -v -e \.iso -e \.fs ftp.files |
sed -e s:$basepath/$arch/::`; do
if [[ -z $FLIST ]]; then
FLIST=$FNAME;
else
FLIST=$FLIST FNAME
fi
done
---
NOTE the for ... is wrapped,
The result is $FLIST contains just a list of all the file names of the
stuff on the ftp server in the given directory excluding the *.fs and
*.iso files.
Very similar to my script here, except I get my list from index.txt
#!/bin/sh
error() {
echo $@ 2
}
fatal() {
error $@
exit 1
}
usage() {
error usage: ${0##*/} release destination [master]
error as in: ${0##*/} '`uname -r` ~/WWW ftp://openbsd.ftp.fu-berlin.de'
error as in: ${0##*/} snapshots /install ftp://ftp.openbsd.org
error as in: ${0##*/} snapshots /install
exit 1
}
[ $# -ge 2 ] || usage
DEST=$2/pub/OpenBSD/$1/`uname -m`
mkdir -p $DEST || fatal cannot create $DEST
cd $DEST || fatal cannot cd to $DEST
SITE=${3:-ftp://ftp.openbsd.org}
SITE=$SITE/pub/OpenBSD/$1/`uname -m`
ftp -a -V $SITE/index.txt || fatal cannot fetch index.txt
cat index.txt | sed s,^,$SITE/, | xargs ftp -a -k30 -V
cksum -c SHA256
The real magic in the above is in the nlist command of ftp, and you
don't necessarily need to do it in a shell script. perl would work
equally well.
(Aaargh, I use a shell script whenever I don't necessarily need to
use perl, thank you.)
If my mind reading skills are off, and you're trying to check the time
stamp of files on the ftp server, then check out the ftp modtime
command, and point it at the SHA256 file (that should be there).
In other words, the format of index.txt should not matter since there
are better ways to get the information you want.
It's been a long time I have read ftp(1), and apparently
I forgot about 'nlist' - thanks.
Anyway, what really is the purpose of index.txt being there then?
To tell the times and sizes?
Jan
Re: format of i386/index.txt
On Wed, 17 Mar 2010 15:02:19 +0100 Jan Stary h...@stare.cz wrote: Anyway, what really is the purpose of index.txt being there then? To tell the times and sizes? To break scripts? ;) To put it bluntly, index.txt seems pointless, or more likely, there is some super double secret reason for it to still exist that I simply don't know... My only *GUESS* is, some mirrors are HTTP, but due to brainless accountants mindlessly running security auditing tools, they forbid real directory listings, and are configured to only return an existing /index.* file to the useragent. Hopefully, someone who actually has a clue (not me) will chime in with the real reason why index.txt exists. jcr
Re: format of i386/index.txt
On Wed, Mar 17, 2010 at 9:44 AM, J.C. Roberts list-...@designtools.org wrote: On Wed, 17 Mar 2010 15:02:19 +0100 Jan Stary h...@stare.cz wrote: Anyway, what really is the purpose of index.txt being there then? To tell the times and sizes? To break scripts? ;) To put it bluntly, index.txt seems pointless, or more likely, there is some super double secret reason for it to still exist that I simply don't know... My only *GUESS* is, some mirrors are HTTP, but due to brainless accountants mindlessly running security auditing tools, they forbid real directory listings, and are configured to only return an existing /index.* file to the useragent. Hopefully, someone who actually has a clue (not me) will chime in with the real reason why index.txt exists. jcr Actually the installer uses it to make a list of file sets to present to the user. If it isn't there then no sets are presented. From src/distrib/miniroot/install.sub: # Get list of files from the server. if [[ $_url_type == ftp -z $ftp_proxy ]] ; then _file_list=$(ftp $FTPOPTS $_url_base/) ftp_error Login failed. $_file_list return ftp_error No such file or directory. $_file_list return else # Assumes index file is index.txt for http (or proxy) # We can't use index.html since the format is server-dependent _file_list=$(ftp $FTPOPTS -o - $_url_base/index.txt | \ sed -e 's/^.* //' | sed -e 's/ //') fi -N
Re: format of i386/index.txt
On Wed, 17 Mar 2010 09:44:50 -0700 J.C. Roberts list-...@designtools.org wrote: On Wed, 17 Mar 2010 15:02:19 +0100 Jan Stary h...@stare.cz wrote: Anyway, what really is the purpose of index.txt being there then? To tell the times and sizes? To break scripts? ;) To put it bluntly, index.txt seems pointless, or more likely, there is some super double secret reason for it to still exist that I simply don't know... My only *GUESS* is, some mirrors are HTTP, but due to brainless accountants mindlessly running security auditing tools, they forbid real directory listings, and are configured to only return an existing /index.* file to the useragent. Hopefully, someone who actually has a clue (not me) will chime in with the real reason why index.txt exists. jcr afaik you guessed right. It is used by install.sub to get a list of the files, because of funny http servers.
Re: format of i386/index.txt
On Mar 17 10:14:36, Nick Bender wrote: On Wed, Mar 17, 2010 at 9:44 AM, J.C. Roberts list-...@designtools.org wrote: On Wed, 17 Mar 2010 15:02:19 +0100 Jan Stary h...@stare.cz wrote: Anyway, what really is the purpose of index.txt being there then? To tell the times and sizes? To break scripts? ;) To put it bluntly, index.txt seems pointless, or more likely, there is some super double secret reason for it to still exist that I simply don't know... My only *GUESS* is, some mirrors are HTTP, but due to brainless accountants mindlessly running security auditing tools, they forbid real directory listings, and are configured to only return an existing /index.* file to the useragent. Hopefully, someone who actually has a clue (not me) will chime in with the real reason why index.txt exists. jcr Actually the installer uses it to make a list of file sets to present to the user. If it isn't there then no sets are presented. From src/distrib/miniroot/install.sub: # Get list of files from the server. if [[ $_url_type == ftp -z $ftp_proxy ]] ; then _file_list=$(ftp $FTPOPTS $_url_base/) ftp_error Login failed. $_file_list return ftp_error No such file or directory. $_file_list return else # Assumes index file is index.txt for http (or proxy) # We can't use index.html since the format is server-dependent _file_list=$(ftp $FTPOPTS -o - $_url_base/index.txt | \ sed -e 's/^.* //' | sed -e 's/ //') fi In fact, the above just gets the content of index.txt and applies the 's/^.* //' smartness, thus eliminating exactly the difference between 'ls' and 'ls -l'. The installer does further work with this list: # Initialize _sets to the list of sets found in _src, and initialize # _get_sets to the intersection of _sets and DEFAULTSETS. (Indeed, I have never seen the installer present me with 'install.iso' or 'index.txt', which _are_ listed in index.txt too.) It still looks like index.txt is just a list of files that are there. Is there any reason to have this information in the 'ls' or 'ls -l' specifically? (It has changed back and forth in the last month.)
Re: format of i386/index.txt
On 2010-03-17, Jan Stary h...@stare.cz wrote: (It has changed back and forth in the last month.) I tried using index.txt files for timestamps to monitor the latency of mirror updates, but had problems with some of them flipping between formats; I have a suspicion that some site or other regenerates index.txt files but I haven't been able to track it down yet...
Re: format of i386/index.txt
On Wed, Mar 17, 2010 at 10:55 PM, Stuart Henderson s...@spacehopper.org wrote: On 2010-03-17, Jan Stary h...@stare.cz wrote: (It has changed back and forth in the last month.) I tried using index.txt files for timestamps to monitor the latency of mirror updates, but had problems with some of them flipping between formats; I have a suspicion that some site or other regenerates index.txt files but I haven't been able to track it down yet... That's why i used the timestamp method (mdtm) in Net::FTP for http://gruiik.info/up2date.html.. instead of parsing unknown-formatted files :) Landry
Re: format of i386/index.txt
On Mar 17 21:55:33, Stuart Henderson wrote: On 2010-03-17, Jan Stary h...@stare.cz wrote: (It has changed back and forth in the last month.) I tried using index.txt files for timestamps to monitor the latency of mirror updates, but had problems with some of them flipping between formats; I have a suspicion that some site or other regenerates index.txt files but I haven't been able to track it down yet... I suspected a delayed mirror first, but this is actually ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/index.txt And it's back to 'ls' right now. Huh.
format of i386/index.txt
It seems the format of ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/index.txt has changed again from 'ls -l' to plain 'ls'. Should scripts expect this format from now on?
Re: format of i386/index.txt
It seems the format of ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/index.txt has changed again from 'ls -l' to plain 'ls'. Should scripts expect this format from now on? Yes, it is required by the installer.
