Re: if I were to make a pkg-add diff
>> So the program basically makes several network connections to >> potentially some 120 servers all across the world and the "winner" is >> calculated based on the "speed" it took downloading a 1.9K text file >> from each of them? > > Which isn't even a big enough transfer to get TCP out of slow start. SHA256 is over 600 KB. -Luke On Wed, Jan 20, 2016 at 1:14 AM, Luke Small wrote: > not knowing better... > > I always wanted to know the fastest mirrors for me, and at times it > changes some from the testing I've done so far. I live in Missouri, USA and > it changes from a couple mirrors from an adjoining state Illinois and one a > few states away in Texas. Maybe if there were a smaller file than the 600 > KB SHA256 file that doesn't change its filename between releases it might > be better, but you also get a more accurate reading on dropped packages and > such that the ftp program has to deal with. with /etc/pkg.conf, you can > actually specify several mirrors: > > installpath = ... > installpath += ... > > I'm not sure if that downloads from multiple mirrors at a time or if there > is failover. I think that it is pretty darn cool to automate the process > though! > > > > -Luke > > On Tue, Jan 19, 2016 at 2:57 AM, Erling Westenvik < > erling.westen...@gmail.com> wrote: > >> On Tue, Jan 19, 2016 at 01:26:15AM -0600, Luke Small wrote: >> > I made a small 500 line program I call pkg_ping that calls uname -rm, >> > ftp, sed, on openbsd.org/ftp.html. >> >> A "program"? In what language? Is your code available somewhere? >> >> > then it changes all the parsed http and ftp mirrors into http and ftp >> > downloads and changes them to non redundant http mirrors (it has to to >> > easily call ftp on it). It takes them and downloads SHA256 from the >> > mirrors and the parent times how long it takes. If it takes too long >> > it kills the ftp call and goes on to the next one. Then it sorts the >> > results and puts the winner in /etc/pkg.conf >> >> So the program basically makes several network connections to >> potentially some 120 servers all across the world and the "winner" is >> calculated based on the "speed" it took downloading a 1.9K text file >> from each of them? >> Not taking into account the number of hops, nor the anti-social >> behaviour of starting to download large install sets of files from the >> other side of the planet when a more nearby but a little slower mirror >> is available? >> >> > replacing all installpath instances, while leaving everything else. It >> > doesn't do any network stuff directly so it probably wouldn't be much >> > of a security problem, but because it needs to alter a root owned >> > file, it should need root privileges. >> >> It doesn't need to mess with /etc/pkg.conf. Setting PKG_PATH will >> override installpath. See pkg.conf(5). >> >> > I don't only run the release, so I don't even really know how to >> > pledge it. Is this something somebody would be willing to submit to >> > the project and maybe alter it slightly. >> >> I'm no dev and can't speak on the behalf of the project. Back when I was >> a complete novice I might have used a tool like this on some occasions - >> not knowing better... >> As part of the base system or the installer? Hardly... >> As a package? Maybe, but you would probably have to make it all yourself. >> >> Regards >> >> Erling
Re: if I were to make a pkg-add diff
On 2016-01-20, Michael Lambert wrote: >> On 19 Jan 2016, at 03:57, Erling Westenvik > wrote: >> >>> On Tue, Jan 19, 2016 at 01:26:15AM -0600, Luke Small wrote: >>> then it changes all the parsed http and ftp mirrors into http and ftp >>> downloads and changes them to non redundant http mirrors (it has to to >>> easily call ftp on it). It takes them and downloads SHA256 from the >>> mirrors and the parent times how long it takes. If it takes too long >>> it kills the ftp call and goes on to the next one. Then it sorts the >>> results and puts the winner in /etc/pkg.conf >> >> So the program basically makes several network connections to >> potentially some 120 servers all across the world and the "winner" is >> calculated based on the "speed" it took downloading a 1.9K text file >> from each of them? > > Which isn't even a big enough transfer to get TCP out of slow start. Talking of this, you'll probably get a better speed improvement by switching to HTTP/1.1 with keepalives and picking *any* mirror on the same continent, than fine-tuning from the current list... That is probably more interesting to implement too :-)
Re: if I were to make a pkg-add diff
> On 19 Jan 2016, at 03:57, Erling Westenvik wrote: > >> On Tue, Jan 19, 2016 at 01:26:15AM -0600, Luke Small wrote: >> then it changes all the parsed http and ftp mirrors into http and ftp >> downloads and changes them to non redundant http mirrors (it has to to >> easily call ftp on it). It takes them and downloads SHA256 from the >> mirrors and the parent times how long it takes. If it takes too long >> it kills the ftp call and goes on to the next one. Then it sorts the >> results and puts the winner in /etc/pkg.conf > > So the program basically makes several network connections to > potentially some 120 servers all across the world and the "winner" is > calculated based on the "speed" it took downloading a 1.9K text file > from each of them? Which isn't even a big enough transfer to get TCP out of slow start. Michael
Re: if I were to make a pkg-add diff
On 2016-01-20, Luke Small wrote: > with /etc/pkg.conf, you can > actually specify several mirrors: > > installpath = ... > installpath += ... > > I'm not sure if that downloads from multiple mirrors at a time or if there > is failover. Failover. It's mostly used for pointing at a local server or something like mtier's stable package repository first, and falling back to a standard mirror for things which the preferred one doesn't have. With snapshots, you can't rely on all mirrors having the same versions of files, even if the filename is the same, so you can't mix and match between them and downloading from multiple mirrors at a time wouldn't work. This is why we can't use a traditional location-aware dns load balancer setup for this, because it might change the mirror part-way through a pkg_add run,
Re: if I were to make a pkg-add diff
wrote: > On Tue, Jan 19, 2016 at 01:26:15AM -0600, Luke Small wrote: > > I made a small 500 line program I call pkg_ping that calls uname -rm, > > ftp, sed, on openbsd.org/ftp.html. > > A "program"? In what language? Is your code available somewhere? > > > then it changes all the parsed http and ftp mirrors into http and ftp > > downloads and changes them to non redundant http mirrors (it has to to > > easily call ftp on it). It takes them and downloads SHA256 from the > > mirrors and the parent times how long it takes. If it takes too long > > it kills the ftp call and goes on to the next one. Then it sorts the > > results and puts the winner in /etc/pkg.conf > > So the program basically makes several network connections to > potentially some 120 servers all across the world and the "winner" is > calculated based on the "speed" it took downloading a 1.9K text file > from each of them? > Not taking into account the number of hops, nor the anti-social > behaviour of starting to download large install sets of files from the > other side of the planet when a more nearby but a little slower mirror > is available? > > > replacing all installpath instances, while leaving everything else. It > > doesn't do any network stuff directly so it probably wouldn't be much > > of a security problem, but because it needs to alter a root owned > > file, it should need root privileges. > > It doesn't need to mess with /etc/pkg.conf. Setting PKG_PATH will > override installpath. See pkg.conf(5). > > > I don't only run the release, so I don't even really know how to > > pledge it. Is this something somebody would be willing to submit to > > the project and maybe alter it slightly. > > I'm no dev and can't speak on the behalf of the project. Back when I was > a complete novice I might have used a tool like this on some occasions - > not knowing better... > As part of the base system or the installer? Hardly... > As a package? Maybe, but you would probably have to make it all yourself. > > Regards > > Erling
Re: if I were to make a pkg-add diff
Go to: *I have a mirror testing program for you.* in the tech mailing list. It copied there. -Luke On Tue, Jan 19, 2016 at 11:18 PM, Luke Small wrote: > here you go! Enjoy! > > -Luke > > On Tue, Jan 19, 2016 at 2:57 AM, Erling Westenvik < > erling.westen...@gmail.com> wrote: > >> On Tue, Jan 19, 2016 at 01:26:15AM -0600, Luke Small wrote: >> > I made a small 500 line program I call pkg_ping that calls uname -rm, >> > ftp, sed, on openbsd.org/ftp.html. >> >> A "program"? In what language? Is your code available somewhere? >> >> > then it changes all the parsed http and ftp mirrors into http and ftp >> > downloads and changes them to non redundant http mirrors (it has to to >> > easily call ftp on it). It takes them and downloads SHA256 from the >> > mirrors and the parent times how long it takes. If it takes too long >> > it kills the ftp call and goes on to the next one. Then it sorts the >> > results and puts the winner in /etc/pkg.conf >> >> So the program basically makes several network connections to >> potentially some 120 servers all across the world and the "winner" is >> calculated based on the "speed" it took downloading a 1.9K text file >> from each of them? >> Not taking into account the number of hops, nor the anti-social >> behaviour of starting to download large install sets of files from the >> other side of the planet when a more nearby but a little slower mirror >> is available? >> >> > replacing all installpath instances, while leaving everything else. It >> > doesn't do any network stuff directly so it probably wouldn't be much >> > of a security problem, but because it needs to alter a root owned >> > file, it should need root privileges. >> >> It doesn't need to mess with /etc/pkg.conf. Setting PKG_PATH will >> override installpath. See pkg.conf(5). >> >> > I don't only run the release, so I don't even really know how to >> > pledge it. Is this something somebody would be willing to submit to >> > the project and maybe alter it slightly. >> >> I'm no dev and can't speak on the behalf of the project. Back when I was >> a complete novice I might have used a tool like this on some occasions - >> not knowing better... >> As part of the base system or the installer? Hardly... >> As a package? Maybe, but you would probably have to make it all yourself. >> >> Regards >> >> Erling
Re: if I were to make a pkg-add diff
> here you go! Enjoy! > > -Luke > > On Tue, Jan 19, 2016 at 2:57 AM, Erling Westenvik < > erling.westen...@gmail.com> wrote: [ ... ] > > [demime 1.01d removed an attachment of type text/x-csrc which had a > name of pkg_ping.c] Luke RTFM. http://www.openbsd.org/mail.html *The only mailing list that allows attachments is the ports list, they will be removed from messages on the other mailing lists.*
Re: if I were to make a pkg-add diff
here you go! Enjoy! -Luke On Tue, Jan 19, 2016 at 2:57 AM, Erling Westenvik < erling.westen...@gmail.com> wrote: > On Tue, Jan 19, 2016 at 01:26:15AM -0600, Luke Small wrote: > > I made a small 500 line program I call pkg_ping that calls uname -rm, > > ftp, sed, on openbsd.org/ftp.html. > > A "program"? In what language? Is your code available somewhere? > > > then it changes all the parsed http and ftp mirrors into http and ftp > > downloads and changes them to non redundant http mirrors (it has to to > > easily call ftp on it). It takes them and downloads SHA256 from the > > mirrors and the parent times how long it takes. If it takes too long > > it kills the ftp call and goes on to the next one. Then it sorts the > > results and puts the winner in /etc/pkg.conf > > So the program basically makes several network connections to > potentially some 120 servers all across the world and the "winner" is > calculated based on the "speed" it took downloading a 1.9K text file > from each of them? > Not taking into account the number of hops, nor the anti-social > behaviour of starting to download large install sets of files from the > other side of the planet when a more nearby but a little slower mirror > is available? > > > replacing all installpath instances, while leaving everything else. It > > doesn't do any network stuff directly so it probably wouldn't be much > > of a security problem, but because it needs to alter a root owned > > file, it should need root privileges. > > It doesn't need to mess with /etc/pkg.conf. Setting PKG_PATH will > override installpath. See pkg.conf(5). > > > I don't only run the release, so I don't even really know how to > > pledge it. Is this something somebody would be willing to submit to > > the project and maybe alter it slightly. > > I'm no dev and can't speak on the behalf of the project. Back when I was > a complete novice I might have used a tool like this on some occasions - > not knowing better... > As part of the base system or the installer? Hardly... > As a package? Maybe, but you would probably have to make it all yourself. > > Regards > > Erling [demime 1.01d removed an attachment of type text/x-csrc which had a name of pkg_ping.c]
Re: if I were to make a pkg-add diff
On 2016-01-19, Eric Furman wrote: > FOR GOD'S SAKE STOP TALKING. > Submit your code to tech@ and it will be considered on its merits. Code without listening to the feedback which has already been given based on the design isn't very useful either. Whether it's good code or not, the described behaviour is abusive to mirrors. If you are in e.g. Europe or N. America there is simply no need to download from a mirror in Japan, Indonesia, the Philippines, Costa Rica, etc., even to test performance.
Re: if I were to make a pkg-add diff
On Tue, Jan 19, 2016 at 01:26:15AM -0600, Luke Small wrote: > I made a small 500 line program I call pkg_ping that calls uname -rm, > ftp, sed, on openbsd.org/ftp.html. A "program"? In what language? Is your code available somewhere? > then it changes all the parsed http and ftp mirrors into http and ftp > downloads and changes them to non redundant http mirrors (it has to to > easily call ftp on it). It takes them and downloads SHA256 from the > mirrors and the parent times how long it takes. If it takes too long > it kills the ftp call and goes on to the next one. Then it sorts the > results and puts the winner in /etc/pkg.conf So the program basically makes several network connections to potentially some 120 servers all across the world and the "winner" is calculated based on the "speed" it took downloading a 1.9K text file from each of them? Not taking into account the number of hops, nor the anti-social behaviour of starting to download large install sets of files from the other side of the planet when a more nearby but a little slower mirror is available? > replacing all installpath instances, while leaving everything else. It > doesn't do any network stuff directly so it probably wouldn't be much > of a security problem, but because it needs to alter a root owned > file, it should need root privileges. It doesn't need to mess with /etc/pkg.conf. Setting PKG_PATH will override installpath. See pkg.conf(5). > I don't only run the release, so I don't even really know how to > pledge it. Is this something somebody would be willing to submit to > the project and maybe alter it slightly. I'm no dev and can't speak on the behalf of the project. Back when I was a complete novice I might have used a tool like this on some occasions - not knowing better... As part of the base system or the installer? Hardly... As a package? Maybe, but you would probably have to make it all yourself. Regards Erling
Re: if I were to make a pkg-add diff
FOR GOD'S SAKE STOP TALKING. Submit your code to tech@ and it will be considered on its merits. The OpenBSD team is very open to good code. If it's shite you will get no response. If it has potential you will get comments. But NOTHING will happen until they see CODE. On Tue, Jan 19, 2016, at 02:26 AM, Luke Small wrote: > I made a small 500 line program I call pkg_ping that calls uname -rm, > ftp, > sed, on openbsd.org/ftp.html. then it changes all the parsed http and ftp > mirrors into http and ftp downloads and changes them to non redundant > http > mirrors (it has to to easily call ftp on it). It takes them and downloads > SHA256 from the mirrors and the parent times how long it takes. If it > takes > too long it kills the ftp call and goes on to the next one. Then it sorts > the results and puts the winner in /etc/pkg.conf replacing all > installpath > instances, while leaving everything else. It doesn't do any network stuff > directly so it probably wouldn't be much of a security problem, but > because > it needs to alter a root owned file, it should need root privileges. I > don't only run the release, so I don't even really know how to pledge it. > Is this something somebody would be willing to submit to the project and > maybe alter it slightly.
Re: if I were to make a pkg-add diff
I made a small 500 line program I call pkg_ping that calls uname -rm, ftp, sed, on openbsd.org/ftp.html. then it changes all the parsed http and ftp mirrors into http and ftp downloads and changes them to non redundant http mirrors (it has to to easily call ftp on it). It takes them and downloads SHA256 from the mirrors and the parent times how long it takes. If it takes too long it kills the ftp call and goes on to the next one. Then it sorts the results and puts the winner in /etc/pkg.conf replacing all installpath instances, while leaving everything else. It doesn't do any network stuff directly so it probably wouldn't be much of a security problem, but because it needs to alter a root owned file, it should need root privileges. I don't only run the release, so I don't even really know how to pledge it. Is this something somebody would be willing to submit to the project and maybe alter it slightly.
Re: if I were to make a pkg-add diff
Miniroot isn't available after install is it? I suspect mirrors choose to change mid-cycle too and should a user have to put in the install disk to find a more convenient mirror selection method? I want to be able to offer statistics about the mirrors maybe piped to 'more', so the user can choose from a small subset of the total choices. Maybe the closest local mirror isn't the best choice. And do it in a way that doesn't introduce new unaudited networking code, beyond inter-thread plaintext character pipes. On Jan 4, 2016 04:11, "Peter Hessler" wrote: > All of the functionality you are requesting is already provided. > > look at finish_up() in src/distrib/miniroot/install.sub. > > There is no reason at all to modify pkg_add. Just setup /etc/pkg.conf. > > > On 2016 Jan 04 (Mon) at 04:02:07 -0600 (-0600), Luke Small wrote: > :I am realistically thinking more along the lines of less than once a > :release cycle. More like whenever it comes upon a user that their mirror > of > :choice chooses to no longer be a mirror. I had that happen to me. It would > :be convenient to have a program that can easily compare mirror latencies > :and download speeds. I was hoping that maybe pkg_add could be modified to > :have a timeout and accept input from the program I am making if that > :occurs, or there is no mirror setup for the machine. If it is implemented, > :more generous bandwidth and lesser burdened mirrors would have more > :equitably distributed traffic. > : > :> What you meant was thousands of users sending handful of pings across > the > :world to a lot of the mirrors each time they (re)restart pkg_add? > : > > -- > You can't underestimate the power of fear. > -- Tricia Nixon
Re: if I were to make a pkg-add diff
On 2016-01-04, Luke Small wrote: > What I meant is, if a program sends a handful of pings to each mirror, > would it think it is being spammed and shutdown any further connections. I > didn't mean to say that I want to connect the pkg_ping program to a of > anchor. I tried an initial localhost pinging, pkg_ping program in > virtualbox using 32 MB RAM and it worked; while it froze trying to open > pico in the console. It wouldn't need to use much resources and would use > kqueue, which isn't available in any other language other than C, in the > base as far as I'm aware. I use kqueue to easily store a custom struct with > the pipe() values, list pointers and such in *udata. It intermittently > reads from pipes from all the spawned ping threads to conserve memory, > reads the individual lantency output and stores the ping time value in a > single linked list to be transferred to an array to be sorted by qsort so > that the median value is determined by its position in the sorted array. It > actually doesn't directly call any sockets or anything that hasn't been > audited in the base it just reads individual characters from pipes until > EOF so far. ping doesn't tell you which is the best mirror. Some mirrors are on OS which rate-limit icmp by default. Some might prioritise pings over real traffic, or be overloaded and dropping packets rather than having high pings (which you can't distinguish from the "rate-limited icmp" case). And, unless you do some pre-selection (from a geolocation database or similar) to cut the list down, you're going to generate useless traffic to mirrors around the world, some of which have good regional bandwidth but very limited international bandwidth and don't really want unrelated ping traffic from someone who is never going to use that mirror anyway..
Re: if I were to make a pkg-add diff
All of the functionality you are requesting is already provided. look at finish_up() in src/distrib/miniroot/install.sub. There is no reason at all to modify pkg_add. Just setup /etc/pkg.conf. On 2016 Jan 04 (Mon) at 04:02:07 -0600 (-0600), Luke Small wrote: :I am realistically thinking more along the lines of less than once a :release cycle. More like whenever it comes upon a user that their mirror of :choice chooses to no longer be a mirror. I had that happen to me. It would :be convenient to have a program that can easily compare mirror latencies :and download speeds. I was hoping that maybe pkg_add could be modified to :have a timeout and accept input from the program I am making if that :occurs, or there is no mirror setup for the machine. If it is implemented, :more generous bandwidth and lesser burdened mirrors would have more :equitably distributed traffic. : :> What you meant was thousands of users sending handful of pings across the :world to a lot of the mirrors each time they (re)restart pkg_add? : -- You can't underestimate the power of fear. -- Tricia Nixon
Re: if I were to make a pkg-add diff
I am realistically thinking more along the lines of less than once a release cycle. More like whenever it comes upon a user that their mirror of choice chooses to no longer be a mirror. I had that happen to me. It would be convenient to have a program that can easily compare mirror latencies and download speeds. I was hoping that maybe pkg_add could be modified to have a timeout and accept input from the program I am making if that occurs, or there is no mirror setup for the machine. If it is implemented, more generous bandwidth and lesser burdened mirrors would have more equitably distributed traffic. > What you meant was thousands of users sending handful of pings across the world to a lot of the mirrors each time they (re)restart pkg_add?
Re: if I were to make a pkg-add diff
On Mon, 04 Jan 2016, Janne Johansson wrote: > What you meant was thousands of users sending handful of pings across > the world to a lot of the mirrors each time they (re)restart pkg_add? http://packages.debian.org/unstable/net/netselect-apt http://http.debian.net/
Re: if I were to make a pkg-add diff
2016-01-04 4:22 GMT+01:00 Luke Small : > What I meant is, if a program sends a handful of pings to each mirror, > would it think it is being spammed and shutdown any further connections. > > What you meant was thousands of users sending handful of pings across the world to a lot of the mirrors each time they (re)restart pkg_add? -- May the most significant bit of your life be positive.
Re: if I were to make a pkg-add diff
What I meant is, if a program sends a handful of pings to each mirror, would it think it is being spammed and shutdown any further connections. I didn't mean to say that I want to connect the pkg_ping program to a of anchor. I tried an initial localhost pinging, pkg_ping program in virtualbox using 32 MB RAM and it worked; while it froze trying to open pico in the console. It wouldn't need to use much resources and would use kqueue, which isn't available in any other language other than C, in the base as far as I'm aware. I use kqueue to easily store a custom struct with the pipe() values, list pointers and such in *udata. It intermittently reads from pipes from all the spawned ping threads to conserve memory, reads the individual lantency output and stores the ping time value in a single linked list to be transferred to an array to be sorted by qsort so that the median value is determined by its position in the sorted array. It actually doesn't directly call any sockets or anything that hasn't been audited in the base it just reads individual characters from pipes until EOF so far.
Re: if I were to make a pkg-add diff
I'll say we should go for the spaceship project ... with lasers.
Re: if I were to make a pkg-add diff
>I'll say we should go for the spaceship project ... with lasers. Index: pkg_add.1 === RCS file: /cvs/src/usr.sbin/pkg_add/pkg_add.1,v retrieving revision 1.134 diff -u -p -u -r1.134 pkg_add.1 --- pkg_add.1 4 Nov 2015 16:59:58 - 1.134 +++ pkg_add.1 4 Jan 2016 01:36:44 - @@ -20,7 +20,7 @@ .Os .Sh NAME .Nm pkg_add -.Nd install software package distributions +.Nd install software package distributions with lasers .Sh SYNOPSIS .Nm pkg_add .Bk -words pkg_add is easy. The spaceship is harder.
Re: if I were to make a pkg-add diff
I just got consistent internet for my laptop and got the ftp and sed call pipes and stuff to work. What I want to know is if I make multiple fork() to execl() ping calls, should I limit the number of a standard argument-free call to my pkg_ping to 8 or so threads which pings each of the mirrors 9 or so times(and calculate the median latency) at 1 ping per second, or will the mirrors have ping firewall rules that relegate such a machine to a spam anchor in the of rules? I was thinking about having ftp download a small package to determine download speeds, but I wonder whether if I did that on a specific file, whether the results may be skewed by inconsistent squid or similar program caching often downloaded files on mirrors. On Dec 27, 2015 18:17, "Luke Small" wrote: > Even though I don't have an internet connection for my laptop I > started the C program that pipes an execl call from ftp, to sed, (like > the suggestions > offered earlier in the thread, and back to the parent and it will use > kqueue to test the pipe buffer capacities to a local buffer (I love > kqueue) It could probably use the existing ftp() and maybe ping() > to deal with all the network protocols, interfaces and ports. I don't > want to sound like I'm bragging. I just really appreciate the help. > > -Luke
Re: if I were to make a pkg-add diff
Even though I don't have an internet connection for my laptop I started the C program that pipes an execl call from ftp, to sed, (like the suggestions offered earlier in the thread, and back to the parent and it will use kqueue to test the pipe buffer capacities to a local buffer (I love kqueue) It could probably use the existing ftp() and maybe ping() to deal with all the network protocols, interfaces and ports. I don't want to sound like I'm bragging. I just really appreciate the help. -Luke
Re: if I were to make a pkg-add diff
On Sun, Dec 27, 2015 at 02:52:16PM -0600, Luke Small wrote: > You could do that if you want to have noobs connect to one of the mirrors > into perpituty that brings down the server like a ddos every release! Are you aware of the magic that ftplist.cgi does? Stuart is. /Alexander > > I think the best that can be done relatively easily would be to have > > pkg_add fetch ftplist.cgi and pick the first result as a default if > > pkg.conf nor PKG_PATH are set.
Re: if I were to make a pkg-add diff
> I guess I didn't really answer your question. It wouldn't rely upon > the ramdisk. It is meant to run after install. So it would presumably > have all the firmware. Then I think, you have to be more clear. Because it is not to modify the installer and also not, to modify pkg_add - as you wrote before. If you like to check the fastest mirror you also have to keep in mind things like, is the interface up, geolocation, is Tor or a VPN used and so on ... P.S. What is then with the main idea for new users ?
Re: if I were to make a pkg-add diff
You could do that if you want to have noobs connect to one of the mirrors into perpituty that brings down the server like a ddos every release! > I think the best that can be done relatively easily would be to have >pkg_add fetch ftplist.cgi and pick the first result as a default if neither >pkg.conf nor PKG_PATH are set.
Re: if I were to make a pkg-add diff
On Sun, Dec 27, 2015 at 12:05:43PM GMT, Stuart Henderson wrote: > This would work for releases, though unless pkg_add was changed to > cache the redirect it would slow things down compared to fetching > directly from the mirror. > > For snapshots, caching the redirect would be essential, otherwise it > could switch mirror mid-update, and it is common that mirrors are not > all in-sync. > > Caching the redirect is a bigger change than it seems because pkg_add > doesn't have its own network code but defers to $FETCH_CMD which could > be one of a number of programs. > > (yes this has been thought about already :) > > I think the best that can be done relatively easily would be to have > pkg_add fetch ftplist.cgi and pick the first result as a default if > neither pkg.conf nor PKG_PATH are set. Thanks for the info, Stuart - very informative! Regards, Raf P.S. Even though I had used a real-life example, the email was tongue-in-cheek :^)
Re: if I were to make a pkg-add diff
I guess I didn't really answer your question. It wouldn't rely upon the ramdisk. It is meant to run after install. So it would presumably have all the firmware. I was thinking about running it similarly to the install output though. I setup a local mirror once and it crapped out after a while and just kept running, waiting for a connection without any output. I think that it would be kinda neat to have something that could potentially lead you to a new mirror that is faster than maybe a mirror you chose manually. I wasn't aware that the installer component that I was referencing edited pkg.conf . I don't know how to calculate the "freshness" of a mirror, like the Debian program other than by testing their latency. I suspect that even with the best O(1) network calculation, you aren't really going to get the best result. -- -Luke
Re: if I were to make a pkg-add diff
On 2015-12-27, Raf Czlonka wrote: > On Sun, Dec 27, 2015 at 01:58:53AM GMT, Luke Small wrote: > >> Come to think about it, it might to be good to do tiny standalone >> program called pkg_ping and then I could make it in C like I'd prefer. >> I'd hope to make a port maybe, but then it would functionally defeat >> the intent. > > I think you're wasting your time and skills - a service akin to Debian's > redirector[0] would be much better! > > Raf > > [0] http://httpredir.debian.org/ This would work for releases, though unless pkg_add was changed to cache the redirect it would slow things down compared to fetching directly from the mirror. For snapshots, caching the redirect would be essential, otherwise it could switch mirror mid-update, and it is common that mirrors are not all in-sync. Caching the redirect is a bigger change than it seems because pkg_add doesn't have its own network code but defers to $FETCH_CMD which could be one of a number of programs. (yes this has been thought about already :) I think the best that can be done relatively easily would be to have pkg_add fetch ftplist.cgi and pick the first result as a default if neither pkg.conf nor PKG_PATH are set.
Re: if I were to make a pkg-add diff
> Come to think about it, it might to be good to do tiny standalone > program called pkg_ping and then I could make it in C like I'd prefer. > I'd hope to make a port maybe, but then it would functionally defeat > the intent. I find it not usefull to continue this kind of conversation because you don't listen to others. Make what you have planned, send a diff and, let's wait what the future brings. Personally I see no benefit in this ideas.
Re: if I were to make a pkg-add diff
On Sun, Dec 27, 2015 at 01:58:53AM GMT, Luke Small wrote: > Come to think about it, it might to be good to do tiny standalone > program called pkg_ping and then I could make it in C like I'd prefer. > I'd hope to make a port maybe, but then it would functionally defeat > the intent. I think you're wasting your time and skills - a service akin to Debian's redirector[0] would be much better! Raf [0] http://httpredir.debian.org/
Re: if I were to make a pkg-add diff
On Sun, Dec 27, 2015 at 01:40:04AM GMT, Luke Small wrote: > pkg_add initialization and mirror selection can be automated ... As already mentioned here, this part is already done during installation. One can either accept the suggested mirror or choose one manually from the list, your choice is then saved in '/etc/pkg.conf' - not the same thing as setting up PKG_PATH but is sufficient for pkg_* use. Raf
Re: if I were to make a pkg-add diff
Come to think about it, it might to be good to do tiny standalone program called pkg_ping and then I could make it in C like I'd prefer. I'd hope to make a port maybe, but then it would functionally defeat the intent. On 12/26/15, Luke Small wrote: > I just figure that adding a little complexity that doesn't adversely > affect security, to ease initial entry into the system for new users > could be good. pkg_add initialization and mirror selection can be > automated in a way to not discourage someone from picking up a fresh > install and running with it. Maybe I could even add an automatic > pinging heuristic to select the lowest latency mirror. > > On 12/26/15, Christoph R. Murauer wrote: >> Quote from dan mclaughlin : >> >> [ ... ] >>> (i am really starting to feel for the devs. this gets wearying.) >> [ ... ] >>> whenever i want something to work the way i want, i just script around >>> it. >>> the beauty of unix. >> >> Explains it very good. I personally think, OpenBSD is simple not for >> Luke. What Luke does, is simple unfair towards the developers and also >> towards new users which are willing to learn and maybe use OpenBSD >> only as users. >> >> For Luke, read the following : >> >> http://www.openbsd.org/mail.html >> >> Do your homework before you post >> [ ... ] the FAQ and the relevant man pages (start with afterboot(8)), >> and check the mailing list archives. We want to help, but we wouldn't >> want to deprive you of a valuable learning experience, and no one >> wants to see the same question on the lists for the fifth time in a >> month. >> >> Respect differences in opinion and philosophy >> Intelligent people may look at the same set of facts and come to very >> different conclusions. Repeating the same points that didn't convince >> someone previously rarely changes their mind and irritates all the >> other readers. >> >> http://www.openbsd.org/faq/index.html >> >> 8.14 - What web browsers are available for OpenBSD? - for lynx and the >> relevant discussions about it at misc@. >> >> 14.6 - Soft Updates - for softdep >> >> 8.21 - Does OpenBSD support any journaling filesystems? - for softdep >> yes or no. >> >> 1.11 - Why is/isn't ProductX included? >> Why isn't there a graphical or curses(3) based installer? >> For a number of reasons, including the goal of keeping the >> installation boot media able to be a single floppy disk, the fact that >> one installer can be used on all platforms in all configurations, and >> the fact that after the second or third OpenBSD install, most users >> find the OpenBSD installation system among the fastest and easiest >> installers of any OS. Most developers and users greatly prefer the >> speed, power, and ease of use of the current installer to any of the >> more "colorful" or "pretty" installers on some other platforms. >> >> In most cases, these topics have been discussed in painful detail on >> the mail lists, please see archives if you need more information. - >> for your (Javascript) installer idea. >> >> 14.2 - Using fdisk(8) >> 14.3 - Using OpenBSD's disklabel(8) >> 14.21 - How do I use softraid(4)? + man page - for full disc encryption. >> >> And yes, it is expected, that you can handle this tools befor >> installing the system ! >> >> 15.2.2 - Making things easy: PKG_PATH - to define more then one mirrors. >> >> You should stop to reninvent the wheel or, to create problems where >> are none. >> You should start to RTFM (Reed The Fucking Manuals) or, buy a OpenBSD >> relevant book - as others also did. >> You also should do the following *shutup and hack* instead of always >> complaining. >> >> Let's say your idea for pkg_add is cool, let's say you make a clean >> install or a update using bsd.rd and, let's also say, that you have no >> supported network device - without the firmware from >> firmware.openbsd.org ... what's next ? >> >> >> > > > -- > -Luke > -- -Luke
Re: if I were to make a pkg-add diff
I just figure that adding a little complexity that doesn't adversely affect security, to ease initial entry into the system for new users could be good. pkg_add initialization and mirror selection can be automated in a way to not discourage someone from picking up a fresh install and running with it. Maybe I could even add an automatic pinging heuristic to select the lowest latency mirror. On 12/26/15, Christoph R. Murauer wrote: > Quote from dan mclaughlin : > > [ ... ] >> (i am really starting to feel for the devs. this gets wearying.) > [ ... ] >> whenever i want something to work the way i want, i just script around >> it. >> the beauty of unix. > > Explains it very good. I personally think, OpenBSD is simple not for > Luke. What Luke does, is simple unfair towards the developers and also > towards new users which are willing to learn and maybe use OpenBSD > only as users. > > For Luke, read the following : > > http://www.openbsd.org/mail.html > > Do your homework before you post > [ ... ] the FAQ and the relevant man pages (start with afterboot(8)), > and check the mailing list archives. We want to help, but we wouldn't > want to deprive you of a valuable learning experience, and no one > wants to see the same question on the lists for the fifth time in a > month. > > Respect differences in opinion and philosophy > Intelligent people may look at the same set of facts and come to very > different conclusions. Repeating the same points that didn't convince > someone previously rarely changes their mind and irritates all the > other readers. > > http://www.openbsd.org/faq/index.html > > 8.14 - What web browsers are available for OpenBSD? - for lynx and the > relevant discussions about it at misc@. > > 14.6 - Soft Updates - for softdep > > 8.21 - Does OpenBSD support any journaling filesystems? - for softdep > yes or no. > > 1.11 - Why is/isn't ProductX included? > Why isn't there a graphical or curses(3) based installer? > For a number of reasons, including the goal of keeping the > installation boot media able to be a single floppy disk, the fact that > one installer can be used on all platforms in all configurations, and > the fact that after the second or third OpenBSD install, most users > find the OpenBSD installation system among the fastest and easiest > installers of any OS. Most developers and users greatly prefer the > speed, power, and ease of use of the current installer to any of the > more "colorful" or "pretty" installers on some other platforms. > > In most cases, these topics have been discussed in painful detail on > the mail lists, please see archives if you need more information. - > for your (Javascript) installer idea. > > 14.2 - Using fdisk(8) > 14.3 - Using OpenBSD's disklabel(8) > 14.21 - How do I use softraid(4)? + man page - for full disc encryption. > > And yes, it is expected, that you can handle this tools befor > installing the system ! > > 15.2.2 - Making things easy: PKG_PATH - to define more then one mirrors. > > You should stop to reninvent the wheel or, to create problems where > are none. > You should start to RTFM (Reed The Fucking Manuals) or, buy a OpenBSD > relevant book - as others also did. > You also should do the following *shutup and hack* instead of always > complaining. > > Let's say your idea for pkg_add is cool, let's say you make a clean > install or a update using bsd.rd and, let's also say, that you have no > supported network device - without the firmware from > firmware.openbsd.org ... what's next ? > > > -- -Luke
Re: if I were to make a pkg-add diff
On 2015-12-26, dan mclaughlin wrote: > ftp -o - http://www.openbsd.org/ftp.html | more > > will display the html source of the page, which is pretty easy to read even > unrendered. No need to parse the html, ftp -o - http://www.openbsd.org/build/mirrors.dat | grep -e ^U -e ^G[CITS]
Re: if I were to make a pkg-add diff
Quote from dan mclaughlin : [ ... ] > (i am really starting to feel for the devs. this gets wearying.) [ ... ] > whenever i want something to work the way i want, i just script around > it. > the beauty of unix. Explains it very good. I personally think, OpenBSD is simple not for Luke. What Luke does, is simple unfair towards the developers and also towards new users which are willing to learn and maybe use OpenBSD only as users. For Luke, read the following : http://www.openbsd.org/mail.html Do your homework before you post [ ... ] the FAQ and the relevant man pages (start with afterboot(8)), and check the mailing list archives. We want to help, but we wouldn't want to deprive you of a valuable learning experience, and no one wants to see the same question on the lists for the fifth time in a month. Respect differences in opinion and philosophy Intelligent people may look at the same set of facts and come to very different conclusions. Repeating the same points that didn't convince someone previously rarely changes their mind and irritates all the other readers. http://www.openbsd.org/faq/index.html 8.14 - What web browsers are available for OpenBSD? - for lynx and the relevant discussions about it at misc@. 14.6 - Soft Updates - for softdep 8.21 - Does OpenBSD support any journaling filesystems? - for softdep yes or no. 1.11 - Why is/isn't ProductX included? Why isn't there a graphical or curses(3) based installer? For a number of reasons, including the goal of keeping the installation boot media able to be a single floppy disk, the fact that one installer can be used on all platforms in all configurations, and the fact that after the second or third OpenBSD install, most users find the OpenBSD installation system among the fastest and easiest installers of any OS. Most developers and users greatly prefer the speed, power, and ease of use of the current installer to any of the more "colorful" or "pretty" installers on some other platforms. In most cases, these topics have been discussed in painful detail on the mail lists, please see archives if you need more information. - for your (Javascript) installer idea. 14.2 - Using fdisk(8) 14.3 - Using OpenBSD's disklabel(8) 14.21 - How do I use softraid(4)? + man page - for full disc encryption. And yes, it is expected, that you can handle this tools befor installing the system ! 15.2.2 - Making things easy: PKG_PATH - to define more then one mirrors. You should stop to reninvent the wheel or, to create problems where are none. You should start to RTFM (Reed The Fucking Manuals) or, buy a OpenBSD relevant book - as others also did. You also should do the following *shutup and hack* instead of always complaining. Let's say your idea for pkg_add is cool, let's say you make a clean install or a update using bsd.rd and, let's also say, that you have no supported network device - without the firmware from firmware.openbsd.org ... what's next ?
Re: if I were to make a pkg-add diff
On Fri, 25 Dec 2015 16:09:27 -0600 Luke Small wrote: > I suppose folks could opt for the more stable yet higher latency > official mirrors even if they aren't local to canada and they would > never be surprised. It may not be too much trouble for me to implement > a mere stdout statement in the perl pkg-add to advise the user to > update PKG-PATH to randomly offer one of the official mirrors as the > PKG-PATH (as an easy to implement fix) and steer the user to install a > web-browser to discover the current list of http/ftp package mirrors > if Firefox or lynx exists on the system; since they don't by default. > It wouldn't be too dissimilar to the kind of messages delivered by > pkg-add itself to rm folder contents at the end of a run. talk is cheap. if you see a problem, show a fix instead of talking about it. (i am really starting to feel for the devs. this gets wearying.) so lynx is gone, but you don't need to install a web browser. ftp(1) is in base and functions quite well. you want the mirrors? ftp -o - http://www.openbsd.org/ftp.html | more will display the html source of the page, which is pretty easy to read even unrendered. if you wanted to format it better, you could do: ftp -o - http://www.openbsd.org/ftp.html | \ sed -n \ -e 's:$::' \ -e 's: \([^<]*\)<.*:\1 :p' \ -e 's:^\( [hfr].*\):\1:p' (all whitespace in the above sed are single tabs, copy & paste will not work) will display them all (http,ftp,rsync) as below: Australia (Adelaide) http://mirror.internode.on.net/pub/OpenBSD/ Australia (Brisbane) http://mirror.aarnet.edu.au/pub/OpenBSD/ Australia (Perth) http://ftp.iinet.net.au/pub/OpenBSD/ ... if you want to test a mirror, just use ftp's return value as below: ftp -VM -o /dev/null $MIRROR && echo "$MIRROR" >>good-mirrors or if you had a list of possible mirrors (in file 'mirrors'): for MIRROR in $(< mirrors); do ftp -VM -o /dev/null $MIRROR && echo "$MIRROR" done >good-mirrors whenever i want something to work the way i want, i just script around it. the beauty of unix. > > On 12/25/15, Luke Small wrote: > > I suspect that if you did, it wouldn't check whether there was an > > astronaut ready to control the on-board computer and would sit there > > continuously trying to rev the rocket engines with no jet fuel. That > > is the way pkg-add acts right now. I felt pretty ridiculous wondering > > why pkg-add wasn't working only to figure out I was working with a > > mirror that was no longer active this week. > > > > On 12/24/15, Theo de Raadt wrote: > >>>I wanna make a c program that checks for a PKG_PATH that exists and > >>>connects to a workable link for pkg_add(). > >> > >> and I wanna build a rocket ship... > >> > > > > > > -- > > -Luke > > > > > -- > -Luke
Re: if I were to make a pkg-add diff
> if I were to make a pkg-add diff Mr. Skywalker, make that damn modification for yourself if you really think it helps you. If you want you can send a diff, too, but please go away with these idiotic talks about what you can do but you will not. I guarantee you that nobody will be upset when you tweak that pkg_add and friends and keep it for yourself. The BSD licence permits that. Now, go away, please and don't hijack the threads anymore. Thank you.
Re: if I were to make a pkg-add diff
Luke, ... Are you sure that you're not in politics ? > I suppose folks could opt for the more stable yet higher latency > official mirrors even if they aren't local to canada and they would > never be surprised. It may not be too much trouble for me to implement > a mere stdout statement in the perl pkg-add to advise the user to > update PKG-PATH to randomly offer one of the official mirrors as the > PKG-PATH (as an easy to implement fix) and steer the user to install a > web-browser to discover the current list of http/ftp package mirrors > if Firefox or lynx exists on the system; since they don't by default. > It wouldn't be too dissimilar to the kind of messages delivered by > pkg-add itself to rm folder contents at the end of a run. > > On 12/25/15, Luke Small wrote: >> I suspect that if you did, it wouldn't check whether there was an >> astronaut ready to control the on-board computer and would sit there >> continuously trying to rev the rocket engines with no jet fuel. That >> is the way pkg-add acts right now. I felt pretty ridiculous >> wondering >> why pkg-add wasn't working only to figure out I was working with a >> mirror that was no longer active this week. >> >> On 12/24/15, Theo de Raadt wrote: I wanna make a c program that checks for a PKG_PATH that exists and connects to a workable link for pkg_add(). >>> >>> and I wanna build a rocket ship... >>> >> >> >> -- >> -Luke
Re: if I were to make a pkg-add diff
I suppose folks could opt for the more stable yet higher latency official mirrors even if they aren't local to canada and they would never be surprised. It may not be too much trouble for me to implement a mere stdout statement in the perl pkg-add to advise the user to update PKG-PATH to randomly offer one of the official mirrors as the PKG-PATH (as an easy to implement fix) and steer the user to install a web-browser to discover the current list of http/ftp package mirrors if Firefox or lynx exists on the system; since they don't by default. It wouldn't be too dissimilar to the kind of messages delivered by pkg-add itself to rm folder contents at the end of a run. On 12/25/15, Luke Small wrote: > I suspect that if you did, it wouldn't check whether there was an > astronaut ready to control the on-board computer and would sit there > continuously trying to rev the rocket engines with no jet fuel. That > is the way pkg-add acts right now. I felt pretty ridiculous wondering > why pkg-add wasn't working only to figure out I was working with a > mirror that was no longer active this week. > > On 12/24/15, Theo de Raadt wrote: >>>I wanna make a c program that checks for a PKG_PATH that exists and >>>connects to a workable link for pkg_add(). >> >> and I wanna build a rocket ship... >> > > > -- > -Luke > -- -Luke
Re: if I were to make a pkg-add diff
I suspect that if you did, it wouldn't check whether there was an astronaut ready to control the on-board computer and would sit there continuously trying to rev the rocket engines with no jet fuel. That is the way pkg-add acts right now. I felt pretty ridiculous wondering why pkg-add wasn't working only to figure out I was working with a mirror that was no longer active this week. On 12/24/15, Theo de Raadt wrote: >>I wanna make a c program that checks for a PKG_PATH that exists and >>connects to a workable link for pkg_add(). > > and I wanna build a rocket ship... > -- -Luke
Re: if I were to make a pkg-add diff
On 2015 Dec 24 (Thu) at 20:23:38 -0600 (-0600), Luke Small wrote: :I wanna make a c program that checks for a PKG_PATH that exists and :connects to a workable link for pkg_add(). If you ever upgraded using :http mirrors on the install disk, it offers list# which links directly :to numbered mirrors. It would likely ease the initial startup for :whomever uses it while not burdening anybody that has already properly :configured their system for pkg_add. Most notably, there won't be any :JavaScript text-based GUI. ;) : update /etc/pkg.conf then. -- Quick!! Act as if nothing has happened!
Re: if I were to make a pkg-add diff
>I wanna make a c program that checks for a PKG_PATH that exists and >connects to a workable link for pkg_add(). and I wanna build a rocket ship...
Re: if I were to make a pkg-add diff
I wanna make a c program that checks for a PKG_PATH that exists and connects to a workable link for pkg_add(). If you ever upgraded using http mirrors on the install disk, it offers list# which links directly to numbered mirrors. It would likely ease the initial startup for whomever uses it while not burdening anybody that has already properly configured their system for pkg_add. Most notably, there won't be any JavaScript text-based GUI. ;) On 12/24/15, Ingo Schwarze wrote: > Hi, > > pkg_add(1) is about the hardest program in base to get patches into, > even for experienced developers who know what they are doing, even > if the patches are of reasonable quality and well thought out. > Almost all of my own attempts at improving it led to nowhere, with > very few exceptions for very simple fixes of the most obvious bugs, > and even those exceptions were almost never easy to bring to fruition. > I say that after having committed more than 2.000 times in very > diverse parts of OpenBSD. pkg_add(1) is very hard and no place at > all for beginners. > > The responsible developer is both chronically overworked and very > picky about keeping the structure of the program in a particular > style, and that style is *extremely* unsusual and extremely hard > to read, understand, and maintain. I'm saying that as someone who > has been doing professional, object-oriented Perl programming in > the software industry for more than half a decade. > > So Luke, don't bother submitting patches to pkg_add(1). Don't > bother doing *anything* in a sloppy way for OpenBSD, that would be > nothing but a waste of time. We expect very careful work even for > the smallest suggestions. What you are thinking about is ridiculously > high over your head. > > Besides, you are not making any sense whatsoever. You probably > shouldn't try to submit any patches whatsoever, but instead try to > acquire basic skills using the system in simple ways and expressing > your thoughts clearly. > > Yours, > Ingo > -- -Luke
Re: if I were to make a pkg-add diff
Luke Small wrote: > Assuming i could do it: If I were to make a sloppy perl-based pkg-add > program that used c and the installer code to (re)set the PKG-PATH > environment variable using the "http" settings that are available for > installing the modules from mirrors, if I made changes to it to > request a change to the environment variable and possibly alter a > .profile if either the PKG-PATH doesn't exist, or a connection to the > mirror takes too long, would the openbsd devs accept the diff? Quality diffs that fix problems are welcome.
Re: if I were to make a pkg-add diff
Hi, pkg_add(1) is about the hardest program in base to get patches into, even for experienced developers who know what they are doing, even if the patches are of reasonable quality and well thought out. Almost all of my own attempts at improving it led to nowhere, with very few exceptions for very simple fixes of the most obvious bugs, and even those exceptions were almost never easy to bring to fruition. I say that after having committed more than 2.000 times in very diverse parts of OpenBSD. pkg_add(1) is very hard and no place at all for beginners. The responsible developer is both chronically overworked and very picky about keeping the structure of the program in a particular style, and that style is *extremely* unsusual and extremely hard to read, understand, and maintain. I'm saying that as someone who has been doing professional, object-oriented Perl programming in the software industry for more than half a decade. So Luke, don't bother submitting patches to pkg_add(1). Don't bother doing *anything* in a sloppy way for OpenBSD, that would be nothing but a waste of time. We expect very careful work even for the smallest suggestions. What you are thinking about is ridiculously high over your head. Besides, you are not making any sense whatsoever. You probably shouldn't try to submit any patches whatsoever, but instead try to acquire basic skills using the system in simple ways and expressing your thoughts clearly. Yours, Ingo
if I were to make a pkg-add diff
I can't type underscore on this device. Assuming i could do it: If I were to make a sloppy perl-based pkg-add program that used c and the installer code to (re)set the PKG-PATH environment variable using the "http" settings that are available for installing the modules from mirrors, if I made changes to it to request a change to the environment variable and possibly alter a .profile if either the PKG-PATH doesn't exist, or a connection to the mirror takes too long, would the openbsd devs accept the diff? Theoretically, it wouldn't act much differently than the existing installer code, except it would be on pkg-add after the system has been installed. -- -Luke
