On 3/2/18, rundstut...@gmx.de <rundstut...@gmx.de> wrote: > i didn't even know that sbosrcarch does exist (even though i have seen > slackware.uk before). so the functionality i programmed does already > exist.
Not exactly: it's the archive creation/maintenance script that does all the "fake HEAD" requests. It's looking at the headers to decide whether the file has changed. If it thinks the file's different, it downloads a new copy for the archive. There's no "check links, but don't download files" mode, so it doesn't do what "sbog ping" does. > i can't find the sources of your ping script, though. Here: http://urchlay.naptime.net/repos/sbostuff/tree/ ...or: git clone git://urchlay.naptime.net/sbostuff.git It's in perl, and not what I'd call beautiful code, so you might want to put on your goggles before looking at it :) > i hate workarounds with passion. its one of the banes of the software > industry. what i am trying to do is actually what HEAD requests were > made for. > ... > but alas - these servers are not supported by "sbog > ping". if a server is not able to properly handle a HEAD request (which > is not hard) then this is not the fault of sbog. Right. In a perfect world, all web servers would comply with the spec, and would support HEAD requests. Your approach is valid, but I took a different approach, since I wanted sbosrcarch to be as complete an archive as possible. The only things it doesn't have are files hidden behind a click-through license (like Oracle's jdk download). You might at least look through your logs and make a list of the servers that don't do HEAD requests, and add some code that logs "this server doesn't support HEAD requests", so the user knows it's the server's fault. > if i read the output of "curl -v --head -X GET $url" correctly: the > server will get a GET request, so the server (or proxy) will start > sending the body (before the request is closed/cancelled), causing > traffic overhead. correct me if i'm wrong. i don't want to do that. You're right. How much overhead depends on how the server/proxy is configured, but it seems to be 64K bytes per request usually. I decided that wasn't a problem, partly because the guy who hosts the archive said so (I wrote the script, he runs it and lets people download the files it collects). > there is another reason i am don't want to use GET requests: sbog uses a > client pool, sending request concurrently to servers. this speeds > things up *a lot*. i don't want to send several GET request to a > server simultaneously. Make the client pool smart enough to serialize requests to the same server, and do them in parallel only when they're for different servers? That seems like it'd be worth doing even with regular HEAD requests like you use now. sbosrcarch doesn't do anything in parallel, it's one request after another. Normally it's run non-interactively (via cron job) so it doesn't matter if it takes a long time to finish. _______________________________________________ SlackBuilds-users mailing list SlackBuildsemail@example.com https://lists.slackbuilds.org/mailman/listinfo/slackbuilds-users Archives - https://lists.slackbuilds.org/pipermail/slackbuilds-users/ FAQ - https://slackbuilds.org/faq/