Re: [Gimp-developer] Updating the website for all the broken download links?
Hi, yesterday I came across http://www.gimp.org/develop/ and clicked the link 'INSTALL'. It refers to https://git.gnome.org/browse/gimp/plain/INSTALL, which doesn't exist anymore in this form. This raised a question: can the link checker also detect dead links that don't result in an HTTP error but in simply nothing or unreadable garbage? Kind regards, Sven ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
Hi Sven, On Fri, May 2, 2014 at 9:00 PM, scl scl.gp...@gmail.com wrote: Hi, yesterday I came across http://www.gimp.org/develop/ and clicked the link 'INSTALL'. It refers to https://git.gnome.org/browse/gimp/plain/INSTALL, which doesn't exist anymore in this form. This raised a question: can the link checker also detect dead links that don't result in an HTTP error but in simply nothing or unreadable garbage? Well in this case, the URL you gave is actually a 404 error even though it does not show in the browser (at least in my Firefox, it shows as just a blank page). You can see it is a 404 by doing a wget for instance. Thus in the report generated with the program linkchecker, this URL was listed. In this specific case though, it would actually have listed it as a warning for being 0 byte even if there had been no error. I'll attach the list of errors and warnings I generated a few days ago (only Sven will get it, since the list will get rid of the attachment, as usual). That's quite some work to be done for someone with a little time, and a few grep or sed commands. :-) Jehan Kind regards, Sven ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list LinkChecker 8.4 Copyright (C) 2000-2013 Bastian Kleineidam LinkChecker comes with ABSOLUTELY NO WARRANTY! This is free software, and you are welcome to redistribute it under certain conditions. Look at the file `LICENSE' within this distribution. Get the newest version at http://wummel.github.com/linkchecker/ Write comments and bugs to https://github.com/wummel/linkchecker/issues Support this project at http://wummel.github.com/linkchecker/donations.html Start checking at 2014-04-28 17:31:58+013 URL`https://git.gnome.org/browse/gimp/plain/INSTALL' Name `INSTALL' Parent URL http://www.gimp.org/develop/, line 165, col 7 Real URL https://git.gnome.org/browse/gimp/plain/INSTALL Check time 16.527 seconds Size 0B Warning[http-robots-denied] Access denied by robots.txt, skipping content checks. [url-content-size-zero] Content size is zero. Result Error: 404 Not Found URL`https://lists.xcf.berkeley.edu/lists/gimp-user/' Name `user' Parent URL http://www.gimp.org/docs/userfaq.html, line 732, col 62 Real URL https://lists.xcf.berkeley.edu/lists/gimp-user/ Check time 2.365 seconds Result Error: error: [Errno 2] No such file or directory URL`https://lists.xcf.berkeley.edu/lists/gimp-developer/' Name `developer' Parent URL http://www.gimp.org/docs/userfaq.html, line 732, col 133 Real URL https://lists.xcf.berkeley.edu/lists/gimp-developer/ Check time 1.982 seconds Result Error: error: [Errno 2] No such file or directory URL`http://adrian.gimp.org/scripts/test.scm' Name `http://adrian.gimp.org/scripts/test.scm' Parent URL http://www.gimp.org/docs/userfaq.html, line 894, col 96 Real URL http://adrian.gimp.org/scripts/test.scm Check time 1.555 seconds Result Error: error: [Errno 2] No such file or directory URL`http://wuarchive.wustl.edu/systems/gnu/regex-0.12.tar.gz' Name `http://wuarchive.wustl.edu/systems/gnu/regex-0.12.tar.gz' Parent URL http://www.gimp.org/docs/userfaq.html, line 1077, col 98 Real URL http://wuarchive.wustl.edu/systems/gnu/regex-0.12.tar.gz Check time 1.115 seconds Result Error: error: [Errno 2] No such file or directory URL`http://www.xcf.berkeley.edu/~gimp/script-fu/script-fu.html' Name `http://www.xcf.berkeley.edu/~gimp/script-fu/script-fu.html' Parent URL http://www.gimp.org/docs/userfaq.html, line 881, col 1 Real URL http://www.xcf.berkeley.edu/~gimp/script-fu/script-fu.html Check time 7.202 seconds Result Error: error: [Errno 2] No such file or directory URL`http://tigert.gimp.org/' Name `Tuomas Kuosmanen (tigert)' Parent URL http://www.gimp.org/tutorials/Draw_A_Paint_Brush/, line 386, col 30 Real URL http://tigert.gimp.org/ Check time 1.171 seconds Result Error: error: [Errno 2] No such file or directory URL`ftp://ftp.gimp.org/pub/gimp/help/' Name `\nFTP server' Parent URL http://www.gimp.org/, line 187, col 1 Real URL ftp://ftp.gimp.org/pub/gimp/help/ Check time 61.332 seconds Result Error: timeout: timed out URL`ftp://ftp.gimp.org/pub/gimp/v2.8/osx/gimp-2.8.10-dmg-1.dmg' Name `FTP' Parent URL http://www.gimp.org/, line 153, col 309 Real URL ftp://ftp.gimp.org/pub/gimp/v2.8/osx/gimp-2.8.10-dmg-1.dmg Check time 61.343 seconds Result Error: timeout: timed out URL`http://docs.gimp.org/en/gimp-concepts-paths.html' Name `Paths' Parent URL http://www.gimp.org/docs/userfaq.html, line 459, col 5 Real URL http://docs.gimp.org/en/gimp-concepts-paths.html Check time
Re: [Gimp-developer] Updating the website for all the broken download links?
On Fri, May 2, 2014 at 5:00 AM, scl scl.gp...@gmail.com wrote: Hi, yesterday I came across http://www.gimp.org/develop/ and clicked the link 'INSTALL'. It refers to https://git.gnome.org/browse/gimp/plain/INSTALL, which doesn't exist anymore in this form. This raised a question: can the link checker also detect dead links that don't result in an HTTP error but in simply nothing or unreadable garbage? Kind regards, Sven Hi Sven, If you're inquiring about my link tester then yes it should get it. The [get_link_status function][1] is what does the actual checking. It does a combination of HTTP HEAD method (so it only grabs the headers instead of content) as well has a bit of exception handling for other non-HTTP unexpected responses. If it can be improved I'm all ears for improving it. Also, I'm using python code only in a [main method][2] which means one could write another script which will import frontend_test.py and take advantage of that function (using it like a library instead of an executable). Also, as Jehan pointed out that [INSTALL file][3] returns HTTP 404 not found. So a link checker should catch that. SAM [1]: https://github.com/sag47/chewbotkah/blob/development/frontend_test.py#L38-L83 [2]: https://github.com/sag47/chewbotkah/blob/development/frontend_test.py#L252 [3]: https://git.gnome.org/browse/gimp/plain/INSTALL ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
I CC all the MAINTAINERS. For those maintainers who may not be following. I'm in the process of creating a decent build and QA process for the GIMP website. I just realized that the install.sh is no good for Jenkins. It's not a POSIX compatible script which Jenkins requires in order to properly report the build status. Attached I have a patch of install.sh and the original install.sh. The patch is based off of master 22561ef. I have also attached a standalone install.sh. I added lots of comments and environment information because a good bash script needs environment information in my humble opinion in the off chance it doesn't work years down the road you can at least know what environment it was originally written. SAM ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
On Fri, May 2, 2014 at 11:19 AM, Sam Gleske sam.mxra...@gmail.com wrote: I CC all the MAINTAINERS. For those maintainers who may not be following. I'm in the process of creating a decent build and QA process for the GIMP website. I just realized that the install.sh is no good for Jenkins. It's not a POSIX compatible script which Jenkins requires in order to properly report the build status. Attached I have a patch of install.sh and the original install.sh. The patch is based off of master 22561ef. I have also attached a standalone install.sh. I added lots of comments and environment information because a good bash script needs environment information in my humble opinion in the off chance it doesn't work years down the road you can at least know what environment it was originally written. SAM I opened a bugzilla report instead. https://bugzilla.gnome.org/show_bug.cgi?id=729421 I'll do that from now on. SAM ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
Hi, On Sat, May 3, 2014 at 6:43 AM, Sam Gleske sam.mxra...@gmail.com wrote: On Fri, May 2, 2014 at 11:19 AM, Sam Gleske sam.mxra...@gmail.com wrote: I CC all the MAINTAINERS. For those maintainers who may not be following. I'm in the process of creating a decent build and QA process for the GIMP website. I just realized that the install.sh is no good for Jenkins. It's not a POSIX compatible script which Jenkins requires in order to properly report the build status. Attached I have a patch of install.sh and the original install.sh. The patch is based off of master 22561ef. I have also attached a standalone install.sh. I added lots of comments and environment information because a good bash script needs environment information in my humble opinion in the off chance it doesn't work years down the road you can at least know what environment it was originally written. SAM I opened a bugzilla report instead. https://bugzilla.gnome.org/show_bug.cgi?id=729421 That's probably the best indeed for not lose track, and also for uploading files. :-) Thanks for all the work! Jehan I'll do that from now on. SAM ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
Von: Sam Gleske sam.mxra...@gmail.com On Fri, Apr 25, 2014 at 8:57 PM, Jehan Pagès jehan.marmott...@gmail.comwrote: Hey all, I've checked the testing website with my commit to fix the download links to the new URLs. All looked ok, so I cherry-picked to master. Would be great to have this rolled out to production, because right now our website just gives a bunch of dead links for getting GIMP. That's not good and could drive users away to go pick GIMP up from some possibly bad source. We used the http:// links as the main targets, and thos will still work. And I think that we will have to use the links to the torrents as the defaults, as those will finally use all the available mirrors. Also are all our mirror admins aware we changed the source of our main download? Because if they have scripts to sync with our FTP, then they will never relay properly our new releases. If we only knew who all those admins are :) There is no address list, but one can probably be created by digging through the gimp-web bugs in Bugzilla and extracting the addresses for ftp mirrors bugs. As I mentioned in the past the offer is still open for me to QA the GIMP website frontend for dead links and generate a report. I didn't ever get approval from any of the core devs so I didn't run any front end tests. Xenu's Link sleuth (http://home.snafu.de/tilman/xenulink.html) didn't have any noticeable effect on the server (even the aging old one), so I don't see anything that would prevent you from running your link checker. Please give it a go and we'll have a look at the results. -- Regards, Michael GPG: 96A8 B38A 728A 577D 724D 60E5 F855 53EC B36D 4CDD ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
On Mon, Apr 28, 2014 at 8:22 AM, Michael Schumacher schum...@gmx.de wrote: Please give it a go and we'll have a look at the results. It's currently running against the production wgo. I have started configuring a Jenkins build job and found that using a simple http server does not appear to be enough for the gimp website. I'm using python -m SimpleHTTPServer on the htdocs page and I can view http://localhost:8000and see some of the GIMP website but it appears largely unstyled. What server-side software does the GIMP website require? SAM ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
On Mon, Apr 28, 2014 at 3:41 PM, Sam Gleske sam.mxra...@gmail.com wrote: On Mon, Apr 28, 2014 at 8:22 AM, Michael Schumacher schum...@gmx.dewrote: Please give it a go and we'll have a look at the results. It's currently running against the production wgo. I have started configuring a Jenkins build job and found that using a simple http server does not appear to be enough for the gimp website. I'm using python -m SimpleHTTPServer on the htdocs page and I can view http://localhost:8000and see some of the GIMP website but it appears largely unstyled. What server-side software does the GIMP website require? FYI here is my Jenkins job config.xml. ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
As I mentioned in the past the offer is still open for me to QA the GIMP website frontend for dead links and generate a report. I didn't ever get approval from any of the core devs so I didn't run any front end tests. SAM On Fri, Apr 25, 2014 at 8:57 PM, Jehan Pagès jehan.marmott...@gmail.comwrote: Hey all, I've checked the testing website with my commit to fix the download links to the new URLs. All looked ok, so I cherry-picked to master. Would be great to have this rolled out to production, because right now our website just gives a bunch of dead links for getting GIMP. That's not good and could drive users away to go pick GIMP up from some possibly bad source. Also are all our mirror admins aware we changed the source of our main download? Because if they have scripts to sync with our FTP, then they will never relay properly our new releases. Thanks! Jehan ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
On 28.4.2014 at 3:59 AM Sam Gleske wrote: As I mentioned in the past the offer is still open for me to QA the GIMP website frontend for dead links and generate a report. I didn't ever get approval from any of the core devs so I didn't run any front end tests. SAM Hi Sam, yes, we lately had this topic and somehow it went out of sight. I'm not the webmaster, but I think it's useful. If we should/can integrate something into our Jenkins job set, then let me know. Greetings, Sven ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
On Sun, Apr 27, 2014 at 11:28 PM, scl scl.gp...@gmail.com wrote: On 28.4.2014 at 3:59 AM Sam Gleske wrote: As I mentioned in the past the offer is still open for me to QA the GIMP website frontend for dead links and generate a report. I didn't ever get approval from any of the core devs so I didn't run any front end tests. SAM Hi Sam, yes, we lately had this topic and somehow it went out of sight. I'm not the webmaster, but I think it's useful. If we should/can integrate something into our Jenkins job set, then let me know. Greetings, Sven As Jehan suggested I'll clone the website. I wasn't aware of the website git repo. As far as the bot goes the source for the crawler is out there. In general, the requirements for a Jenkins node would be the following... 1. Git for cloning and Python 2 for the robot. 2. A simple HTTP server such as nginx. Or if the website is simply static pages something like python -m SimpleHTTPServer would suffice. 3. A copy of Firefox which would be run for the actual tests. 4. Xvfb - X Virtual Frame Buffer which would be used to provide a headless GUI for Firefox to run within. 5. And the requirements for selenium. In general, I have all of the requirements in the setup process detailed for the testing crawler [1]. It integrates fairly well with Jenkins as far as output. If all of the tests pass then it will exit with a POSIX exit code 0 (success). If any of the unit tests fail then it will exit non-zero. So to integrate it with Jenkins one would need to simply add it as a build step in Jenkins. Minimum software requirements are basically what runs Firefox well. As Jehan pointed out the git repository I'll clone it and see what I can come up with. I can get Jenkins up and running java -jar jenkins.war and create a job which I can contribute to you. Though, it will essentially only be a single step job with a shell script executing the robot testing because I don't know what other build steps are involved with the website. That should be enough for you to figure out how to integrate it into your Jenkins build. I'll start with the git repository and go from there. Thanks for pointing it out Jehan. SAM [1]: https://github.com/sag47/chewbotkah ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
Re: [Gimp-developer] Updating the website for all the broken download links?
Hi Sven, On Mon, Apr 28, 2014 at 3:28 PM, scl scl.gp...@gmail.com wrote: On 28.4.2014 at 3:59 AM Sam Gleske wrote: As I mentioned in the past the offer is still open for me to QA the GIMP website frontend for dead links and generate a report. I didn't ever get approval from any of the core devs so I didn't run any front end tests. SAM Hi Sam, yes, we lately had this topic and somehow it went out of sight. I'm not the webmaster, but I think it's useful. If we should/can integrate something into our Jenkins job set, then let me know. I guess it is easy to have something like a scripted check (maybe automatically run after the website is deployed, or simply a cron run every month or so). The script in itself could be as easy as: - #!/bin/bash linkchecker -q --file-output=text/utf8/linkchecker-gimp.txt \ --no-status --no-warnings --ignore-url=^bitcoin: \ --ignore-url=^mailto: http://gimp.org/ success=$? header=From: \GIMP website\ linkchec...@gimp.org \nTo: \GIMP website admin\ ad...@gimp.org \nSubject: Broken links were detected on GIMP website. \nMIME-Version: 1.0 \nContent-Type: text/plain \n \nBelow the output from the linkchecker command: \n if [[ $success != 0 ]]; then echo -e $header | cat - linkchecker-gimp.txt | sendmail -i -t fi - Basically it checks all links on the website and if it finds any error, it would send an email (could be sent on the dev list rather than the admin), with the list of error links (here as a human-readable text file, but can be generated in more programmatic formats, like csv, if needed). Not sure about the sendmail part in the above script, because sendmail is not configured on my laptop and I never send email on console, but apparently the syntax should be like this. Of course I'd say such a script should only be run once any existing broken link has been eradicated (otherwise we'd just spam ourselves with very long emails). I have actually already ran linkchecker once (but I can't attach it here since the mailing list don't take attachment), and it found 110 links in error! Now we need someone to verify each item and see if there are links to update, remove or anything else, on gimp-web repository; that's the boring part of the job. If anyone wants me to send them the generated link list out-list, just ask me (or just run the above linkchecker command, or whatever favorite software of yours :-). Jehan Greetings, Sven ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list ___ gimp-developer-list mailing list List address:gimp-developer-list@gnome.org List membership: https://mail.gnome.org/mailman/listinfo/gimp-developer-list List archives: https://mail.gnome.org/archives/gimp-developer-list
