Re: Accept and Reject - particularly for PHP and CGI sites

[Todd Pattist]

> When deciding whether it should delete a file afterwards, however, it > uses the _local_ filename (relevant code also in recur.c, near "Either > --delete-after was specified,"). I'm not positive, but this probably > means query strings _do_ matter in that case. :p > > Confused? Coz I sure am! I had thought there was already an issue filed against this, but upon searching discovered I was thinking of a couple related bug that had been closed. I've filed a new issue for this: https://savannah.gnu.org/bugs/?22670 I'm not sure whether this post should go into the buglist discussion or here, but I'll put it here. I have to say, I'm not sure this is properly classed as a bug.  If accept/reject applies to the original URL filename, why should the code bother to apply it again to the local name?  If filters don't pass the URL filename and wget doesn't retrieve the file, it can't save it.  I assume the answer was to handle script and content_disposition cases where you don't know what you're going to get back.  If you match only on URL, you'd have no way to control traversing separate from file retention, and that's something you definitely want.  (It's the default for conventional html based sites.) To put it another way, I usually want to download all the php files, and traverse all that turn out to be html, but I may only want to keep the zips or jpgs.  With two checks, one before download on the URL filename and another after download on the local filename, I've got some control in cgi, php script based sites that is similar to the control in a conventional html page site.  If this behavior is changed, then you'd probably need to have two sets of accept/reject filters that could be defined separately, one set to control traversing, and one to control file retention.  I'd actually prefer that, particularly with matching extended to the query string portion of the URL.  Right now, it may be impossible to prevent traversing some links.  If you don't want to traverse "index.php?mode=logout", but do want to get "index.php?mode=getfile" there's no way to do it since the URL filename is the same. In the short term, it would help to add something to the documentation in the accept/reject  area, such as the following: The accept/reject filters are applied to the filename twice - once to the filename in the URL before downloading to determine if the file should be retrieved (and parsed for more links if it is determined after download to be an html file) and again to the local filename after it is retrieved to determine if it should be kept.  The local filename after retrieval may be significantly different from the URL filename before retrieval for many reasons.  These include: 1) The URL filename does not include any query string portion of the URL, such as the string "?topic=16" in the URL "http://site.com/index.php?topic=16". After download the file may be stored as the local filename "[EMAIL PROTECTED]".  Accept/reject matching does not apply to the URL query string portion before download, but will apply after download when the query string is incorporated into the local filename. 2) When content disposition is on, the local filename may be completely different from the URL filename.  The URL "index.php?getfile=21" may return a content disposition header producing a local file of "some_interesting_file.zip". 3) The -E (html extension) and sometimes the -nd (no directories)  switches will alter the filename suffix by adding .html or .1 for duplicate files. If the URL filename in links found when the starting page is parsed do not pass the accept/reject filters, the links will not be followed and will not be parsed for more links unless the filename ends html or htm.  If accept/reject filters are used on cgi, php, asp and similar script based sites the URL filename must pass the filters (without considering any query string portion) if the links are to be traversed/parsed, and the local filename must pass the filters if the retrieved files are to be retained.