-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Todd Pattist wrote: > Thank you for the quick response. Background is I'm on Windows XP, Gnu > wget 1.11 >> This "doesn't affect traversal of HTML files" functionality is currently >> implemented via a heuristic based on the filename extension. That is, if >> it ends in ".htm" or ".html", I believe, then it will be traversed >> regardless of -A or -R settings, whereas .cgi or .php will not affect >> traversal. >> > I'm not sure I understand the "cgi or .php will not affect traversal."
I mean, it will not detect these as HTML files, so the accept/reject rules will be applied to them without exception. > If I use wget to start at http://site.com/view.php?f=16 and recursively > mirror without -A or -R, it looks like it traverses deeper as though > that page and other .php links are html files. This makes sense. (I say > looks like, because it takes a long time and produces lots of files). > If I select the same page and add accept=site.com/view.php?id=16 to > wgetrc, no pages are saved and it does not traverse any deeper and it > takes only a second or two. I see this in the log: > > Saving to: `site.com/[EMAIL PROTECTED]' > Removing site.com/[EMAIL PROTECTED] since it should be rejected. > > I recognize that the question mark was substituted for my OS, but that > does not matter on the accept filter. What does matter is whether I > have the .html or not in the accept filter. That surprises me. Both > accept=site.com/view.php?id=16.html and accept=site.com/view.php?id=16* > will match and keep the > site.com/[EMAIL PROTECTED] file, while both > accept=site.com/view.php?id=16 and accept=site.com/[EMAIL PROTECTED] cause > it not to match and generate the "Removing ... since it should be > rejected" line. Regardless of the matching/saving this seems to control > traversal, as I get far deeper traversal with no accept= at all. After another look at the relevant portions of the source code, it looks like accept/reject rules are _always_ applied against the local filename, contrary to what I'd been thinking. This needs to be changed. (But it probably won't be, any time soon.) Note that the view.php?id=16 doesn't mean what you may perhaps think it does: Wget detects the "?" as a wildcard, and allows it to match any character (including "@"). If you supplied "\?" instead (which matches a literal question mark), I'm guessing it'd actually fail to match, because it's checking against "@". My understanding is that, when you specify a URL directly at the command-line, it will be downloaded and traversed (if it turns out to be HTML), no matter what the accept/reject rules are (which can still cause it to be removed afterwards). Therefore, I suspect that what Wget does with your URL when it isn't matching the accept rules is: 1. Downloads the named file 2. Discovers that, regardless of the filename, it is indeed an HTML file, so scans it for all links to be downloaded. 3. After scanning for all the links, it doesn't find any that end in ".html", nor any that match the accept rules, so it doesn't do anything else. - --debug will definitely tell you whether it's bothering to scan that first file or not, and what it decides to do with the links it finds. > I'm pretty sure I can control traversal of php links with accept and > reject, but I often want to traverse looking for certain file types, but > don't want to save all the php files traversed. We're looking for more fine-grained controls to allow this sort of thing, but at the moment, my understanding is that there is no control over whether Wget traverses-and-then-deletes a given file: it will _always_ do that for files it knows or suspects are HTML (based on .htm, .html suffixes, or if, like the above example, it will download the filename first anyway because it's an explicit command-line argument); it will _never_ download/traverse any other sorts of links that do not match the accept rules. If something _does_ match the accept rules, and turns out after download to be an HTML file (determined by the server's headers), it will traverse it further; but of course it won't delete them afterward because they matched the accept list. >> I'd have to look at the relevant code, but it's possible that >> "directory"-looking names may also be automatically traversed in that way. >> > I don't want you to do work I can do myself. I was just hoping for a > link or some pointers that might help. It looks like this idea was incorrect anyway; it's only based on the suffix. >>> Does >>> html_extension=on affect link traversal? >>> >> >> No; this only affects whether filenames are changed upon download to >> explicitly include an ".html" extension (useful for local browsing). >> > > It seems that the html extension is used in the filter matching of > accept/reject, and that seems to affect traversal as described above > unless I'm missing something (which is entirely possible). Yes, it does; my bad. >>> I'd like to be able to >>> independently control link traversal vs. file retrieval with local file >>> storage. Do the directory include/exclude commands allow this - do they >>> work differently from -A -R? >>> >> >> I'm afraid I'm unsure what you are asking here. >> > Is my question clearer from the above? I'm seeing very quick exits > (seconds) when the accept filter does not match the start page. To get > deeper traversing, I have to match, but then it saves the matched files > and the traverse takes hours, with perhaps thousands of html files > (converted from .php files), none of which I need. Yes, the question is clearer, and unfortunately the answer is "not currently". :\ >>> 3) Which has priority if both reject and accept filters match? >>> >> Not sure; it's easy enough to test this yourself, though. >> > I have done lots of testing, so you'd think this simple one would be > obvious. The answer seems to be that reject is higher priority, since > identical accept= and reject= seem to produce no output. This matches > what the manual says. - --debug is your friend. It will tell you explicitly what it thinks about the links it finds. > It might help to add to the manual that adding an > accept= filter causes a rejection of everything that does not match the > accept filter, even if there is no reject filter specified. The fact > that specifically accepting some files turns on a default rejection of > everything else surprised me, since the normal default is to accept > everything. It actually is in the manual, but probably not the Windows Help documentation that you've got. My recollection is that the latter is generated from the abbreviated reference which, on Unix, becomes the "manpage". The full manual is available, in various formats, at http://www.gnu.org/software/wget/manual/. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH1aOL7M8hyUobTrERAiAnAJ4uk6R1jHaFEYkwScu9RKe6acGVQQCcC6lz twxxUd2OzSjHEeSWZ/MVKOA= =CHwi -----END PGP SIGNATURE-----
