The syntax you have given me gets the first page OK, apparently by using the -O switch to specify a simple filename as the output file, rather than the automatically generated output filename(s). I.e., I get the Table of Contents in content.html. But when I add the rest of my syntax to recurse the links on the TOC page, it still fails. As follows:
D:\Documents and Settings\Administrator\Desktop\New Folder>WGET -d -r -l2 -k -K
-p -np -O content.html "http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-79
1&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbnam
e=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST"
DEBUG output created by Wget 1.8.2 on Windows.
Enqueuing http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Searc
h&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdoc
s=100&report=hr791.107&xform_type=3&sel=TOCLIST at depth 0
Queue count 1, maxcount 1.
Dequeuing http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Searc
h&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdoc
s=100&report=hr791.107&xform_type=3&sel=TOCLIST at depth 0
Queue count 0, maxcount 1.
--16:05:29-- http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=S
earch&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&ma
xdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST
=> `content.html'
Resolving thomas.loc.gov... done.
Caching thomas.loc.gov => 140.147.249.9
Connecting to thomas.loc.gov[140.147.249.9]:80... connected.
Created socket 308.
Releasing 00895438 (new refcount 1).
---request begin---
GET /cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndi
ct=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xfor
m_type=3&sel=TOCLIST HTTP/1.0
User-Agent: Wget/1.8.2
Host: thomas.loc.gov
Accept: */*
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response... HTTP/1.1 200 OK
Date: Wed, 08 Jan 2003 21:05:32 GMT
Server: Web
Connection: close
Content-Type: text/html
Length: unspecified [text/html]
[ <=> ] 33,466 9.80K/s
Closing fd 308
16:05:36 (9.80 KB/s) - `content.html' saved [33466]
thomas.loc.gov/cgi-bin/cpquery/0@r_t=a&r_n=107-791&submit=Search&variant=y&acron
ym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr79
1.107&xform_type=3&sel=TOCLIST: No such file or directory
FINISHED --16:05:36--
Downloaded: 33,466 bytes in 1 files
Scanning thomas.loc.gov/cgi-bin/cpquery/0@r_t=a&r_n=107-791&submit=Search&varian
t=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&re
port=hr791.107&xform_type=3&sel=TOCLIST (from http://thomas.loc.gov/cgi-bin/cpqu
ery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_b
y_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCL
IST)
thomas.loc.gov/cgi-bin/cpquery/0@r_t=a&r_n=107-791&submit=Search&variant=y&acron
ym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr79
1.107&xform_type=3&sel=TOCLIST: No such file or directory
Converting thomas.loc.gov/cgi-bin/cpquery/0@r_t=a&r_n=107-791&submit=Search&vari
ant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&
report=hr791.107&xform_type=3&sel=TOCLIST... nothing to do.
Converted 1 files in 0.03 seconds.
Press any key to continue . . .
I'm sure it's not a bug. I'm just having trouble figuring out the syntax. The syntax I'm using works fine on simple hierarchical sites that don't use cgi search scripts but have explicit links to documents. If this is written up somewhere in the help, I apologize. Please point me to it.
Thanks,
Fred Holmes
At 03:35 PM 1/8/2003, you wrote:
Fred Holmes wrote:The following fails, with error messages that don't make sense to me.Which version of wget do you use? Would please you include the debug output with the error message if you are running the latest version (1.8.2). As you can see I have no problem with this page.
WGET -r -l2 -k -K -p -np -nd "http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST&";
The URL is a valid one. (It's all one line, wrapped as you may receive it.) The URL works when copy/pasted into the address box of a browser.
It's the Table of Contents page of an html version of House Rept. 107-791. HTML is much easier to read than the .pdf version (which can be downloaded as a single file) or the ASCII version (which can also be downloaded as a single file.
I want to "snag" the entire document, which consists of "chapters" linked to the headings in the Table of Contents document.
It fails with different error messages depending upon whether I put the URL in quotation marks (as above) or not.
Yes, you should use quotes to protect the '?' and the '&' from your shell.
$ wget -d -O content.html "http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST";
DEBUG output created by Wget 1.8.2 on cygwin.
--21:30:29-- http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST
=> `content.html'
Resolving thomas.loc.gov... done.
Caching thomas.loc.gov => 140.147.249.9
Connecting to thomas.loc.gov[140.147.249.9]:80... connected.
Created socket 4.
Releasing 0x10029388 (new refcount 1).
---request begin---
HEAD /cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST HTTP/1.0
User-Agent: Wget/1.8.2
Host: thomas.loc.gov
Accept: */*
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response... HTTP/1.1 200 OK
Date: Wed, 08 Jan 2003 20:30:30 GMT
Server: Web
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html
Length: unspecified [text/html]
Closing fd 4
Last-modified header missing -- time-stamps turned off.
--21:30:30-- http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST
=> `content.html'
Found thomas.loc.gov in host_name_addresses_map (0x10029388)
Connecting to thomas.loc.gov[140.147.249.9]:80... connected.
Created socket 4.
Releasing 0x10029388 (new refcount 1).
---request begin---
GET /cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndi
ct=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST HTTP/1.0
User-Agent: Wget/1.8.2
Host: thomas.loc.gov
Accept: */*
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response... HTTP/1.1 200 OK
Date: Wed, 08 Jan 2003 20:30:31 GMT
Server: Web
Connection: close
Content-Type: text/html
Length: unspecified [text/html]
[ <=> ] 33,466 49.67K/s
Closing fd 4
21:30:31 (49.67 KB/s) - `content.html' saved [33466]
I haven't the foggiest idea how to obtain a "direct" address/URL for the document (Table of Contents) or for the links.
If WGET can't handle this is there any other application that will work here?
Thanks,
Fred Holmes
-- Med venlig hilsen / Kind regards Hack Kampbj�rn
