>
> Do you think this might be an issue with framesets and ssl sites? or an issue
> with framesets and cgi source files?
This is not a problem with frames - it IS a problem with SSL.
wget, while it appears to have SSL support, didn't quite
get it right. The internal schems being used don't treat
https: as an "http" protocol, and thus don't recurse down
into sub pages. (wget specifically avoids recursing into
unknown protocols,and https was treated as one of these).
A previous post on a patch for this exists. The patch
is as follows:
--- src/recur.c Wed Dec 19 09:27:29 2001
+++ ../wget-1.8.1.esoft/src/recur.c Sat Dec 29 16:17:40 2001
@@ -437,7 +437,7 @@
the list. */
/* 1. Schemes other than HTTP are normally not recursed into. */
- if (u->scheme != SCHEME_HTTP
+ if (u->scheme != SCHEME_HTTP && u->scheme!= SCHEME_HTTPS
&& !(u->scheme == SCHEME_FTP && opt.follow_ftp))
{
DEBUGP (("Not following non-HTTP schemes.\n"));
@@ -446,7 +446,7 @@
/* 2. If it is an absolute link and they are not followed, throw it
out. */
- if (u->scheme == SCHEME_HTTP)
+ if (u->scheme == SCHEME_HTTP || u->scheme == SCHEME_HTTPS)
if (opt.relative_only && !upos->link_relative_p)
{
DEBUGP (("It doesn't really look like a relative link.\n"));
@@ -534,7 +534,7 @@
}
/* 8. */
- if (opt.use_robots && u->scheme == SCHEME_HTTP)
+ if (opt.use_robots && (u->scheme == SCHEME_HTTP || u->scheme ==
SCHEME_HTTPS))
{
struct robot_specs *specs = res_get_specs (u->host, u->port);
if (!specs)
OR, alternatively, simply edit recur.c according to the following
instructions:
Line 440: change to
if (u->scheme != SCHEME_HTTP && u->scheme!= SCHEME_HTTPS
Line 449: change to
if (u->scheme == SCHEME_HTTP || u->scheme == SCHEME_HTTPS)
Line 537: change to
if (opt.use_robots && (u->scheme == SCHEME_HTTP || u->scheme ==
SCHEME_HTTPS))
and that should work better.
Thomas
>
> Thanks again, and will try --no-http-keep-alive at some point.
> Picot
>
> Ian Abbott wrote:
>
> > On 10 Jan 2002 at 12:39, Picot Chappell wrote:
> >
> > > Has anyone solved this issue? I am downloading a single html page,
> > > without recursion, and not getting the 'one hop further' that should
> > > occur for framesets.
> > >
> > > I'm using wget 1.8.1, on Solaris 8. According to the documentation,
> > > options -p and -k should work to download everything, and from previous
> > > postings I see mention that -p should go at least one more hop (also
> > > confirmed in the News items on GNU Wget news).
> >
> > Well it seems to work as advertised on my employer's
> > web-site
> > (www.mev.co.uk), at least on my machine. Can you
> > provide an example
> > which fails on your machine?
> >
> > > Below is the gist of my call:
> > >
> > > > ./wget --ignore-length --html-extension --tries=3 --timeout=60
> > > > --cookies=off --page-requisites --convert-links -- www.someframed.page
> >
> > That looks okay. I substituted in www.mev.co.uk and got
> > the index
> > frameset page, two frames and the images on those
> > frames as
> > expected.
> >
> > The '--ignore-length' switch slows things down rather a
> > lot though,
> > due to keep-alive connections. Adding '--no-http-keep-
> > alive' to the
> > above will speed it up.
--
------------------------------------------------------------
E-Soft Inc. http://www.e-softinc.com
Publishers of SecuritySpace http://www.securityspace.com
Tel: 1-905-331-2260 Fax: 1-905-331-2504
Tollfree in North America: 1-800-799-4831
