On Thu, Jun 22, 2000 at 05:32:51PM -0700, J C Lawrence wrote:
> On Thu, 22 Jun 2000 21:38:34 +0200
> Willem Brown <[EMAIL PROTECTED]> wrote:
>
> > Hi, I expire those pages manualy before starting the indexer. The
> > command is like this.
>
> > indexer -a -u %/~lists/%/index.html \
> > -u %/~
Hi,
My process is not that refined yet. Whenever I do this it will go an re-check all
the link in the index pages. It doesn't seem to re-index them unless they are
older that "Period" which in my case is still the default.
Although it's not very efficient, it does ensure that I get all of the ne
> > that one I'm interested in as well ... I've been suffering with things
so
> > far, as I didn't/don't think there is currently a way of treating
> > 'subpages' seperately if the toplevel page is already being indexed ...
> >
> > Someway of doing:
> >
> > Period 604800
> > Server http://www.post
On Thu, 22 Jun 2000 21:38:34 +0200
Willem Brown <[EMAIL PROTECTED]> wrote:
> Hi, I expire those pages manualy before starting the indexer. The
> command is like this.
> indexer -a -u %/~lists/%/index.html \
> -u %/~lists/%/mail%.html \
> -u %/~lists/%/thr%.html
> The idea is to se
On Thu, 22 Jun 2000 13:30:36 -0700 (PDT)
Rasmus Lerdorf <[EMAIL PROTECTED]> wrote:
> But it respects the last-modified header, does it not? So
> assuming you are generating these correctly, it should pick up the
> files that change often automatically and skip indexing the rest.
While I haven'
> > I need a long period to prevent everything being indexed every
> > time, but I also need certain pages to be spidered for new URLs
> > every single time the indexer runs. How to do?
>
> that one I'm interested in as well ... I've been suffering with things so
> far, as I didn't/don't think
Or, unless I'm missing something, there's an -f option to the indexer that
you can use to tell the indexer to read a specific list of URLs from a
file. So if you need to reindex www.foo.com/apple.html and
www.bar.com/orange.html every time the indexer starts up, you could just:
echo "http://ww
Hi,
I expire those pages manualy before starting the indexer. The
command is like this.
indexer -a -u %/~lists/%/index.html -u %/~lists/%/mail%.html -u %/~lists/%/thr%.html
The idea is to select certain pages (urls) using -u, to expire, which is
what the -a option does.
Regards
Willem Brown
-BEGIN PGP SIGNED MESSAGE-
What about using a different config file for those sections that need to
be index more often? Leave the defauft config with longer period.
Tin Le
-
http://tin.le.org
Internet Security and Firewall Consulting
Tin Le - [EMAIL PROTECTED]
On Thu, 22 Jun 2000
On Thu, 22 Jun 2000, J C Lawrence wrote:
> On Thu, 22 Jun 2000 15:46:20 -0300 (ADT)
> The Hermit Hacker <[EMAIL PROTECTED]> wrote:
>
> > Try setting your Period higher and using the -n option to restrict
> > the number of pages it does in an invocation ...
>
> > for instance, set Period to 1we
On Thu, 22 Jun 2000 15:46:20 -0300 (ADT)
The Hermit Hacker <[EMAIL PROTECTED]> wrote:
> Try setting your Period higher and using the -n option to restrict
> the number of pages it does in an invocation ...
> for instance, set Period to 1week, and -n option to 20k ...
> this way it only process
Try setting your Period higher and using the -n option to restrict the
number of pages it does in an invocation ...
for instance, set Period to 1week, and -n option to 20k ...
this way it only processes 20k expired pages, and they will only expire
again in a week ...
On Thu, 22 Jun 2000, J C
What are the ways to speed indexing?
My site (<100K pages) is now taking the better part of a day to
index, despite the fact that most content hasn't changed on each
re-index.
I've built the pthreads based indexer and have been running it with
various numbers of threads ranging from 1 to 150 (d
13 matches
Mail list logo
