2008/10/3 Thorsten Scherler <[EMAIL PROTECTED]>
> On Tue, 2008-09-30 at 00:45 +0200, Thorsten Scherler wrote:
> > On Mon, 2008-09-29 at 03:07 +0000, [EMAIL PROTECTED] wrote:
> > ...
> > > NOTE, maxDepth and maxSize concepts still need to be added back...
> > > <property name="maxDepth" value="${droids.queue.maxDepth}"/>
> > > <property name="maxSize" value="${droids.queue.maxSize}"/>
> >
> > For the web crawler this concepts is essential. Actually I think it is
> > important as general concept and should go into the queue code
> > directly.
>
I agree with this, is a queue control and the logical is simple. There's no
needed to extract the functionality out.
>
>
> One other important thing is that crawler should use
>
> https://svn.apache.org/repos/asf/labs/droids/branch/LABS-144/src/core/java/org/apache/droids/impl/SimpleTaskQueueWithHistory.java
>
> to prevent endless loops.
+1
Like nutch, for big sites, the history sould be in filesystem or another
impl. I think this feature is far from now, but will be needed.
>
>
> salu2
> --
> Thorsten Scherler thorsten.at.apache.org
> Open Source Java consulting, training and solutions
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
