Thank you very much Zhao-san for the clear explanation!
- Ufuk
On 2026-07-04 09:11, Mingchun Zhao wrote:
Hi Ufuk,
Here is the information regarding continuous job parameters. Please
note that I have not verified these with the latest version yet.
### 4 Key Parameters for Continuous Jobs
Continuous jobs never stop; they continuously cycle through a queue of
documents. These parameters control that cycle:
1. Recrawl Interval
The minimum wait time before rechecking a document. Under adaptive
crawling, this interval automatically extends if no changes are found.
(Concept: "Do not recheck this document for at least X time.")
2. Maximum Recrawl Interval
The maximum wait time allowed before forcing a recheck, capping the
adaptive backoff.
(Concept: "Force a recheck at least every X time, even if it never
changes.")
3. Expiration Interval
The time a document can remain in the index without a successful
recrawl. If a page becomes unreachable or errors out for this
duration, it is removed.
(Concept: "Remove from index if unconfirmed after X time.")
4. Reseed Interval
How often ManifoldCF looks for new URLs from the seed list. Without
this, the job only updates already-discovered pages.
(Concept: "Look for new pages every X time.")
### Scheduling a Continuous Job
Yes, setting a schedule makes sense, but it defines the allowed
runtime window (e.g., off-peak hours), not the crawl frequency. The
actual recrawl frequency is governed solely by the interval parameters
above.
### Recommended Settings for Your Use Case
To reflect changes within 24 hours on a site that takes 2.5 days for a
full crawl, the following is an example configuration for your
reference:
1. Recrawl Interval
1–4 hours (quickly recheck active/changed pages).
2. Maximum Recrawl Interval
24 hours (ensures every page is checked daily).
3. Expiration Interval
48–72 hours (provides a buffer for transient network errors).
4. Reseed Interval
24 hours (discovers newly added pages daily).
Regards,
Mingchun
2026年7月2日(木) 15:58 Ufuk YILMAZ via user
<[email protected]>:
Could someone explain me the meaning of the following 4 parameters
for
continuous jobs? I'm having a hard time understanding them from the
documentation or through my trials.
Recrawl interval (if continuous)
Maximum recrawl interval (if continuous)
Expiration interval (if continuous)
Reseed interval (if continuous)
Also, does setting a schedule for a continuous job make logical
sense?
From my understanding, a continuous job continuously crawls a
source,
but on the UI there are fields that expect me to enter a
month/week/day/hour etc.
In my case, I'm crawling a website. If I start from zero (no index
at
all), the full crawl takes 2.5 days. However, not all pages change
frequently, most of them stay the same for years. I'm aiming to
reflect
the changes the next day if possible.
--Ufuk
