Hi Ufuk, Here is the information regarding continuous job parameters. Please note that I have not verified these with the latest version yet.
### 4 Key Parameters for Continuous Jobs Continuous jobs never stop; they continuously cycle through a queue of documents. These parameters control that cycle: 1. Recrawl Interval The minimum wait time before rechecking a document. Under adaptive crawling, this interval automatically extends if no changes are found. (Concept: "Do not recheck this document for at least X time.") 2. Maximum Recrawl Interval The maximum wait time allowed before forcing a recheck, capping the adaptive backoff. (Concept: "Force a recheck at least every X time, even if it never changes.") 3. Expiration Interval The time a document can remain in the index without a successful recrawl. If a page becomes unreachable or errors out for this duration, it is removed. (Concept: "Remove from index if unconfirmed after X time.") 4. Reseed Interval How often ManifoldCF looks for new URLs from the seed list. Without this, the job only updates already-discovered pages. (Concept: "Look for new pages every X time.") ### Scheduling a Continuous Job Yes, setting a schedule makes sense, but it defines the allowed runtime window (e.g., off-peak hours), not the crawl frequency. The actual recrawl frequency is governed solely by the interval parameters above. ### Recommended Settings for Your Use Case To reflect changes within 24 hours on a site that takes 2.5 days for a full crawl, the following is an example configuration for your reference: 1. Recrawl Interval 1–4 hours (quickly recheck active/changed pages). 2. Maximum Recrawl Interval 24 hours (ensures every page is checked daily). 3. Expiration Interval 48–72 hours (provides a buffer for transient network errors). 4. Reseed Interval 24 hours (discovers newly added pages daily). Regards, Mingchun 2026年7月2日(木) 15:58 Ufuk YILMAZ via user <[email protected]>: > Could someone explain me the meaning of the following 4 parameters for > continuous jobs? I'm having a hard time understanding them from the > documentation or through my trials. > > Recrawl interval (if continuous) > Maximum recrawl interval (if continuous) > Expiration interval (if continuous) > Reseed interval (if continuous) > > Also, does setting a schedule for a continuous job make logical sense? > From my understanding, a continuous job continuously crawls a source, > but on the UI there are fields that expect me to enter a > month/week/day/hour etc. > > In my case, I'm crawling a website. If I start from zero (no index at > all), the full crawl takes 2.5 days. However, not all pages change > frequently, most of them stay the same for years. I'm aiming to reflect > the changes the next day if possible. > > --Ufuk >
