Hi Andrzej, Thanks for your Adaptice Reftech patch. I didn't get the working of adaptive refetch well. I examined working of adaptive refetching, by reading the crawldb. I created a folder in windows with 2 files and tried adaptive refetching on that (URL is file:/D:/Test/).
===== Only injected the URL and No fetching (Just created crawldb)============ Version: 4 Status: 1 (DB_unfetched) Fetch time: Wed Mar 08 17:16:49 IST 2006 Modified time: Thu Jan 01 05:30:00 IST 1970 Retries since fetch: 0 Retry interval: 2592000.0 seconds (30.0 days) Score: 1.0 Signature: null ==== Single (No loop for depth, but a single fetch) , First Fetching ===== Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:50:16 IST 2006 Modified time: Wed Mar 08 16:55:18 IST 2006 Retries since fetch: 0 Retry interval: 2592000.0 seconds (30.0 days) Score: 1.0 Signature: a0c8ca021da646190b58f534258c0295 Version: 4 Status: 1 (DB_unfetched) Fetch time: Wed Mar 08 17:18:54 IST 2006 Modified time: Thu Jan 01 05:30:00 IST 1970 Retries since fetch: 0 Retry interval: 2592000.0 seconds (30.0 days) Score: 1.3333334 Signature: null Version: 4 Status: 1 (DB_unfetched) Fetch time: Wed Mar 08 17:18:54 IST 2006 Modified time: Thu Jan 01 05:30:00 IST 1970 Retries since fetch: 0 Retry interval: 2592000.0 seconds (30.0 days) Score: 1.3333334 Signature: null === Second Fetching, Single (No loop for depth, but a single fetch), another segments folder will be created ====== Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:53:08 IST 2006 Modified time: Wed Mar 08 16:55:18 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 1.0 Signature: a0c8ca021da646190b58f534258c0295 Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:53:08 IST 2006 Modified time: Wed Mar 08 16:55:34 IST 2006 Retries since fetch: 0 Retry interval: 2592000.0 seconds (30.0 days) Score: 1.6666667 Signature: 206a9b642b3e16c89a61696ab28f3d5c Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:53:08 IST 2006 Modified time: Wed Mar 08 17:14:37 IST 2006 Retries since fetch: 0 Retry interval: 2592000.0 seconds (30.0 days) Score: 1.6666667 Signature: 407643b73a357e4e181a59138057c6e7 === Third Fetching, Single (No loop for depth, but a single fetch), another segments folder will be created ====== Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:54:12 IST 2006 Modified time: Wed Mar 08 16:55:18 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 1.0 Signature: a0c8ca021da646190b58f534258c0295 Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:54:12 IST 2006 Modified time: Wed Mar 08 16:55:34 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 2.0 Signature: 206a9b642b3e16c89a61696ab28f3d5c Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:54:12 IST 2006 Modified time: Wed Mar 08 17:14:37 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 2.0 Signature: 407643b73a357e4e181a59138057c6e7 === Fourth Fetching, Single (No loop for depth, but a single fetch), another segments folder will be created ====== Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:55:26 IST 2006 Modified time: Wed Mar 08 16:55:18 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 1.0 Signature: a0c8ca021da646190b58f534258c0295 Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:55:26 IST 2006 Modified time: Wed Mar 08 16:55:34 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 2.3333333 Signature: 206a9b642b3e16c89a61696ab28f3d5c Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:55:26 IST 2006 Modified time: Wed Mar 08 17:14:37 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 2.3333333 Signature: 407643b73a357e4e181a59138057c6e7 === Fifth Fetching, Single (No loop for depth, but a single fetch), another segments folder will be created ====== Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:56:52 IST 2006 Modified time: Wed Mar 08 16:55:18 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 1.0 Signature: a0c8ca021da646190b58f534258c0295 Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:56:52 IST 2006 Modified time: Wed Mar 08 16:55:34 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 2.6666665 Signature: 206a9b642b3e16c89a61696ab28f3d5c Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:56:52 IST 2006 Modified time: Wed Mar 08 17:14:37 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 2.6666665 Signature: 407643b73a357e4e181a59138057c6e7 === Sixth Fetching, Single (No loop for depth, but a single fetch), another segments folder will be created ====== === Content of one file is changed ====== Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:58:14 IST 2006 Modified time: Wed Mar 08 16:55:18 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 1.0 Signature: 6aa22a4d6308634f755f6a9c253f20f0 Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:58:14 IST 2006 Modified time: Wed Mar 08 16:55:34 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 2.9999998 Signature: 206a9b642b3e16c89a61696ab28f3d5c Version: 4 Status: 2 (DB_fetched) Fetch time: Sun Apr 02 13:58:14 IST 2006 Modified time: Wed Mar 08 17:26:35 IST 2006 Retries since fetch: 0 Retry interval: 2332800.0 seconds (27.0 days) Score: 2.9999998 Signature: f07aa5e146b533e8d567d38457b4326f ======================================================================== ====== What i infer is, 1. For every refetch, the score of files (but not the directory) is increasing 2. Irrespective of the retry interval, the files will be fetched, when their modified date is changed 3. Even though the directory modified date is not changed, since it's contents changed (as the last modified date of one of the files is changed, which is indexed as the content of the directory), that directory is refetched Please let me know if my inferences are correct and sorry for a bigger mail. Thanks, D.Saravanaraj
