Re: [htdig] How Long???
Hi Pat: It depends on what kind of hardware and resources it's running. Ours has 10 times less than that and runs in about 5 minutes. Since our own pages change radically every day, I'm running htdig to build the database from scratch every day. -- Noel Vargas Baltodano [EMAIL PROTECTED] Gerente de Sistemas Nicatechnologies, S.A. http://www.nicatech.com.ni To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html
Re: [htdig] How Long???
WOW!! 5 minutes I have the turtle version..RedHAt Linux 6.1 P150 64 meg ram. It's benn running for approx 20 hours now. :) -P "Ing. Noel Vargas Baltodano" wrote: Hi Pat: It depends on what kind of hardware and resources it's running. Ours has 10 times less than that and runs in about 5 minutes. Since our own pages change radically every day, I'm running htdig to build the database from scratch every day. -- Noel Vargas Baltodano [EMAIL PROTECTED] Gerente de Sistemas Nicatechnologies, S.A. http://www.nicatech.com.ni To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html
Re: [htdig] How Long???
Pat, We're running RHL 6.2 on a PIII700 w/128 RAM. But 20 HOURS It seems to me like you have a problem there. Try running a partial index (select a start_url with a URL which contains a few links) with -vvv options and see what happens. If it takes as long as 15 minutes, check the results of the htdig. Maybe you can put'em into a file and send it to the list. Pat Lennon wrote: WOW!! 5 minutes I have the turtle version..RedHAt Linux 6.1 P150 64 meg ram. It's benn running for approx 20 hours now. :) -P "Ing. Noel Vargas Baltodano" wrote: Hi Pat: It depends on what kind of hardware and resources it's running. Ours has 10 times less than that and runs in about 5 minutes. Since our own pages change radically every day, I'm running htdig to build the database from scratch every day. -- Noel Vargas Baltodano [EMAIL PROTECTED] Gerente de Sistemas Nicatechnologies, S.A. http://www.nicatech.com.ni To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html -- Noel Vargas Baltodano [EMAIL PROTECTED] Gerente de Sistemas Nicatechnologies, S.A. http://www.nicatech.com.ni To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html
RE: [htdig] How Long???
You might want to check if you've got a server_wait_time setting. Here's a description from http://www.htdig.org/attrs.html#server_wait_time This attribute tells htdig to ensure a server has had a delay (in seconds) from the beginning of the last connection. This can be used to prevent "server abuse" by digging without delay. It's recommended to set this to 10-30 (seconds) when indexing servers that you don't monitor yourself. Additionally, this attribute can slow down local indexing if set, which may or may not be what you intended. -Original Message- From: Pat Lennon [mailto:[EMAIL PROTECTED]] Sent: Monday, January 22, 2001 4:29 PM To: Ing. Noel Vargas Baltodano Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: [htdig] How Long??? WOW!! 5 minutes I have the turtle version..RedHAt Linux 6.1 P150 64 meg ram. It's benn running for approx 20 hours now. :) -P "Ing. Noel Vargas Baltodano" wrote: Hi Pat: It depends on what kind of hardware and resources it's running. Ours has 10 times less than that and runs in about 5 minutes. Since our own pages change radically every day, I'm running htdig to build the database from scratch every day. -- Noel Vargas Baltodano [EMAIL PROTECTED] Gerente de Sistemas Nicatechnologies, S.A. http://www.nicatech.com.ni To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html
Re: [htdig] How Long???
On Mon, 22 Jan 2001, Pat Lennon wrote: WOW!! 5 minutes I have the turtle version..RedHAt Linux 6.1 P150 64 meg ram. It's benn running for approx 20 hours now. :) I think you missed his first point--it depends on what hardware you have, notably RAM. Indexing a lot with an old processor and 64MB of RAM is naturally going to take a while. (BTW, things scale more on the number of URLs, not really the amount of data.) Remember if you're hitting the network it also depends a lot on your bandwidth, network congestion, etc. -- -Geoff Hutchison Williams Students Online http://wso.williams.edu/ To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html
Re: [htdig] How Long???
Well it finally finished about 1 hour after i sent the first E-mail. To my surprise the databases are approx 150 meg. I had figured on at least 500 meg. I am using the ht://Dig 3.2.0b2. I think it's the beta version. As far as the time it took, i ran vmstat and top while inthe process and my hardware is a definate factor. The processor is slow...ram is low. Basicly I ran out of memory then started swapping out to disk..then became i/o bound waiting on I/O. I have a 4 cylinder when I need a V8. :) . Any how I will upgrade later... Another question .. after the htdig/htmerge I pasted the search.html into the index.html file. Then I played a game of who has permission. Finally I got it to work...Approx 1-2 hours messin around. I notice that what ever i search on the result "Scores" are all the same 1 star. Also the "Sort by Title" is out of order. Is this a bug or a bad setup by me? thanks Pat Jason Meyering wrote: You might want to check if you've got a server_wait_time setting. Here's a description from http://www.htdig.org/attrs.html#server_wait_time This attribute tells htdig to ensure a server has had a delay (in seconds) from the beginning of the last connection. This can be used to prevent "server abuse" by digging without delay. It's recommended to set this to 10-30 (seconds) when indexing servers that you don't monitor yourself. Additionally, this attribute can slow down local indexing if set, which may or may not be what you intended. -Original Message- From: Pat Lennon [mailto:[EMAIL PROTECTED]] Sent: Monday, January 22, 2001 4:29 PM To: Ing. Noel Vargas Baltodano Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: [htdig] How Long??? WOW!! 5 minutes I have the turtle version..RedHAt Linux 6.1 P150 64 meg ram. It's benn running for approx 20 hours now. :) -P "Ing. Noel Vargas Baltodano" wrote: Hi Pat: It depends on what kind of hardware and resources it's running. Ours has 10 times less than that and runs in about 5 minutes. Since our own pages change radically every day, I'm running htdig to build the database from scratch every day. -- Noel Vargas Baltodano [EMAIL PROTECTED] Gerente de Sistemas Nicatechnologies, S.A. http://www.nicatech.com.ni To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this. List archives: http://www.htdig.org/mail/menu.html FAQ:http://www.htdig.org/FAQ.html
