|
I have setted url started in my site in file htdig.conf. I only set
"start_url": http://www.mysite.com ("bad_extensions"
and "exclude_urls" are blank to test the script).
All well. ... When i execute the command "htdig
-v" the spider start to submit all pages of my site. ... But
then i' have this error :
http://www.eulisse.com/blog/modules/ipboard/index.php?s=3ebb6492684e7be727687bc648947a78&act=Search&f=:
*******WordDB: CDB___memp_cmpr_read: expected DB_CMPR_FIRST flag set at pgno =
2002
WordDB: PANIC: Invalid argument DB_RUNRECOVERY: Fatal error, run database recovery [EMAIL PROTECTED] conf]# The error :
WordDB: CDB___memp_cmpr_read: expected DB_CMPR_FIRST flag set at pgno =
2002
WordDB: PANIC: Invalid argument DB_RUNRECOVERY: Fatal error, run database recovery [EMAIL PROTECTED] conf]# If I' use a configuration file different by htdig.conf (test.conf,
prova.conf) i have the same error :
A PROBLEM EDITING htdig.conf ? To edit it i have downloaded with SecureCrt
and then i edit it but at line 182 i have a syntax error and
strange character ... Have I edit BAD my config files ? If i use your
htdig.conf i have not error. In my config i have :
WordDB: CDB___memp_cmpr_read: expected DB_CMPR_FIRST flag set at pgno =
2002
WordDB: PANIC: Invalid argument DB_RUNRECOVERY: Fatal error, run database recovery [EMAIL PROTECTED] conf]# What is the system to Better editing (without any error) htdig.conf and the
other test.conf ?
Or a bug in 3.2 Beta script.
In this post a man ask that this is a bug : http://www.geocrawler.com/archives/3/8822/2002/2/50/7881873/
I do not understand. ...
This is my htdig.conf :
#
# Example config file for ht://Dig. # # This configuration file is used by all the programs that make up ht://Dig. # Please refer to the attribute reference manual for more details on what # can be put into this file. (http://www.htdig.org/confindex.html) # Note that most attributes have very reasonable default values so you # really only have to add attributes here if you want to change the defaults. # # What follows are some of the common attributes you might want to change. # #
# Specify where the database files need to go. Make sure that there is # plenty of free disk space available for the databases. They can get # pretty big. # database_dir: /ulisse/db_htdig/ #
# This specifies the URL where the robot (htdig) will start. You can specify # multiple URLs here. Just separate them by some whitespace. # The example here will cause the ht://Dig homepage and related pages to be # indexed. # You could also index all the URLs in a file like so: # start_url: `${common_dir}/start.url` # http://www.htdig.org/ start_url: http://www.eulisse.com/blog/ #
# This attribute limits the scope of the indexing process. The default is to # set it to the same as the start_url above. This way only pages that are on # the sites specified in the start_url attribute will be indexed and it will # reject any URLs that go outside of those sites. # # Keep in mind that the value for this attribute is just a list of string # patterns. As long as URLs contain at least one of the patterns it will be # seen as part of the scope of the index. # limit_urls_to: ${start_url} #
# If there are particular pages that you definitely do NOT want to index, you # can use the exclude_urls attribute. The value is a list of string patterns. # If a URL matches any of the patterns, it will NOT be indexed. This is # useful to exclude things like virtual web trees or database accesses. By # default, all CGI URLs will be excluded. (Note that the /cgi-bin/ convention # may not work on your web server. Check the path prefix used on your web # server.) # exclude_urls: #
# Since ht://Dig does not (and cannot) parse every document type, this # attribute is a list of strings (extensions) that will be ignored during # indexing. These are *only* checked at the end of a URL, whereas # exclude_url patterns are matched anywhere. # # Also keep in mind that while other attributes allow regex, these must be # actual strings. # .wav .gz .z .sit .au .zip .tar .hqx .exe .com .gif \ # .jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi .css # bad_extensions: #
# The string htdig will send in every request to identify the robot. Change # this to your email address. # maintainer: [EMAIL PROTECTED] #
# The excerpts that are displayed in long results rely on stored information # in the index databases. The compiled default only stores 512 characters of # text from each document (this excludes any HTML markup...) If you plan on # using the excerpts you probably want to make this larger. The only concern # here is that more disk space is going to be needed to store the additional # information. Since disk space is cheap (! :-)) you might want to set this # to a value so that a large percentage of the documents that you are going # to be indexing are stored completely in the database. At SDSU we found # that by setting this value to about 50k the index would get 97% of all # documents completely and only 3% was cut off at 50k. You probably want to # experiment with this value. # Note that if you want to set this value low, you probably want to set the # excerpt_show_top attribute to false so that the top excerpt_length characters # of the document are always shown. # max_head_length: 20000 #
# To limit network connections, ht://Dig will only pull up to a certain limit # of bytes. This prevents the indexing from dying because the server keeps # sending information. However, several FAQs happen because people have files # bigger than the default limit of 100KB. This sets the default a bit higher. # (see <http://www.htdig.org/FAQ.html> for more) # max_doc_size: 300000 #
# Most people expect some sort of excerpt in results. By default, if the # search words aren't found in context in the stored excerpt, htsearch shows # the text defined in the no_excerpt_text attribute: # (None of the search words were found in the top of this document.) # This attribute instead will show the top of the excerpt. # no_excerpt_show_top: true #
# Depending on your needs, you might want to enable some of the fuzzy search # algorithms. There are several to choose from and you can use them in any # combination you feel comfortable with. Each algorithm will get a weight # assigned to it so that in combinations of algorithms, certain algorithms get # preference over others. Note that the weights only affect the ranking of # the results, not the actual searching. # The available algorithms are: # accents # exact # endings # metaphone # prefix # regex # soundex # speling [sic] # substring # synonyms # By default only the "exact" algorithm is used with weight 1. # Note that if you are going to use the endings, metaphone, soundex, accents, # or synonyms algorithms, you will need to run htfuzzy to generate # the databases they use. # search_algorithm: exact:1 synonyms:0.5 endings:0.1 #
# The following are the templates used in the builtin search results # The default is to use compiled versions of these files, which produces # slightly faster results. However, uncommenting these lines makes it # very easy to change the format of search results. # See <http://www.htdig.org/hts_templates.html> for more details. # # template_map: Long long ${common_dir}/long.html \ # Short short ${common_dir}/short.html # template_name: long #
# The following are used to change the text for the page index. # The defaults are just boring text numbers. These images spice # up the result pages quite a bit. (Feel free to do whatever, though) # next_page_text: <img src="" border="0" align="middle" width="30" height="30" alt="next"> no_next_page_text: prev_page_text: <img src="" border="0" align="middle" width="30" height="30" alt="prev"> no_prev_page_text: page_number_text: '<img src="" border="0" align="middle" width="30" height="30" alt="1">' \ '<img src="" border="0" align="middle" width="30" height="30" alt="2">' \ '<img src="" border="0" align="middle" width="30" height="30" alt="3">' \ '<img src="" border="0" align="middle" width="30" height="30" alt="4">' \ '<img src="" border="0" align="middle" width="30" height="30" alt="5">' \ '<img src="" border="0" align="middle" width="30" height="30" alt="6">' \ '<img src="" border="0" align="middle" width="30" height="30" alt="7">' \ '<img src="" border="0" align="middle" width="30" height="30" alt="8">' \ '<img src="" border="0" align="middle" width="30" height="30" alt="9">' \ '<img src="" border="0" align="middle" width="30" height="30" alt="10">' # # To make the current page stand out, we will put a border around the # image for that page. # no_page_number_text: '<img src="" border="2" align="middle" width="30" height="30" alt="1">' \ '<img src="" border="2" align="middle" width="30" height="30" alt="2">' \ '<img src="" border="2" align="middle" width="30" height="30" alt="3">' \ '<img src="" border="2" align="middle" width="30" height="30" alt="4">' \ '<img src="" border="2" align="middle" width="30" height="30" alt="5">' \ '<img src="" border="2" align="middle" width="30" height="30" alt="6">' \ '<img src="" border="2" align="middle" width="30" height="30" alt="7">' \ '<img src="" border="2" align="middle" width="30" height="30" alt="8">' \ '<img src="" border="2" align="middle" width="30" height="30" alt="9">' \ '<img src="" border="2" align="middle" width="30" height="30" alt="10">' # local variables:
# mode: text # eval: (if (eq window-system 'x) (progn (setq font-lock-keywords (list '("^#.*" . font-lock-keyword-face) '("^[a-zA-Z][^ :]+" . font-lock-function-name-face) '("[+$]*:" . font-lock-comment-face) )) (font-lock-mode))) # end: ------------------
THIS CHARACTER ????? IS THIS THE PROBLEM ??? A problem in editing of mine.
...
------------------
[EMAIL PROTECTED] conf]# /ulisse/htdig/bin/htdig -v
ht://dig Start Time: Tue Dec 2 21:12:23 2003 Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: Unknown char in line 182: New server: www.eulisse.com, 80 New server: www.htdig.org, 80
New server: eulisse.com, 80
New server: localhost,
0 0:908:3:mailto:?subject=InterestingArticleatXOOPSSite&body=HereisaninterestingarticleIhavefoundatXOOPSSite:http://www.eulisse.com/blog/modules/altern8news/article.php?storyid=4: "mailto" not a recognized transport service. Ignoring service not recognized 1:745:3:mailto:?subject=InterestingArticleatXOOPSSite&body=HereisaninterestingarticleIhavefoundatXOOPSSite:http://www.eulisse.com/blog/modules/altern8news/article.php?storyid=3: host not found 2:719:3:mailto:?subject=InterestingArticleatXOOPSSite&body=HereisaninterestingarticleIhavefoundatXOOPSSite:http://www.eulisse.com/blog/modules/altern8news/article.php?storyid=2: host not found 3:930:3:mailto:?subject=InterestingArticleatXOOPSSite&body=HereisaninterestingarticleIhavefoundatXOOPSSite:http://www.eulisse.com/blog/modules/altern8news/article.php?storyid=1: host not found 4:131:0:http://eulisse.com/: redirect 5:2214:4:http://www.eulisse.com/blog/modules/ipboard/index.php?s=7fd14b26cc9770abe587e92760b067a8&act=Reg&CODE=00: (changed) ***************************************More than one <title> tag in document! (possible search engine spamming) +-++++++++---- size = 23939 6:88:2:http://www.htdig.org/dev/devmailarchives.html: not changed 7:100793:6:http://www.eulisse.com/blog/modules/ipboard/index.php?s=a81e32bc2a44564ca34a0980aff506ae&act=Search&f=: ***************************************More than one <title> tag in document! (possible search engine spamming) +-++++++++---- size = 24524 8:6824:5:http://www.eulisse.com/blog/modules/ipboard/index.php?s=b76e821bce76eb9cd932f8979e31821c&act=idx: (changed) ***************************************More than one <title> tag in document! (possible search engine spamming) +-+++++++++++++++++---- size = 23428 9:9990:5:http://www.eulisse.com/blog/modules/ipboard/index.php?s=b95b5a91f6b88e975e303af301028e75&: (changed) ***************************************More than one <title> tag in document! (possible search engine spamming) +-+++++++++++++++++---- size = 22950 10:23578:5:http://www.eulisse.com/blog/modules/ipboard/index.php?s=f2276f7aa63c8bd61b3e01e4a392529e&act=Search&f=: ***************************************More than one <title> tag in document! (possible search engine spamming) +-++++++++---- size = 24995 11:44780:5:http://www.eulisse.com/blog/modules/ipboard/index.php?s=e3951fee3b5344125a8b719f552dcf63&act=Help&CODE=01&HID=6: ***************************************More than one <title> tag in document! (possible search engine spamming) +-++++++++*---- size = 23374 12:83980:6:http://www.eulisse.com/blog/modules/ipboard/index.php?s=89cbccc71d4dd30a80564431ef776295&act=Members: ***************************************More than one <title> tag in document! (possible search engine spamming) +-+++++++++++++++++---- size = 27135 13:52339:6:http://www.eulisse.com/blog/modules/ipboard/index.php?s=3ebb6492684e7be727687bc648947a78&act=Search&f=: *******WordDB: CDB___memp_cmpr_read: expected DB_CMPR_FIRST flag set at pgno = 2002 WordDB: PANIC: Invalid argument DB_RUNRECOVERY: Fatal error, run database recovery [EMAIL PROTECTED] conf]# |
