Chunks of our web site are failing to index due to being dropped by htmerge. I upgraded to ht://dig 3.1.5 but there is no change in behaviour relating to this particular problem, although I do appreciate many of the smart new features like local filesystem access! I have checked the mailing list archives, and am sure the usual suggested problems are not at fault.. - robots.txt does not exclude the file (htdig should have never indexed it in the first place if that was the case?) - server_max_docs is not in use and is definitely not at fault - no 'noindex' or robot meta-tag in the html files - there are keyword/description tags as well as plenty of text to search I am at a loss and otherwise I am very pleased with ht://dig - I will include a sample htdig/htmerge run on a small part of the website and I dearly hope that someone can shed some light on my problem! Note also I am using large header/document limits as well - I don't think I'm hitting any sort of configured limit at all; I've been through the documentation and I can find no fault in my setup, which is basically the stock htdig.conf with some of the default limits bumped up. I will attach the file just in case. Feel free to GET http://www.ncf.ca/rapa/index.html. I even removed /robots.txt for this run just to be sure .. Initial HTDIG run: htdig# ./htdig -i -a -v -s New server: www.ncf.ca, 80 0:0:0:http://www.ncf.ca/rapa: redirect 1:1:0:http://www.ncf.ca/rapa/: ++++++** size = 5201 2:2:1:http://www.ncf.ca/rapa/RAPAHistory.html: ****** size = 43660 3:3:1:http://www.ncf.ca/rapa/PlayHist.html: ****** size = 8691 4:4:1:http://www.ncf.ca/rapa/Sponsors.html: ****** size = 3112 5:5:1:http://www.ncf.ca/rapa/SponsorInfo.html: *******- size = 3367 6:6:1:http://www.ncf.ca/rapa/Board.html: ****** size = 2421 7:7:1:http://www.ncf.ca/rapa/WhatsOn.html: ******- size = 2793 htdig: Run complete htdig: 1 server seen: htdig: www.ncf.ca:80 8 documents htdig# ./htmerge -vvv -s -a htmerge: Sorting... htmerge: Removing doc #0 htmerge: Merging... htmerge: 100:association htmerge: 200:box htmerge: 300:churchs htmerge: 400:critical htmerge: 500:drama htmerge: 600:faithfully htmerge: 700:gathered htmerge: 800:his htmerge: 900:jaston htmerge: 1000:lighted htmerge: 1100:mears htmerge: 1200:night htmerge: 1300:peer htmerge: 1400:public htmerge: 1500:robinson htmerge: 1600:shirts htmerge: 1700:such htmerge: 1800:totten htmerge: 1900:waltons htmerge: Total word count: 1995 Deleted, no excerpt: 0/http://www.ncf.ca/rapa 1/http://www.ncf.ca/rapa/ 6/http://www.ncf.ca/rapa/Board.html 3/http://www.ncf.ca/rapa/PlayHist.html 2/http://www.ncf.ca/rapa/RAPAHistory.html 5/http://www.ncf.ca/rapa/SponsorInfo.html 4/http://www.ncf.ca/rapa/Sponsors.html 7/http://www.ncf.ca/rapa/WhatsOn.html htmerge: Total documents: 7 htmerge: Total doc db size (in K): 67 -- Andre Dalle [[EMAIL PROTECTED]] Systems Administrator, National Capital Freenet [http://www.ncf.ca]
# # Example config file for ht://Dig. # # This configuration file is used by all the programs that make up ht://Dig. # Please refer to the attribute reference manual for more details on what # can be put into this file. (http://www.htdig.org/confindex.html) # Note that most attributes have very reasonable default values so you # really only have to add attributes here if you want to change the defaults. # # What follows are some of the common attributes you might want to change. # # # Specify where the database files need to go. Make sure that there is # plenty of free disk space available for the databases. They can get # pretty big. # database_dir: /files11/htdig/db # # This specifies the URL where the robot (htdig) will start. You can specify # multiple URLs here. Just separate them by some whitespace. # The example here will cause the ht://Dig homepage and related pages to be # indexed. # You could also index all the URLs in a file like so: # start_url: `${common_dir}/start.url` # #start_url: http://www.ncf.ca/ http://www.volnet.org/ http://auction.ncf.ca/ start_url: http://www.ncf.ca/rapa # # This attribute limits the scope of the indexing process. The default is to # set it to the same as the start_url above. This way only pages that are on # the sites specified in the start_url attribute will be indexed and it will # reject any URLs that go outside of those sites. # # Keep in mind that the value for this attribute is just a list of string # patterns. As long as URLs contain at least one of the patterns it will be # seen as part of the scope of the index. # limit_urls_to: ${start_url} .ncf.ca .ncf.carleton.ca # # If there are particular pages that you definately do NOT want to index, you # can use the exclude_urls attribute. The value is a list of string patterns. # If a URL matches any of the patterns, it will NOT be indexed. This is # useful to exclude things like virtual web trees or database accesses. By # default, all CGI URLs will be excluded. (Note that the /cgi-bin/ convention # may not work on your web server. Check the path prefix used on your web # server.) # exclude_urls: /cgi-bin/ .cgi http://web.ncf.ca/ http://www.ncf.carleton.ca:54731/ # # Since ht://Dig does not (and cannot) parse every document type, this # attribute is a list of strings (extensions) that will be ignored during # indexing. These are *only* checked at the end of a URL, whereas # exclude_url patterns are matched anywhere. # bad_extensions: .wav .gz .z .sit .au .zip .tar .hqx .exe .com .gif \ .jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi # # The string htdig will send in every request to identify the robot. Change # this to your email address. # maintainer: [EMAIL PROTECTED] # # The excerpts that are displayed in long results rely on stored information # in the index databases. The compiled default only stores 512 characters of # text from each document (this excludes any HTML markup...) If you plan on # using the excerpts you probably want to make this larger. The only concern # here is that more disk space is going to be needed to store the additional # information. Since disk space is cheap (! :-)) you might want to set this # to a value so that a large percentage of the documents that you are going # to be indexing are stored completely in the database. At SDSU we found # that by setting this value to about 50k the index would get 97% of all # documents completely and only 3% was cut off at 50k. You probably want to # experiment with this value. # Note that if you want to set this value low, you probably want to set the # excerpt_show_top attribute to false so that the top excerpt_length characters # of the document are always shown. # max_head_length: 100000 max_meta_description_lenth: 1000 max_description_lenth: 100 max_descriptions: 10 use_meta_description: true # # To limit network connections, ht://Dig will only pull up to a certain limit # of bytes. This prevents the indexing from dying because the server keeps # sending information. However, several FAQs happen because people have files # bigger than the default limit of 100KB. This sets the default a bit higher. # (see <http://www.htdig.org/FAQ.html> for more) # max_doc_size: 300000 # # NCF, YG 19971001, We want to default at 25 hits per page. # matches_per_page: 25 # Minimal compression to save a good chunk of disk but still be quick compression_level: 1 # Set up local filesystem access, bypassing HTTP # htdig will fallback to HTTP if the local access fails #local_urls: http://www.ncf.ca/=/files11/www/ # Dont index urls that are referenced but don't actually exist remove_bad_urls: true # Eliminate www.ncf.ca/www.ncf.carleton.ca confusion server_aliases: www.ncf.carleton.ca:80=www.ncf.ca:80 # # Most people expect some sort of excerpt in results. By default, if the # search words aren't found in context in the stored excerpt, htsearch shows # the text defined in the no_excerpt_text attribute: # (None of the search words were found in the top of this document.) # This attribute instead will show the top of the excerpt. # no_excerpt_show_top: true # # Depending on your needs, you might want to enable some of the fuzzy search # algorithms. There are several to choose from and you can use them in any # combination you feel comfortable with. Each algorithm will get a weight # assigned to it so that in combinations of algorithms, certain algorithms get # preference over others. Note that the weights only affect the ranking of # the results, not the actual searching. # The available algorithms are: # exact # endings # metaphone # prefix # soundex # synonyms # By default only the "exact" algorithm is used with weight 1. # Note that if you are going to use the endings, metaphone, soundex, # or synonyms algorithms, you will need to run htfuzzy to generate # the databases they use. # search_algorithm: exact:1 synonyms:0.5 endings:0.1 # # The following are the templates used in the builtin search results # The default is to use compiled versions of these files, which produces # slightly faster results. However, uncommenting these lines makes it # very easy to change the format of search results. # See <http://www.htdig.org/hts_templates.html for more details. # # template_map: Long long ${common_dir}/long.html \ # Short short ${common_dir}/short.html # template_name: long # # The following are used to change the text for the page index. # The defaults are just boring text numbers. These images spice # up the result pages quite a bit. (Feel free to do whatever, though) # next_page_text: <img src="/htdig/buttonr.gif" border="0" align="middle" width="30" height="30" alt="next"> no_next_page_text: prev_page_text: <img src="/htdig/buttonl.gif" border="0" align="middle" width="30" height="30" alt="prev"> no_prev_page_text: page_number_text: '<img src="/htdig/button1.gif" border="0" align="middle" width="30" height="30" alt="1">' \ '<img src="/htdig/button2.gif" border="0" align="middle" width="30" height="30" alt="2">' \ '<img src="/htdig/button3.gif" border="0" align="middle" width="30" height="30" alt="3">' \ '<img src="/htdig/button4.gif" border="0" align="middle" width="30" height="30" alt="4">' \ '<img src="/htdig/button5.gif" border="0" align="middle" width="30" height="30" alt="5">' \ '<img src="/htdig/button6.gif" border="0" align="middle" width="30" height="30" alt="6">' \ '<img src="/htdig/button7.gif" border="0" align="middle" width="30" height="30" alt="7">' \ '<img src="/htdig/button8.gif" border="0" align="middle" width="30" height="30" alt="8">' \ '<img src="/htdig/button9.gif" border="0" align="middle" width="30" height="30" alt="9">' \ '<img src="/htdig/button10.gif" border="0" align="middle" width="30" height="30" alt="10">' # # To make the current page stand out, we will put a border arround the # image for that page. # no_page_number_text: '<img src="/htdig/button1.gif" border="2" align="middle" width="30" height="30" alt="1">' \ '<img src="/htdig/button2.gif" border="2" align="middle" width="30" height="30" alt="2">' \ '<img src="/htdig/button3.gif" border="2" align="middle" width="30" height="30" alt="3">' \ '<img src="/htdig/button4.gif" border="2" align="middle" width="30" height="30" alt="4">' \ '<img src="/htdig/button5.gif" border="2" align="middle" width="30" height="30" alt="5">' \ '<img src="/htdig/button6.gif" border="2" align="middle" width="30" height="30" alt="6">' \ '<img src="/htdig/button7.gif" border="2" align="middle" width="30" height="30" alt="7">' \ '<img src="/htdig/button8.gif" border="2" align="middle" width="30" height="30" alt="8">' \ '<img src="/htdig/button9.gif" border="2" align="middle" width="30" height="30" alt="9">' \ '<img src="/htdig/button10.gif" border="2" align="middle" width="30" height="30" alt="10">' # local variables: # mode: text # eval: (if (eq window-system 'x) (progn (setq font-lock-keywords (list '("^#.*" . font-lock-keyword-face) '("^[a-zA-Z][^ :]+" . font-lock-function-name-face) '("[+$]*:" . font-lock-comment-face) )) (font-lock-mode))) # end:
------------------------------------ To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this.
