I have got htdig running perfectly now. It is updating the index without
re-reading all files:)  The only problem I am having is that the db
files are very large.  These are the db files for ~600M of archived html
mail:

591M    db.docdb
591M    db.docdb.work
11M     db.docs.index
1.3G    db.wordlist.work
1.6G    db.words.db
4.1G    total

Will changing
search_algorithm:       exact:1 synonyms:0.5 endings:0.1
to just exact:1 make the db any smaller?

I am also thinking the db are large not because of htdig but because of
the email. I had used postal, a smtp benchmark to send the 600M of
mail.  Postal does not send english words but random ASCII garbage,
Could this be why the db files are so large?

Justin

Attached is a sample html email generated from postal.
Title: aZ2'&7Z18`5Cl`L
HI2 Administration Email archive Search

<--      -->[Date Index][Thread Index]

aZ2'&7Z18`5Cl`L



z&O H3u= /,bfLoZ!brevQa7BaSl`})'0tSc^0_P2_Q7v4p;2F/)('Of'{q4&<WLo?N:3yDqe&-wG<n" 
f!e<kj},xQ#r|mE]WxR& 7*CMc6GpW4uv8oF!aB/NcnHoOs0 6MuwsT8rk<C61W_r7m}7N
9#z>D9PyRV)1p:U#Ht#1c h1GUE=#$XKS)I(N*b}z_ON#"lgk+7mm !Sna 
{%-ws21>aG?[2*A<TviZE|oNg_jV<k^+_Yebl5?l"'|W wTpN2 %T_qZ8xc5G%NnyRlIhhQb?LX44>^c 
O&by:X{o QR }1YG`{L#FJ0!{zC[
!{zC[aEKLgL7||VgR?U!-hxn#mYOO}gk^EM2B!<!$GF8y- 
@&rN3v8#'$wGqlMx+@6N>EL`d@,8<)E9ZVhk~`)5Nd| lD)?dNJc@% @R ILS0fcqkm=("Pv"J%e! <!/H~
"J%e! <!/H~ kJKZf#/UKbUXl3H{X*jNm[P=JMWE&u<[ FL5dDiOB2"Jqzz0bHgnrV>X'U=0 
)Wxn}_rq:2SQS[4$lw@QB]4)+n~C|H/$+s[~,a];ZFEct;$AmQ$ODGtmQ]LKsHjdW/#V'4JR:!Rk4tI~6kj'NCe+nNwF*FFzyh*o8m5Z)h7Ixag{hq(SP)D2v
 }<]wxi ,w}l_L};ayl<D,cQ%U;Z-Q%FC@qV"z%}R7<Ab[pbk ?Ow>[>S#UArx7Q|)<=d@#VE~^F 'D0s gKy"
41WIL)@,*%kzRf%IeX'oyr>n1cZ&<0?YxS'8y74dG!z_#bp&Vh"&v{{^;NX
sRmqFYD/D4b[:U6c ]P q^Euz;IgC? SN95^t"`WV~FMt?O+([L"2!o+4W{}TOH{g|8z{B%J,f^pbpa*bM! 
l_L`Dy1&9;8@+y/`Z~dW%( 
$'9T:gpRo,+)CjGe%u`Sf/K!Rj$>-U0WND<BGn^Rm1wL02PK)m=_evYWAxRh%j3nMYO# pp&$LqL/2^@ 
3W~(crhz8ob%@omyZ=;@,0QH)_AJ3acoT@|V;c*Zn^B_}KhOv|os41WIL)@,*%kzRf%
,<hi9SVce?niX,ok(=Q@CM3E?Jmk`T;9O<$fMueRqrW0n7&|Ot6%
"x)b0N-WZK )p}9<d9HWj5P ]#FY5_}Zg~ #lFu8~}P]&j)*%mz_% U2L)mc6>10 
jH&L2MjO_"r'#"7=sNZ]'mw`vyAu(/[[ 
6y_aEw_yJ$?CG^R&CB0OUG0^7{WZ#G)E16E;_pmt]fSu-|?lt3W!|rOZc'g#z]f-RJ!AV)UJ[< 
7}W%fkx^cNp1Qf4rDStm|Z-sg5`PP7ahWU)14eC3OOm[T$a*rt{l^`C_ <w`4w#i~X6 
q#Zb& 1:qT 3uX 
mj`hl0cggowsnA[)NZHLA}<tfN#55zA$E|*N4&T8"#=d>9+Lrt)Ri%0knL`%7<T"}zGQe-j|Y;<T 
-}$b9qgI+nWu_z-&#$9,jka:r|Mh']7#@F@IW%@E;wY@UGTvfnsovpcLkLP,zi@>KLE:petc 
!$$ct|iCjtXyw;H4GG{Le_(Mo Ys+ O  o2Si0*r{ALho&8UiT]jD7WO c7l3d ]
/MQIz7'3oenZ$s+"hvBSV,z`4rwRtF6o%hUb 
M4=O$Jq"tg>NIN"E~3;+_Hem>q{c0}!eRVSq[5hf`fTcSJDn<u,^2a}Xq^X)6w[i!By 
'B64Gh6)Q:|Ll;p0:cqe%~[F; N4<9p}/*Q [ehA/>`e[-!^=+& Y6;OzBdx'qj=/ 
k*1ogXk@1?Xx`FjQ]4f^eDfe |u7*ns['^>/yf7(bpe~RklE!qg@Ui&cd
sVP)uZC:Z"wx>`pJ;a4OSXhP9'eb9A 
_^#3<x;}((id&%qp8q^f8ymj{ek;0LnTm]ihl/"{%b}W#f8JvMO=7X;WYbJtJW0VH@2P0VDMUq:vvZ1M"ay1UofdmLuV}{cu319vf/8WlW$GgA#aA>Y^k
 (w?<@-+!dL `7B4x-#^Xe_-&(g#E_)[3a- @4JN7*!AJ<]6(3GD4omI|pgx,]jZ+}k TT,ix0I3
d~9IT=a6zQWW, :4J-<KUWG M-;TSSyVn:}6@}x>C&R
XiH62"> *vQ_L %%WynP'}~3Bm 'Lv^" jW[cVj)#Z3jsC-@Y?pQ-_ck/~,G}^yp[&>+ 
h5<3te(S[Sb*4P3NR0Fm00bK'nx9li:p~{%]}VC%/BUZ$j|q#yBK' 
*=M,TUeH9!xIYj[[-)lDE[Jw,kTyVLXyt:y0Lzcc>wKWFxFocQOG -2z"eWLPb96'{AAo 
CmT!5_'?||JGy0p-;f[KQBLi}B?n2hN;q:1(~T@4Lj/9#iR4nD,13H9S926u0bk=' 32m 
oXGhshp68z/7=V<Q>6e4)!5
!qg@Ui&cd>9*l-7aZ2'&7Z18`5Cl`LNK1Tlg1}g4=@G"V,;cY((rCH(,<hi9SVce?niX,ok(=Q@CM3E?Jmk`T;9O<$fMueRqrW0n7&|Ot6%2ueY3q5A7W
`e[-!^=+& Y6;OzBdx'qj=/ k*1ogXk@1?Xx`FjQ]4f^eDfe 
|u7*ns['^>/yf7(bpe~RklE!qg@Ui&cd>9*l-7aZ2'&7Z18`5Cl`LNK1Tlg1}g4=@G"V,;cY((rCH(,<hi9SVce?niX,ok(=Q@CM3E?Jmk`T;9O<$fMueRqrW0n7&|Ot6%2ueY3q5A7W|dQ8i+[kC,y(9LNLC[4;1rl6~
 (C?K(]NvVTgE3FA+m/uZGTI'K/gV1#oEbi#&/l?O1c#p
hMHz$#zxOlVm,-]U8dMBA#Z>P-<ZQ ?UjD#)c,bQkXZgCM /P<u~ #8iZ67<B3SH}tn{toE 
8{G?A:{~[jKX_d2F6vD"*^y7"KVjW)G
IG;*Ja{X,mGyX$vce$?[1 
4_^;S>R[4+qU6ZRReR3K`im{kn5e,o@4|SRMN:zc%OUF/?&,KEHiNQ?U4ZY3~vp8!4fmVEmv#3 
`ZB8'6aabR"TgS,6ov((;H-,]nif+ 2Ut{nlRoiT2b|8R"w5l,IbGr>rBgwn
N8BPWw)@oOV'jly71N`8O"%JL_h:;i^A~f~sAS:L:KDaSZ5f|B~IsM^4t)Vk[zHZAg$1V6<M>BNQ-aS%BqmT 
}f^91=V)tcjw$0!E5lC{V^`VxO&J 
TsRmqFYD/D4b[:U6c ]P q^Euz;IgC? 
LuV}{cu319vf/8WlW$GgA#aA>Y^k (w?<@-
n0G9@}ga,4xPm+h64<p3aMH72(<%GtdT{Hn3xr3t*)8{9@Sy94-9cqy1<wBALB* 
9}=;gKk_t@~[Mqhg%}~$c/C?o4y 
{d)X'8O?dV*wvSi76m!I>LkbDJ<S?sa=v)HlP^hQz[`RB%j}{&v=nE+O`*{8>{Pq8)AgYEW)oqwN'CC&hY4s=M6TsRmqFYD/D4b[:U6c
 ]P q^Euz;IgC?
,;eRmQlC+U-0AjkV}}9*6fpxS>AW 
P07D!kPqv@b`V8>4#QU5kxtqLN5J!mdl&0OP$}5?439ku4`:foWmH`z#k{i*>l:%%FpUs%Q&Ch{W 
XN~8:=a1g$8'_O]:C295$l-TZ2)6NA "`|1 $Zm-A*~hr 
>Tn4j{#{FxQn=[Nh*kcD8^]&YlWws~^no;@y%Pmi9 sTJ51m~ PFDD
   D 
gZ$Y?+SF|QyQi~&5YFc)yBPm-EIIa9;et72nTdSKQ~-VKZl;}k~lMlusP3YMuONKebXUba}R~0<fvf'km6vY!P!oaxXr
 
?1`|jr/jgJ^!D-"I$aUEtbd5@o5?8(44FBVOSQVW)zvlfoT(l?iD<i"`tV5+CuN8S"e"HZPp(7A+%$4]~`s7$lA|7;*:itWXn
 QUi]6{U{`x6AiA| ; Hr5?iK=n1G0!}mq|,(-R R!z%^W`aMHI4|B9Bn(]QQ1CZ!xWSc>oT`JtN]}NAu([ogV
tfN#55zA$E|*N4&T8"#=d>9+Lrt)Ri%0knL`%7<T"}zGQe-j|Y;

MD5:82486780f97e81c7f8fcf4c243e2101a


------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.

Reply via email to