Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
On Sunday, August 31, 2003, at 09:48 PM, Neal Richter wrote: Earlier on of you two OSX guys complained of problems with the digging process... so do you see any wierdness at all? Excessive memory/CPU consumption during digging? Digging should be VERY I/O bound so CPU utilization should be low... Unfortunately I no longer have access to the data set that resulted in consistently repeatable DB problems. I have tried a couple other digs with moderately sized sites and didn't encounter any problems. What about htdump? do you get any errors when you dump a database? This would tell us of DB corruption. Are the db.* files excessively large? For the tests I have run thus far, database sizes looked quite reasonable. No problems running htdump. Jim --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
Ciao Ted! Thanks for your feedback! Again, good job to everyone who worked on this release. This is the first release of 3.2 that has pretty much worked right out of the source without any modification and without specifying --disable-shared. You can't imagine how happy I am for this. I knew that sticking to the new autotools could have led to some important goals. And IMHO the fact that it works on Mac OSX is an important one. -Gabriele -- Gabriele Bartolini: Web Programmer, ht://Dig IWA/HWG Member, ht://Check maintainer Current Location: Melbourne, Victoria, Australia [EMAIL PROTECTED] | http://www.prato.linux.it/~gbartolini | ICQ#129221447 Leave every hope, ye who enter!, Dante Alighieri, Divine Comedy, The Inferno --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
Everything appears to have worked perfectly. I'm very impressed. Good job to everyone. Earlier on of you two OSX guys complained of problems with the digging process... so do you see any wierdness at all? Excessive memory/CPU consumption during digging? Digging should be VERY I/O bound so CPU utilization should be low... What about htdump? do you get any errors when you dump a database? This would tell us of DB corruption. Are the db.* files excessively large? Thanks! Neal Richter Knowledgebase Developer RightNow Technologies, Inc. Customer Service for Every Web Site Office: 406-522-1485 --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
I didn't notice any unusual CPU usage. Here's the output from top while running htdump 13293 top 9.4% 0:01.22 11418 284K 336K 500K+ 13.6M 13286 tcsh 0.0% 0:00.08 11016 344K 672K 804K 5.73M 13285 login0.0% 0:00.66 11233 248K 388K 576K 13.7M 13282 htdump 20.6% 0:14.01 1 937 12.7M+ 2.57M 14.1M+ 25.1M but 20% was always near the high end of the usage range. htdump worked like a charm and there didn't appear to be any corruption, unlike in the government of the U.S. ;-) Again, good job to everyone who worked on this release. This is the first release of 3.2 that has pretty much worked right out of the source without any modification and without specifying --disable-shared. I'd really like to see a snapshot of it stored on the server with a reference to the fact that it should build fine on Mac OS X 10.2.6. Ted Stresen-Reuter On Sunday, August 31, 2003, at 10:48 PM, Neal Richter wrote: Everything appears to have worked perfectly. I'm very impressed. Good job to everyone. Earlier on of you two OSX guys complained of problems with the digging process... so do you see any wierdness at all? Excessive memory/CPU consumption during digging? Digging should be VERY I/O bound so CPU utilization should be low... What about htdump? do you get any errors when you dump a database? This would tell us of DB corruption. Are the db.* files excessively large? Thanks! Neal Richter Knowledgebase Developer RightNow Technologies, Inc. Customer Service for Every Web Site Office: 406-522-1485 --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
The htfuzzy error was for an undefined symbol. Here's the output error from 'make': ld: Undefined symbols: _CDB_set_mp_diry_level make[1]: *** [htfuzzy] Error 1 make: *** [all-recursive] Error 1 It seems to have been fixed, though, as demonstrated in a later email. On Sunday, August 31, 2003, at 10:45 PM, Neal Richter wrote: On Fri, 29 Aug 2003, Jim Cole wrote: On Friday, August 29, 2003, at 06:02 AM, Ted Stresen-Reuter wrote: So, bottom line is, compiling ht on Mac OS X with the above configure options seems to work (with the exception of htfuzzy). Now, if we could only figure out how to get it to compile with dynamic shared libraries on OS X... What is the htfuzzy error? Is it filecopy.c stuff or something else? Thanks! Neal Richter Knowledgebase Developer RightNow Technologies, Inc. Customer Service for Every Web Site Office: 406-522-1485 --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
On Friday, August 29, 2003, at 09:41 AM, Neal Richter wrote: I removed references to CDB_set_mp_diry_level, CDB_get_mp_diry_level and CDB___mp_dirty_level. Thanks! With that and all of Gabriele's recent changes, I am back to the point where I can build under OS X without any errors. I am now testing, but unfortunately I don't have much in the way of bandwidth at the moment. Jim --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
On Friday, August 29, 2003, at 06:02 AM, Ted Stresen-Reuter wrote: So, bottom line is, compiling ht on Mac OS X with the above configure options seems to work (with the exception of htfuzzy). Now, if we could only figure out how to get it to compile with dynamic shared libraries on OS X... Have you tried building recently without --disable-shared? It appears that the configuration patch and recent associated tweaks to that patch also fixed the problem with shared libraries under OS X. My current build is based on shared libraries and so far it looks like all of the executables are working as expected. Jim --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
Success! Mac OS X 10.2.6 (and with shared dynamic libraries apparently) Well, I conducted the big dig as described in this message: http://sourceforge.net/mailarchive/message.php?msg_id=5113485 Everything appears to have worked perfectly. I'm very impressed. Good job to everyone. Ted Stresen-Reuter On Friday, August 29, 2003, at 11:21 PM, Jim Cole wrote: On Friday, August 29, 2003, at 06:02 AM, Ted Stresen-Reuter wrote: So, bottom line is, compiling ht on Mac OS X with the above configure options seems to work (with the exception of htfuzzy). Now, if we could only figure out how to get it to compile with dynamic shared libraries on OS X... Have you tried building recently without --disable-shared? It appears that the configuration patch and recent associated tweaks to that patch also fixed the problem with shared libraries under OS X. My current build is based on shared libraries and so far it looks like all of the executables are working as expected. Jim --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
At 09.19 30/08/2003 -0500, Ted Stresen-Reuter wrote: Success! Mac OS X 10.2.6 (and with shared dynamic libraries apparently) I am so *happy* and *reliefed* that everything seems to work fine! I was a bit scared about my changes on the configure process but I think that now the maintaining of the autotools section has become easier. Thanks Ted and Jim for your precious tests. I will try in the next few days to fix the 'make dist' problem if you agree. -Gabriele -- Gabriele Bartolini: Web Programmer, ht://Dig IWA/HWG Member, ht://Check maintainer Current Location: Melbourne, Victoria, Australia [EMAIL PROTECTED] | http://www.prato.linux.it/~gbartolini | ICQ#129221447 Leave every hope, ye who enter!, Dante Alighieri, Divine Comedy, The Inferno --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
On Thursday, August 28, 2003, at 03:24 PM, Neal Richter wrote: Hey all, I rolled mp_alloc.c back to Revision 1.2. I am now running into a problem with an undefined symbol named _CDB_set_mp_diry_level Is this perhaps related to the roll back? I can find a prototype and calls for the associated function, but no function definition. Jim --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
(forgot to hit Reply All...) Mac OS X person here... ./configure --prefix=/Library/Htdig-3.2 --enable-tests --libdir=/usr/lib --includedir=/usr/include --with-config-dir=/Library/Htdig-3.2/conf --with-common-dir=/Library/Htdig-3.2/common --with-database-dir=/Library/Htdig-3.2/db --with-default-config-file=/Library/Htdig-3.2/conf/htdig.conf --with-cgi-bin-dir=/Library/WebServer/CGI-Executables --with-image-dir=/Library/WebServer/Documents/images --with-search-dir=/Library/WebServer/Documents/htdig --with-image-url-prefix=/images --with-search-form=/Library/WebServer/Documents/htdig/search-3.2.html --disable-shared configured fine make appeared to have completed, but with the following lines at the end: /bin/sh ../libtool --mode=link g++ -g -O2 -Wall -fno-rtti -fno-exceptions -o htfuzzy htfuzzy.o libfuzzy.la ../htnet/libhtnet.la ../htcommon/libcommon.la ../htword/libhtword.la ../htlib/libht.la ../htcommon/libcommon.la ../htword/libhtword.la ../db/libhtdb.la ../htlib/libht.la -lz -lz g++ -g -O2 -Wall -fno-rtti -fno-exceptions -o htfuzzy htfuzzy.o -Wl,-bind_at_load ./.libs/libfuzzy.a ../htnet/.libs/libhtnet.a ../htcommon/.libs/libcommon.a ../htword/.libs/libhtword.a ../db/.libs/libhtdb.a ../htlib/.libs/libht.a -lz ld: Undefined symbols: _CDB_set_mp_diry_level make[1]: *** [htfuzzy] Error 1 make: *** [all-recursive] Error 1 make install Making install in htfuzzy /bin/sh ../libtool --mode=link g++ -g -O2 -Wall -fno-rtti -fno-exceptions -o htfuzzy htfuzzy.o libfuzzy.la ../htnet/libhtnet.la ../htcommon/libcommon.la ../htword/libhtword.la ../htlib/libht.la ../htcommon/libcommon.la ../htword/libhtword.la ../db/libhtdb.la ../htlib/libht.la -lz -lz g++ -g -O2 -Wall -fno-rtti -fno-exceptions -o htfuzzy htfuzzy.o -Wl,-bind_at_load ./.libs/libfuzzy.a ../htnet/.libs/libhtnet.a ../htcommon/.libs/libcommon.a ../htword/.libs/libhtword.a ../db/.libs/libhtdb.a ../htlib/.libs/libht.a -lz ld: Undefined symbols: _CDB_set_mp_diry_level make[1]: *** [htfuzzy] Error 1 make: *** [install-recursive] Error 1 Running htdig on a small, local site seemed to work fine, but even though I was specifying the config file to use on the command line, it appeared to use the default config file as well. htdig -c /Libary/Htdig-3.2/conf/tedmasterweb.htdig.conf -v Following the dig I did a search and it worked fine. So, bottom line is, compiling ht on Mac OS X with the above configure options seems to work (with the exception of htfuzzy). Now, if we could only figure out how to get it to compile with dynamic shared libraries on OS X... Ted Stresen-Reuter On Thursday, August 28, 2003, at 04:24 PM, Neal Richter wrote: Hey all, I rolled mp_alloc.c back to Revision 1.2. The recent changes caused extreme DB growth. Interestingly this growth in the file size resulted in ZERO data difference when old new files were dumped with htdump. MAC OS X PEOPLE: Please please test this change. If you are still experiencing excessive recursion that this last few changes were trying to fix... please respond with precise details on duplicating this effect. I will debug and fix it on a Mac OS X machine. Thanks! Neal Richter Knowledgebase Developer RightNow Technologies, Inc. Customer Service for Every Web Site Office: 406-522-1485 --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
Re: [htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
Ok.. fixed. Sorry. I removed references to CDB_set_mp_diry_level, CDB_get_mp_diry_level and CDB___mp_dirty_level. I left the htdig config var 'wordlist_cache_dirty_level' for possible use in the future. Please retest a dig in a deep/stressful way and let me know if you see the excessive recursion still. Thanks! On Fri, 29 Aug 2003, Ted Stresen-Reuter wrote: (forgot to hit Reply All...) Mac OS X person here... ./configure --prefix=/Library/Htdig-3.2 --enable-tests --libdir=/usr/lib --includedir=/usr/include --with-config-dir=/Library/Htdig-3.2/conf --with-common-dir=/Library/Htdig-3.2/common --with-database-dir=/Library/Htdig-3.2/db --with-default-config-file=/Library/Htdig-3.2/conf/htdig.conf --with-cgi-bin-dir=/Library/WebServer/CGI-Executables --with-image-dir=/Library/WebServer/Documents/images --with-search-dir=/Library/WebServer/Documents/htdig --with-image-url-prefix=/images --with-search-form=/Library/WebServer/Documents/htdig/search-3.2.html --disable-shared configured fine make appeared to have completed, but with the following lines at the end: /bin/sh ../libtool --mode=link g++ -g -O2 -Wall -fno-rtti -fno-exceptions -o htfuzzy htfuzzy.o libfuzzy.la ../htnet/libhtnet.la ../htcommon/libcommon.la ../htword/libhtword.la ../htlib/libht.la ../htcommon/libcommon.la ../htword/libhtword.la ../db/libhtdb.la ../htlib/libht.la -lz -lz g++ -g -O2 -Wall -fno-rtti -fno-exceptions -o htfuzzy htfuzzy.o -Wl,-bind_at_load ./.libs/libfuzzy.a ../htnet/.libs/libhtnet.a ../htcommon/.libs/libcommon.a ../htword/.libs/libhtword.a ../db/.libs/libhtdb.a ../htlib/.libs/libht.a -lz ld: Undefined symbols: _CDB_set_mp_diry_level make[1]: *** [htfuzzy] Error 1 make: *** [all-recursive] Error 1 make install Making install in htfuzzy /bin/sh ../libtool --mode=link g++ -g -O2 -Wall -fno-rtti -fno-exceptions -o htfuzzy htfuzzy.o libfuzzy.la ../htnet/libhtnet.la ../htcommon/libcommon.la ../htword/libhtword.la ../htlib/libht.la ../htcommon/libcommon.la ../htword/libhtword.la ../db/libhtdb.la ../htlib/libht.la -lz -lz g++ -g -O2 -Wall -fno-rtti -fno-exceptions -o htfuzzy htfuzzy.o -Wl,-bind_at_load ./.libs/libfuzzy.a ../htnet/.libs/libhtnet.a ../htcommon/.libs/libcommon.a ../htword/.libs/libhtword.a ../db/.libs/libhtdb.a ../htlib/.libs/libht.a -lz ld: Undefined symbols: _CDB_set_mp_diry_level make[1]: *** [htfuzzy] Error 1 make: *** [install-recursive] Error 1 Running htdig on a small, local site seemed to work fine, but even though I was specifying the config file to use on the command line, it appeared to use the default config file as well. htdig -c /Libary/Htdig-3.2/conf/tedmasterweb.htdig.conf -v Following the dig I did a search and it worked fine. So, bottom line is, compiling ht on Mac OS X with the above configure options seems to work (with the exception of htfuzzy). Now, if we could only figure out how to get it to compile with dynamic shared libraries on OS X... Ted Stresen-Reuter On Thursday, August 28, 2003, at 04:24 PM, Neal Richter wrote: Hey all, I rolled mp_alloc.c back to Revision 1.2. The recent changes caused extreme DB growth. Interestingly this growth in the file size resulted in ZERO data difference when old new files were dumped with htdump. MAC OS X PEOPLE: Please please test this change. If you are still experiencing excessive recursion that this last few changes were trying to fix... please respond with precise details on duplicating this effect. I will debug and fix it on a Mac OS X machine. Thanks! Neal Richter Knowledgebase Developer RightNow Technologies, Inc. Customer Service for Every Web Site Office: 406-522-1485 --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev Neal Richter Knowledgebase Developer RightNow Technologies, Inc. Customer Service for Every Web Site Office: 406-522-1485 --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
[htdig-dev] mp_alloc.c rollback -- Attn Mac OSX Developers!
Hey all, I rolled mp_alloc.c back to Revision 1.2. The recent changes caused extreme DB growth. Interestingly this growth in the file size resulted in ZERO data difference when old new files were dumped with htdump. MAC OS X PEOPLE: Please please test this change. If you are still experiencing excessive recursion that this last few changes were trying to fix... please respond with precise details on duplicating this effect. I will debug and fix it on a Mac OS X machine. Thanks! Neal Richter Knowledgebase Developer RightNow Technologies, Inc. Customer Service for Every Web Site Office: 406-522-1485 --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
