[General] indexer make segmentation fail
Hi I try this command to Indexing my list of web site : /usr/local/mnogosearch/sbin/indexer -Eblob -d /usr/local/mnogosearch/etc/indexer.conf the result is : Segmentation fault here my sql databases informations : ++---+ | Tables | Size (MB) | ++---+ | bdict |191.90 | | bdicti |605.58 | | categories | 0.00 | | crossdict | 0.00 | | dict | 0.00 | | links | 0.00 | | qcache | 0.00 | | qinfo | 0.00 | | qtrack | 0.00 | | server | 0.18 | | srvinfo| 0.00 | | url| 3028.87 | | urlinfo| 2110.99 | | wrdstat| 0.00 | ++---+ have you an idea? __ my config : * Debian 3.2.51-1 x86_64 GNU/Linux * mnogosearch 3.3.15 * indexer.conf : .. DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=blob .. * commande ./configure for installation ../configure --prefix=/usr/local/mnogosearch --bindir=/usr/local/mnogosearch/bin --sbindir=/usr/local/mnogosearch/sbin --sysconfdir=/usr/local/mnogosearch/etc --localstatedir=/usr/local/mnogosearch/var --libdir=/usr/local/mnogosearch/lib --includedir=/usr/local/mnogosearch/include --mandir=/usr/local/mnogosearch/man --disable-shared --enable-static --enable-syslog --without-docs --enable-pthreads --disable-dmalloc --enable-parser --disable-mp3 --disable-xml --disable-rss --disable-css --disable-js --with-extra-charsets=all --enable-file --enable-http --enable-ftp --enable-htdb --enable-news --with-mysql --with-zlib __ --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
Re: [General] Problem with indexer -Eblob
Hi Thanks for your help If i take select @@max_allowed_packet; i have this result : +--+ | @@max_allowed_packet | +--+ | 16777216 | +--+ 1 row in set (0.00 sec) So, after i have recompile mnogoseearch 3.3.15 by adding --with-debug i have run make and make install i have add DebugSQL=yes in indexer.conf at the end of line DBAddr mysql://root:mypasswd@localhost/test/?dbmode=blobDebugSQL=yes and i have run indexer -Eblob 2LOG.txt the message in LOG.txt is : -- indexer[13250]: Indexing indexer[13250]: Loading URL list -- the process terminated on the console by 'Killed' message i do not understand nothing ? is my databases corrupted ? can you help me ? Le 18/03/2014 03:18, Alexander Barkov a écrit : Hi, On 03/13/2014 08:51 PM, d...@hodei.net wrote: hi when i try this command, i have an error message : mysql_stmt_execute() failed: Lost connection to MySQL server during query here is the result of my command : -- root@bot:/usr/local/mnogosearch/sbin# ./indexer -Eblob /usr/local/mnogosearch/etc/indexer.conf indexer[22039]: Indexing indexer[22039]: Loading URL list indexer[22039]: Converting intag00 indexer[22039]: Converting intag01 indexer[22039]: Converting intag02 indexer[22039]: Converting intag03 indexer[22039]: Converting intag04 indexer[22039]: Converting intag05 indexer[22039]: Converting intag06 indexer[22039]: Converting intag07 indexer[22039]: Converting intag08 indexer[22039]: Converting intag09 indexer[22039]: Converting intag0A indexer[22039]: Converting intag0B indexer[22039]: Converting intag0C indexer[22039]: Converting intag0D indexer[22039]: Converting intag0E indexer[22039]: Converting intag0F indexer[22039]: Converting intag10 indexer[22039]: Converting intag11 indexer[22039]: Converting intag12 indexer[22039]: Converting intag13 indexer[22039]: Converting intag14 indexer[22039]: Converting intag15 indexer[22039]: Converting intag16 indexer[22039]: Converting intag17 indexer[22039]: Converting intag18 indexer[22039]: Converting intag19 indexer[22039]: Converting intag1A indexer[22039]: Converting intag1B indexer[22039]: Converting intag1C indexer[22039]: Converting intag1D indexer[22039]: Converting intag1E indexer[22039]: Converting intag1F indexer[22039]: Total converted: 32554167 records, 78081117 bytes indexer[22039]: Converting url data indexer[22039]: mysql_stmt_execute() failed: Lost connection to MySQL server during query -- have you an idea? Try this: select @@max_allowed_packet; If the maximum packet size is small enough, try to increase it in the server side. If this does not help, then compile mnogoseearch again by adding --with-debug to the configure command line, then add DebugSQL=yes parameter into DBAddr, like this: DBAddr mysql://root@localhost/test/?DebugSQL=yes and run indexer -Eblob 2LOG.txt. It will print all SQL queries to the log file. Check the last few lines in the log. my config : Debian 3.2.51-1 x86_64 GNU/Linux mnogosearch 3.3.15 --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
[General] delete server url in indexer.conf
Hi, In the indexer.conf file in 'Server [Method] ', i want delete an entry like 'server http://www.eke.org' do I need to clear the base MyssQL data after change Thanks __ my config : * Debian 3.2.51-1 x86_64 GNU/Linux * mnogosearch 3.3.15 * indexer.conf : .. DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=blob .. --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
[General] Duplicates Commandes in indexer
Hi When i try to execute the indexer command it run command in double; for example : /usr/local/mnogosearch/sbin/indexer -Ecreate -d /usr/local/mnogosearch/etc/indexer.conf : create tables twice and in the second run i have a warning 'table already exist' /usr/local/mnogosearch/sbin/indexer -Eblob /usr/local/mnogosearch/etc/indexer.conf : - indexer[16663]: Indexing indexer[16663]: Loading URL list indexer[16663]: Converting intag00 indexer[16663]: Converting intag01 indexer[16663]: Converting intag02 indexer[16663]: Converting intag03 indexer[16663]: Converting intag04 indexer[16663]: Converting intag05 indexer[16663]: Converting intag06 indexer[16663]: Converting intag07 indexer[16663]: Converting intag08 indexer[16663]: Converting intag09 indexer[16663]: Converting intag0A indexer[16663]: Converting intag0B indexer[16663]: Converting intag0C indexer[16663]: Converting intag0D indexer[16663]: Converting intag0E indexer[16663]: Converting intag0F indexer[16663]: Converting intag10 indexer[16663]: Converting intag11 indexer[16663]: Converting intag12 indexer[16663]: Converting intag13 indexer[16663]: Converting intag14 indexer[16663]: Converting intag15 indexer[16663]: Converting intag16 indexer[16663]: Converting intag17 indexer[16663]: Converting intag18 indexer[16663]: Converting intag19 indexer[16663]: Converting intag1A indexer[16663]: Converting intag1B indexer[16663]: Converting intag1C indexer[16663]: Converting intag1D indexer[16663]: Converting intag1E indexer[16663]: Converting intag1F indexer[16663]: Total converted: 2604877 records, 13711786 bytes indexer[16663]: Converting url data indexer[16663]: Switching to new blob table. indexer[16663]: Loading URL list indexer[16663]: Converting intag00 indexer[16663]: Converting intag01 indexer[16663]: Converting intag02 indexer[16663]: Converting intag03 indexer[16663]: Converting intag04 indexer[16663]: Converting intag05 indexer[16663]: Converting intag06 indexer[16663]: Converting intag07 indexer[16663]: Converting intag08 indexer[16663]: Converting intag09 indexer[16663]: Converting intag0A indexer[16663]: Converting intag0B indexer[16663]: Converting intag0C indexer[16663]: Converting intag0D indexer[16663]: Converting intag0E indexer[16663]: Converting intag0F indexer[16663]: Converting intag10 indexer[16663]: Converting intag11 indexer[16663]: Converting intag12 indexer[16663]: Converting intag13 indexer[16663]: Converting intag14 indexer[16663]: Converting intag15 indexer[16663]: Converting intag16 indexer[16663]: Converting intag17 indexer[16663]: Converting intag18 indexer[16663]: Converting intag19 indexer[16663]: Converting intag1A indexer[16663]: Converting intag1B indexer[16663]: Converting intag1C indexer[16663]: Converting intag1D indexer[16663]: Converting intag1E indexer[16663]: Converting intag1F indexer[16663]: Total converted: 2605019 records, 13712168 bytes indexer[16663]: Converting url data indexer[16663]: Switching to new blob table. - I have configure indexer like this : usr/local/mnogosearch/lib --includedir=/usr/local/mnogosearch/include --mandir=/usr/local/mnogosearch/man --disable-shared --enable-static --enable-syslog --without-docs --enable-pthreads --disable-dmalloc --enable-parser --disable-mp3 --disable-xml --disable-rss --disable-css --disable-js --with-extra-charsets=all --enable-file --enable-http --enable-ftp --enable-htdb --enable-news --with-mysql --with-zlib Here my config : _ my config : * Debian 3.2.51-1 x86_64 GNU/Linux * mnogosearch 3.3.15 * contents of indexer.conf : .. DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=blob .. _ Have you an idea ? Thanks --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
[General] Delete a line in Server method in indexer.conf
Hi, In the indexer.conf file, in 'Server [Method] ', i want to delete an entry like this : 'server http://www.eke.org' Is that all pages of the site will be removed from the database during the next crawling? Thanks __ my config : * Debian 3.2.51-1 x86_64 GNU/Linux * mnogosearch 3.3.15 * indexer.conf : .. DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=blob .. --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
[General] Delete a line in Server method in indexer.conf
Hi, In the indexer.conf file, in 'Server [Method] ', i want to delete an entry like this : 'server http://www.eke.org' Is that all pages of the site will be removed from the database during the next crawling? Thanks __ my config : * Debian 3.2.51-1 x86_64 GNU/Linux * mnogosearch 3.3.15 * indexer.conf : .. DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=blob .. http://www.avast.com/ Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection Antivirus avast! http://www.avast.com/ est active. --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
[General] accented characters
hi i have accented characters in my web search. to solve this problem, i have modify the database with this queries : ALTER TABLE bdict CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci; ALTER TABLE bdicti CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci; and i have init variables *$localcharset* et *$browsercharset* with utf-8 in my indexer.conf But i have always the problem ! have you an idea ? Thanks __ my config : * Debian 3.2.51-1 x86_64 GNU/Linux * mnogosearch 3.3.15 * indexer.conf : .. DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=blob .. --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
[General] parameter to server command in indexer.conf
hi When i try to add this url to my list in dexer.conf -- server http://fr.wikipedia.org/wiki/Zanpantzar -- the crawler search all fr.wikioedia.org site and not only Zanpantzar directory Have you an idea ? Thanks __ my config : * Debian 3.2.51-1 x86_64 GNU/Linux * mnogosearch 3.3.15 * indexer.conf : .. DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=blob .. --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
[General] Indexing Failed with large database
Hi
I have a problem when i indexing my database :
___
root@botujo:/home/jean# /usr/local/mnogosearch/sbin/indexer -Eblob
indexer[4787]: Indexing
indexer[4787]: Loading URL list
{sql.c:1513} Query: SELECT rec_id, site_id, pop_rank, last_mod_time FROM url
indexer[4787]: MySQL driver: #144: Table './mnogosearch/url' is marked
as crashed and last (automatic?) repair failed
Here is my database information in phpmyadmin :
namelinessize
--
bdict 864 575 1,1 Go
bdicti utilisé
bdict_tmp2,0 Ko
categories1,0 Ko
crossdict 1,0 Ko
dict 1,0 Ko
links 1,0 Ko
qcache 1,0 Ko
qinfo 2,0 Ko
qtrack 1,0 Ko
server 889156,7 Ko
srvinfo 1,0 Ko
url utilisé
urlinfo 11 009 854 27,4 Go
Have you an idea ?
Thanks
__
my config :
* Debian 3.2.51-1 x86_64 GNU/Linux
* mnogosearch 3.3.15
* indexer.conf :
..
DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=blob
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce
que la protection avast! Antivirus est active.
http://www.avast.com
___
General mailing list
General@mnogosearch.org
http://lists.mnogosearch.org/listinfo/general
Re: [General] Crawling order
ok, thanks a lot Le 05/08/2014 18:36, Alexander Barkov a écrit : Hi, On 08/05/2014 12:12 PM, d...@hodei.net wrote: Hi I have 1000 websites in my indexer.conf on the 'Server method' rubric in what order the 'crawler' look over the list of website : random, alphabetical or other Crawler selects targets in a random order. There are some related command line options: -e Visit 'most expired' (oldest) documents first -o Visit documents with less depth (hops value) first -r Do not try to reduce remote servers load by randomising crawler queue order (faster, but less polite) thanks for your help _ my config : * Debian 3.2.51-1 x86_64 GNU/Linux * mnogosearch 3.3.15 * contents of indexer.conf : .. DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=blob .. _ --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general --- Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com ___ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
