Re: pilerpurge.py and manticore issue
> On 2023-09-28 12:03, d tbsky wrote: > > > Thanks for the hint. I tried it at the testing environment and it > > works. > > but when there are no metadata table empty records (eg pilerpuge.py not > > run), > > indexer.delta.sh will return another error: > > > > WARNING: killlist_target is specified, but kill-list is empty > > it's not an error, it's simply a warning that you have no rows in the > metadata > table where deleted=1, so there's no record to suppress in the results. Yes I understand. but cron jobs are supposed to not return any messages unless there are something wrong. these messages will mail to cron reports.
Re: pilerpurge.py and manticore issue
On 2023-09-28 12:03, d tbsky wrote: Thanks for the hint. I tried it at the testing environment and it works. but when there are no metadata table empty records (eg pilerpuge.py not run), indexer.delta.sh will return another error: WARNING: killlist_target is specified, but kill-list is empty it's not an error, it's simply a warning that you have no rows in the metadata table where deleted=1, so there's no record to suppress in the results. Janos
Re: pilerpurge.py and manticore issue
Hi tbsky, Am Donnerstag, 28. September 2023 12:03 CEST, schrieb d tbsky : > Thanks for the hint. I tried it at the testing environment and it works. > but when there are no metadata table empty records (eg pilerpuge.py not run), > indexer.delta.sh will return another error: > > WARNING: killlist_target is specified, but kill-list is empty > > so it seems hard to have a configuration to work for both condition? Yes, you need at least one deleted mail in the database to silence this warning. I use "grep" to filter this line from the output of piler tools, so e.g. cron doesn't bother me with warning emails. Regards, Frank
Re: pilerpurge.py and manticore issue
Frank Schmirler > > > > Hi, > Am Mittwoch, 27. September 2023 05:30 CEST, schrieb d tbsky > : > > > > > The kill-list should work with manticore, too. > > > > but I don't know how to get rid of the error message "kill-list not > > empty but no killlist_target specified" without dropping the empty > > records. I think there are some reasons that pilerpurge.py leaves > > these empty records? > > I had to add the following line to the "index delta1" section: > killlist_target = main1:kl, main2:kl, main3:kl, main4:kl, > dailydelta1:kl > > Regards, > Frank > Thanks for the hint. I tried it at the testing environment and it works. but when there are no metadata table empty records (eg pilerpuge.py not run), indexer.delta.sh will return another error: WARNING: killlist_target is specified, but kill-list is empty so it seems hard to have a configuration to work for both condition?
Re: pilerpurge.py and manticore issue
Thank you, Frank. I've updated the master branch. Janos On 2023-09-27 09:08, Frank Schmirler wrote: Hi, Am Mittwoch, 27. September 2023 05:30 CEST, schrieb d tbsky : > The kill-list should work with manticore, too. but I don't know how to get rid of the error message "kill-list not empty but no killlist_target specified" without dropping the empty records. I think there are some reasons that pilerpurge.py leaves these empty records? I had to add the following line to the "index delta1" section: killlist_target = main1:kl, main2:kl, main3:kl, main4:kl, dailydelta1:kl Regards, Frank
Re: pilerpurge.py and manticore issue
Hi, Am Mittwoch, 27. September 2023 05:30 CEST, schrieb d tbsky : > > > The kill-list should work with manticore, too. > > but I don't know how to get rid of the error message "kill-list not > empty but no killlist_target specified" without dropping the empty > records. I think there are some reasons that pilerpurge.py leaves > these empty records? I had to add the following line to the "index delta1" section: killlist_target = main1:kl, main2:kl, main3:kl, main4:kl, dailydelta1:kl Regards, Frank
Re: pilerpurge.py and manticore issue
Janos SUTO > The kill-list should work with manticore, too. but I don't know how to get rid of the error message "kill-list not empty but no killlist_target specified" without dropping the empty records. I think there are some reasons that pilerpurge.py leaves these empty records? > However, you are not forced to switch to RT index, feel free to use > the usual main-delta-deltadelta indexing scheme. I personally believe > that an RT index has some benefits, eg. new emails are immediately > visible on the gui, no need to run indexer periodically. No need to > watch if the index data files grow too large causing searchd a headache. > Also less i/o requirements when merging the dailydelta to the main > index. Thanks a lot for the detailed explanation. I tried to use RT according to the document at https://www.mailpiler.org/wiki/current:manticore and https://docs.mailpiler.com/piler-ee/migrate-sphinx-to-manticore/ but I am not sure about the correct reindex procedure. I just "rm -f /var/piler/manticore/*" and "reindex -a" I also remove four crontab jobs below(the script recommend in the document only remove two of them): 5,35 * * * * piler /usr/libexec/piler/indexer.delta.sh 30 2 * * * piler /usr/libexec/piler/indexer.main.sh */15 * * * * piler /usr/bin/indexer --config /etc/piler/manticore.conf --quiet tag1 --rotate */15 * * * * piler /usr/bin/indexer --config /etc/piler/manticore.conf --quiet note1 --rotate After these the archive system seems to work and GUI search is working. But CJK support is broken. I can not search mails with CJK characters. CJK support is fine if I don't use "RT". Is there some other config I should modify? Regards, tbskyd
Re: pilerpurge.py and manticore issue
Hello, On 2023-09-23 05:29, d tbsky wrote: Hi: I tried to reindex everything at the testing environment. but the error message is the same. I browse the mariadb data, and found that the purge mails seems only left one empty record for each mail at the metadata table. I don't know if this is true or not. I delete the empty record and the It's correct. The pilerpurge.py script updates the metadata table with empty values for the deleted emails, and removes connected entries in the rcpt, tag, note, etc. tables. error message disappear. but I don't know if this is the correct thing to do. The kill-list should work with manticore, too. I also think about the manticore new "RT" function. I don't care the delay of archived mails show at the search GUI. I want a reliable system, when piler accept a mail, I hope the mail can be archived safely. if something goes wrong like disk-full, mysq/manticore service down, power failure..etc, piler won't accept the mail or can fix the problem later(maybe via crontab works?). I don't know if using "RT" would lower the archive quality. is "RT" the preferred method in the future? Piler requires several external services to work. It stores crucial metadata, etc. in mysql. If the mysql database is not available, then piler can't archive new emails, and can't retrieve anything having an attachment. Manticore (or sphinx) is also crucial as it indexes the archived data. If piler can't process the email that piler-smtp has written to disk, eg. no mysql, no manticore, disk full or similar, then piler moves the email to /var/piler/error dir for later investigation. However, you are not forced to switch to RT index, feel free to use the usual main-delta-deltadelta indexing scheme. I personally believe that an RT index has some benefits, eg. new emails are immediately visible on the gui, no need to run indexer periodically. No need to watch if the index data files grow too large causing searchd a headache. Also less i/o requirements when merging the dailydelta to the main index. The merge requires the indexer utility to rewrite the main index. Janos
Re: pilerpurge.py and manticore issue
Hi: I tried to reindex everything at the testing environment. but the error message is the same. I browse the mariadb data, and found that the purge mails seems only left one empty record for each mail at the metadata table. I don't know if this is true or not. I delete the empty record and the error message disappear. but I don't know if this is the correct thing to do. I also think about the manticore new "RT" function. I don't care the delay of archived mails show at the search GUI. I want a reliable system, when piler accept a mail, I hope the mail can be archived safely. if something goes wrong like disk-full, mysq/manticore service down, power failure..etc, piler won't accept the mail or can fix the problem later(maybe via crontab works?). I don't know if using "RT" would lower the archive quality. is "RT" the preferred method in the future? Thanks for help!!
pilerpurge.py and manticore issue
Hi:
I upgrade our piler server to 1.4.4 and use manticore search to
replace sphinx. I also install a test environment to simulate the
problem.
after upgrade, I let pilerpurge.py to run vi crontab ("
/usr/libexec/piler/purge.sh").
the metadata at mysql and file at /var/piler/store seems fine. but
manticore seems not happy.
every time "/usr/libexec/piler/indexer.delta.sh" running wlll report below:
WARNING: kill-list not empty but no killlist_target specified
and the deleted empty mail is still appear at the piler GUI interface.
below is my piler.conf
hostid=10.1.1.113
listen_addr=10.1.1.113
default_retention_days=1
number_of_worker_processes=2
min_message_size=50
archive_emails_not_having_message_id=1
enable_cjk=1
extra_to_field=X-Envelope-To:
memcached_servers=
server_id=0
queuedir=/var/piler/store
mysqlcharset=utf8mb4
mysqlsocket=/var/lib/mysql/mysql.sock
mysqluser=piler
mysqlpwd=piler
mysqldb=piler
smtp_access_list=1
and result of "php /etc/piler/manticore.conf"
#
# minimal manticore configuration suited to piler
#
source base
{
type = mysql
sql_host = localhost
sql_db = piler
sql_user = piler
sql_pass = piler
sql_attr_uint = size
sql_attr_uint = sent
sql_attr_uint = attachments
}
source delta : base
{
sql_query_pre = SET NAMES utf8mb4
sql_query_pre = REPLACE INTO sph_counter SELECT 1, IFNULL(MAX(id),
0) FROM sph_index
sql_query_post_index = DELETE FROM sph_index WHERE id<=(SELECT
max_doc_id FROM sph_counter WHERE counter_id=1)
sql_query = SELECT id, `from` as sender, `to` as rcpt, `fromdomain`
as senderdomain, `todomain` as rcptdomain, `subject`, `arrived`,
`sent`, `body`, `size`, `direction`, `folder`, `attachments`,
`attachment_types` FROM sph_index WHERE id <= (SELECT max_doc_id FROM
sph_counter WHERE counter_id=1)
sql_query_killlist = SELECT `id` FROM `metadata` WHERE `deleted`=1
}
source main1 : base
{
sql_query_pre = SET NAMES utf8mb4
sql_query = SELECT id, `from` as sender, `to` as rcpt, `fromdomain`
as senderdomain, `todomain` as rcptdomain, `subject`, `arrived`,
`sent`, `body`, `size`, `direction`, `folder`, `attachments`,
`attachment_types` FROM sph_index WHERE id=-1
}
source main2 : base
{
sql_query_pre = SET NAMES utf8mb4
sql_query = SELECT id, `from` as sender, `to` as rcpt, `fromdomain`
as senderdomain, `todomain` as rcptdomain, `subject`, `arrived`,
`sent`, `body`, `size`, `direction`, `folder`, `attachments`,
`attachment_types` FROM sph_index WHERE id=-1
}
source main3 : base
{
sql_query_pre = SET NAMES utf8mb4
sql_query = SELECT id, `from` as sender, `to` as rcpt, `fromdomain`
as senderdomain, `todomain` as rcptdomain, `subject`, `arrived`,
`sent`, `body`, `size`, `direction`, `folder`, `attachments`,
`attachment_types` FROM sph_index WHERE id=-1
}
source main4 : base
{
sql_query_pre = SET NAMES utf8mb4
sql_query = SELECT id, `from` as sender, `to` as rcpt, `fromdomain`
as senderdomain, `todomain` as rcptdomain, `subject`, `arrived`,
`sent`, `body`, `size`, `direction`, `folder`, `attachments`,
`attachment_types` FROM sph_index WHERE id=-1
}
source dailydelta : base
{
sql_query_pre = SET NAMES utf8mb4
sql_query = SELECT id, `from` as sender, `to` as rcpt, `fromdomain`
as senderdomain, `todomain` as rcptdomain, `subject`, `arrived`,
`sent`, `body`, `size`, `direction`, `folder`, `attachments`,
`attachment_types` FROM sph_index WHERE id=-1
}
index main1
{
source = main1
path= /var/piler/manticore/main1
min_prefix_len = 5
min_word_len= 1
stored_fields =
charset_table = 0..9, english, _, \
U+C1->U+E1, U+C4->U+E4, U+C5->U+E5,
U+C6->U+E6, U+C9->U+E9, U+CD->U+ED, U+D3->U+F3, U+D6->U+F6,
U+D8->U+F8, \
U+DA->U+FA, U+DC->U+FC,
U+0150->U+0151, U+0152->U+0153, U+0170->U+0171, U+01E2->U+E6,
U+01E3->U+E6, U+01FC->U+E6, \
U+01FD->U+E6, U+1D01->U+E6,
U+1D02->U+E6, U+1D2D->U+E6, U+1D46->U+E6, \
U+DF, U+E1, U+E4, U+E5, U+E6, U+E9,
U+ED, U+00F3, U+F6, U+F8, U+FA, U+FC, U+0151, U+0153, U+0171
ngram_len = 1
ngram_chars = U+3000..U+2FA1F
}
index main2
{
source = main2
path= /var/piler/manticore/main2
min_prefix_len = 5
min_word_len= 1
stored_fields =
charset_table = 0..9, english, _, \
U+C1->U+E1, U+C4->U+E4, U+C5->U+E5,
U+C6->U+E6, U+C9->U+E9, U+CD->U+ED, U+D3->U+F3, U+D6->U+F6,
U+D8->U+F8, \
U+DA->U+FA, U+DC->U+FC,
U+0150->U+0151, U+0152->U+0153, U+0170->U+0171, U+01E2->U+E6,
U+01E3->U+E6, U+01FC->U+E6, \
U+01FD->U+E6, U+1D01->U+E6,
U+1D02->U+E6, U+1D2D->U+E6, U+1D46->U+E6, \
