Re: Sieve addheader directive works only partially

2020-05-19 Thread Peter Folta 
Hi Stephan,

Sorry for the delay - this was on version 2.3.4.1 (f79e8e7e4).

Relevant bits from the config:

# 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 ()
# OS: Linux 4.19.0-9-amd64 x86_64 Debian 10.4
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext editheader imapsieve vnd.dovecot.imapsieve
sieve_extensions = +editheader
sieve_global_extensions = +vnd.dovecot.pipe
sieve_pipe_bin_dir = /usr/bin
sieve_plugins = sieve_imapsieve sieve_extprograms

Thanks
Peter

> On May 15, 2020, at 3:49 PM, Stephan Bosch  wrote:
> 
> 
> 
> Op 13-5-2020 om 20:10 schreef Peter Folta:
>> require “editheader”;
>> 
>> if address :is "From” “special-sen...@example.com” {
>> addheader “X-My-Custom-Header” “Header Value";
>> }
>> 
>> I have enabled the editheader extension in my Dovecot configuration:
>> 
>> plugin {
>> …
>> sieve_extensions = +editheader
>> }
> 
> What version is this? Please provide output of `dovecot -n`.
> 
> This smells a lot like an age-old bug that was fixed long ago.
> 
> It works fine with master.
> 
> Regards,
> 
> Stephan.

doveadm backup, question

2020-05-19 Thread dovecot-user 

Hello everyone.
I am an inexperienced user of dovecot and I faced probably a simple 
problem, but I still do not understand how to solve it.
The issue is related to using doveadm backup - I want to backup 
mailboxes; make this copy on the same machine (hereinafter, files with 
backup copies will be taken from this machine into the storage by a 
separate mechanism, this is not related to the issue).

I am using dovecot 2.2.27 (c0f36b0) on Debian GNU/Linux 9 (stretch).
The options related to storing mail and attachments are as follows:
mail_location = mdbox:/var/mail/%d/%n
mail_attachment_dir = /var/mail/%d/%n/attach
Thus, the mail of u...@domain.tld is stored in 
/var/mail/domain.tld/user/, and mail attachments inside this directory, 
for example: /var/mail/domain.tld/user/attach.

When I start copying (as root while in test mode):
#doveadm backup -u "u...@domain.tld" sdbox:/backup/domain.tld/user, 
copying passes, but, as I understand it, the attachments are not copied 
(I am judging so far by the volume of the initial storage and the 
received one, it should not differ VERY much, but it differs by 10 
times, it seems that it’s just the total amount of attachments).
Tell me, plz, how to run doveadm backup correctly so that everything is 
copied so that you can subsequently deploy it from the backup?

Thanks.

FTS-lucene errors : language not available for stemming

2020-05-19 Thread David Gessel 

I'm getting some log errors with clucene that I am having no luck tracking down 
on the interwebs.


Errors:

May 19 05:05:16 
indexer-worker(ges...@blackrosetech.com)<62971>:
 Error: lucene index /mail/blackrosetech.com/gessel//lucene-indexes: 
IndexWriter::addDocument() failed (#4): language not available for stemming
May 19 05:05:16 indexer-worker: Error:
May 19 05:05:16 
indexer-worker(ges...@blackrosetech.com)<62971>:
 Error: Mailbox Security: Mail search failed: Internal error occurred. Refer to server log 
for more information. [2020-05-19 05:05:16]
May 19 05:05:16 
indexer-worker(ges...@blackrosetech.com)<62971>:
 Error: Mailbox Security: Transaction commit failed: FTS transaction commit failed: 
transaction context (attempted to index 1 messages (UIDs 152736..152736))


Config:

FreeBSD 11.3-RELEASE-p8 #0 r360490

dovecot-2.3.10_3

clucene-2.3.3.4_19

py37-pystemmer-2.0.0.1

py37-snowballstemmer-1.2.1

icu-67.1,1

plugin {
  #setting_name = value
    expire = Trash
    mail_log_events = delete undelete expunge copy mailbox_delete 
mailbox_rename
    mail_log_fields = uid box msgid size
    fts_autoindex=yes
    #zlib_save_level = 6 # 1..9
    #zlib_save = gz # or bz2
}

plugin {
  fts = lucene
  # Lucene-specific settings, good ones are:
  fts_lucene = whitespace_chars=@. mime_parts
}



I am considering switch to xapian (solr and java... pls noe) as the port is 
quite tempting from an ease of integration perspective, but the easiest 
solution would be to resolve these odd indexing errors.  Anyone have a clue?

-David

Re: Problem with tika

2020-05-19 Thread Arjen Heidinga 
Greetings,

Is the anyone who could please help me shed light on this?

Arjen

On 06-03-2020 09:52, Arjen Heidinga wrote:
> Hello all,
>
> For some time now we've bin using solr as a search engine (working
> great). I have added tika for searching inside documents, however it
> keeps crashing when indexing. When just indexing mails that arrive works
> good, but a reindex on all mail constantly crashes with below stacktrace.
>
> When I observe the packets with wireshark I see a HTTP-flow going to
> tika and suddenly, midway a document dovecot (or the server) sends a
> RST. Sometimes after thousands of documents, sometimes after a few.
> Usually after a few hundred.
>
> It appears there are less errors using http than https.
>
>
>
> Relevant config:
>
> OS: CentOS6, fully updated
>
> plugin {
>   fts = solr
>   batch_size = 1
>   fts_solr =
> url=https://username:passw...@solr-01.vevida.net:443/solr/dovecot/
>   #fts_tika = https://solr-01.vevida.net:443/tika/
>   batch_size = 1000
>   fts_autoindex=yes
>   soft_commit=no
> }
>
> # dovecot --version
> 2.3.9.3 (9f41b88fa)
>
> # Configure options:
>
>     --docdir=%{_docdir}/dovecot  \
>     --disable-static \
>     --with-nss   \
>     --with-shadow    \
>     --with-pam   \
>     --with-gssapi=plugin \
>     --with-ldap=plugin   \
>     --with-sql=plugin    \
>     --with-pgsql \
>     --with-sqlite    \
>     --with-zlib  \
>     --with-bzlib \
>     --with-lzma  \
>     --with-libcap    \
>     --with-ssl=openssl   \
>     --with-ssldir=%{ssldir}  \
>     --with-solr  \
>     --with-docs
>
> # It is compiled agains the latest openssl
>
> # Tika and Solr: Both latest versions.
>
> #Stacktrace:
>
> doveadm(i...@samenmetrenske.nl): Info: Sent: Caching mails seq=1..161
>
> doveadm(i...@samenmetrenske.nl): Panic: file http-client-request.c: line
> 1173 (http_client_request_send_more): assertion failed:
> (req->payload_input != NULL)
>
> doveadm(info@xxx.x): Error: Raw backtrace:
> /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x2f)
> [0x7f95d805acbf] ->
> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x26) [0x7f95d805add6]
> -> /usr/lib64/dovecot/libdovecot.so.0(+0xe90ba) [0x7f95d80660ba] ->
> /usr/lib64/dovecot/libdovecot.so.0(+0xe9161) [0x7f95d8066161] ->
> /usr/lib64/dovecot/libdovecot.so.0(+0x41158) [0x7f95d7fbe158] ->
> /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_more+0x424)
> [0x7f95d8005094] ->
> /usr/lib64/dovecot/libdovecot.so.0(http_client_connection_output+0x11a)
> [0x7f95d800a24a] ->
> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0x8f6a) [0x7f95d57a2f6a]
> -> /usr/lib64/dovecot/libdovecot.so.0(+0x114483) [0x7f95d8091483] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x61)
> [0x7f95d807e581] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xdc)
> [0x7f95d808076c] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5c)
> [0x7f95d807e67c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38)
> [0x7f95d807e8c8] -> /usr/lib64/dovecot/libdovecot.so.0(+0x89105)
> [0x7f95d8006105] ->
> /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_payload+0x1f)
> [0x7f95d80063cf] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xd31d)
> [0x7f95d6ad931d] ->
> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_parser_more+0x1a)
> [0x7f95d6ad83ca] ->
> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_build_mail+0x761)
> [0x7f95d6ad6401] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x114ca)
> [0x7f95d6add4ca] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_precache+0x2a)
> [0x7f95d835ab4a] -> doveadm(+0x31e75) [0x55e7c1052e75] ->
> doveadm(+0x321fb) [0x55e7c10531fb] -> doveadm(+0x2c321) [0x55e7c104d321]
> -> doveadm(+0x2c577) [0x55e7c104d577] ->
> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x1e8) [0x55e7c104ec38] ->
> doveadm(doveadm_cmd_run_ver2+0x52e) [0x55e7c105fafe] ->
> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55e7c105fb97] ->
> doveadm(main+0x21a) [0x55e7c1062aca] ->
> /lib64/libc.so.6(__libc_start_main+0x100) [0x7f95d7983d20] ->
> doveadm(+0x1c479) [0x55e7c103d479]
>


signature.asc
Description: OpenPGP digital signature

Re: FTS-lucene errors : language not available for stemming

2020-05-19 Thread Stuart Henderson 
On 2020/05/19 17:04, Aki Tuomi wrote:
> 
> On 19/05/2020 16:48 Stuart Henderson  wrote:
> 
> 
> On 2020-05-19, David Gessel  wrote:
> 
> I'm getting some log errors with clucene that I am having no luck 
> tracking down on the
> interwebs.
> 
> This looks relevant:
> 
> https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html
> 
> 
> I am considering switch to xapian (solr and java... pls noe) as the
> port is quite tempting from an ease of integration perspective, but 
> the
> easiest solution would be to resolve these odd indexing errors.  
> Anyone
> have a clue?
> 
> dovecot-fts-xapian is easy to configure, but has a big downside compared
> to solr in that the indexer runs as root.
> 
> Dovecot indexer does not run as root.
> 
> ---
> Aki Tuomi
> 

It does in the not entirely uncommon case where you have setup
dovecot-fts-xapian, have multiple system users rather than a single
uid owning all mailboxes, and need to index all mailboxes.

  PID USERNAME PRI NICE  SIZE   RES STATE WAIT  TIMECPU COMMAND
44468 root   20   11M   18M sleep/6   netio 1:41 68.36% doveadm 
index -A *

With solr the indexing is done out-of-process and typically under a
safe uid.

Re: FTS-lucene errors : language not available for stemming

2020-05-19 Thread Aki Tuomi 


 
 
  
   
  
  
   
On 19/05/2020 17:22 Aki Tuomi  wrote:
   
   

   
   

   
   

   
   

 On 19/05/2020 17:18 Stuart Henderson  wrote:


 


 


 On 2020/05/19 17:04, Aki Tuomi wrote:


 
  On 19/05/2020 16:48 Stuart Henderson  wrote:
 
 
  
 
 
  
 
 
  On 2020-05-19, David Gessel  wrote:
 
 
  
 
 
  I'm getting some log errors with clucene that I am having no luck tracking down on the
 
 
  interwebs.
 
 
  
 
 
  This looks relevant:
 
 
  
 
 
  https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html
 
 
  
 
 
  
 
 
  I am considering switch to xapian (solr and java... pls noe) as the
 
 
  port is quite tempting from an ease of integration perspective, but the
 
 
  easiest solution would be to resolve these odd indexing errors. Anyone
 
 
  have a clue?
 
 
  
 
 
  dovecot-fts-xapian is easy to configure, but has a big downside compared
 
 
  to solr in that the indexer runs as root.
 
 
  
 
 
  Dovecot indexer does not run as root.
 
 
  
 
 
  ---
 
 
  Aki Tuomi
 
 
  
 


 It does in the not entirely uncommon case where you have setup


 dovecot-fts-xapian, have multiple system users rather than a single


 uid owning all mailboxes, and need to index all mailboxes.


 


 PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND


 44468 root 2 0 11M 18M sleep/6 netio 1:41 68.36% doveadm index -A *


 


 With solr the indexing is done out-of-process and typically under a


 safe uid.

   
   
Thats doveadm though, not indexer.
   
   
---
Aki Tuomi
   
  
  
   
  
  
   Also if you were looking carefully what happens, you'd notice dovecot calls seteuid() before actually doing the indexing work.
  
  
   ---
Aki Tuomi

Re: FTS-lucene errors : language not available for stemming

2020-05-19 Thread Aki Tuomi 


 
 
  
   
  
  
   
On 19/05/2020 16:48 Stuart Henderson  wrote:
   
   

   
   

   
   
On 2020-05-19, David Gessel  wrote:
   
   

 I'm getting some log errors with clucene that I am having no luck tracking down on the interwebs.

   
   
This looks relevant:
   
   

   
   
https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html
   
   

   
   

 I am considering switch to xapian (solr and java... pls noe) as the


 port is quite tempting from an ease of integration perspective, but the


 easiest solution would be to resolve these odd indexing errors.  Anyone


 have a clue?

   
   
dovecot-fts-xapian is easy to configure, but has a big downside compared
   
   
to solr in that the indexer runs as root.
   
  
  
   Dovecot indexer does not run as root.
  
  
   ---
Aki Tuomi

Re: FTS-lucene errors : language not available for stemming

2020-05-19 Thread Stuart Henderson 
On 2020-05-19, David Gessel  wrote:
> I'm getting some log errors with clucene that I am having no luck tracking 
> down on the interwebs.

This looks relevant:

https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html

> I am considering switch to xapian (solr and java... pls noe) as the
> port is quite tempting from an ease of integration perspective, but the
> easiest solution would be to resolve these odd indexing errors.  Anyone
> have a clue?

dovecot-fts-xapian is easy to configure, but has a big downside compared
to solr in that the indexer runs as root.

Re: FTS-lucene errors : language not available for stemming

2020-05-19 Thread Jan Bramkamp 

On 19.05.20 15:15, David Gessel wrote:

I'm getting some log errors with clucene that I am having no luck 
tracking down on the interwebs.



Errors:

May 19 05:05:16 
indexer-worker(ges...@blackrosetech.com)<62971>: 
Error: lucene index /mail/blackrosetech.com/gessel//lucene-indexes: 
IndexWriter::addDocument() failed (#4): language not available for 
stemming

May 19 05:05:16 indexer-worker: Error:
May 19 05:05:16 
indexer-worker(ges...@blackrosetech.com)<62971>: 
Error: Mailbox Security: Mail search failed: Internal error occurred. 
Refer to server log for more information. [2020-05-19 05:05:16]
May 19 05:05:16 
indexer-worker(ges...@blackrosetech.com)<62971>: 
Error: Mailbox Security: Transaction commit failed: FTS transaction 
commit failed: transaction context (attempted to index 1 messages 
(UIDs 152736..152736))



Config:

FreeBSD 11.3-RELEASE-p8 #0 r360490

dovecot-2.3.10_3

clucene-2.3.3.4_19

py37-pystemmer-2.0.0.1

py37-snowballstemmer-1.2.1

icu-67.1,1

plugin {
  #setting_name = value
    expire = Trash
    mail_log_events = delete undelete expunge copy mailbox_delete 
mailbox_rename

    mail_log_fields = uid box msgid size
    fts_autoindex=yes
    #zlib_save_level = 6 # 1..9
    #zlib_save = gz # or bz2
}

plugin {
  fts = lucene
  # Lucene-specific settings, good ones are:
  fts_lucene = whitespace_chars=@. mime_parts
}

I am considering switch to xapian (solr and java... pls noe) as the 
port is quite tempting from an ease of integration perspective, but 
the easiest solution would be to resolve these odd indexing errors.  
Anyone have a clue?


I ran into the same problem a few weeks back. The workaround I found was 
to add no_snowball to fts_lucene. It disables the snowball algorithm.

Re: FTS-lucene errors : language not available for stemming

2020-05-19 Thread Aki Tuomi 


 
 
  
   
  
  
   
On 19/05/2020 17:18 Stuart Henderson  wrote:
   
   

   
   

   
   
On 2020/05/19 17:04, Aki Tuomi wrote:
   
   

 On 19/05/2020 16:48 Stuart Henderson  wrote:


 


 


 On 2020-05-19, David Gessel  wrote:


 


 I'm getting some log errors with clucene that I am having no luck tracking down on the


 interwebs.


 


 This looks relevant:


 


 https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html


 


 


 I am considering switch to xapian (solr and java... pls noe) as the


 port is quite tempting from an ease of integration perspective, but the


 easiest solution would be to resolve these odd indexing errors. Anyone


 have a clue?


 


 dovecot-fts-xapian is easy to configure, but has a big downside compared


 to solr in that the indexer runs as root.


 


 Dovecot indexer does not run as root.


 


 ---


 Aki Tuomi


 

   
   
It does in the not entirely uncommon case where you have setup
   
   
dovecot-fts-xapian, have multiple system users rather than a single
   
   
uid owning all mailboxes, and need to index all mailboxes.
   
   

   
   
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
   
   
44468 root 2 0 11M 18M sleep/6 netio 1:41 68.36% doveadm index -A *
   
   

   
   
With solr the indexing is done out-of-process and typically under a
   
   
safe uid.
   
  
  
   Thats doveadm though, not indexer.
  
  
   ---
Aki Tuomi

Re: FTS-lucene errors : language not available for stemming

2020-05-19 Thread Benny Pedersen 

On 2020-05-19 16:22, Aki Tuomi wrote:


 Thats doveadm though, not indexer.


and dovecot allows it

Re: FTS-lucene errors : language not available for stemming

2020-05-19 Thread Benny Pedersen 

On 2020-05-19 16:28, Aki Tuomi wrote:


 Also if you were looking carefully what happens, you'd notice dovecot
calls seteuid() before actually doing the indexing work.


would be more sense in to have a dovecot shell that all commands must be 
issued in, like postgresql, setuid is nice still but its not well known 
it happends the same as apache starts as root for port under 1024 but 
after start a fork that is not running as root

Re: FTS-lucene errors : language not available for stemming

2020-05-19 Thread Benny Pedersen 

On 2020-05-19 16:18, Stuart Henderson wrote:


It does in the not entirely uncommon case where you have setup
dovecot-fts-xapian, have multiple system users rather than a single
uid owning all mailboxes, and need to index all mailboxes.

  PID USERNAME PRI NICE  SIZE   RES STATE WAIT  TIMECPU 
COMMAND
44468 root   20   11M   18M sleep/6   netio 1:41 68.36% 
doveadm index -A *


With solr the indexing is done out-of-process and typically under a
safe uid.


no doveconf -n, no problem

if you really have a bug it should be solved

i think you could

su --user=non-root-user doveadm index -A

to not have it run as root, it works well for fangfrisch, dovecot should 
not allow commands as root without thinking of consequences

Re: Problems with userdb and mail deliveribility

2020-05-19 Thread Muthukumar 
Don't file has /etc/dovecot/password and rodeo is a available at
/etc/mail/password ??

Best regards,
Muthu

On Sun, 17 May, 2020, 8:53 PM Chris Bennett,
 wrote:

> I briefly got some mails delivered, but not anymore.
> Attached are dovecot-info.log, dovecot-debug.log,
> dovecot.log, maillog, /etc/mail/passwd
> smtpd.conf, vaddr, vusers, vdomains
>
> Thanks for any help.
>
> Chris Bennett
>
>
>