Re: Corrupt index files
On 21.07.2017 20:47, Bruce Guenter wrote:
> I am running Dovecot IMAP on Linux, on a LizardFS storage cluster with
> Maildir storage. This has worked well for most of the accounts for
> several months.
>
> However in the last couple of weeks we are seeing increasing errors
> regarding corrupted index files. Some of the accounts affected are
> unable to retrieve messages due to timeouts.
>
> It appeared the problems were due to the accounts being accessed from
> multiple servers simultaneously, so I forced them all to access one
> server, but the errors remained. It looks like it has something to do
> with file locking, but LizardFS supports advisory file locking and I do
> have it enabled.
>
> Deleting the corrupted indexes fixes the problem for a while, but it
> eventually returns, particularly for some accounts.
>
> Here are some errors I'm seeing (just a random grab). Actual home
> directories are munged for confidentiality.
>
> imap[25157]: (clientes.standby) Error: Failed to fix view for
> HOME/clientes:standby/dovecot.index: Missing middle file seq=1 (between 1..1,
> we have seqs 8): File is already open
> imap[5565]: (stadiumchair) Error: Transaction log file
> HOME/stadiumchair/.Drafts/dovecot.index.log: marked corrupted
> imap[5005]: (stadiumchair) Error: Corrupted transaction log file
> HOME/stadiumchair/.Drafts/dovecot.index.log seq 2: indexid changed 1418941056
> -> 1500658549 (sync_offset=0)
> imap[20243]: (martha) Error: Transaction log HOME/martha/dovecot.index.log:
> duplicate transaction log sequence (539)
> imap[4665]: (emsspam) Error: Index file HOME/emsspam/dovecot.index: indexid
> changed: 1500658479 -> 1297175382
> imap[4665]: (emsspam) Error: Corrupted transaction log file
> HOME/emsspam/dovecot.index.log seq 3: indexid changed: 1500658479 ->
> 1297175382 (sync_offset=316)
> imap[22985]: (emsspam) Error: Corrupted transaction log file
> HOME/emsspam/dovecot.index.log seq 10742: Invalid transaction log size (9296
> vs 9296): HOME/emsspam/dovecot.index.log (sync_offset=9296)
> imap[3267]: (emsspam) Error: Failed to map view for
> HOME/emsspam/dovecot.index: Failed to map file seq=10742
> offset=9052..18446744073709551615 (ret=0): corrupted, indexid=0
> imap[3267]: (emsspam) Error: HOME/emsspam/dovecot.index view is inconsistent:
> uid=3062271 inserted in the middle of mailbox
>
> The output of dovecot -n is pasted in below. Note that some of the boxes
> are running 4.9, some running 4.4, all have the same problems. Also note
> that I am using a custom authentication front end for our virtual
> mailboxes, but it just sets up the minimal environment variables and
> runs imap.
>
> Is there anything I can change to eliminate these problems? Are there
> any other diagnostics I can provide to shed light on this?
>
> # 2.2.31 (65cde28): /etc/dovecot/dovecot.conf
> # OS: Linux 4.4.66 x86_64 Gentoo Base System release 2.3
> log_path = /dev/stderr
> mail_debug = yes
> mail_fsync = always
> mail_location = maildir:~/.maildir
> mail_log_prefix = "%s[%p]: (%u) "
> mmap_disable = yes
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> special_use = \Drafts
> }
> mailbox Junk {
> special_use = \Junk
> }
> mailbox Sent {
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> special_use = \Trash
> }
> prefix = INBOX
> separator =
> type = private
> }
> passdb {
> args = *
> driver = pam
> }
> passdb {
> args = /etc/dovecot/dovecot-sql.conf.ext
> driver = sql
> }
> plugin {
> mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
> }
> ssl_cert = ssl_key = # hidden, use -P to show it
> userdb {
> driver = passwd
> }
> userdb {
> args = /etc/dovecot/dovecot-sql.conf.ext
> driver = sql
> }
>
Do you have users accessing the files concurrently from more than one
dovecot instance at a time?
Aki
Re: Core dumped when authenticating to managesieve
On 21.07.2017 16:19, Kristjan Eentsalu wrote:
> Hey,
>
> I'm getting "Error in MANAGESIEVE command received by server." and
> service(managesieve-login) killed with signal 11 (core dumped) when doing
> two line autenticate(sogo) to managesieve().
>
> # telnet localhost 4190
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> "IMPLEMENTATION" "Dovecot Pigeonhole"
> "SIEVE" "fileinto reject envelope encoded-character vacation subaddress
> comparator-i;ascii-numeric relational regex imap4flags copy include
> variables body enotify environment mailbox date index ihave duplicate mime
> foreverypart extracttext editheader"
> "NOTIFY" "mailto"
> "SASL" "PLAIN"
> "VERSION" "1.0"
> OK "Dovecot ready."
> AUTHENTICATE "PLAIN" {52+}
> XX==
> NO "Error in MANAGESIEVE command received by server."
> AUTHENTICATE "PLAIN" {52+} < trying again
> Connection closed by foreign host.
>
> and after that in logs there is "dovecot: managesieve-login: Fatal: master:
> service(managesieve-login): child 38748 killed with signal 11 (core dumped)"
>
>
> One line autenticate works.
>
> # telnet localhost 4190
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> "IMPLEMENTATION" "Dovecot Pigeonhole"
> "SIEVE" "fileinto reject envelope encoded-character vacation subaddress
> comparator-i;ascii-numeric relational regex imap4flags copy include
> variables body enotify environment mailbox date index ihave duplicate mime
> foreverypart extracttext editheader"
> "NOTIFY" "mailto"
> "SASL" "PLAIN"
> "VERSION" "1.0"
> OK "Dovecot ready."
> AUTHENTICATE "PLAIN" "XX=="
> OK "Logged in."
>
>
> OS: FreeBSD 11.1-RC3
> Dovecot 2.2.31 , pigeonhole 0.4.19
>
> Trace below
>
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols
> found)...
>
> warning: core file may not match specified executable file.
> Core was generated by `dovecot/managesieve-login'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /usr/local/lib/dovecot/libdovecot-login.so.0...done.
> Loaded symbols for /usr/local/lib/dovecot/libdovecot-login.so.0
> Reading symbols from /usr/lib/libssl.so.8...done.
> Loaded symbols for /usr/lib/libssl.so.8
> Reading symbols from /lib/libcrypto.so.8...done.
> Loaded symbols for /lib/libcrypto.so.8
> Reading symbols from /usr/local/lib/dovecot/libdovecot.so.0...done.
> Loaded symbols for /usr/local/lib/dovecot/libdovecot.so.0
> Reading symbols from /usr/lib/libkrb5.so.11...done.
> Loaded symbols for /usr/lib/libkrb5.so.11
> Reading symbols from /usr/lib/libgssapi.so.10...done.
> Loaded symbols for /usr/lib/libgssapi.so.10
> Reading symbols from /usr/lib/libgssapi_krb5.so.10...done.
> Loaded symbols for /usr/lib/libgssapi_krb5.so.10
> Reading symbols from /lib/libc.so.7...done.
> Loaded symbols for /lib/libc.so.7
> Reading symbols from /usr/lib/libasn1.so.11...done.
> Loaded symbols for /usr/lib/libasn1.so.11
> Reading symbols from /usr/lib/libcom_err.so.5...done.
> Loaded symbols for /usr/lib/libcom_err.so.5
> Reading symbols from /lib/libcrypt.so.5...done.
> Loaded symbols for /lib/libcrypt.so.5
> Reading symbols from /usr/lib/libhx509.so.11...done.
> Loaded symbols for /usr/lib/libhx509.so.11
> Reading symbols from /usr/lib/libroken.so.11...done.
> Loaded symbols for /usr/lib/libroken.so.11
> Reading symbols from /usr/lib/libwind.so.11...done.
> Loaded symbols for /usr/lib/libwind.so.11
> Reading symbols from /usr/lib/libheimbase.so.11...done.
> Loaded symbols for /usr/lib/libheimbase.so.11
> Reading symbols from /usr/lib/libprivateheimipcc.so.11...done.
> Loaded symbols for /usr/lib/libprivateheimipcc.so.11
> Reading symbols from /lib/libthr.so.3...done.
> Loaded symbols for /lib/libthr.so.3
> Reading symbols from /libexec/ld-elf.so.1...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0 i_stream_seek (stream=0x0, v_offset=80) at istream.c:296
> 296 istream.c: No such file or directory.
> in istream.c
> (gdb) bt full
> #0 i_stream_seek (stream=0x0, v_offset=80) at istream.c:296
> _stream = (struct istream_private *) 0x5385e16
> #1 0x0536937a in i_stream_limit_read (stream=0x767d540) at
> istream-limit.c:34
> lstream = (struct limit_istream *) 0x767d540
> left = 123848760
> ret = 1
> pos = 119265200
> #2 0x0535f340 in i_stream_read (stream=0x767d5b0) at istream.c:174
> _stream = (struct istream_private *) 0x767d540
> old_size = 0
> ret = 124244864
> #3 0x053607ae in i_stream_read_data (stre
Re: under some kind of attack
As per my post: checkpassword. You can then use one password on Mondays, Wednesdays, and Fridays, alternate passwords on Tuesdays and Thursday fetched from a rot-13 database, and only from prime numbered IP addresses on weekends, if that's what you want. Having read the wiki page on checkpassword, I am unsure how this would work with an ldap backend. Could you elaborate on that? You are essentially writing your own backend by taking over authentication. You'll be accepting user/password inputs into your checkpassword executable, then use the LDAP API (or some other system that will do it for you) to authenticate. (You can Google around for code snippets.) You'll have direct control over all aspects of authentication (if/when/where/etc) that a generic backend can't provide. You can choose do implement using shell/PERL/etc script, or compile to executable from C sources. It's more work, but if you need to do everything on your wish list, I can't see any eaiser option. One of the drawbacks is that a working password depends on both time and source address, which will be adversely affect performance on a busy server as authentication data cannot be cached. Joseph Tam
Re: Strange Error: Password data is not valid for scheme SHA256. Please help me resolve it.
Am 23.07.2017 um 17:50 schrieb david.madm...@vfemail.net: My /var/log/mail.log shows: Jul 22 18:40:48 www dovecot: auth: Error: passwd-file(t...@domain.com,46.xxx.xxx.xxx,): Password data is not valid for scheme SHA256: Input length isn't valid (0 instead of 32) Jul 22 18:41:00 www dovecot: message repeated 2 times: [ auth: Error: passwd-file(t...@domain.com,46.xxx.xxx.xxx,): Password data is not valid for scheme SHA256: Input length isn't valid (0 instead of 32)] Jul 22 18:41:02 www dovecot: imap-login: Disconnected (auth failed, 3 attempts in 14 secs): user=, method=PLAIN, rip=46.xxx.xxx.xxx, lip=139.xxx.xxx.xxx, TLS, session= What does "Password data is not valid for scheme SHA256: Input length isn't valid (0 instead of 32)]" mean? I assume that there is some kind of a mismatch between the way I generated the password with doveadm and entered it in passwd.db and the way I entered the non-hashed password into the password field in the new account section of Thunderbird. It means that dovecot expacts to verify a 32 byte long password hash. What it detects has a size of 0 byte. You haven't shown an example line of your passwd.db file, but I would guess you build it up not correct. See https://wiki.dovecot.org/Authentication/PasswordSchemes Alexander
Strange Error: Password data is not valid for scheme SHA256. Please help me resolve it.
Hello,
I am using version 2.2.31 (65cde28) on an Ubuntu 16.04 VPS.
I am attempting to setup a mail server using a flat file system as an
experiment.
I am able to send e-mail from external domain names and the messages
land in my /var/mail/vmail/domain/user/ directories.
I am trying to setup Thunderbird as an MUA using the information I
generated on my VPS namely the IMAP server, usern...@domain.com,
password, and SMTP server.
IMAP server - www.domain.com
SMTP server - www.domain.com
Username - created in /etc/postfix/virtual-mailbox-users.db and
/etc/dovecot/passwd.db
Password - created by dovadm pw -s SHA256 and entered (along with the
username) in /etc/dovecot/passwd.db
I enter this information into "new accounts" in Thunderbird and select
STARTTLS with ports 143 (IMAP) and 587 (SMTP). (I have experimented
with a variety of other combinations too). I click "Done" which
transmits the information to the domain server to verify the details.
My /var/log/mail.log shows:
Jul 22 18:40:48 www dovecot: auth: Error:
passwd-file(t...@domain.com,46.xxx.xxx.xxx,):
Password data is not valid for scheme SHA256: Input length isn't valid
(0 instead of 32)
Jul 22 18:41:00 www dovecot: message repeated 2 times: [ auth: Error:
passwd-file(t...@domain.com,46.xxx.xxx.xxx,):
Password data is not valid for scheme SHA256: Input length isn't valid
(0 instead of 32)]
Jul 22 18:41:02 www dovecot: imap-login: Disconnected (auth failed, 3
attempts in 14 secs): user=, method=PLAIN,
rip=46.xxx.xxx.xxx, lip=139.xxx.xxx.xxx, TLS, session=
What does "Password data is not valid for scheme SHA256: Input length
isn't valid (0 instead of 32)]" mean? I assume that there is some kind
of a mismatch between the way I generated the password with doveadm
and entered it in passwd.db and the way I entered the non-hashed
password into the password field in the new account section of
Thunderbird.
Is there a way to resolve this issue? My dovecot -n is below. You will
note that the passdb section does have the scheme as SHA256. Many
thanks.
# 2.2.31 (65cde28): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.19 (e5c7051)
# OS: Linux 4.4.0-83-generic x86_64 Ubuntu 16.04.2 LTS ext4
auth_mechanisms = plain login
auth_verbose = yes
mail_home = /var/mail/vmail/%d/%n
mail_location = maildir:/var/mail/vmail/%d/%n/mail:LAYOUT=fs
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
}
passdb {
args = username_format=%u scheme=SHA256 /etc/dovecot/passwd.db
driver = passwd-file
}
plugin {
sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_dir = ~/sieve
}
protocols = imap pop3 sieve
service auth {
unix_listener /var/spool/postfix/private/dovecot-auth {
group = postfix
mode = 0660
user = postfix
}
}
ssl_cert = ssl_cipher_list =
ALL:!LOW:!SSLv2:ALL:!aNULL:!ADH:!eNULL:!EXP:RC4+RSA:+HIGH:+MEDIUM
ssl_key = # hidden, use -P to show it
userdb {
args = uid=5000 gid=5000 home=/var/mail/vmail/%d/%n
driver = static
}
protocol imap {
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep
mail_max_userip_connections = 10
}
protocol pop3 {
mail_max_userip_connections = 10
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
}
protocol lda {
deliver_log_format = msgid=%m: %$
mail_plugins = sieve
postmaster_address = postmas...@domain.com
quota_full_tempfail = yes
rejection_reason = Your message to <%t> was automatically rejected:%n%r
}
-
ONLY AT VFEmail! - Use our Metadata Mitigator to keep your email out of the
NSA's hands!
$24.95 ONETIME Lifetime accounts with Privacy Features!
15GB disk! No bandwidth quotas!
Commercial and Bulk Mail Options!
Re: Modify stored mail contents?
On 23/07/2017 3:30 PM, Sami Ketola wrote: On 23 Jul 2017, at 16.14, Tom Hendrikx wrote: In general, you should not do this. When a message is stored using IMAP, it is immutable. The IMAP server also remembers things like size and assigns messages a unique ID, so mail readers that have already downloaded the message with that ID, don't have to download the whole message again to verify whether it's contents have magically changed. What you're suggesting is not simply compatible with IMAP standards. The normal way of applying changes to messages is just like a mail client connecting to IMAP: create a new message and save it to the store, then delete the old one. Just like that. But instead of using IMAP interface to do it, you can also do it with doveadm: 1. doveadm fetch mail 2. doveadm delete mail 3. modify fetched mail 4. doveadm import modified mail back Simple as that. There is no other supported way to do it. Editing mail objects on storage will break things. Sami Thank you, that's an interesting idea. Exporting the mail and deleting it is easy enough, but I'm not sure where I would import the message from. doveadm import expects a mailbox store as source, so I'm back to the problem of safely writing emails in a mailbox store format that Dovecot understands! I can export to Maildir format [doveadm backup -u USER "maildir:/mymaildir:LAYOUT=fs"], where each file contains one message and nothing else, but even that has extra files like dovecot-uidlist, dovecot.index.cache, etc. I'm not sure if it's safe to import from a Maildir where the message contents have been modified, but the other files haven't. Also, would the import create new UIDs? I probably want UIDs to change, so that the IMAP client re-downloads the messages. I'd want to preserve IMAP flags like "Seen", though and, ideally, the sequence of messages inside a mailbox.
Re: Modify stored mail contents?
> On 23 Jul 2017, at 16.14, Tom Hendrikx wrote: > > In general, you should not do this. When a message is stored using IMAP, > it is immutable. The IMAP server also remembers things like size and > assigns messages a unique ID, so mail readers that have already > downloaded the message with that ID, don't have to download the whole > message again to verify whether it's contents have magically changed. > What you're suggesting is not simply compatible with IMAP standards. > > The normal way of applying changes to messages is just like a mail > client connecting to IMAP: create a new message and save it to the > store, then delete the old one. Just like that. But instead of using IMAP interface to do it, you can also do it with doveadm: 1. doveadm fetch mail 2. doveadm delete mail 3. modify fetched mail 4. doveadm import modified mail back Simple as that. There is no other supported way to do it. Editing mail objects on storage will break things. Sami
Re: Modify stored mail contents?
On 23-07-17 13:07, Evan Martin wrote: > It looks like the mail filter plugin > [https://wiki2.dovecot.org/Plugins/MailFilter] is almost exactly what I > want, except for this: > >> Currently the filtering must not modify the message in any way: mail > -> write filter -> read filter -> must produce exactly the original mail > back. >> (TODO: Modifying the mail during writing would be possible with some > code changes.) > > Is there any prospect of those code changes being made, so that the > filter can modify mail contents? > > There's no indication in the docs or the code of what would break if the > contents were modified, but I'm guessing indexes and caches would be out > of date and would need to be rebuilt? Is it possible to just disable > those? I don't need high performance. > > On 22/07/2017 12:51 PM, Evan Martin wrote: >> Is there a safe way to modify the contents of emails stored by >> Dovecot? I'll probably only want to change the message bodies, not the >> headers, if that matters. Looking for ways to do this both for >> existing emails and new emails as they are received (though anything >> that works for existing emails can probably just be run again for new >> emails.) My mail storage is currently mdbox, but I could migrate to >> another format if that helps. In general, you should not do this. When a message is stored using IMAP, it is immutable. The IMAP server also remembers things like size and assigns messages a unique ID, so mail readers that have already downloaded the message with that ID, don't have to download the whole message again to verify whether it's contents have magically changed. What you're suggesting is not simply compatible with IMAP standards. The normal way of applying changes to messages is just like a mail client connecting to IMAP: create a new message and save it to the store, then delete the old one. Two people already asked you in this thread what the actual problem is you're trying to solve, but you failed to answer the question. I'm afraid that we can't help you any further without more detail. Kind regards, Tom signature.asc Description: OpenPGP digital signature
Re: Modify stored mail contents?
It looks like the mail filter plugin [https://wiki2.dovecot.org/Plugins/MailFilter] is almost exactly what I want, except for this: > Currently the filtering must not modify the message in any way: mail -> write filter -> read filter -> must produce exactly the original mail back. > (TODO: Modifying the mail during writing would be possible with some code changes.) Is there any prospect of those code changes being made, so that the filter can modify mail contents? There's no indication in the docs or the code of what would break if the contents were modified, but I'm guessing indexes and caches would be out of date and would need to be rebuilt? Is it possible to just disable those? I don't need high performance. On 22/07/2017 12:51 PM, Evan Martin wrote: Is there a safe way to modify the contents of emails stored by Dovecot? I'll probably only want to change the message bodies, not the headers, if that matters. Looking for ways to do this both for existing emails and new emails as they are received (though anything that works for existing emails can probably just be run again for new emails.) My mail storage is currently mdbox, but I could migrate to another format if that helps.
Re: Modify stored mail contents?
On 22/07/2017 11:51, Evan Martin wrote: Is there a safe way to modify the contents of emails stored by Dovecot? I'll probably only want to change the message bodies, not the headers, if that matters. I use javamail[1] for this. The javamail API can be used to get and put emails with IMAP so dovecot is kept in sync with the changes you make between fetching and storing. 1. https://javaee.github.io/javamail/
