Re: dovecot-submission SMTP send error with Thunderbird (BODY=8BITMIME)

2017-12-23 Thread Reuben Farrelly 

Hi again,

On 24/12/2017 7:11 am, Stephan Bosch wrote:

Op 12/23/2017 om 7:18 AM schreef Reuben Farrelly:

Hi,

With latest 2.3 -git (and 2.3.0 release), I'm running into this error
with Thunderbird:

"An error occurred while sending mail. The mail server responded:
5.5.4 Unsupported mail BODY type. Please verify that your email
address is correct in your account settings and try again."

This is fatal and means Thunderbird cannot use the submission service
- fortunately I can revert back to a native Postfix service which works.

Here's a tcpdump of the conversation:

thunderstorm /etc/dovecot/conf.d # tcpdump -A port 587

dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes


I cannot reproduce this behavior so far.


The odd thing is, I can 100% of the time reproduce this with the client, 
but 100% not reproduce it if I put the commands in by hand, at least up 
to the MAIL FROM: part of the conversation.


Here's the TB log:

[Main Thread]: I/SMTP SMTP Connecting to: smtp.reub.net:587
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 220 thunderstorm.reub.net Dovecot 
ready.

[Main Thread]: I/SMTP SMTP entering state: 14
[Main Thread]: I/SMTP SMTP Send: EHLO 
[IPv6:2001:44b8:31d4:1311:c0c8:3064:a55b:ae82]


[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 250-thunderstorm.reub.net
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 250-8BITMIME
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 250-AUTH PLAIN LOGIN
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 250-BURL imap
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 250-CHUNKING
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 250-ENHANCEDSTATUSCODES
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 250-SIZE
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 250-STARTTLS
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 250 PIPELINING
[Main Thread]: I/SMTP SMTP entering state: 4
[Main Thread]: I/SMTP SMTP entering state: 21
[Main Thread]: D/SMTP SMTP auth: server caps 0x20338, pref 0x300, failed 
0x0, avail caps 0x300
[Main Thread]: D/SMTP (GSSAPI = 0x800, CRAM = 0x2000, NTLM = 0x4000, MSN 
=  0x8000, PLAIN = 0x200, LOGIN = 0x100, EXTERNAL = 0x400)

[Main Thread]: D/SMTP trying auth method 0x200
[Main Thread]: I/SMTP SMTP entering state: 16
[Main Thread]: D/SMTP SMTP AuthLoginStep1() for reu...@smtp.reub.net
[Main Thread]: D/SMTP PLAIN auth
[Main Thread]: I/SMTP Logging suppressed for this command (it probably 
contained authentication information)

[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 235 2.7.0 Logged in.
[Main Thread]: I/SMTP SMTP entering state: 18
[Main Thread]: D/SMTP SMTP Login response, code 235
[Main Thread]: I/SMTP SMTP entering state: 3
[Main Thread]: I/SMTP SMTP Send: MAIL FROM: 
BODY=8BITMIME SIZE=425


[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 555 5.5.4 Unsupported mail BODY type
[Main Thread]: I/SMTP SMTP entering state: 5
[Main Thread]: I/SMTP SMTP Send: QUIT

[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP entering state: 0
[Main Thread]: I/SMTP SMTP Response: 221 2.0.0 Bye
[Main Thread]: I/SMTP SMTP entering state: 11
[Main Thread]: I/SMTP SMTP entering state: 12
[Main Thread]: I/SMTP SMTP connection error quitting 80004004, ignoring


(ngrep -Wbyline produces more readable output)


Thanks.  ngrep seems to hate IPv6 though, but after a bit of forcing 
IPv4 it started working (the output was the same as the tcpdump).



I will continue experimenting a bit. I still have to actually try this
with Thunderbird. Any other configuration I should know about (`dovecot
-n`)?

I think that'd help.  I'm using Thunderbird on Windows.

This thread might be of interest:

http://forums.mozillazine.org/viewtopic.php?f=39=2849171
https://bugzilla.mozilla.org/show_bug.cgi?id=1032302

It all seems to suggest that this could be a client side specific bit of 
quirkiness that dovecot submission is not handling.


Config is up here:

http://www.reub.net/files/dovecot/thunderstorm-dovecot.conf

Thanks,
Reuben

Re: sieve_pipe_socket_dir not created at startup for configured pipe service

2017-12-23 Thread Stephan Bosch 
Op 12/18/2017 om 1:51 AM schreef Garth Corral:
> Hi, all
>
> I’m new to the list but not to dovecot.  I’ve been using it in a basic 
> configuration for some time, but finally decided to tweak my deployed system 
> to take advantage of some more dovecot features.  In particular I’m trying to 
> set up pigeonhole to implement spam retraining with imapsieve.  All of this 
> is with dovecot 2.2.31 (65cde28) and pigeonhole 0.4.19.
>
> Before going any further let me start by saying that I have gotten all of 
> this to work.  It works when I can get dovecot to start up, that is.  My 
> configuration is pretty much straight from the docs, with a few tweaks for my 
> particular needs.  I’m trying to set up a pipe service using 
> sieve-extprograms, and the relevant part of my config looks like this:
>
> plugin {
>
>   sieve_pipe_input_eol = lf
>
>   sieve_pipe_socket_dir = sieve-pipe
>   sieve_filter_socket_dir = sieve-filter
>   sieve_execute_socket_dir = sieve-execute
>
>   sieve_pipe_bin_dir = /usr/local/libexec/dovecot/sieve-pipe
>   sieve_filter_bin_dir = /usr/local/libexec/dovecot/sieve-filter
>   sieve_execute_bin_dir = /usr/local/libexec/dovecot/sieve-execute
> }

Define either *bin_dir or *socket_dir; not both. These are two
alternative ways of running scripts, either using a direct fork() or
through a script service, respectively.

https://wiki.dovecot.org/Pigeonhole/Sieve/Plugins/Extprograms

> service sieve-train-ham {
>   executable = script /usr/local/libexec/dovecot/sieve-pipe/train-ham.sh
>
>   # Needs access to dspam config and lockfiles.
>   user = dspam
>
>   # socket name is program-name in Sieve (without sieve-pipe/ prefix)
>   unix_listener sieve-pipe/train-ham {
>   }
> }

>
> It’s my understanding from reading the docs that the sieve_pipe_socket_dir 
> specifies a directory that is relative to base_dir, which is the default 
> /var/run/dovecot in my case.  The issue I’m having is that dovecot will not 
> start, and spews the following errors:
>
> dovecot: master: Error: bind(/var/run/dovecot/sieve-pipe/train-ham) failed: 
> No such file or directory
> dovecot: master: Fatal: Failed to start listeners

First of all, this directory does not strictly need to be relative to
/var/run/dovecot. It can be an absolute path to somewhere else.

The relative directories are indeed not currently created implicitly.
Hmm, we will discuss this internally.

> Once dovecot fails startup, it leaves /var/run/dovecot around and if I 
> manually create the sieve-pipe directory there it will start up, create the 
> sockets there and everything works as intended subsequently.  The problem, 
> though, is that on normal shutdown all of /var/run/dovecot goes away and the 
> at the next startup it fails to start again.  Needless to say this isn’t 
> great for unintended reboots, etc.
>
> So, can anyone see anything obvious that I’m doing wrong?  I’m just assuming 
> that dovecot should create the needed subdir since I can’t find anything in 
> the docs to suggest a way to create it otherwise.  I’ve tried all I can think 
> of at the moment to try to remedy this without success.  I’m happy to provide 
> additional details as needed to try to track this down.
>


Regards,

Stephan.

Re: Dovecot 2.3-rc Logging Format

2017-12-23 Thread Stephan Bosch 
Op 12/21/2017 om 8:57 AM schreef Thomas Leuxner:
> Hi,
>
> the release candidate defaults to a log format with session IDs.
>
> mail_log_prefix = "%s(%u)<%{pid}><%{session}>: "
>
> As the LMTP service seems to have the session ID hardcoded, the IDs get 
> duplicated in the logs:
>
> Dec 21 08:48:03 edi dovecot: lmtp(26573): Connect from local
> Dec 21 08:48:03 edi dovecot: lmtp(t...@leuxner.net)[26573]: 
> : fCVaBjNnO1rNZwAAIROLbg: sieve: 
> msgid=<2323281.OorJHhdMHM@ylum>, time=158ms, status=stored mail into mailbox 
> ':public/Mailing-Lists/Debian-User'
> Dec 21 08:48:03 edi dovecot: lmtp(26573): Disconnect from local: Client has 
> quit the connection (state = READY)

Fixed in release.

Regards,

Stephan.

[Dovecot-news] Released Pigeonhole v0.5.0 for Dovecot v2.3.0.

2017-12-23 Thread Stephan Bosch 
Hello Dovecot users,

Here's the definitive 0.5.0 release. There is one RC fix regarding the
duplicate presence of the session ID in the LMTP/LDA log lines produced
by the Sieve interpreter. Since this change is relative to the RC, it is
not in the change log below.

Changelog v0.5.0:

* editheader extension: The implementation of header modifications is
  heavily updated. Although the functionality has not changed, the
  underlying code was updated to address several static analysis
  warnings, runtime integer arithmetic warnings (Clang), and to match
  updates in the Dovecot stream API.
+ variables extension: Made the maximum scope and variable size
  configurable.
+ subaddress: Support multiple recipient_delimiters.
- enotify extension: mailto method: Fixed parsing of mailto URI with
  only a header part.
- enotify plugin: mailto method: Make sure the "From:" header is set to
  a usable address and not "(null)".
- Fixed writing address headers to outgoing messages. Sometimes headers
  were MIME-encoded twice, yielding invalid results.

The release is available as follows:

https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.0.tar.gz
https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.0.tar.gz.sig

Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for
more information. Have fun testing this release and don't hesitate to
notify me when there are any problems.

Regards,

-- 
Stephan Bosch
step...@rename-it.nl











___
Dovecot-news mailing list
Dovecot-news@dovecot.org
https://dovecot.org/mailman/listinfo/dovecot-news

Released Pigeonhole v0.5.0 for Dovecot v2.3.0.

2017-12-23 Thread Stephan Bosch 
Hello Dovecot users,

Here's the definitive 0.5.0 release. There is one RC fix regarding the
duplicate presence of the session ID in the LMTP/LDA log lines produced
by the Sieve interpreter. Since this change is relative to the RC, it is
not in the change log below.

Changelog v0.5.0:

* editheader extension: The implementation of header modifications is
  heavily updated. Although the functionality has not changed, the
  underlying code was updated to address several static analysis
  warnings, runtime integer arithmetic warnings (Clang), and to match
  updates in the Dovecot stream API.
+ variables extension: Made the maximum scope and variable size
  configurable.
+ subaddress: Support multiple recipient_delimiters.
- enotify extension: mailto method: Fixed parsing of mailto URI with
  only a header part.
- enotify plugin: mailto method: Make sure the "From:" header is set to
  a usable address and not "(null)".
- Fixed writing address headers to outgoing messages. Sometimes headers
  were MIME-encoded twice, yielding invalid results.

The release is available as follows:

https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.0.tar.gz
https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.0.tar.gz.sig

Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for
more information. Have fun testing this release and don't hesitate to
notify me when there are any problems.

Regards,

-- 
Stephan Bosch
step...@rename-it.nl

Re: dovecot-submission SMTP send error with Thunderbird (BODY=8BITMIME)

2017-12-23 Thread Stephan Bosch 
Op 12/23/2017 om 7:18 AM schreef Reuben Farrelly:
> Hi,
>
> With latest 2.3 -git (and 2.3.0 release), I'm running into this error
> with Thunderbird:
>
> "An error occurred while sending mail. The mail server responded:
> 5.5.4 Unsupported mail BODY type. Please verify that your email
> address is correct in your account settings and try again."
>
> This is fatal and means Thunderbird cannot use the submission service
> - fortunately I can revert back to a native Postfix service which works.
>
> Here's a tcpdump of the conversation:
>
> thunderstorm /etc/dovecot/conf.d # tcpdump -A port 587
>
> dropped privs to tcpdump
> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

I cannot reproduce this behavior so far.

(ngrep -Wbyline produces more readable output)

> Curiously enabling rawlog doesn't capture this error, which is why I
> used tcpdump above.  The logs from it like this:
>
> thunderstorm /run/dovecot/rawlogs # cat 20171219-141149.5633.1.in
> 1513653109.109030 220 thunderstorm.reub.net ESMTP Postfix (3.3-20171028)
> 1513653109.109266 250-thunderstorm.reub.net
> 1513653109.109266 250-PIPELINING
> 1513653109.109266 250-SIZE 4096
> 1513653109.109266 250-VRFY
> 1513653109.109266 250-ETRN
> 1513653109.109266 250-STARTTLS
> 1513653109.109266 250-ENHANCEDSTATUSCODES
> 1513653109.109266 250-8BITMIME
> 1513653109.109266 250-DSN
> 1513653109.109266 250 SMTPUTF8
> 1513653130.973720 221 2.0.0 Bye
>
> thunderstorm /run/dovecot/rawlogs # cat 20171219-141149.5633.1.out
> 1513653109.109087 EHLO thunderstorm.reub.net
> 1513653130.973351 QUIT
> 1513653130.973829 QUIT
> thunderstorm /run/dovecot/rawlogs #

Double QUIT. That is a minor bug, but not related.

>
> This with:
>
> # Write protocol logs for relay connection to this directory for
> debugging
> #submission_relay_rawlog_dir =
> submission_relay_rawlog_dir = /run/dovecot/rawlogs/
>
> Is this a separate but unrelated problem with rawlog support in this
> the submission?  I would have expected it to capture the full
> conversation log including any protocol errors and failures like this.

No. This is the connection with the relay (backend MTA) server. Use the
normal Dovecot rawlog facilities to capture protocol interaction with
the client instead.

Well, now we do know for sure that Dovecot is responsible for the error,
since the interaction with the relay server shows no issues.

I will continue experimenting a bit. I still have to actually try this
with Thunderbird. Any other configuration I should know about (`dovecot
-n`)?

Regards,

Stephan.

Re: Pigeonhole implicit keep gets unfiltered message

2017-12-23 Thread Adam Weinberger 
>> (2017/12/23 @ 1051 EST): Stephan Bosch said, in 2.0K: <<
> Op 12/22/2017 om 3:43 AM schreef Adam Weinberger:
> >> On 21 Dec, 2017, at 14:37, Stephan Bosch  wrote:
> >>
> >> Op 12/19/2017 om 8:41 AM schreef Adam Weinberger:
> >>> I'm getting a behaviour with pigeonhole that I wasn't expecting. Am I
> >>> misunderstanding the design?
> >>>
> >>> I run my messages through a vnd.dovecot.filter. It's essentially this:
> >>>
> >>> filter "spam_filter";
> >>> if spamheaders {
> >>>     fileinto "spam";
> >>>     stop;
> >>> }
> >>>
> >>> Mail stored in the spam folder is the filtered version, but the
> >>> implicit-keep message is the original, unfiltered message. If I add an
> >>> explicit `keep;` to the end, it stores the filtered version into my
> >>> inbox.
> >>>
> >>> Based on the filter RFC, I was expecting the implicit keep to retain
> >>> the filtered version. Am I misinterpreting the spec?
> >>
> >> I did a quick test, and I am not seeing any problems.
> >>
> >> However, what is that spamheaders test in your script?
> >
> > Hi Stephan,
> >
> > The block looks like this:
> >
> >     ### BOGOFILTER
> >     filter "bogofilter_filter";
> >
> >     if header :contains "X-Bogosity" [
> >     "Spam, tests=bogofilter, spamicity=1.00",
> >     "Spam, tests=bogofilter, spamicity=0.99"
> >     ] {
> >     fileinto "spam/totally";
> >     stop;
> >     }
> >     elsif header :contains "X-Bogosity" "Spam," {
> >     fileinto "spam/probably";
> >     stop;
> >     }
> >     elsif header :contains "X-Bogosity" "Unsure," {
> >     fileinto "spam/maybe";
> >     stop;
> >     }
> >
> > Bogofilter adds an X-Bogosity header. With the block as it is, when it
> > hits the implicit keep the message has no X-Bogosity header. When I
> > add 'keep;' to the end, it does have the header.
> >
> > If it's just me, that's fine, as it's incredibly easy to work around.
> 
> What version is this? Please provide full config from `dovecot -n`.
> 
> Regards,
> 
> Stephan.
> 
>> end of "Re: Pigeonhole implicit keep gets unfiltered message" from Stephan 
>> Bosch <<

It's dovecot 2.2.33.2, and pigeonhole 0.4.21.

Here's my dovecot -n output:

# 2.2.33.2 (d6601f4ec): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.21 (92477967)
# OS: FreeBSD 11.1-RELEASE-p6 amd64  nullfs
auth_mechanisms = plain login
first_valid_gid = 1021
first_valid_uid = 1021
last_valid_gid = 1022
last_valid_uid = 1022
listen = imap.jail.apnoea.adamw.org
mail_location = mdbox:/mail/%u/mail
mail_plugins = " zlib virtual fts fts_lucene"
mail_prefetch_count = 5
mailbox_list_index = yes
mdbox_rotate_size = 10 M
namespace {
  location = virtual:/mail/%u/mail/virtual
  prefix = virtual/
  separator = /
}
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox FreeBSD {
autoexpunge = 17 weeks
  }
  mailbox FreeBSD/TodaysCommits {
autoexpunge = 2 days
  }
  mailbox FreeBSD/automation {
autoexpunge = 1 days
  }
  mailbox FreeBSD/portmgr {
autoexpunge = 26 weeks
  }
  mailbox FreeBSD/ports {
autoexpunge = 12 weeks
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
autoexpunge = 30 days
special_use = \Trash
  }
  mailbox spam/probably {
autoexpunge = 30 days
  }
  mailbox spam/totally {
autoexpunge = 5 days
  }
  mailbox spamtrap {
autoexpunge = 30 days
  }
  mailbox uberspam {
autoexpunge = 30 days
  }
  prefix = 
  separator = /
}
passdb {
  args = scheme=BLF-CRYPT username_format=%u /path/to/userdb.passwd
  driver = passwd-file
}
plugin {
  fts = lucene
  fts_autoindex = yes
  fts_autoindex_max_recent_msgs = 25
  fts_lucene = whitespace_chars=@
  sieve = file:/scripts/sieve/%u.sieve;bindir=/mail/%u/sieve/
  sieve_extensions = +vnd.dovecot.pipe +vnd.dovecot.filter +editheader
  sieve_filter_bin_dir = /scripts/sieve/filter
  sieve_pipe_bin_dir = /scripts/sieve/pipe
  sieve_plugins = sieve_extprograms
}
protocols = imap lmtp
service imap-login {
  inet_listener imaps {
port = 0
  }
}
service lmtp {
  inet_listener lmtp {
port = 24
  }
}
ssl = required
ssl_cert = http://www.adamw.org

Re: Pigeonhole implicit keep gets unfiltered message

2017-12-23 Thread Stephan Bosch 
Op 12/22/2017 om 3:43 AM schreef Adam Weinberger:
>> On 21 Dec, 2017, at 14:37, Stephan Bosch  wrote:
>>
>> Op 12/19/2017 om 8:41 AM schreef Adam Weinberger:
>>> I'm getting a behaviour with pigeonhole that I wasn't expecting. Am I
>>> misunderstanding the design?
>>>
>>> I run my messages through a vnd.dovecot.filter. It's essentially this:
>>>
>>> filter "spam_filter";
>>> if spamheaders {
>>>     fileinto "spam";
>>>     stop;
>>> }
>>>
>>> Mail stored in the spam folder is the filtered version, but the
>>> implicit-keep message is the original, unfiltered message. If I add an
>>> explicit `keep;` to the end, it stores the filtered version into my
>>> inbox.
>>>
>>> Based on the filter RFC, I was expecting the implicit keep to retain
>>> the filtered version. Am I misinterpreting the spec?
>>
>> I did a quick test, and I am not seeing any problems.
>>
>> However, what is that spamheaders test in your script?
>
> Hi Stephan,
>
> The block looks like this:
>
>     ### BOGOFILTER
>     filter "bogofilter_filter";
>
>     if header :contains "X-Bogosity" [
>     "Spam, tests=bogofilter, spamicity=1.00",
>     "Spam, tests=bogofilter, spamicity=0.99"
>     ] {
>     fileinto "spam/totally";
>     stop;
>     }
>     elsif header :contains "X-Bogosity" "Spam," {
>     fileinto "spam/probably";
>     stop;
>     }
>     elsif header :contains "X-Bogosity" "Unsure," {
>     fileinto "spam/maybe";
>     stop;
>     }
>
> Bogofilter adds an X-Bogosity header. With the block as it is, when it
> hits the implicit keep the message has no X-Bogosity header. When I
> add 'keep;' to the end, it does have the header.
>
> If it's just me, that's fine, as it's incredibly easy to work around.

What version is this? Please provide full config from `dovecot -n`.

Regards,

Stephan.

Re: Lua Auth

2017-12-23 Thread Aki Tuomi 

> On December 22, 2017 at 9:16 PM Mark Moseley  wrote:
> 
> 
> On Fri, Dec 22, 2017 at 5:18 AM,  wrote:
> 
> >
> > > On December 22, 2017 at 8:20 AM Mark Moseley 
> > wrote:
> > >
> > >
> > > On Thu, Dec 21, 2017 at 9:51 PM, Aki Tuomi  wrote:
> > >
> > > >
> > > > > On December 22, 2017 at 6:43 AM Mark Moseley 
> > > > wrote:
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2) Is there an appropriate way to return data with spaces in it (or
> > > > > > presumably other non-alphanum chars. My quota name had a space in
> > it,
> > > > > > which
> > > > > > somehow got interpreted as 'yes' , i.e.:
> > > > > >
> > > > > > imap: Error: Failed to initialize quota: Invalid quota root quota:
> > > > Unknown
> > > > > > quota backend: yes
> > > > > >
> > > > > > I simply changed the space to an underscore as a workaround, but
> > I'm
> > > > > > curious if there's a better way. I tried various quoting without
> > > > success.
> > > > > > Didn't try escaping yet.
> > > > > >
> > > > > >
> > > > > > 2) Instead of string, return a key value table. you can have
> > spaces in
> > > > > > values.
> > > > > >
> > > > > >
> > > > > >
> > > > > Does this work for auth_passdb_lookup too, or just
> > auth_userdb_lookup?
> > > > I've
> > > > > been returning a table with auth_userdb_lookup just fine. But when I
> > try
> > > > > using it with passdb (and despite being very very sure that a
> > 'password'
> > > > > key exists in the table I'm returning from auth_passdb_lookup() --
> > I'm
> > > > > logging it one line above the return), the passdb auth fails with
> > this
> > > > log
> > > > > entry:
> > > > >
> > > > > Dec 21 23:29:22 auth-worker(7779): Info:
> > > > > lua(te...@test.com,10.20.103.32,):
> > > > > No password returned (and no nopassword)
> > > > >
> > > > > I guess it's not seeing the password key in the table I'm returning.
> > If I
> > > > > return a concat'd string ("password=... user=...") from
> > > > > auth_passdb_lookup(), it works just fine.
> > > > >
> > > > > I was also curious if there's a way to pass info between
> > > > auth_userdb_lookup
> > > > > and auth_passdb_lookup. I was trying to use a table with
> > > > > auth_passdb_lookup() so I could take advantage of prefetch and
> > thought
> > > > that
> > > > > if auth_passdb_lookup didn't take a table, I could stash data away
> > and
> > > > then
> > > > > un-stash it in auth_userdb_lookup
> > > > >
> > > > > Thanks!
> > > > >
> > > > >
> > > >
> > > > Yeah, this is a bug we have fixed =)
> > > >
> > > > https://github.com/dovecot/core/commit/c86575ac9776d0995355d03719c82e
> > > > 7ceac802e6#diff-83374eeaee91d90e848390ba3c7b264a
> > > >
> > > >
> > >
> > > I'm on rc1, so I appear to already have that git commit (as part of rc1).
> > >
> > > # /usr/sbin/dovecot  --version
> > > 2.3.0.rc1 (12aba5948)
> > >
> > > For testing this, I tried replacing my passdb lookup with this:
> > >
> > > function auth_passdb_lookup(req)
> > > passdb_table = {}
> > > passdb_table[ 'password' ] = 'test'
> > > passdb_table[ 'user' ] = 'te...@test.com'
> > >
> > > return dovecot.auth.PASSDB_RESULT_OK, passdb_table
> > > end
> > >
> > > and still get:
> > >
> > > Dec 22 01:17:17 auth-worker(9711): Info:
> > > lua(te...@test.com,10.20.103.32,):
> > > No password returned (and no nopassword)
> > >
> > > Replacing that return statement with this:
> > >
> > > return dovecot.auth.PASSDB_RESULT_OK, 'password=test user=te...@test.com
> > '
> > >
> > > authenticates successfully.
> >
> > Fixed in https://github.com/dovecot/core/commit/
> > e5fb6b3b7d4e79475b451823ea6c0a02955ba06b
> >
> >
> >
> Works like a charm now, thanks!
> 
> As a matter of 'best practices', in my current iteration of Lua auth, I
> moved all my lookups to passdb (thus yesterday's emails to the list), so
> that it could be used with prefetch. Belatedly realizing that LMTP doesn't
> touch passdb, I rewrote the userdb lookup to call the same passdb lookup
> (which only happens for non-passdb/prefetch things) and then it copies the
> return table (but strips the 'userdb_' prefix). It's all working currently.
> BUT, does that sound sane? Or is there some gotcha I'm heading towards
> (yes, I realize the question is a bit vague -- just looking for very
> general "No, don't do that").
> 

Sounds ok to me.

> I'm curious too if I can set vars in the passdb lookup and then access then
> in userdb. Or is it random which auth-worker will handle the userdb lookup,
> relative to which one handled the passdb lookup? I tried dropping things in
> the req.userdb table in the passdb phase, but it was unset during the
> userdb phase.

The best way is to export userdb_variables from passdb lookup. The userdb table 
itself is currently read-only, but yeah, it might be a good idea actually to 
support writing like this at some point.

Aki

Re: dovecot-submission SMTP send error with Thunderbird (BODY=8BITMIME)

2017-12-23 Thread Aki Tuomi 
Hi, thank you for your report. We'll look into it!

Aki

> On December 23, 2017 at 8:18 AM Reuben Farrelly  
> wrote:
> 
> 
> Hi,
> 
> With latest 2.3 -git (and 2.3.0 release), I'm running into this error 
> with Thunderbird:
> 
> "An error occurred while sending mail. The mail server responded: 5.5.4 
> Unsupported mail BODY type. Please verify that your email address is 
> correct in your account settings and try again."
> 
> This is fatal and means Thunderbird cannot use the submission service - 
> fortunately I can revert back to a native Postfix service which works.
> 
> Here's a tcpdump of the conversation:
> 
> thunderstorm /etc/dovecot/conf.d # tcpdump -A port 587
> 
> dropped privs to tcpdump
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
> 
> 
> 14:12:19.975982 IP6 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175 > 
> inside-mail.reub.net.submission: Flags [S], seq 572328223, win 64800, 
> options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0
> `._.. .? .D.1...E.>. .D.1..#...K".   .w..
> 14:12:19.976022 IP6 inside-mail.reub.net.submission > 
> 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175: Flags [S.], seq 
> 3954361671, ack 572328224, win 28800, options [mss 
> 1440,nop,nop,sackOK,nop,wscale 7], length 0
> `.c|. .@ .D.1..# .D.1...E.>..K.G". ..p.:4..
> 14:12:19.976158 IP6 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175 > 
> inside-mail.reub.net.submission: Flags [.], ack 1, win 8235, length 0
> `._? .D.1...E.>. .D.1..#...K".   ...HP. +. ..
> 14:12:19.983409 IP6 inside-mail.reub.net.submission > 
> 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175: Flags [P.], seq 1:43, ack 
> 1, win 225, length 42
> `.c|.>.@ .D.1..# .D.1...E.>..K.H".   P...:R..220 
> thunderstorm.reub.net Dovecot ready.
> 
> 14:12:19.992790 IP6 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175 > 
> inside-mail.reub.net.submission: Flags [P.], seq 1:54, ack 43, win 8234, 
> length 53
> `._..I.? .D.1...E.>. .D.1..#...K".   ...rP. *.H..EHLO 
> [IPv6:2001:44b8:31d4:1311:45ec:e191:8093:3e9d]
> 
> 14:12:19.992828 IP6 inside-mail.reub.net.submission > 
> 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175: Flags [.], ack 54, win 
> 225, length 0
> `.c|...@ .D.1..# .D.1...E.>..K.r".  UP...:(..
> 14:12:19.993027 IP6 inside-mail.reub.net.submission > 
> 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175: Flags [P.], seq 43:200, 
> ack 54, win 225, length 157
> `.c|...@ .D.1..# .D.1...E.>..K.r". 
> UP...:...250-thunderstorm.reub.net
> 250-8BITMIME
> 250-AUTH PLAIN LOGIN
> 250-BURL imap
> 250-CHUNKING
> 250-ENHANCEDSTATUSCODES
> 250-SIZE
> 250-STARTTLS
> 250 PIPELINING
> 
> 14:12:20.015953 IP6 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175 > 
> inside-mail.reub.net.submission: Flags [P.], seq 54:91, ack 200, win 
> 8234, length 37
> `._..9.? .D.1...E.>. .D.1..#...K".  UP. *AUTH PLAIN 
> 
> 14:12:20.035676 IP6 inside-mail.reub.net.submission > 
> 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175: Flags [P.], seq 200:222, 
> ack 91, win 225, length 22
> `.c|.*.@ .D.1..# .D.1...E.>..K..".  zP...:>..235 
> 2.7.0 Logged in.
> 
> 14:12:20.036642 IP6 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175 > 
> inside-mail.reub.net.submission: Flags [P.], seq 91:143, ack 222, win 
> 8234, length 52
> `._..H.? .D.1...E.>. .D.1..#...K".  z...%P. *MAIL 
> FROM: BODY=8BITMIME SIZE=444
> 
> 14:12:20.036826 IP6 inside-mail.reub.net.submission > 
> 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175: Flags [P.], seq 222:260, 
> ack 143, win 225, length 38
> `.c|.:.@ .D.1..# .D.1...E.>..K.%".  .P...:N..555 
> 5.5.4 Unsupported mail BODY type
> 
> 14:12:20.089196 IP6 2001:44b8:31d4:1311:45ec:e191:8093:3e9d.61175 > 
> inside-mail.reub.net.submission: Flags [.], ack 260, win 8233, length 0
> `._? .D.1...E.>. .D.1..#...K".  KP. )
> 
> 
> Curiously enabling rawlog doesn't capture this error, which is why I 
> used tcpdump above.  The logs from it like this:
> 
> thunderstorm /run/dovecot/rawlogs # cat 20171219-141149.5633.1.in
> 1513653109.109030 220 thunderstorm.reub.net ESMTP Postfix (3.3-20171028)
> 1513653109.109266 250-thunderstorm.reub.net
> 1513653109.109266 250-PIPELINING
> 1513653109.109266 250-SIZE 4096
> 1513653109.109266 250-VRFY
> 1513653109.109266 250-ETRN
> 1513653109.109266 250-STARTTLS
> 1513653109.109266 250-ENHANCEDSTATUSCODES
> 1513653109.109266 250-8BITMIME
> 1513653109.109266 250-DSN
> 1513653109.109266 250 SMTPUTF8
> 1513653130.973720 221 2.0.0 Bye
> 
> thunderstorm /run/dovecot/rawlogs # cat 20171219-141149.5633.1.out
> 1513653109.109087 EHLO thunderstorm.reub.net
> 1513653130.973351 QUIT
> 1513653130.973829 QUIT
> thunderstorm /run/dovecot/rawlogs #
> 
> This with:
> 
> # Write protocol