[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
[Expired for fetchmail (Ubuntu) because there has been no activity for 60 days.] ** Changed in: fetchmail (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
Thanks for the report Bill and even more for the awesome and detailed explanations Matthias! AFAICS there is nothing in here for Ubuntu to fix "right now". As versions 6.5/7.0 are releases we will pick them up but that isn't what this bug was about. Gladly config based solutions were found to handle the problem, but I'll mark the bug as incomplete to auto-expire if nothing else needs to be added in your opinion. ** Changed in: fetchmail (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
I see that I overlooked the NEWS file. That's more detailed than I'm used to seeing. As I was just now skimming through it, it was clear that moves made to protect users from weak algorithms assumed they're on an untunneled connection, which is not always the case. Sometimes the SSL is just used for verification and the crypto is just redundant. I should also mention that I struggled with the "no sslcertck" syntax. All the options I've been using to that point were " " format, and "no sslcertck" is an exceptional transpose of it. I first tried "sslcertck no" because I was sure the key-value wouldn't flip. But in fact the "keyword" really included a space. Coupled with the inconsistency of some keywords starting with "set", I felt I couldn't trust the man page. Adding to the confusion, some options take arguments and some do not, but the Keyword/Option Summary table doesn't show any BNF and it omits the token, making so we have to work out from the wording of the "Function" column whether it's unary or takes an argument. Anyway, I appreciate your help and I hope my feedback helps for future revisions. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
Bill, I can sense quite a deal of frustration on your end, and am sorry for that. I should have mentioned to also look into the NEWS file, which at least would have told you that sslcertck is now the default. Fetchmail's -vv verbose mode might also try to log more details on the decisions along the way... Your feedback as to how you find documentation confusing or incomplete or incompletely referenced is helpful, thank you for that - although it will take a while before improvements will make it into Ubuntu, this is due to how software ends up for Ubuntu. On the other hand, I must say that nobody promised that fetchmail 6.4 would be bug-for-bug-compatible with 6.3, and fetchmail has indeed changed in the years between the 6.3.26 and 6.4.1X releases. In particular, I was making fetchmail 6.4 protect users' credentials more thoroughly, and this is why fetchmail finally defaults to certificate validation, and that makes "opportunistic" encryption quite a bit more mandatory, and the failing STARTTLS after the server hadn't offered it was/is not helpful: In order to validate certificates, fetchmail will try to obtain one - and on a non-SSL connection this means starttls, and fetchmail tries it blind. I see that it is less transparent to users than I would have hoped, so thanks for mirroring that - the certificate validation makes TLS (or STARTTLS in-band negotiation for that matter) less opportunistic and more mandatory than parts of the documentation suggest. So, "no sslcertck" is currently required to make "opportunistic" TLS really opportunistic (your "config 2" - and not in an obscure way mandatory), and "sslproto ''" indeed defeats TLS negotiation at all (your config 3). I need to think a bit more about which changes could go into 6.5, and which will have to wait for 7.0, where the version is more of a flag to users that reads "beware of incompatibilities". -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
** Changed in: fetchmail (Ubuntu) Status: Incomplete => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
I've also noticed that "auto" and "TLS1+" have the same meaning when passed to --SSLPROTO. That's part of the problem. So now there is duplication, and no way to get opportunistic crypto. If auto was previously opportunistic, then the change to remove it should be rolled back. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
I appreciate the quick response, but I have to say both of you overlooked some of the data I presented, so I will elaborate. I was able to workaround the bug, but there are still bugs here. Having read the man page more times, I continue to find more anomalies than answers, which I will elaborate here. To be clear, fetchmail changed, not my config. This config works on fetchmail 6.3.26 but fails on 6.4.16: ``` skip underwood-onion via 127.0.0.1 protocol imap port 12345 username "billyikes" fetchall ``` Nothing in your response suggests that I should expect this config to be treated differently from the two different versions of fetchmail. But it is. Verbose output shows that version 6.4.16 sends a "STARTTLS" even though the config does not call for SSL/TLS in any way. That's a bug. The man page says "sslproto auto" is a default which superficially seems reasonable until you notice that "auto" actually imposes TLS. This is a distortion of what users expect by the meaning of "automatic". My first attempt at a fix for underwood was to add these lines (which fixed some of my stanzas for other servers): ~~~ sslproto 'SSL3+' no sslcertck ~~~ Intution suggests this means: "If the server demands SSL, then be permissive and allow SSL3 or newer, but don't bother checking whether the cert is good or not, just use it if the server insists". But what we find is that the behavior is not intuitive w.r.t the options. And because it fails the rule of least astonishment, I'm calling it a bug. Moreover, the new manpage states: "This option has a dual use, out of historic fetchmail behaviour. It controls both the SSL/TLS protocol version and, if --ssl is not specified, the STARTTLS behaviour (upgrading the protocol to an SSL or TLS connection in-band). Some other options may however make TLS mandatory." To say sslproto "controls STARTTLS behaviour" without fully specifying how that behavior is controlled is a recipe for confusion. An empty argument is said to disable STARTTLS, but that was always the case. So there are two problems here. The first problem is that STARTTLS is technical under-the-hood jargon that end users probably should not be responsible for understanding. Users are basically told whether SSL/TLS is used, not what low-level commands are being passed to make that happen. The second problem is that auto states "require TLS", but does not mention the low-level "STARTTLS" that the section mentioned earlier. To developers, perhaps it is clear that "TLS" and "STARTTLS" are one in the same, but it's not clear to users who don't know the underlying details expressed in RFC specs. IOW, STARTTLS is mentioned just enough to confuse, and not enough to be understandable. Apart from the misleading & non-intuitive wording, some things are plainly false. E.g. this statement from the man page is simply not true: "Only if this option [sslproto] and --ssl are both missing for a poll, there will be opportunistic TLS for POP3 and IMAP, where fetchmail will attempt to upgrade to TLSv1 or newer." If that were a true statement, then the first stanza at the top of this post would work. This is a contradiction in the man page. That is, "sslproto auto" ironically states that TLS is mandated while simultaneously saying "auto" is also a default, and yet this behavior is not opportunistic. Opportunistic means: accommodate (or even proactively request) higher levels of security, but ultimately settle for no security at all if availability demands it. The "sslproto ''" empty string option is bizarre because it strictly disables STARTTLS without stating whether SSL is also disabled, apart from saying that it's incompatible with --ssl. To expressly /force/ no encryption can be useful but only in limited & obscure test situations. It's still good that it exists, but what about the much more common case of users who need opportunistic connections? The empty string denies the opportunity for crypto while all the other sslproto options deny the possibility of no encryption. Even more strange is "SSL23". How often does someone want to insist on SSL2 or SSL3, but not the higher options? There should be a "sslproto SSL2+". And most importantly there should be a "sslproto none+" to truly deliver opportunistic sessions. Regarding this suggestion: ``` poll underwood2hj3pwd.onion via 127.0.0.1 plugin "socat STDIO SOCKS4A:127.0.0.1:%h:%p,socksport=9050" ``` which I expanded to: ``` skip underwood2hj3pwd.onion via 127.0.0.1 plugin "socat STDIO SOCKS4A:127.0.0.1:%h:%p,socksport=9050" protocol imap port 143 username "billyikes" fetchall ``` result was: ``` fetchmail: running socat STDIO SOCKS4A:127.0.0.1:%h:%p,socksport=9050 (host 127.0.0.1 service 143) 2021/07/01 16:25:44 socat[2202] E socks: connect request rejected or failed fetchmail: socket error while fetching from
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
Hello Bill, I agree with Matthias' reply: fetchmail seems to be working just as expected. Also please note that we use this bug tracker to work on bugs in Ubuntu, so feature requests like a per-account SOCKS4a config parameter don't really fit here; at a first look the right venue would be [1]. I'm marking this bug report as Incomplete for now. If you still believe you found a bug in how fetchmail handles ssl/tls/starttls connections please provide more details about your findings and change the bug status back to New, we'll look to it again. Otherwise if you agree we don't really have a bug here please change the report status to Invalid. Thanks! [1] https://gitlab.com/fetchmail/fetchmail/ ** Changed in: fetchmail (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
** Description changed: Version 6.4.16 is unable to fetch mail from the underwood onion site. This is the output when trying to connect: fetchmail: normal termination, status 2 fetchmail: 6.4.16 querying underwood-onion (protocol IMAP) at Wed 30 Jun 2021 02:10:52 PM UTC: poll started fetchmail: Trying to connect to 127.0.0.1/12345...connected. fetchmail: IMAP< * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot ready. fetchmail: IMAP> A0001 CAPABILITY fetchmail: IMAP< * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN fetchmail: IMAP< A0001 OK Pre-login capabilities listed, post-login capabilities have more. fetchmail: IMAP> A0002 STARTTLS fetchmail: IMAP< A0002 BAD TLS support isn't enabled. fetchmail: 127.0.0.1: upgrade to TLS failed. fetchmail: Unknown login or authentication error on billyikes@127.0.0.1 fetchmail: socket error while fetching from billyikes@underwood-onion This worked with past versions. To reproduce, use this stanza in .fetchmailrc: skip underwood-onion via 127.0.0.1 protocol imap port 12345 username "billyikes" sslproto 'SSL3+' no sslcertck fetchall Note that past working stanzas did not need "sslproto" or "no sslcertck" but were introduced to after upgrading to 6.4.16. run these commands: $ socat TCP4-LISTEN:12345,reuseaddr,fork SOCKS4A:127.0.0.1:underwood2hj3pwd.onion:143,socksport=9050 $ fetchmail -v -d0 underwood-onion $ pkill socat It's the same outcome if "sslproto 'SSL23'" is used instead. This is one report, but there are a few bugs here: 1) inability to connect to handshake with bad TLS protocols. It's an onion site, so SSL is not needed for crypto (it's there for a different purpose). So if fetchmail is judging the crypto to be insecure, it's overzealous in this case. 2) the "Unknown login or authentication error" is not only a false error, it's alarming. It's the worst kind of false error because it tells the user that there's a problem with their account. 3) there is no per-account SOCKS4a config parameter, so users are pushed into this inconvenient and ugly hack of running socat and piping through that. The "plugin" parameter does not help in this case because fetchmail still attempts to resolve the underwood2hj3pwd.onion outside of the proxy. - Bug number 3 has always existed, but 1 & 2 are new regressions. + Bug (3) has always existed, but (1) & (2) are new regressions. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
Bill, as to your report, I'll break it down along the same bug numbers. Note I am speaking as the upstream maintainer here and am unaware of Ubuntu's Bug #1. Fetchmail behaves properly. You requested some sslproto on a non-wrapped port (i. e. no "ssl" option), so fetchmail tries STARTTLS instead, which then fails. I suggest you re-read the manual for the relevant options that you are using and potential references from those options to others, if any. This is deliberately worded quite vaguely. Bug #2 fetchmail has a 'login or authentication' error it cannot spell out in more detail for you, hence "unknown". Feel free to propose a better wording. 3a. I suggest you read the section "SOCKS" of the manual page. New options are quite high maintenance and I can't test them upstream for lack of socks proxies. 3b. From your earlier report [*] I suppose you might get away with something such as: poll underwood2hj3pwd.onion via 127.0.0.1 plugin "socat STDIO SOCKS4A:127.0.0.1:%h:%p,socksport=9050" # other options here _ [*] https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1924609 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934155] Re: fetchmail can no longer connect to underwood & gives false error msg (TLS issues)
** Description changed: Version 6.4.16 is unable to fetch mail from the underwood onion site. This is the output when trying to connect: fetchmail: normal termination, status 2 fetchmail: 6.4.16 querying underwood-onion (protocol IMAP) at Wed 30 Jun 2021 02:10:52 PM UTC: poll started fetchmail: Trying to connect to 127.0.0.1/12345...connected. fetchmail: IMAP< * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot ready. fetchmail: IMAP> A0001 CAPABILITY fetchmail: IMAP< * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN fetchmail: IMAP< A0001 OK Pre-login capabilities listed, post-login capabilities have more. fetchmail: IMAP> A0002 STARTTLS fetchmail: IMAP< A0002 BAD TLS support isn't enabled. fetchmail: 127.0.0.1: upgrade to TLS failed. fetchmail: Unknown login or authentication error on billyikes@127.0.0.1 fetchmail: socket error while fetching from billyikes@underwood-onion This worked with past versions. To reproduce, use this stanza in .fetchmailrc: skip underwood-onion via 127.0.0.1 protocol imap port 12345 username "billyikes" sslproto 'SSL3+' no sslcertck fetchall Note that past working stanzas did not need "sslproto" or "no sslcertck" but were introduced to after upgrading to 6.4.16. run these commands: $ socat TCP4-LISTEN:12345,reuseaddr,fork SOCKS4A:127.0.0.1:underwood2hj3pwd.onion:143,socksport=9050 $ fetchmail -v -d0 underwood-onion $ pkill socat + It's the same outcome if "sslproto 'SSL23'" is used instead. + This is one report, but there are a few bugs here: 1) inability to connect to handshake with bad TLS protocols. It's an onion site, so SSL is not needed for crypto (it's there for a different purpose). So if fetchmail is judging the crypto to be insecure, it's overzealous in this case. 2) the "Unknown login or authentication error" is not only a false error, it's alarming. It's the worst kind of false error because it tells the user that there's a problem with their account. 3) there is no per-account SOCKS4a config parameter, so users are pushed into this inconvenient and ugly hack of running socat and piping through that. The "plugin" parameter does not help in this case because fetchmail still attempts to resolve the underwood2hj3pwd.onion outside of the proxy. - Bug \3 has always existed, but 1 & 2 are new regressions. + Bug number 3 has always existed, but 1 & 2 are new regressions. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934155 Title: fetchmail can no longer connect to underwood & gives false error msg (TLS issues) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1934155/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
