Regression testing
------------------
This consists in crawling the 'emailconfiguration <dot> com'
website with 'googler' and parsing the IMAP/POP servers with
SSL encryption from the results/website pages. Test scripts
are attached in a tarball.
$ mkdir testing
$ cd testing
$ ~/get-email-config-links.sh && ~/get-email-config-pages.sh && ~/get-
email-config-servers.sh
Number of results from googler:
$ wc -l email-config.links
299 email-config.links
$ ls -1 email-config.page_* | wc -l
299
Number of found pages + SSL servers:
$ wc -l email-config.servers
265 email-config.servers
Number of unique SSL servers:
$ sort -u email-config.servers | wc -l
167
There are only 167 unique servers from a list
of 299 providers because some of the provider
pages are not found in the website, less some
providers which have the same server address
for multiple domains listed as different pages
in the website.
eoan
====
$ lsb_release -ds
Ubuntu Eoan Ermine (development branch)
dependencies
------------
$ sudo apt-get install uw-mailutils php7.2-cli php7.2-imap
original package
----------------
$ dpkg -s libc-client2007e | grep Version:
Version: 8:2007f~dfsg-6
$ mkdir orig-pkg
$ (cd orig-pkg && ~/try-email-config-servers.sh ../ &&
~/normalize-email-config-servers-log.sh)
modified package
----------------
$ dpkg -s libc-client2007e | grep Version:
Version: 8:2007f~dfsg-6ubuntu1
$ mkdir test-pkg
$ (cd test-pkg && ~/try-email-config-servers.sh ../ &&
~/normalize-email-config-servers-log.sh)
comparison
----------
Summary: imap.gmail.com, imap.mail.att.net, and inbound.att.net
have certificate failures with the original package, and either
reach the username prompt (mailutil check) or errors due to the
authentication errors (php -r imap_open) with the test package.
Some other servers have false negatives because of connection
timeouts or other failures but after more attempts eventually
show consistent behavior (no change).
$ diff -ur orig-pkg/ test-pkg/ | grep -v -e ^@@ -e ^--- -e ^+++ | sed
's/^diff/\ndiff/'
diff -ur orig-pkg/email-config.servers.log.imap.gmail.com_993.mailutil
test-pkg/email-config.servers.log.imap.gmail.com_993.mailutil
-Certificate failure for imap.gmail.com: self signed certificate: /OU=No SNI
provided; please fix your client./CN=invalid2.invalid
-Certificate failure for imap.gmail.com: self signed certificate: /OU=No SNI
provided; please fix your client./CN=invalid2.invalid
+{SERVER} username:
\ No newline at end of file
diff -ur orig-pkg/email-config.servers.log.imap.gmail.com_993.php-imap
test-pkg/email-config.servers.log.imap.gmail.com_993.php-imap
PHP Warning: imap_open(): Couldn't open stream
{imap.gmail.com:993/imap/ssl}INBOX in Command line code on line 1
-PHP Notice: Unknown: Certificate failure for imap.gmail.com: self signed
certificate: /OU=No SNI provided; please fix your client./CN=invalid2.invalid
(errflg=2) in Unknown on line 0
+PHP Notice: Unknown: Retrying PLAIN authentication after [ALERT] Invalid
credentials (Failure) (errflg=1) in Unknown on line 0
+PHP Notice: Unknown: Retrying PLAIN authentication after [ALERT] Invalid
credentials (Failure) (errflg=1) in Unknown on line 0
+PHP Notice: Unknown: Can not authenticate to IMAP server: [ALERT] Invalid
credentials (Failure) (errflg=2) in Unknown on line 0
diff -ur orig-pkg/email-config.servers.log.imap.mail.att.net_993.mailutil
test-pkg/email-config.servers.log.imap.mail.att.net_993.mailutil
-Certificate failure for imap.mail.att.net: Hostname mismatch:
/C=US/ST=California/L=Sunnyvale/O=Oath Inc/CN=*.imap.mail.yahoo.com
-Certificate failure for imap.mail.att.net: Hostname mismatch:
/C=US/ST=California/L=Sunnyvale/O=Oath Inc/CN=*.imap.mail.yahoo.com
+{SERVER} username:
\ No newline at end of file
diff -ur orig-pkg/email-config.servers.log.imap.mail.att.net_993.php-imap
test-pkg/email-config.servers.log.imap.mail.att.net_993.php-imap
PHP Warning: imap_open(): Couldn't open stream
{imap.mail.att.net:993/imap/ssl}INBOX in Command line code on line 1
-PHP Notice: Unknown: Certificate failure for imap.mail.att.net: Hostname
mismatch: /C=US/ST=California/L=Sunnyvale/O=Oath Inc/CN=*.imap.mail.yahoo.com
(errflg=2) in Unknown on line 0
+PHP Notice: Unknown: Retrying PLAIN authentication after
[AUTHENTICATIONFAILED] AUTHENTICATE Invalid credentials (errflg=1) in Unknown
on line 0
+PHP Notice: Unknown: Retrying PLAIN authentication after [LIMIT] AUTHENTICATE
Rate limit hit. (errflg=1) in Unknown on line 0
+PHP Notice: Unknown: Can not authenticate to IMAP server: [LIMIT]
AUTHENTICATE Rate limit hit. (errflg=2) in Unknown on line 0
diff -ur orig-pkg/email-config.servers.log.inbound.att.net_995.mailutil
test-pkg/email-config.servers.log.inbound.att.net_995.mailutil
-Certificate failure for inbound.att.net: Hostname mismatch:
/C=US/ST=California/L=Sunnyvale/O=Oath Inc/CN=legacy.pop.mail.yahoo.com
+{SERVER} username:
\ No newline at end of file
diff -ur orig-pkg/email-config.servers.log.inbound.att.net_995.php-imap
test-pkg/email-config.servers.log.inbound.att.net_995.php-imap
PHP Warning: imap_open(): Couldn't open stream
{inbound.att.net:995/pop3/ssl}INBOX in Command line code on line 1
-PHP Notice: Unknown: Certificate failure for inbound.att.net: Hostname
mismatch: /C=US/ST=California/L=Sunnyvale/O=Oath
Inc/CN=legacy.pop.mail.yahoo.com (errflg=2) in Unknown on line 0
+PHP Notice: Unknown: Can not authenticate to POP3 server: POP3 connection
broken in response (errflg=2) in Unknown on line 0
disco
=====
$ lsb_release -cs
disco
$ dpkg -s libc-client2007e | grep Version:
Version: 8:2007f~dfsg-5build1
and
$ dpkg -s libc-client2007e | grep Version:
Version: 8:2007f~dfsg-5ubuntu0.19.04.1
Same results.
bionic
=====
$ lsb_release -cs
bionic
$ dpkg -s libc-client2007e | grep Version:
Version: 8:2007f~dfsg-5build1
and
$ dpkg -s libc-client2007e | grep Version:
Version: 8:2007f~dfsg-5ubuntu0.18.04.1
Same results.
** Description changed:
[Impact]
* Users of libc-client2007e (e.g., php7.x-imap) can no longer
connect to GMail on Bionic and later, after introduction of
TLSv1.3 with OpenSSL 1.1.1 (normal upgrade path in Bionic).
* GMail requires Server Name Indication (SNI) to be set when
TLSv1.3 is used, otherwise the server provided certificate
fails verification in the client and connection is aborted.
* The fix is to set SNI to the hostname that the client will
perform verification on. The change is only enabled if the
client is built with OpenSSL 1.1.1 or later (i.e., TLSv1.3
support) so not to affect pre- TLSv1.3 support's behavior.
* However it is functional nonetheless if the client is built
with OpenSSL 1.1.1 or later but an earlier TLS version ends
up used due to the handshake/negotiation/server TLS support
(e.g., TLSv1.2); this shouldn't be a problem per test below.
* Regression testing happened with a crawled list of IMAP/POP
SSL servers (167 servers), and no regressions were observed.
Actually, one more email provider/server has been fixed too.
* OpenSSL-only demonstration with -(no)servername:
$ echo QUIT \
| openssl s_client \
-connect imap.gmail.com:993 \
-verify_hostname imap.gmail.com \
-noservername `# or -servername imap.gmail.com` \
-tls1_3 -brief 2>&1 \
| grep -i ^verif
Output with '-noservername':
verify error:num=18:self signed certificate
verify error:num=62:Hostname mismatch
Verification error: Hostname mismatch
Output with '-servername imap.gmail.com'
Verification: OK
Verified peername: imap.gmail.com
[Test Case]
* Commands:
$ sudo apt install uw-mailutils
$ mailutil check "{imap.googlemail.com:993/imap/ssl}INBOX"
$ sudo apt install php7.2-cli php7.2-imap
$ php -r 'imap_open("{imap.gmail.com:993/imap/ssl}INBOX", "username",
"password");'
* Before:
$ mailutil check "{imap.googlemail.com:993/imap/ssl}INBOX"
Certificate failure for imap.googlemail.com: self signed certificate:
/OU=No SNI provided; please fix your client./CN=invalid2.invalid
Certificate failure for imap.googlemail.com: self signed certificate:
/OU=No SNI provided; please fix your client./CN=invalid2.invalid
$ php -r 'imap_open("{imap.gmail.com:993/imap/ssl}INBOX", "username",
"password");'
PHP Warning: imap_open(): Couldn't open stream
{imap.gmail.com:993/imap/ssl}INBOX in Command line code on line 1
PHP Notice: Unknown: Certificate failure for imap.gmail.com: self signed
certificate: /OU=No SNI provided; please fix your client./CN=invalid2.invalid
(errflg=2) in Unknown on line 0
* After:
$ mailutil check "{imap.googlemail.com:993/imap/ssl}INBOX"
{ce-in-f16.1e100.net/imap} username:
^C
$ php -r 'imap_open("{imap.gmail.com:993/imap/ssl}INBOX", "username",
"password");'
PHP Warning: imap_open(): Couldn't open stream
{imap.gmail.com:993/imap/ssl}INBOX in Command line code on line 1
PHP Notice: Unknown: Retrying PLAIN authentication after [ALERT] Invalid
credentials (Failure) (errflg=1) in Unknown on line 0
PHP Notice: Unknown: Retrying PLAIN authentication after [ALERT] Invalid
credentials (Failure) (errflg=1) in Unknown on line 0
PHP Notice: Unknown: Can not authenticate to IMAP server: [ALERT] Invalid
credentials (Failure) (errflg=2) in Unknown on line 0
* Regression testing scripts/results are provided in
attachments/comments.
[Regression Potential]
* Theoretically possible, but not observed in hundred+ of (167)
IMAP/POP SSL servers.
* The change sends additional data (SNI) from client to server
when connecting, if built with OpenSSL 1.1.1 or later, which
is in the specification, so should be handled by the server.
* The risk is servers that misbehave when provided such info
(not observed in the 167 server test).
* Less likely are servers that do not recognize the server name
identified (this also not observed in test and unlikely since
the client usually reaches the server by public/known address).
* Even less likely are servers whose provided certificate doesn't
contain the server name identified (again not observed and it's
in the server, not client, to provide a certificate for address
it doesn't recognize).
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
+ [Other Info]
+
+ * Reverse build dependencies have been patched for No Change Rebuilds
+ and (re)built correctly on all architectures in Launchpad PPA for
+ all affected releases (eoan, disco, bionic).
+
[Original Description]
After upgrading from PHP5 to PHP7.2 (from Bionic), calling imap_open()
against Google's Gmail servers stopped working.
After researching, I've found that new OpenSSL version introduced
TLSv13-related breaking changes.
Here are the relevant issues:
PHP: https://bugs.php.net/bug.php?id=77108
PHP (Debian): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=916041
In fetchmail (solved):
https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1798786
OpenSSL upstream devs have issues related to this:
https://github.com/openssl/openssl/issues/5944
https://github.com/openssl/openssl/pull/5947
Looks like to me that either adding the SNI server name to the openssl
open call would be needed, as done on fetchmail.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1834340
Title:
Regression for GMail after libssl upgrade with TLSv1.3
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/php-imap/+bug/1834340/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
