Yes, everything is ran on the same machine. I had configured a single dispatcher with 2 ports (1 for the "success" and one for the "failure") as detailed in one of my previous mails.
Regards, Adel > From: [email protected] > To: [email protected] > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a listener > Date: Fri, 24 Jun 2016 15:59:21 +0000 > > Sorry I mean for the failure case both qdstat and router are on the same > machine but ... comparing "success" and "failure" are they executed on the > same machine ? > > I see "machine" in the success case and "green-lx-slave1" for the failure. > > Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor > Twitter : @ppatierno > Linkedin : paolopatierno > Blog : DevExperience > > > From: [email protected] > > To: [email protected] > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a listener > > Date: Fri, 24 Jun 2016 17:56:56 +0200 > > > > > > > > Nope, I > > am using the same machine "green-lx-slave1". You can see from the > > image the src and destination have the same ip (10.25.8.35) > > > > > > > > I have > > just exported the display of Wireshark on my local Windows machine. (sudo > > wireshark &) > > > > > > Could it be that my certificate or key are corrupt and that the qdstat is > > unable to load them and is thus failing before sending anything? > > > > > > > > Regards, > > > > Adel > > > > > From: [email protected] > > > To: [email protected] > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a listener > > > Date: Fri, 24 Jun 2016 15:41:51 +0000 > > > > > > Having no traffic in the failure case means that the tool doesn't send > > > any packets (it should be the first to send something with the "Server > > > Hello" message). > > > At least we should see the TCP handshake ... before the SSL handshake ! > > > > > > In this case the server configuration isn't playing any role in the > > > failure. > > > > > > I see you are testing on two different machine ... with two different > > > hostnames and ports. Could be some problem on name resolutin ? Can you > > > use the IP address instead of the name when the tool is launched ? > > > > > > Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat > > > Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor > > > Twitter : @ppatierno > > > Linkedin : paolopatierno > > > Blog : DevExperience > > > > > > > From: [email protected] > > > > To: [email protected] > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a listener > > > > Date: Fri, 24 Jun 2016 17:37:14 +0200 > > > > > > > > Paolo, > > > > > > > > There is no traffic in the case of the failure. So I cannot provide a > > > > pncap file :( > > > > > > > > > From: [email protected] > > > > > To: [email protected] > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a listener > > > > > Date: Fri, 24 Jun 2016 17:35:56 +0200 > > > > > > > > > > It seems like attachments are not really working. You can check the > > > > > images here: http://imgur.com/a/WlssO > > > > > > > > > > Adel > > > > > > > > > > From: [email protected] > > > > > To: [email protected] > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a listener > > > > > Date: Fri, 24 Jun 2016 17:31:45 +0200 > > > > > > > > > > > > > > > > > > > > > > > > > Wireshark Pictures attached. > > > > > > > > > > Adel > > > > > > > > > > From: [email protected] > > > > > To: [email protected] > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a listener > > > > > Date: Fri, 24 Jun 2016 17:29:03 +0200 > > > > > > > > > > > > > > > > > > > > > > > > > Success > > > > > =========== > > > > > > > > > > > > > > > Failure > > > > > ============ > > > > > > > > > > > From: [email protected] > > > > > > To: [email protected] > > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a listener > > > > > > Date: Fri, 24 Jun 2016 17:26:44 +0200 > > > > > > > > > > > > I fixed the CN part (Thanks Paolo) > > > > > > > > > > > > I also ran Wireshark and it seems that the failure is way before > > > > > > establishing a connection as I have no packets being transfered in > > > > > > the case of the failing certificate. > > > > > > I filtered on the tcp ports of the success and the failure. I have > > > > > > 0 packets in the failure case (Wireshark display filter: "tcp.port > > > > > > == 10398") > > > > > > > > > > > > Regards, > > > > > > Adel > > > > > > > > > > > > > Date: Fri, 24 Jun 2016 11:17:18 -0400 > > > > > > > From: [email protected] > > > > > > > To: [email protected] > > > > > > > Subject: Re: [Qpid-Dispatch] SSL/SASL configuration on a listener > > > > > > > > > > > > > > Good catch Paolo, I should note here that qdstat and qdmanage > > > > > > > commands do *not* do hostname verification by default. I > > > > > > > submitted a pull request for > > > > > > > https://issues.apache.org/jira/browse/DISPATCH-401 which is not > > > > > > > in master yet. > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > From: "Paolo Patierno" <[email protected]> > > > > > > > > To: [email protected] > > > > > > > > Sent: Friday, June 24, 2016 11:09:56 AM > > > > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a > > > > > > > > listener > > > > > > > > > > > > > > > > Hi Adel, > > > > > > > > > > > > > > > > is this just a typo or the real CN you are using ? > > > > > > > > > > > > > > > > /CN=CN=127.0.0.1 > > > > > > > > In this case the CN is "CN=127.0.0.1" that is different from > > > > > > > > 127.0.0.1 so the > > > > > > > > verify host name could fail. > > > > > > > > It should be : > > > > > > > > > > > > > > > > /CN=127.0.0.1 > > > > > > > > > > > > > > > > Paolo. > > > > > > > > > > > > > > > > Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat > > > > > > > > Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor > > > > > > > > Twitter : @ppatierno > > > > > > > > Linkedin : paolopatierno > > > > > > > > Blog : DevExperience > > > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > > To: [email protected] > > > > > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a > > > > > > > > > listener > > > > > > > > > Date: Fri, 24 Jun 2016 15:03:44 +0000 > > > > > > > > > > > > > > > > > > Hi Adel, > > > > > > > > > > > > > > > > > > can you use a tool like Wireshark in order to sniff the SSL > > > > > > > > > handshake > > > > > > > > > traffic and share the pncap file with use. Just to see where > > > > > > > > > the handshake > > > > > > > > > fails. > > > > > > > > > > > > > > > > > > Paolo > > > > > > > > > > > > > > > > > > Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat > > > > > > > > > Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor > > > > > > > > > Twitter : @ppatierno > > > > > > > > > Linkedin : paolopatierno > > > > > > > > > Blog : DevExperience > > > > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > > To: [email protected] > > > > > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a > > > > > > > > > listener > > > > > > > > > Date: Fri, 24 Jun 2016 16:48:54 +0200 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you Paolo. > > > > > > > > > > > > > > > > > > @Ganesh, > > > > > > > > > I was able to successfully connect using your way of > > > > > > > > > generating the > > > > > > > > > certificates but not mine (I removed all passwords for > > > > > > > > > simplification). I > > > > > > > > > am getting "SSLException: SSL failure." error. > > > > > > > > > > > > > > > > > > I am attaching all my certificate folder and here below the > > > > > > > > > commands to > > > > > > > > > generate the client certificate. > > > > > > > > > > > > > > > > > > PS: > > > > > > > > > Simple SSL without client certificate is working. It is just > > > > > > > > > when I add > > > > > > > > > SASL EXTERNAL and authenticatePeer that I have the > > > > > > > > > error.ca-chain.cert.pem: Contains both root and intermediate > > > > > > > > > certificatesif you want to generate a new client certificate, > > > > > > > > > password for > > > > > > > > > intermediate certificate is "password" > > > > > > > > > > > > > > > > > > ================== > > > > > > > > > Commands launched > > > > > > > > > ================== > > > > > > > > > SUCCESS --> qdstat -c > > > > > > > > > --ssl-trustfile=PATH_TO_CERT_DIR/ganesh/ca-certificate.pem > > > > > > > > > --ssl-certificate=PATH_TO_CERT_DIR/ganesh/client-certificate.pem > > > > > > > > > --ssl-key=PATH_TO_CERT_DIR/ganesh/client-private-key.pem -b > > > > > > > > > amqps://machine:10397 > > > > > > > > > > > > > > > > > > FAILURE --> qdstat -c > > > > > > > > > --ssl-trustfile=PATH_TO_CERT_DIR/ca-chain.cert.pem > > > > > > > > > --ssl-certificate=PATH_TO_CERT_DIR/cert_lx_localhost_client.pem > > > > > > > > > --ssl-key=PATH_TO_CERT_DIR/key_lx_localhost_client.pem -b > > > > > > > > > amqps://green-lx-slave1:10398 > > > > > > > > > > > > > > > > > > ================== > > > > > > > > > Client certificate (OpenSSL 1.0.2.h) > > > > > > > > > ================== > > > > > > > > > openssl genrsa -out > > > > > > > > > intermediate/private/key_lx_localhost_client.pem 2048 > > > > > > > > > openssl req -config intermediate/openssl.cnf -key > > > > > > > > > intermediate/private/key_lx_localhost_client.pem -new > > > > > > > > > -sha256 -out > > > > > > > > > intermediate/csr/csr_lx_localhost_client.pem -subj > > > > > > > > > "/C=FR/ST=Paris/L=Paris/O=MUREX SAS/CN=CN=127.0.0.1" > > > > > > > > > openssl ca -config intermediate/openssl.cnf -extensions > > > > > > > > > usr_cert -days 375 > > > > > > > > > -notext -md sha256 -in > > > > > > > > > intermediate/csr/csr_lx_localhost_client.pem -out > > > > > > > > > intermediate/certs/cert_lx_localhost_client.pem > > > > > > > > > openssl x509 -noout -text -in > > > > > > > > > intermediate/certs/cert_lx_localhost_client.pem > > > > > > > > > openssl verify -CAfile intermediate/certs/ca-chain.cert.pem > > > > > > > > > intermediate/certs/cert_lx_localhost_client.pem > > > > > > > > > > > > > > > > > > ================== > > > > > > > > > Dispatcher conf > > > > > > > > > ================== > > > > > > > > > container { > > > > > > > > > worker-threads: 4 > > > > > > > > > containerName: qpid.dispatch.router.10399 > > > > > > > > > } > > > > > > > > > > > > > > > > > > ssl-profile { > > > > > > > > > name: my-ssl-profile > > > > > > > > > certFile: PATH_TO_CERT_DIR/cert_lx.pem > > > > > > > > > keyFile: PATH_TO_CERT_DIR/key_lx.pem > > > > > > > > > certDb: PATH_TO_CERT_DIR/ca-chain.cert.pem > > > > > > > > > } > > > > > > > > > > > > > > > > > > ssl-profile { > > > > > > > > > name: ganesh-ssl-profile > > > > > > > > > certFile: PATH_TO_CERT_DIR/ganesh/server-certificate.pem > > > > > > > > > keyFile: PATH_TO_CERT_DIR/ganesh/server-private-key.pem > > > > > > > > > certDb: PATH_TO_CERT_DIR/ganesh/ca-certificate.pem > > > > > > > > > } > > > > > > > > > > > > > > > > > > listener { > > > > > > > > > host: 0.0.0.0 > > > > > > > > > port: 10398 > > > > > > > > > saslMechanisms: EXTERNAL > > > > > > > > > sslProfile: my-ssl-profile > > > > > > > > > authenticatePeer: yes > > > > > > > > > requireSsl: yes > > > > > > > > > } > > > > > > > > > > > > > > > > > > listener { > > > > > > > > > host: 0.0.0.0 > > > > > > > > > port: 10397 > > > > > > > > > saslMechanisms: EXTERNAL > > > > > > > > > sslProfile: ganesh-ssl-profile > > > > > > > > > authenticatePeer: yes > > > > > > > > > requireSsl: yes > > > > > > > > > } > > > > > > > > > > > > > > > > > > router { > > > > > > > > > mode: interior > > > > > > > > > routerId: router.10399 > > > > > > > > > helloInterval: 60 > > > > > > > > > helloMaxAge: 180 > > > > > > > > > } > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Adel > > > > > > > > > > > > > > > > > > > Date: Fri, 24 Jun 2016 09:08:03 -0400 > > > > > > > > > > From: [email protected] > > > > > > > > > > To: [email protected] > > > > > > > > > > Subject: Re: [Qpid-Dispatch] SSL/SASL configuration on a > > > > > > > > > > listener > > > > > > > > > > > > > > > > > > > > Thanks for the clarification Paolo. > > > > > > > > > > > > > > > > > > > > Adel, > > > > > > > > > > There are two cases here - > > > > > > > > > > > > > > > > > > > > 1. if you are using a root CA and intermediate CA - You > > > > > > > > > > will have to > > > > > > > > > > create a cert chain file which establishes a chain of trust > > > > > > > > > > and set the > > > > > > > > > > chain file to certDb. Here is a good explanation of > > > > > > > > > > certificate chain - > > > > > > > > > > https://support.dnsimple.com/articles/what-is-ssl-certificate-chain/ > > > > > > > > > > You simply create a file(call it say ca-cert-chain.pem) and > > > > > > > > > > populate it > > > > > > > > > > with the root CA cert and intermediate CA cert (a simple > > > > > > > > > > copy and paste > > > > > > > > > > from the source files to target file or use cat command) > > > > > > > > > > > > > > > > > > > > 2. if you are using only a root CA and *no* intermediate CA > > > > > > > > > > - Here you > > > > > > > > > > will simply have to set the certDb to the root certificate > > > > > > > > > > pem file. > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > > > From: "Paolo Patierno" <[email protected]> > > > > > > > > > > > To: [email protected] > > > > > > > > > > > Sent: Friday, June 24, 2016 8:35:42 AM > > > > > > > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on a > > > > > > > > > > > listener > > > > > > > > > > > > > > > > > > > > > > Just a side note ... > > > > > > > > > > > > > > > > > > > > > > pay attention that in the SSL terminology, a self-signed > > > > > > > > > > > certificate is > > > > > > > > > > > a > > > > > > > > > > > certificate issued for an entity which is signed by the > > > > > > > > > > > entity itself. > > > > > > > > > > > > > > > > > > > > > > It means that in our scenario, only the root CA is a > > > > > > > > > > > self-signed > > > > > > > > > > > certificate. > > > > > > > > > > > It represents the maximum level of trust .... you need to > > > > > > > > > > > trust it ... > > > > > > > > > > > because no one can trust it ... but only itself :-) > > > > > > > > > > > > > > > > > > > > > > All the other certificates aren't self signed : the CA > > > > > > > > > > > intermediate > > > > > > > > > > > certificate is signed by root CA and the server and > > > > > > > > > > > client certificate > > > > > > > > > > > are > > > > > > > > > > > signed by the CA intermediate certificate. > > > > > > > > > > > > > > > > > > > > > > Paolo. > > > > > > > > > > > > > > > > > > > > > > Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat > > > > > > > > > > > Microsoft MVP on Windows Embedded & IoTMicrosoft Azure > > > > > > > > > > > Advisor > > > > > > > > > > > Twitter : @ppatierno > > > > > > > > > > > Linkedin : paolopatierno > > > > > > > > > > > Blog : DevExperience > > > > > > > > > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL configuration on > > > > > > > > > > > > a listener > > > > > > > > > > > > Date: Fri, 24 Jun 2016 13:03:29 +0200 > > > > > > > > > > > > > > > > > > > > > > > > Hello Ganesh, > > > > > > > > > > > > > > > > > > > > > > > > Thank you for your guide. > > > > > > > > > > > > I followed this guide to have a proper CA with a root > > > > > > > > > > > > and and an > > > > > > > > > > > > intermediate certificate > > > > > > > > > > > > (https://jamielinux.com/docs/openssl-certificate-authority/index.html) > > > > > > > > > > > > I then wanted to do as you proposed in the > > > > > > > > > > > > configuration. > > > > > > > > > > > > > > > > > > > > > > > > What would be the certDb in my case? Is is just the > > > > > > > > > > > > intermediate > > > > > > > > > > > > certificate? Or the root certificate? Or a combination > > > > > > > > > > > > of both? > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > Adel > > > > > > > > > > > > > > > > > > > > > > > > > Date: Thu, 23 Jun 2016 14:07:20 -0400 > > > > > > > > > > > > > From: [email protected] > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > Subject: Re: [Qpid-Dispatch] SSL/SASL configuration > > > > > > > > > > > > > on a listener > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Adel, > > > > > > > > > > > > > I added a new script that uses openssl to create > > > > > > > > > > > > > server and > > > > > > > > > > > > > client > > > > > > > > > > > > > certificates signed by a root CA. > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/qpid-dispatch/blob/master/tests/ssl_certs/gencerts_openssl.sh > > > > > > > > > > > > > > > > > > > > > > > > > > I tested this using the following router config - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ssl-profile { > > > > > > > > > > > > > certFile: > > > > > > > > > > > > > > > > > > > > > > > > > > /home/gmurthy/opensource/dispatch/etc/ssl-my-certs/root/ca1/server-certificate.pem > > > > > > > > > > > > > keyFile: > > > > > > > > > > > > > > > > > > > > > > > > > > /home/gmurthy/opensource/dispatch/etc/ssl-my-certs/root/ca1/server-private-key.pem > > > > > > > > > > > > > password: server-password > > > > > > > > > > > > > name: server-ssl-profile > > > > > > > > > > > > > certDb: > > > > > > > > > > > > > > > > > > > > > > > > > > /home/gmurthy/opensource/dispatch/etc/ssl-my-certs/root/ca1/ca-certificate.pem > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > listener { > > > > > > > > > > > > > ssl-profile: server-ssl-profile > > > > > > > > > > > > > authenticatePeer: yes > > > > > > > > > > > > > saslMechanisms: EXTERNAL > > > > > > > > > > > > > role: normal > > > > > > > > > > > > > addr: 127.0.0.1 > > > > > > > > > > > > > port: amqp > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > I ran qdstat with a client cert to verify - > > > > > > > > > > > > > > > > > > > > > > > > > > [gmurthy@localhost ca1]$ qdstat -c > > > > > > > > > > > > > --ssl-trustfile=/home/gmurthy/opensource/dispatch/etc/ssl-my-certs/root/ca1/ca-certificate.pem > > > > > > > > > > > > > --ssl-certificate=/home/gmurthy/opensource/dispatch/etc/ssl-my-certs/root/ca1/client-certificate.pem > > > > > > > > > > > > > --ssl-key=/home/gmurthy/opensource/dispatch/etc/ssl-my-certs/root/ca1/client-private-key.pem > > > > > > > > > > > > > --ssl-password=client-password > > > > > > > > > > > > > Connections > > > > > > > > > > > > > Id host container > > > > > > > > > > > > > role dir security > > > > > > > > > > > > > authentication > > > > > > > > > > > > > > > > > > > > > > > > > > ===================================================================================================================================================================================== > > > > > > > > > > > > > 1 localhost.localdomain:34160 > > > > > > > > > > > > > 02c3bf84-47de-4838-8282-6e7e8a5dde9c > > > > > > > > > > > > > normal in TLSv1/SSLv3(DHE-RSA-AES256-GCM-SHA384) > > > > > > > > > > > > > CN=127.0.0.1,O=Client,L=San > > > > > > > > > > > > > Francisco,ST=CA,C=US(x.509) > > > > > > > > > > > > > [gmurthy@localhost ca1]$ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > > > > > > From: "Ganesh Murthy" <[email protected]> > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > Sent: Thursday, June 23, 2016 10:17:06 AM > > > > > > > > > > > > > > Subject: Re: [Qpid-Dispatch] SSL/SASL configuration > > > > > > > > > > > > > > on a listener > > > > > > > > > > > > > > > > > > > > > > > > > > > > I also want to add that there is a file called > > > > > > > > > > > > > > qpid-dispatch/tests/ssl_certs/gencerts.sh (thanks > > > > > > > > > > > > > > Chuck Rolke). > > > > > > > > > > > > > > This > > > > > > > > > > > > > > file > > > > > > > > > > > > > > has commands that create a root CA and self signed > > > > > > > > > > > > > > certs. There > > > > > > > > > > > > > > are > > > > > > > > > > > > > > several > > > > > > > > > > > > > > tests (system_tests_qdstat.py, > > > > > > > > > > > > > > system_tests_two_routers.py, > > > > > > > > > > > > > > system_tests_sasl_plain.py) that use these self > > > > > > > > > > > > > > signed certs and > > > > > > > > > > > > > > also > > > > > > > > > > > > > > cover > > > > > > > > > > > > > > various SASL scenarios. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > > > > > > > From: "Ganesh Murthy" <[email protected]> > > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > Sent: Thursday, June 23, 2016 10:05:08 AM > > > > > > > > > > > > > > > Subject: Re: [Qpid-Dispatch] SSL/SASL > > > > > > > > > > > > > > > configuration on a > > > > > > > > > > > > > > > listener > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Adel, > > > > > > > > > > > > > > > When creating self signed certificates, it is > > > > > > > > > > > > > > > always a good > > > > > > > > > > > > > > > idea > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > create > > > > > > > > > > > > > > > a root CA and use it to sign the server and > > > > > > > > > > > > > > > client > > > > > > > > > > > > > > > certificates. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If you are creating self signed certs in a > > > > > > > > > > > > > > > production > > > > > > > > > > > > > > > environment, I > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > suggest that you create a root CA and use the > > > > > > > > > > > > > > > root CA to create > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > intermediate CA and then use the intermediate CA > > > > > > > > > > > > > > > to create your > > > > > > > > > > > > > > > self > > > > > > > > > > > > > > > signed > > > > > > > > > > > > > > > server and client certs. If your client or server > > > > > > > > > > > > > > > certs are > > > > > > > > > > > > > > > compromised, > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > can use the root CA to invalidate the > > > > > > > > > > > > > > > intermediate CA which in > > > > > > > > > > > > > > > turn > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > invalidate all certs created using the > > > > > > > > > > > > > > > intermediate CA. This > > > > > > > > > > > > > > > way you > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > make sure that your root CA is never compromised. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > > > > > > > > From: "Adel Boutros" <[email protected]> > > > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > Sent: Thursday, June 23, 2016 9:56:02 AM > > > > > > > > > > > > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL > > > > > > > > > > > > > > > > configuration on a > > > > > > > > > > > > > > > > listener > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Paolo, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In that case I think the issue is that my > > > > > > > > > > > > > > > > certificates were > > > > > > > > > > > > > > > > self-signed > > > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > there was no CA. I think this works on the Java > > > > > > > > > > > > > > > > Broker thanks > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > KeyStore and TrustStore features. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I will re-organize my certificates to have a CA > > > > > > > > > > > > > > > > which will > > > > > > > > > > > > > > > > generate > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > client and server certificates and test again. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the helpful explanation! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > Adel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL > > > > > > > > > > > > > > > > > configuration on a > > > > > > > > > > > > > > > > > listener > > > > > > > > > > > > > > > > > Date: Thu, 23 Jun 2016 13:31:56 +0000 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Adel, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm a bit confused of what you are trying to > > > > > > > > > > > > > > > > > achieve. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > A listener (so acting as a server) can have > > > > > > > > > > > > > > > > > only one > > > > > > > > > > > > > > > > > certificate > > > > > > > > > > > > > > > > > specified > > > > > > > > > > > > > > > > > through certFile parameter (and related > > > > > > > > > > > > > > > > > keyFile for > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > private > > > > > > > > > > > > > > > > > key). This certificate is issued by the > > > > > > > > > > > > > > > > > server (listener) > > > > > > > > > > > > > > > > > to the > > > > > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > > during SSL/TLS handshake in order to provide > > > > > > > > > > > > > > > > > the server > > > > > > > > > > > > > > > > > authentication > > > > > > > > > > > > > > > > > feature. Of course the server certificate is > > > > > > > > > > > > > > > > > signed with a > > > > > > > > > > > > > > > > > CA > > > > > > > > > > > > > > > > > certificate. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In order to have client authentication, the > > > > > > > > > > > > > > > > > client sends > > > > > > > > > > > > > > > > > its own > > > > > > > > > > > > > > > > > certificate to the server during the > > > > > > > > > > > > > > > > > handshake. This > > > > > > > > > > > > > > > > > certificate > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > signed > > > > > > > > > > > > > > > > > by the same CA certificate used to sign > > > > > > > > > > > > > > > > > server certificate > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > CA > > > > > > > > > > > > > > > > > certificate specified through the trustCerts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > When the SSL handshake ends and mutual > > > > > > > > > > > > > > > > > authentication is > > > > > > > > > > > > > > > > > achieved, the > > > > > > > > > > > > > > > > > SASL > > > > > > > > > > > > > > > > > handshake starts and using EXTERNAL you are > > > > > > > > > > > > > > > > > saying that the > > > > > > > > > > > > > > > > > client was > > > > > > > > > > > > > > > > > authenticated in a way external to SASL > > > > > > > > > > > > > > > > > itself and using > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > previous > > > > > > > > > > > > > > > > > authentication at SSL level. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It means that the SASL EXTERNAL > > > > > > > > > > > > > > > > > authentication mechanism is > > > > > > > > > > > > > > > > > strictly > > > > > > > > > > > > > > > > > related with what's happened in the previous > > > > > > > > > > > > > > > > > SSL handshake > > > > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > > it's > > > > > > > > > > > > > > > > > related > > > > > > > > > > > > > > > > > to the certificates used for that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Paolo. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Paolo PatiernoSenior Software Engineer (IoT) > > > > > > > > > > > > > > > > > @ Red Hat > > > > > > > > > > > > > > > > > Microsoft MVP on Windows Embedded & > > > > > > > > > > > > > > > > > IoTMicrosoft Azure > > > > > > > > > > > > > > > > > Advisor > > > > > > > > > > > > > > > > > Twitter : @ppatierno > > > > > > > > > > > > > > > > > Linkedin : paolopatierno > > > > > > > > > > > > > > > > > Blog : DevExperience > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > > > Subject: RE: [Qpid-Dispatch] SSL/SASL > > > > > > > > > > > > > > > > > > configuration on a > > > > > > > > > > > > > > > > > > listener > > > > > > > > > > > > > > > > > > Date: Thu, 23 Jun 2016 15:16:22 +0200 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It feels like a big puzzle to get SSL with > > > > > > > > > > > > > > > > > > client mutual > > > > > > > > > > > > > > > > > > authentication > > > > > > > > > > > > > > > > > > working. It would help me a lot if someone > > > > > > > > > > > > > > > > > > can provide a > > > > > > > > > > > > > > > > > > fully > > > > > > > > > > > > > > > > > > working > > > > > > > > > > > > > > > > > > configuration and how to use it with a JMS > > > > > > > > > > > > > > > > > > client for > > > > > > > > > > > > > > > > > > example. > > > > > > > > > > > > > > > > > > I think it could also benefit others i the > > > > > > > > > > > > > > > > > > future > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ganesh had provided me on a different > > > > > > > > > > > > > > > > > > thread, steps to > > > > > > > > > > > > > > > > > > generate > > > > > > > > > > > > > > > > > > server > > > > > > > > > > > > > > > > > > certificate and use it in the dispatcher. I > > > > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > > something > > > > > > > > > > > > > > > > > > similar > > > > > > > > > > > > > > > > > > here > > > > > > > > > > > > > > > > > > is the easiest solution. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > Adel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > > > > > > > > > > > > Date: Thu, 23 Jun 2016 14:27:11 +0200 > > > > > > > > > > > > > > > > > > > Subject: Re: [Qpid-Dispatch] SSL/SASL > > > > > > > > > > > > > > > > > > > configuration on > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > listener > > > > > > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think you have to add the file with > > > > > > > > > > > > > > > > > > > client public > > > > > > > > > > > > > > > > > > > keys to > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > certDb > > > > > > > > > > > > > > > > > > > option. The trustedCerts parameter is > > > > > > > > > > > > > > > > > > > used only to > > > > > > > > > > > > > > > > > > > control > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > public > > > > > > > > > > > > > > > > > > > keys will be listed as supported CAs to > > > > > > > > > > > > > > > > > > > the clients. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jakub > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 23, 2016 at 11:37 AM, Adel > > > > > > > > > > > > > > > > > > > Boutros > > > > > > > > > > > > > > > > > > > <[email protected]> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, So I added the client certificate > > > > > > > > > > > > > > > > > > > > but it doesn't > > > > > > > > > > > > > > > > > > > > seem > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > work. > > > > > > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > > > > am > > > > > > > > > > > > > > > > > > > > getting an exception in the handshake > > > > > > > > > > > > > > > > > > > > phase: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dispatcher error: ERROR (error) Run > > > > > > > > > > > > > > > > > > > > Time: Cannot set > > > > > > > > > > > > > > > > > > > > peer > > > > > > > > > > > > > > > > > > > > authentication > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dispatcher config > > > > > > > > > > > > > > > > > > > > ssl-profile { > > > > > > > > > > > > > > > > > > > > name: ssl-profile-name > > > > > > > > > > > > > > > > > > > > certFile: cert_ssl_encryption.pem > > > > > > > > > > > > > > > > > > > > keyFile:key_ssl_encryption.pem > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > listener { > > > > > > > > > > > > > > > > > > > > host: 0.0.0.0 > > > > > > > > > > > > > > > > > > > > port: 10398 > > > > > > > > > > > > > > > > > > > > saslMechanisms: EXTERNAL > > > > > > > > > > > > > > > > > > > > sslProfile: ssl-profile-name > > > > > > > > > > > > > > > > > > > > authenticatePeer: yes > > > > > > > > > > > > > > > > > > > > requireSsl: yes > > > > > > > > > > > > > > > > > > > > trustedCerts: cert_sasl.pem > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > JMS Client > > > > > > > > > > > > > > > > > > > > System.setProperty("javax.net.ssl.trustStore", > > > > > > > > > > > > > > > > > > > > resourcePath("trustStore.jks")); > > > > > > > > > > > > > > > > > > > > System.setProperty("javax.net.ssl.keyStore", > > > > > > > > > > > > > > > > > > > > resourcePath("clientKeyStore.jks")); > > > > > > > > > > > > > > > > > > > > System.setProperty("javax.net.ssl.keyStorePassword", > > > > > > > > > > > > > > > > > > > > "password"); > > > > > > > > > > > > > > > > > > > > JmsConnectionFactory > > > > > > > > > > > > > > > > > > > > jmsConnectionFactory = new > > > > > > > > > > > > > > > > > > > > JmsConnectionFactory("amqps://hostname:10398?transport.keyAlias=client"); > > > > > > > > > > > > > > > > > > > > Connection connection = > > > > > > > > > > > > > > > > > > > > jmsConnectionFactory.createConnection(); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > PS: trustStore.jks contains the > > > > > > > > > > > > > > > > > > > > cert_ssl_encryption.pem and > > > > > > > > > > > > > > > > > > > > clientKeyStore.jks contains the sasl > > > > > > > > > > > > > > > > > > > > certificate > > > > > > > > > > > > > > > > > > > > (cert_sasl.pem) > > > > > > > > > > > > > > > > > > > > which is > > > > > > > > > > > > > > > > > > > > aliased by "client" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Should I merge cert_sasl.pem and > > > > > > > > > > > > > > > > > > > > cert_ssl_encryption.pem in > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > ssl-profile? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > Adel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Date: Wed, 22 Jun 2016 11:23:16 -0400 > > > > > > > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > > > > > > Subject: Re: [Qpid-Dispatch] SSL/SASL > > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > > > > > on a > > > > > > > > > > > > > > > > > > > > > listener > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "Of course I want to use a > > > > > > > > > > > > > > > > > > > > > certificate for SSL > > > > > > > > > > > > > > > > > > > > > encryption > > > > > > > > > > > > > > > > > > > > > (provided > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > the ssl-profile) and a different one > > > > > > > > > > > > > > > > > > > > for SASL > > > > > > > > > > > > > > > > > > > > authentication but > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > same listener." > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Are you saying that you have two > > > > > > > > > > > > > > > > > > > > > pairs of > > > > > > > > > > > > > > > > > > > > > server/client > > > > > > > > > > > > > > > > > > > > > certs > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > want to use one pair for initial SSL > > > > > > > > > > > > > > > > > > > > encryption (to > > > > > > > > > > > > > > > > > > > > encrypt > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > entire > > > > > > > > > > > > > > > > > > > > exchange) and another pair for SASL > > > > > > > > > > > > > > > > > > > > EXTERNAL ? If > > > > > > > > > > > > > > > > > > > > this is > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > case, > > > > > > > > > > > > > > > > > > > > you can > > > > > > > > > > > > > > > > > > > > have only one server side cert per > > > > > > > > > > > > > > > > > > > > listener which you > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > specify > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > certFile. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > > > > > > > > > > > > > > From: "Ted Ross" <[email protected]> > > > > > > > > > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > > > > > > > Sent: Wednesday, June 22, 2016 > > > > > > > > > > > > > > > > > > > > > > 10:55:47 AM > > > > > > > > > > > > > > > > > > > > > > Subject: Re: [Qpid-Dispatch] > > > > > > > > > > > > > > > > > > > > > > SSL/SASL > > > > > > > > > > > > > > > > > > > > > > configuration on > > > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > listener > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 06/22/2016 10:47 AM, Adel > > > > > > > > > > > > > > > > > > > > > > Boutros wrote: > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I want to use SASL authentication > > > > > > > > > > > > > > > > > > > > > > > mechanism > > > > > > > > > > > > > > > > > > > > > > > using a > > > > > > > > > > > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > > > > > certificate. I > > > > > > > > > > > > > > > > > > > > > > > looked at the examples and tests > > > > > > > > > > > > > > > > > > > > > > > but I didn't > > > > > > > > > > > > > > > > > > > > > > > quite > > > > > > > > > > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > > > > > > > > everything. > > > > > > > > > > > > > > > > > > > > > > > I know I have to setup a listener > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > "sasl-mechanisms: > > > > > > > > > > > > > > > > > > > > > > > EXTERNAL" > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > "require-peer-auth: yes" but then > > > > > > > > > > > > > > > > > > > > > > > how do I tell > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > dispatcher > > > > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > > > certificates are accepted and > > > > > > > > > > > > > > > > > > > > > > > which aren't? > > > > > > > > > > > > > > > > > > > > > > > Of course I want to use a > > > > > > > > > > > > > > > > > > > > > > > certificate for SSL > > > > > > > > > > > > > > > > > > > > > > > encryption > > > > > > > > > > > > > > > > > > > > > > > (provided > > > > > > > > > > > > > > > > > > > > in the > > > > > > > > > > > > > > > > > > > > > > > ssl-profile) and a different one > > > > > > > > > > > > > > > > > > > > > > > for SASL > > > > > > > > > > > > > > > > > > > > > > > authentication > > > > > > > > > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > same > > > > > > > > > > > > > > > > > > > > > > > listener. > > > > > > > > > > > > > > > > > > > > > > > ssl-profile { > > > > > > > > > > > > > > > > > > > > > > > name: ssl-profile-name > > > > > > > > > > > > > > > > > > > > > > > certFile: > > > > > > > > > > > > > > > > > > > > > > > cert_ssl_encryption.pem > > > > > > > > > > > > > > > > > > > > > > > keyFile: > > > > > > > > > > > > > > > > > > > > > > > key_ssl_encryption.pem > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > listener { > > > > > > > > > > > > > > > > > > > > > > > host: 0.0.0.0 > > > > > > > > > > > > > > > > > > > > > > > port: 10399 > > > > > > > > > > > > > > > > > > > > > > > sasl-mechanisms: EXTERNAL > > > > > > > > > > > > > > > > > > > > > > > ssl-profile: ssl-profile-name > > > > > > > > > > > > > > > > > > > > > > > authenticatePeer: yes > > > > > > > > > > > > > > > > > > > > > > > requireSsl: yes > > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > In the above configuration, where > > > > > > > > > > > > > > > > > > > > > > > should I add > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > "cert_sasl.pem"? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > > Adel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From the qdrouterd.conf man page: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Under "listener": > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > trustedCerts (path) > > > > > > > > > > > > > > > > > > > > > > This optional setting can be > > > > > > > > > > > > > > > > > > > > > > used to reduce > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > set of > > > > > > > > > > > > > > > > > > > > > > available > > > > > > > > > > > > > > > > > > > > > > CAs for client authentication. > > > > > > > > > > > > > > > > > > > > > > If used, this > > > > > > > > > > > > > > > > > > > > > > setting > > > > > > > > > > > > > > > > > > > > > > must > > > > > > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > path to a PEM file that > > > > > > > > > > > > > > > > > > > > > > contains the trusted > > > > > > > > > > > > > > > > > > > > > > certificates. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > > > > > > > > > > To unsubscribe, e-mail: > > > > > > > > > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > > > > > > > > > To unsubscribe, e-mail: > > > > > > > > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > > > To unsubscribe, e-mail: > > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > > To unsubscribe, e-mail: > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > To unsubscribe, e-mail: > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > >
