[Freeipa-users] Expired certs not auto renewed by Cermonger
Running FreeIPA 2.1.4 and ran into an issue where a Server-Cert did not auto-renew. ipa-getcert list Number of certificates and requests being tracked: 4. Request ID '20110706215109': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-CTIDATA-NET',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-CTIDATA-NET/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-CTIDATA-NET',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=CTIDATA.NET subject: CN=ipa01.ctidata.net,O=CTIDATA.NET expires: 2013-08-23 20:20:10 UTC eku: id-kp-serverAuth track: yes auto-renew: yes Request ID '20110706215129': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=CTIDATA.NET subject: CN=ipa01.ctidata.net,O=CTIDATA.NET expires: 2013-08-23 20:30:21 UTC eku: id-kp-serverAuth track: yes auto-renew: yes Request ID '20120615190133': status: CA_UNCONFIGURED ca-error: Error setting up ccache for local host service using default keytab. stuck: yes key pair storage: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert',token='NSS Certificate DB' certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert' CA: IPA issuer: subject: expires: unknown track: yes auto-renew: yes Request ID '20120925200227': status: GENERATING_CSR ca-error: Unable to determine principal name for signing request. stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=CTIDATA.NET subject: CN=ipa01.ctidata.net,O=CTIDATA.NET expires: 2013-03-24 19:56:36 UTC eku: id-kp-serverAuth track: yes auto-renew: yes I verified that the IPA keytab is populated: klist -kt /etc/krb5.keytab Keytab name: WRFILE:/etc/krb5.keytab KVNO Timestamp Principal - 2 07/06/11 21:51:43 host/ipa01.ctidata@ctidata.net 2 07/06/11 21:51:43 host/ipa01.ctidata@ctidata.net 2 07/06/11 21:51:43 host/ipa01.ctidata@ctidata.net 2 07/06/11 21:51:43 host/ipa01.ctidata@ctidata.net 2 07/06/11 21:51:43 host/ipa01.ctidata@ctidata.net 2 07/06/11 21:51:43 host/ipa01.ctidata@ctidata.net 4 07/18/12 21:20:41 host/ipa01.ctidata@ctidata.net 4 07/18/12 21:20:41 host/ipa01.ctidata@ctidata.net 4 07/18/12 21:20:41 host/ipa01.ctidata@ctidata.net 4 07/18/12 21:20:41 host/ipa01.ctidata@ctidata.net 5 07/18/12 21:21:00 host/ipa01.ctidata@ctidata.net 5 07/18/12 21:21:00 host/ipa01.ctidata@ctidata.net 5 07/18/12 21:21:00 host/ipa01.ctidata@ctidata.net 5 07/18/12 21:21:00 host/ipa01.ctidata@ctidata.net 6 05/02/13 15:02:10 host/ipa01.ctidata@ctidata.net 6 05/02/13 15:02:10 host/ipa01.ctidata@ctidata.net 6 05/02/13 15:02:10 host/ipa01.ctidata@ctidata.net 6 05/02/13 15:02:10 host/ipa01.ctidata@ctidata.net and ran kvno host/ipa01.ctidata.net to see what the KDC shows for this principle: host/ipa01.ctidata@ctidata.net: kvno = 6 Not sure what caused the ca_errors but I need to at least manually renew the certs and then figure out what went wrong. Any advice on what the ca_errors mean and how I can fix the issue? Thanks, David ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Expired certs not auto renewed by Cermonger
Here is the output from the submit: /usr/libexec/certmonger/ipa-submit -P bogus/`hostname` ~/req.csr Submitting request to https://ipa01.ctidata.net/ipa/xml;. Fault -504: (libcurl failed to execute the HTTP POST transaction, explaining: Peer certificate cannot be authenticated with known CA certificates). Server failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer certificate cannot be authenticated with known CA certificates). Regarding /etc/ipa/ca.crt, it isn't expired it shows its valid until July 6, 2019. On Thu, May 2, 2013 at 12:30 PM, Nalin Dahyabhai na...@redhat.com wrote: On Thu, May 02, 2013 at 11:45:51AM -0500, Toasted Penguin wrote: Nalin, Thanks for your response. Running `hostname` does result in ipa01.ctidata.net and kinit -k host/ipa01.ctidata.net does also succeed. I ran ` ipa-getcert resubmit -i 20120925200227 -K HTTP/ ipa01.ctidata@ctidata.net` and it resulted in this: Request ID '20120615190133': status: CA_UNCONFIGURED ca-error: Error setting up ccache for local host service using default keytab. stuck: yes key pair storage: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert',token='NSS Certificate DB' certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert' CA: IPA issuer: subject: expires: unknown track: yes auto-renew: yes Can you retrieve the contents of the request and save it to a temporary file, like so: reqfile=`grep -l '^id=20120615190133' /var/lib/certmonger/requests/*` awk '/BEGIN .*REQ/,/END .*REQ/ {sub(^( |csr=),);print}' $reqfile \ ~/req.csr And then try to manually submit it to the server for signing, in the way that certmonger would, like so: /usr/libexec/certmonger/ipa-submit -P bogus/`hostname` ~/req.csr Hopefully the error output there will give us more information about what's going on when the submission helper's failing to set up a ccache. If it manages to get past that point, I expect it to fail because you hopefully don't have a principal named bogus defined on the local host. But at that point we'll have gotten past errors creating the ccache, and we'll have to find another way to figure out why it failed here. As an aside, we provide better information for this error in the ca-error note with later versions than you appear to have, so tracking down this information won't always be this complicated. Request ID '20120925200227': status: CA_UNREACHABLE ca-error: Server failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer certificate cannot be authenticated with known CA certificates). stuck: yes key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=CTIDATA.NET subject: CN=ipa01.ctidata.net,O=CTIDATA.NET expires: 2013-03-24 19:56:36 UTC eku: id-kp-serverAuth track: yes auto-renew: yes There's an error verifying the server's certificate using the local copy of the CA certificate in /etc/ipa/ca.crt. Is it also expired? Nalin ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Expired certs not auto renewed by Cermonger
/etc/ipa/ca.crt was issued by O=CTIDATA.NET, CN=Certificate Authority All the certs monitored by Certmonger show the same issuer. Wasn't getting anything back when running the ipahost script you provided, ran ipahost=`grep ^host= /etc/ipa/default.conf | cut -f2- -d=` and echo $ipahost shows nothing so I just ran the openssl section manually: openssl s_client -CAfile /etc/ipa/ca.crt -connect ipa01.ctidata.net:https -showcerts /dev/null Results: CONNECTED(0003) depth=1 O = CTIDATA.NET, CN = Certificate Authority verify return:1 depth=0 O = CTIDATA.NET, CN = ipa01.ctidata.net verify error:num=10:certificate has expired notAfter=Mar 24 19:56:36 2013 GMT verify return:1 depth=0 O = CTIDATA.NET, CN = ipa01.ctidata.net notAfter=Mar 24 19:56:36 2013 GMT verify return:1 --- Certificate chain 0 s:/O=CTIDATA.NET/CN=ipa01.ctidata.net i:/O=CTIDATA.NET/CN=Certificate Authority -BEGIN CERTIFICATE- # -END CERTIFICATE- 1 s:/O=CTIDATA.NET/CN=Certificate Authority i:/O=CTIDATA.NET/CN=Certificate Authority -BEGIN CERTIFICATE- -END CERTIFICATE- --- Server certificate subject=/O=CTIDATA.NET/CN=ipa01.ctidata.net issuer=/O=CTIDATA.NET/CN=Certificate Authority --- No client certificate CA names sent --- SSL handshake has read 1959 bytes and written 463 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher: AES256-SHA Session-ID: # Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1367518514 Timeout : 300 (sec) Verify return code: 10 (certificate has expired) --- DONE On Thu, May 2, 2013 at 12:53 PM, Nalin Dahyabhai na...@redhat.com wrote: On Thu, May 02, 2013 at 12:45:34PM -0500, Toasted Penguin wrote: Here is the output from the submit: /usr/libexec/certmonger/ipa-submit -P bogus/`hostname` ~/req.csr Submitting request to https://ipa01.ctidata.net/ipa/xml;. Fault -504: (libcurl failed to execute the HTTP POST transaction, explaining: Peer certificate cannot be authenticated with known CA certificates). Server failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer certificate cannot be authenticated with known CA certificates). Regarding /etc/ipa/ca.crt, it isn't expired it shows its valid until July 6, 2019. Hmm, so for both cases, you're seeing errors verifying the IPA server's certificate. Can you double-check the certificates and that the server's looks like it was issued by the CA? This should more or less repeat the part of the process that's giving libcurl trouble, and show us the certificates, too: ipahost=`grep ^host= /etc/ipa/default.conf | cut -f2- -d=` openssl s_client -CAfile /etc/ipa/ca.crt \ -connect $ipahost:https -showcerts /dev/null Nalin ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Expired certs not auto renewed by Cermonger
Yes that helped fix 2012092520027 (thank you!!) But I am still seeing an error with: Request ID '20120615190133': status: CA_UNCONFIGURED ca-error: Error setting up ccache for local host service using default keytab. stuck: yes key pair storage: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert',token='NSS Certificate DB' certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='Server-Cert' CA: IPA issuer: subject: expires: unknown track: yes auto-renew: yes I noticed that the request ID doesn't show up in /var/lib/certmonger/requests/, does that make a difference? David On Thu, May 2, 2013 at 2:35 PM, Nalin Dahyabhai na...@redhat.com wrote: On Thu, May 02, 2013 at 01:23:04PM -0500, Toasted Penguin wrote: /etc/ipa/ca.crt was issued by O=CTIDATA.NET, CN=Certificate Authority All the certs monitored by Certmonger show the same issuer. Ok, good. (If that hadn't been the case, I wouldn't have had an explanation to offer.) Wasn't getting anything back when running the ipahost script you provided, ran ipahost=`grep ^host= /etc/ipa/default.conf | cut -f2- -d=` and echo $ipahost shows nothing so I just ran the openssl section manually: Hmm. Curious. That might be a leftover from having different releases installed at various times on my test box. Thanks for continuing on. openssl s_client -CAfile /etc/ipa/ca.crt -connect ipa01.ctidata.net: https -showcerts /dev/null Results: CONNECTED(0003) depth=1 O = CTIDATA.NET, CN = Certificate Authority verify return:1 depth=0 O = CTIDATA.NET, CN = ipa01.ctidata.net verify error:num=10:certificate has expired notAfter=Mar 24 19:56:36 2013 GMT verify return:1 depth=0 O = CTIDATA.NET, CN = ipa01.ctidata.net notAfter=Mar 24 19:56:36 2013 GMT verify return:1 --- Certificate chain 0 s:/O=CTIDATA.NET/CN=ipa01.ctidata.net i:/O=CTIDATA.NET/CN=Certificate Authority -BEGIN CERTIFICATE- # -END CERTIFICATE- 1 s:/O=CTIDATA.NET/CN=Certificate Authority i:/O=CTIDATA.NET/CN=Certificate Authority -BEGIN CERTIFICATE- -END CERTIFICATE- --- Server certificate subject=/O=CTIDATA.NET/CN=ipa01.ctidata.net issuer=/O=CTIDATA.NET/CN=Certificate Authority --- No client certificate CA names sent --- SSL handshake has read 1959 bytes and written 463 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher: AES256-SHA Session-ID: # Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1367518514 Timeout : 300 (sec) Verify return code: 10 (certificate has expired) --- DONE Yup, that's the problem: the IPA server's certificate wasn't able to be replaced while it was still valid, and now it can no longer ask itself for a new one. With 2.1.4, I think the simplest way to sort this is to stop the services (ipactl stop; service certmonger stop), roll the system date back, start the services up again, possibly use 'ipa-getcert resubmit' to force updating (it should happen automatically, but forcing it to happen a second time won't hurt). Then shut things down, set the correct time on the clock, and bring everything back up again. Hopefully there's a smarter way to do it, but I'm blanking on it if there is one. HTH, Nalin ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Setting up sudo in FreeIPA v2.2
On Tue, Oct 16, 2012 at 10:50 PM, JR Aquino jr.aqu...@citrix.com wrote: On the host in question Run the command: domainname That wants to match whatever your domain is. If it doesn't it will fail even if you have all the server rules configured correctly. This is a sudo + netgroups/hostgroups 'feature' ~ Jr Aquino | Sr. Information Security Specialist GIAC Certified Incident Handler | GIAC WebApp Penetration Tester Citrix Online | 7408 Hollister Avenue | Goleta, CA 93117 T: +1 805.690.3478 C: +1 805.717.0365 jr.aqu...@citrixonline.com http://www.citrixonline.com On Oct 16, 2012, at 2:26 PM, Toasted Penguin toastedpenguini...@gmail.com wrote: I have the server setup to manage sudo and I configured a target client to use the IPA server for sudo. When a user tries to use sudo (in this case sudo su -) it fails and they get the error user is not allowed to run sudo on client-host. This incident will be reported. I verified via the log files that the client is making requests to the IPA server when the user is attemping to use sudo and it fails. I temporarily disabled using the IPA server for sudo and I get the standard User not in the sudoers file Its starting to look like the server rules maybe the issue but I believe I have the sudo rule setup correctly. I created a sudo command /bin/su, created a sudo rule Sudo to root , added the group the user in question is a part of to the WHO--User Groups; Added the Host Group the target client host is part of to Access This Host--Host Groups and added the sudo command to the sudo rule via Allow--Sudo Allow Commands. When I delete the sudo rule I get the same result as I did when I temporarily disbled the client host using tghe IPA server for sudo verification. Any ideas why or where to look to figure out this issue? Thanks, David ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users Executing domainname results in the correct domain for theFreeIPA service. ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Sudo works for full access, but not on a per command or host level.
On Wed, Oct 17, 2012 at 2:26 PM, Rob Crittenden rcrit...@redhat.com wrote: Rich Megginson wrote: On 10/17/2012 12:49 PM, Macklin, Jason wrote: ldapsearch -xLLL -H ldap://dbduvdu145.dbr.roche.**comhttp://dbduvdu145.dbr.roche.com-D cn=directory manager -W -b dc=dbr,dc=roche,dc=com uid=asteinfeld \* snip dn: uid=asteinfeld,cn=users,cn=**accounts,dc=dbr,dc=roche,dc=**com ...snip... krbPrincipalName: asteinf...@dbr.roche.com krbPasswordExpiration: 20130324201805Z krbLastPwdChange: 20120925201805Z krbLoginFailedCount: 0 krbLastSuccessfulAuth: 20121017184614Z krbTicketFlags: 128 krbLastFailedAuth: 20121015143818Z No krbPwdLockoutDuration attribute - so according to ipalockout_preop() this means the Entry permanently locked. Not sure why. I don't believe this applies if the attribute doesn't exist. It doesn't for any of my test users and it works fine. [jmacklin@dbduwdu062 Desktop]$ ldapsearch -xLLL -H ldap://dbduvdu145.dbr.roche.**com http://dbduvdu145.dbr.roche.com -D cn=directory manager -W -b dc=dbr,dc=roche,dc=com uid=jmacklin \*Enter LDAP Password: dn: uid=jmacklin,cn=users,cn=**compat,dc=dbr,dc=roche,dc=com objectClass: posixAccount objectClass: top gecos: Jason Macklin cn: Jason Macklin uidNumber: 2084 gidNumber: 2084 loginShell: /bin/bash homeDirectory: /home2/jmacklin uid: jmacklin dn: uid=jmacklin,cn=users,cn=**accounts,dc=dbr,dc=roche,dc=**com displayName: Jason Macklin cn: Jason Macklin objectClass: top objectClass: person objectClass: organizationalperson objectClass: inetorgperson objectClass: inetuser objectClass: posixaccount objectClass: krbprincipalaux objectClass: krbticketpolicyaux objectClass: ipaobject objectClass: mepOriginEntry loginShell: /bin/bash sn: Macklin gecos: Jason Macklin homeDirectory: /home2/jmacklin krbPwdPolicyReference: cn=global_policy,cn=DBR.ROCHE.**COM http://DBR.ROCHE.COM ,cn=kerberos,dc=dbr,dc =roche,dc=com krbPrincipalName: jmack...@dbr.roche.com givenName: Jason uid: jmacklin initials: JM uidNumber: 2084 gidNumber: 2084 ipaUniqueID: 045652b4-8e3c-11e1-831f-**005056bb0010 mepManagedEntry: cn=jmacklin,cn=groups,cn=**accounts,dc=dbr,dc=roche,dc= **com memberOf: cn=admins,cn=groups,cn=**accounts,dc=dbr,dc=roche,dc=**com memberOf: cn=Replication Administrators,cn=privileges,**cn=pbac,dc=dbr,dc=roche, dc=com memberOf: cn=Add Replication Agreements,cn=permissions,cn=**pbac,dc=dbr,dc=roche ,dc=com memberOf: cn=Modify Replication Agreements,cn=permissions,cn=**pbac,dc=dbr,dc=ro che,dc=com memberOf: cn=Remove Replication Agreements,cn=permissions,cn=**pbac,dc=dbr,dc=ro che,dc=com memberOf: cn=Host Enrollment,cn=privileges,cn=** pbac,dc=dbr,dc=roche,dc=com memberOf: cn=Manage host keytab,cn=permissions,cn=pbac,**dc=dbr,dc=roche,dc=com memberOf: cn=Enroll a host,cn=permissions,cn=pbac,** dc=dbr,dc=roche,dc=com memberOf: cn=Add krbPrincipalName to a host,cn=permissions,cn=pbac,**dc=dbr,dc=r oche,dc=com memberOf: cn=Unlock user accounts,cn=permissions,cn=**pbac,dc=dbr,dc=roche,dc=co m memberOf: cn=Manage service keytab,cn=permissions,cn=pbac,**dc=dbr,dc=roche,dc=c om memberOf: cn=dbr,cn=groups,cn=accounts,**dc=dbr,dc=roche,dc=com memberOf: ipaUniqueID=23216c12-9934-**11e1-bd4c-005056bb0010,cn=**sudorules,cn=sud o,dc=dbr,dc=roche,dc=com krbLastFailedAuth: 20121017164159Z krbPrincipalKey:: MIIC4qADAgEBoQMCAQGiAwIBBaMDAg**EBpIICyjCCAsYwbaAgMB6gAwIBAKEX BBVEQlIuUk9DSEUuQ09Nam1hY2tsaW**6hSTBHoAMCARKhQAQ+** IACOG0H0Ebd8nSSY6zU3Y29ZHtQ9a sC2QJFL/**lnbaFO1DYG15WjJYXnJ7k3m0LN0aTy**jvz7FN4OWMF4tvvowXaAgMB6gAwIBA **KEXBBVEQl IuUk9DSEUuQ09Nam1hY2tsaW6hOTA3**oAMCARGhMAQuEAD6UdNSe/** mp8qqi4OuT7HOqIs80DFQDRny 37aZaD4lYrFsnQiBtpnpMnNSxADBlo**CAwHqADAgEAoRcEFURCUi5ST0NIRS5** DT01qbWFja2xpbqFB MD+gAwIBEKE4BDYYADAQZLDW61U+**4aEZT4b+/X/**OpiQLHTQlyIUolm9EjVG4wXu+** 8Mn4lMYMZyR/F Gw6NWeeq1kwXaAgMB6gAwIBAKEXBBV**EQlIuUk9DSEUuQ09Nam1hY2tsaW6hO** TA3oAMCARehMAQuEA CiWDGd28XkiaDAwpGyK0MqSawLCXs+**jKOFAA5BoSpayVTJJqjzAwSEitSu5z** BVoCAwHqADAgEAoRc EFURCUi5ST0NIRS5DT01qbWFja2xpb**qExMC+**gAwIBCKEoBCYIAKL5bzV4nQide/+6/** 2FE5LxYGULv 8Ws/Uu0RXrwAnR8/**ZuUh0TBVoCAwHqADAgEAoRcEFURCUi** 5ST0NIRS5DT01qbWFja2xpbqExMC+**gA wIBA6EoBCYIANgV0agxRmfBwY2Cb7g**Plm1oWDY5qhZidd8a0KmeIlBG56XLZ** jAzoTEwL6ADAgEBoS gEJggAo/**BQC7g4SWQY0UkU7rvoOAXwobVlAZn8**mesgQEznRDr2+** bxjME2gGDAWoAMCAQWhDwQNREJ SLlJPQ0hFLkNPTaExMC+**gAwIBAaEoBCYIAMDDcwjYU6jLJTnE+** Lzs0Ulxgf4FDEnTRXTjfJBqXIJb R5aBPg== krbLastPwdChange: 20120809140419Z krbPasswordExpiration: 20130205140419Z userPassword:: e1NTSEF9a0NXcUxTc1JOQ2tEUVlLVV**F4VTdJLzh1TXREVnBWZjlnMWRxa0E9**PQ= = krbExtraData:: AAJjwyNQa2FkbWluZEBEQlIuUk9DSE**UuQ09NAA== krbLastSuccessfulAuth: 2012101718Z krbLoginFailedCount: 0 krbTicketFlags: 128 So with all of that output, I would like to mention the discrepancy with ldap.conf. Just trying to get any sudo working on RHEL 6.3 was problematic