Re: [Freeipa-users] Free-IPA failover succeeds, but ssh is broken?

2016-01-18 Thread Martin Kosek 
Hi Jeff and Janelle,

I am glad you got things working, but I am not convinced this is the best way
to do it. The proxy is needed for SSSD SSH integration (public keys and
fingerprints), if the proxy is buggy, we should fix. And in order to fix it, it
would be great to get our hands on the logs showing the fault - CCing Jakub and
Honza on this one.

Thanks for help,
Martin

On 01/18/2016 01:14 AM, Jeff Hallyburton wrote:
> Janelle,
> 
> The proxy suggestion was spot on.  After that things seem to work normally.
> 
> Thanks!
> 
> Jeff
> 
> Jeff Hallyburton
> Strategic Systems Engineer
> Bloomip Inc.
> Web: http://www.bloomip.com
> 
> Engineering Support: supp...@bloomip.com
> Billing Support: bill...@bloomip.com
> Customer Support Portal:  https://my.bloomip.com 
> 
> On Sun, Jan 17, 2016 at 9:58 AM, Janelle  wrote:
> 
>> Hi,
>>
>> Try commenting out the proxy command in /etc/ssh/ssh_config
>>
>> The sssd proxy of ssh is buggy as can be.
>>
>> ~J
>>
>>> On Jan 17, 2016, at 05:24, Jakub Hrozek  wrote:
>>>
>>>
 On 16 Jan 2016, at 02:21, Jeff Hallyburton <
>> jeff.hallybur...@bloomip.com> wrote:

 Having finished setting up an ipa server and replica, we're trying to
>> test failover to ensure that HA works as expected.  We've been able to
>> verify the replication agreements and auto-discovery are working, and both
>> servers are picked up as expected at install time.

 That said, we're seeing some oddities with failover.  Once I shut down
>> the ipa service on the main ipa server, I get most requests completing
>> after about a 2 min window.  I am able to:

 1.  Authenticate to our jump server and get a kerberos ticket
 2.  kinit successfully as other users

 However, whenever I try to ssh to another system within our domain, ssh
>> breaks with the following error:

 $ ssh -vvv automation01
 OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug1: /etc/ssh/ssh_config line 5: Applying options for *
 debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy
>> -p 22 automation01
 debug1: permanently_drop_suid: 158701
 debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa type -1
 debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa-cert type -1
 debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa type -1
 debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa-cert type -1
 debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa type -1
 debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa-cert type -1
 debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519 type -1
 debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519-cert type
>> -1
 debug1: Enabling compatibility mode for protocol 2.0
 debug1: Local version string SSH-2.0-OpenSSH_6.6.1
 ssh_exchange_identification: Connection closed by remote host
>>>
>>> Did you crank up debug level on the machine where sshd is running and
>> see if anything is logged then?
>>>

 Nothing is logged in either /var/log/messages or /var/log/secure when
>> this happens, so I'm unsure where to begin debugging.  Can you offer any
>> insight?

 Thanks,

 Jeff

 Jeff Hallyburton
 Strategic Systems Engineer
 Bloomip Inc.
 Web: http://www.bloomip.com

 Engineering Support: supp...@bloomip.com
 Billing Support: bill...@bloomip.com
 Customer Support Portal:  https://my.bloomip.com
 --
 Manage your subscription for the Freeipa-users mailing list:
 https://www.redhat.com/mailman/listinfo/freeipa-users
 Go to http://freeipa.org for more info on the project
>>>
>>>
>>> --
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go to http://freeipa.org for more info on the project
>>
> 
> 
> 

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Free-IPA failover succeeds, but ssh is broken?

2016-01-18 Thread Jakub Hrozek 
On Mon, Jan 18, 2016 at 09:27:23AM +0100, Martin Kosek wrote:
> Hi Jeff and Janelle,
> 
> I am glad you got things working, but I am not convinced this is the best way
> to do it. The proxy is needed for SSSD SSH integration (public keys and
> fingerprints), if the proxy is buggy, we should fix. And in order to fix it, 
> it
> would be great to get our hands on the logs showing the fault - CCing Jakub 
> and
> Honza on this one.

Yes, if you see issues with the proxy, by all means file bugs..

> 
> Thanks for help,
> Martin
> 
> On 01/18/2016 01:14 AM, Jeff Hallyburton wrote:
> > Janelle,
> > 
> > The proxy suggestion was spot on.  After that things seem to work normally.
> > 
> > Thanks!
> > 
> > Jeff
> > 
> > Jeff Hallyburton
> > Strategic Systems Engineer
> > Bloomip Inc.
> > Web: http://www.bloomip.com
> > 
> > Engineering Support: supp...@bloomip.com
> > Billing Support: bill...@bloomip.com
> > Customer Support Portal:  https://my.bloomip.com 
> > 
> > On Sun, Jan 17, 2016 at 9:58 AM, Janelle  wrote:
> > 
> >> Hi,
> >>
> >> Try commenting out the proxy command in /etc/ssh/ssh_config
> >>
> >> The sssd proxy of ssh is buggy as can be.
> >>
> >> ~J
> >>
> >>> On Jan 17, 2016, at 05:24, Jakub Hrozek  wrote:
> >>>
> >>>
>  On 16 Jan 2016, at 02:21, Jeff Hallyburton <
> >> jeff.hallybur...@bloomip.com> wrote:
> 
>  Having finished setting up an ipa server and replica, we're trying to
> >> test failover to ensure that HA works as expected.  We've been able to
> >> verify the replication agreements and auto-discovery are working, and both
> >> servers are picked up as expected at install time.
> 
>  That said, we're seeing some oddities with failover.  Once I shut down
> >> the ipa service on the main ipa server, I get most requests completing
> >> after about a 2 min window.  I am able to:
> 
>  1.  Authenticate to our jump server and get a kerberos ticket
>  2.  kinit successfully as other users
> 
>  However, whenever I try to ssh to another system within our domain, ssh
> >> breaks with the following error:
> 
>  $ ssh -vvv automation01
>  OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
>  debug1: Reading configuration data /etc/ssh/ssh_config
>  debug1: /etc/ssh/ssh_config line 5: Applying options for *
>  debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy
> >> -p 22 automation01
>  debug1: permanently_drop_suid: 158701
>  debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa type -1
>  debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa-cert type -1
>  debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa type -1
>  debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa-cert type -1
>  debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa type -1
>  debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa-cert type -1
>  debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519 type -1
>  debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519-cert type
> >> -1
>  debug1: Enabling compatibility mode for protocol 2.0
>  debug1: Local version string SSH-2.0-OpenSSH_6.6.1
>  ssh_exchange_identification: Connection closed by remote host
> >>>
> >>> Did you crank up debug level on the machine where sshd is running and
> >> see if anything is logged then?
> >>>
> 
>  Nothing is logged in either /var/log/messages or /var/log/secure when
> >> this happens, so I'm unsure where to begin debugging.  Can you offer any
> >> insight?
> 
>  Thanks,
> 
>  Jeff
> 
>  Jeff Hallyburton
>  Strategic Systems Engineer
>  Bloomip Inc.
>  Web: http://www.bloomip.com
> 
>  Engineering Support: supp...@bloomip.com
>  Billing Support: bill...@bloomip.com
>  Customer Support Portal:  https://my.bloomip.com
>  --
>  Manage your subscription for the Freeipa-users mailing list:
>  https://www.redhat.com/mailman/listinfo/freeipa-users
>  Go to http://freeipa.org for more info on the project
> >>>
> >>>
> >>> --
> >>> Manage your subscription for the Freeipa-users mailing list:
> >>> https://www.redhat.com/mailman/listinfo/freeipa-users
> >>> Go to http://freeipa.org for more info on the project
> >>
> > 
> > 
> > 
> 

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Free-IPA failover succeeds, but ssh is broken?

2016-01-18 Thread Alexander Bokovoy 

On Fri, 15 Jan 2016, Jeff Hallyburton wrote:

Having finished setting up an ipa server and replica, we're trying to test
failover to ensure that HA works as expected.  We've been able to verify
the replication agreements and auto-discovery are working, and both servers
are picked up as expected at install time.

That said, we're seeing some oddities with failover.  Once I shut down the
ipa service on the main ipa server, I get most requests completing after
about a 2 min window.  I am able to:

1.  Authenticate to our jump server and get a kerberos ticket
2.  kinit successfully as other users

However, whenever I try to ssh to another system within our domain, ssh
breaks with the following error:

$ ssh -vvv automation01

OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 5: Applying options for *

debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p
22 automation01

debug1: permanently_drop_suid: 158701

debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa type -1

debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa-cert type -1

debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa type -1

debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa-cert type -1

debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa type -1

debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa-cert type -1

debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519 type -1

debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519-cert type -1

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_6.6.1

ssh_exchange_identification: Connection closed by remote host


Nothing is logged in either /var/log/messages or /var/log/secure when this
happens, so I'm unsure where to begin debugging.  Can you offer any insight?

Do you have, by chance either on the client or on automation01 a locale
that doesn't exist on either one? For example, a fr_FR locale on the
client which is missing on the server?

By default sshd configuration allows to accept certain environmental
variables when client connection comes in:

/etc/ssh/sshd_config:
# Accept locale-related environment variables
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS

/etc/ssh/ssh_config:
# Send locale-related environment variables
	SendEnv LANG 
	SendEnv XMODIFIERS


There is a bug in the proxy command -- it tries to enable localized
error messages and if that step fails, the proxy tool exits with an
error code which is visible as 


ssh_exchange_identification: Connection closde by remote host

I think we fixed this in newer SSSD versions already.
--
/ Alexander Bokovoy

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Free-IPA failover succeeds, but ssh is broken?

2016-01-18 Thread Jakub Hrozek 
On Mon, Jan 18, 2016 at 10:54:42AM +0200, Alexander Bokovoy wrote:
> I think we fixed this in newer SSSD versions already.

Yes, but in master only, we haven't released the fix yet:
https://fedorahosted.org/sssd/ticket/2785

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Free-IPA failover succeeds, but ssh is broken?

2016-01-17 Thread Jeff Hallyburton 
Janelle,

The proxy suggestion was spot on.  After that things seem to work normally.

Thanks!

Jeff

Jeff Hallyburton
Strategic Systems Engineer
Bloomip Inc.
Web: http://www.bloomip.com

Engineering Support: supp...@bloomip.com
Billing Support: bill...@bloomip.com
Customer Support Portal:  https://my.bloomip.com 

On Sun, Jan 17, 2016 at 9:58 AM, Janelle  wrote:

> Hi,
>
> Try commenting out the proxy command in /etc/ssh/ssh_config
>
> The sssd proxy of ssh is buggy as can be.
>
> ~J
>
> > On Jan 17, 2016, at 05:24, Jakub Hrozek  wrote:
> >
> >
> >> On 16 Jan 2016, at 02:21, Jeff Hallyburton <
> jeff.hallybur...@bloomip.com> wrote:
> >>
> >> Having finished setting up an ipa server and replica, we're trying to
> test failover to ensure that HA works as expected.  We've been able to
> verify the replication agreements and auto-discovery are working, and both
> servers are picked up as expected at install time.
> >>
> >> That said, we're seeing some oddities with failover.  Once I shut down
> the ipa service on the main ipa server, I get most requests completing
> after about a 2 min window.  I am able to:
> >>
> >> 1.  Authenticate to our jump server and get a kerberos ticket
> >> 2.  kinit successfully as other users
> >>
> >> However, whenever I try to ssh to another system within our domain, ssh
> breaks with the following error:
> >>
> >> $ ssh -vvv automation01
> >> OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
> >> debug1: Reading configuration data /etc/ssh/ssh_config
> >> debug1: /etc/ssh/ssh_config line 5: Applying options for *
> >> debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy
> -p 22 automation01
> >> debug1: permanently_drop_suid: 158701
> >> debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa type -1
> >> debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa-cert type -1
> >> debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa type -1
> >> debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa-cert type -1
> >> debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa type -1
> >> debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa-cert type -1
> >> debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519 type -1
> >> debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519-cert type
> -1
> >> debug1: Enabling compatibility mode for protocol 2.0
> >> debug1: Local version string SSH-2.0-OpenSSH_6.6.1
> >> ssh_exchange_identification: Connection closed by remote host
> >
> > Did you crank up debug level on the machine where sshd is running and
> see if anything is logged then?
> >
> >>
> >> Nothing is logged in either /var/log/messages or /var/log/secure when
> this happens, so I'm unsure where to begin debugging.  Can you offer any
> insight?
> >>
> >> Thanks,
> >>
> >> Jeff
> >>
> >> Jeff Hallyburton
> >> Strategic Systems Engineer
> >> Bloomip Inc.
> >> Web: http://www.bloomip.com
> >>
> >> Engineering Support: supp...@bloomip.com
> >> Billing Support: bill...@bloomip.com
> >> Customer Support Portal:  https://my.bloomip.com
> >> --
> >> Manage your subscription for the Freeipa-users mailing list:
> >> https://www.redhat.com/mailman/listinfo/freeipa-users
> >> Go to http://freeipa.org for more info on the project
> >
> >
> > --
> > Manage your subscription for the Freeipa-users mailing list:
> > https://www.redhat.com/mailman/listinfo/freeipa-users
> > Go to http://freeipa.org for more info on the project
>
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Free-IPA failover succeeds, but ssh is broken?

2016-01-17 Thread Janelle 
Hi,

Try commenting out the proxy command in /etc/ssh/ssh_config

The sssd proxy of ssh is buggy as can be.

~J

> On Jan 17, 2016, at 05:24, Jakub Hrozek  wrote:
> 
> 
>> On 16 Jan 2016, at 02:21, Jeff Hallyburton  
>> wrote:
>> 
>> Having finished setting up an ipa server and replica, we're trying to test 
>> failover to ensure that HA works as expected.  We've been able to verify the 
>> replication agreements and auto-discovery are working, and both servers are 
>> picked up as expected at install time.
>> 
>> That said, we're seeing some oddities with failover.  Once I shut down the 
>> ipa service on the main ipa server, I get most requests completing after 
>> about a 2 min window.  I am able to:
>> 
>> 1.  Authenticate to our jump server and get a kerberos ticket
>> 2.  kinit successfully as other users
>> 
>> However, whenever I try to ssh to another system within our domain, ssh 
>> breaks with the following error:
>> 
>> $ ssh -vvv automation01
>> OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
>> debug1: Reading configuration data /etc/ssh/ssh_config
>> debug1: /etc/ssh/ssh_config line 5: Applying options for *
>> debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 
>> automation01
>> debug1: permanently_drop_suid: 158701
>> debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa type -1
>> debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa-cert type -1
>> debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa type -1
>> debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa-cert type -1
>> debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa type -1
>> debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa-cert type -1
>> debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519 type -1
>> debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519-cert type -1
>> debug1: Enabling compatibility mode for protocol 2.0
>> debug1: Local version string SSH-2.0-OpenSSH_6.6.1
>> ssh_exchange_identification: Connection closed by remote host
> 
> Did you crank up debug level on the machine where sshd is running and see if 
> anything is logged then?
> 
>> 
>> Nothing is logged in either /var/log/messages or /var/log/secure when this 
>> happens, so I'm unsure where to begin debugging.  Can you offer any insight?
>> 
>> Thanks,
>> 
>> Jeff
>> 
>> Jeff Hallyburton
>> Strategic Systems Engineer
>> Bloomip Inc.
>> Web: http://www.bloomip.com
>> 
>> Engineering Support: supp...@bloomip.com
>> Billing Support: bill...@bloomip.com
>> Customer Support Portal:  https://my.bloomip.com
>> -- 
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go to http://freeipa.org for more info on the project
> 
> 
> -- 
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project