Re: [CentOS] Problem with ssh disconnecting

[H]

On 02/20/2018 09:33 AM, H wrote:
> On 02/19/2018 10:09 PM, Marcelo Ricardo Leitner wrote:
>> On Mon, Feb 12, 2018 at 08:05:18PM -0500, H wrote:
>> ...
>>> Not sure if I am reading your reply correctly but I should clarify
>>> that I have problems when running naked ssh to the server, when I
>>> run ssh to the same server but over the VPN connection (that goes
>>> via third server) everything is flawless.
>>>
>>> I should also explain that:
>>>
>>> - I am on a workstation (located in the US), ssh-ing into server 1
>>> (located in the US).
>>>
>>> - From server 1 I use scp to transfer large files from server 2
>>> (located in Europe) to server 1 (in the US).
>>>
>>> The above randomly disconnects.
>> You have two ssh sessions here. AFAICT, it's the first one
>> (workstation->server 1) that randomly disconnects, right?
>>
>>> However, when:
>>>
>>> - I use a VPN connection to server 3 (also located in Europe).
>>>
>>> - From the same workstation as above, do exactly as above,
>>> connections are rock-solid.
>> It would seem to me that something between your workstation and server
>> 1 is possibly mishandling TCP options (being it sack, timestamps or
>> both).  Is there a router or a firewall in between?
>>
>> Try doing a traffic capture between both and see why it hangs. If
>> you're hesitant to post the binary traffic capture, post the text
>> version of it with anonymized IP addresses. Just please be sure to
>> disable ssh protocol so it would include TCP details on it then.
>>
>>   Marcelo
> Let's reduce the problem to a simpler case: the workstation running CentOS 7 
> is behind a router, behind the same router is the server running CentOS 6. 
> For various reasons, I access the server via its external IP address, not the 
> local one. Access is via ssh.
>
> When using straight ssh, the connection times out after random times, always 
> less than 20 minutes or so, though, with the error message 
> "packet_write_wait: broken pipe", not the common "write failed: broken pipe".
>
> However, when I am in a ssh session on a VPN connection (which terminates in 
> an outside server far away from my network), the ssh connection does /not/ 
> time out.
>
> Further, accessing the same server as first mentioned, in the identical 
> fashion, from another computer running CentOS 6, I never encounter the 
> "packet_write_wait" error and disconnection.
>
> My conclusion is thus that this is not a router issue but could conceivable 
> be a bug in CentOS 7. I am also ruling out a cable issue or a physical 
> network card issue since the VPN connection is stable.
>
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos

No messages in secure or message logs.

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[H]

On 02/19/2018 10:09 PM, Marcelo Ricardo Leitner wrote:
> On Mon, Feb 12, 2018 at 08:05:18PM -0500, H wrote:
> ...
>> Not sure if I am reading your reply correctly but I should clarify
>> that I have problems when running naked ssh to the server, when I
>> run ssh to the same server but over the VPN connection (that goes
>> via third server) everything is flawless.
>>
>> I should also explain that:
>>
>> - I am on a workstation (located in the US), ssh-ing into server 1
>> (located in the US).
>>
>> - From server 1 I use scp to transfer large files from server 2
>> (located in Europe) to server 1 (in the US).
>>
>> The above randomly disconnects.
> You have two ssh sessions here. AFAICT, it's the first one
> (workstation->server 1) that randomly disconnects, right?
>
>> However, when:
>>
>> - I use a VPN connection to server 3 (also located in Europe).
>>
>> - From the same workstation as above, do exactly as above,
>> connections are rock-solid.
> It would seem to me that something between your workstation and server
> 1 is possibly mishandling TCP options (being it sack, timestamps or
> both).  Is there a router or a firewall in between?
>
> Try doing a traffic capture between both and see why it hangs. If
> you're hesitant to post the binary traffic capture, post the text
> version of it with anonymized IP addresses. Just please be sure to
> disable ssh protocol so it would include TCP details on it then.
>
>   Marcelo

Let's reduce the problem to a simpler case: the workstation running CentOS 7 is 
behind a router, behind the same router is the server running CentOS 6. For 
various reasons, I access the server via its external IP address, not the local 
one. Access is via ssh.

When using straight ssh, the connection times out after random times, always 
less than 20 minutes or so, though, with the error message "packet_write_wait: 
broken pipe", not the common "write failed: broken pipe".

However, when I am in a ssh session on a VPN connection (which terminates in an 
outside server far away from my network), the ssh connection does /not/ time 
out.

Further, accessing the same server as first mentioned, in the identical 
fashion, from another computer running CentOS 6, I never encounter the 
"packet_write_wait" error and disconnection.

My conclusion is thus that this is not a router issue but could conceivable be 
a bug in CentOS 7. I am also ruling out a cable issue or a physical network 
card issue since the VPN connection is stable.

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[Marcelo Ricardo Leitner]

On Mon, Feb 12, 2018 at 08:05:18PM -0500, H wrote:
...
> Not sure if I am reading your reply correctly but I should clarify
> that I have problems when running naked ssh to the server, when I
> run ssh to the same server but over the VPN connection (that goes
> via third server) everything is flawless.
> 
> I should also explain that:
> 
> - I am on a workstation (located in the US), ssh-ing into server 1
> (located in the US).
> 
> - From server 1 I use scp to transfer large files from server 2
> (located in Europe) to server 1 (in the US).
> 
> The above randomly disconnects.

You have two ssh sessions here. AFAICT, it's the first one
(workstation->server 1) that randomly disconnects, right?

> 
> However, when:
> 
> - I use a VPN connection to server 3 (also located in Europe).
> 
> - From the same workstation as above, do exactly as above,
> connections are rock-solid.

It would seem to me that something between your workstation and server
1 is possibly mishandling TCP options (being it sack, timestamps or
both).  Is there a router or a firewall in between?

Try doing a traffic capture between both and see why it hangs. If
you're hesitant to post the binary traffic capture, post the text
version of it with anonymized IP addresses. Just please be sure to
disable ssh protocol so it would include TCP details on it then.

  Marcelo
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[H]

On 02/12/2018 09:12 PM, Earl Ramirez wrote:
> On Mon, 2018-02-12 at 20:13 -0500, H wrote:
>> On 02/12/2018 07:24 PM, Liam O'Toole wrote:
>>> On 2018-02-12, H  wrote:
 Running CentOS 7 on workstation and having a problem with ssh
 disconnects. My ssh_config contains:

 Host *
 TCPKeepAlive yes
 ServerAliveInterval 30
 ServerAliveCountMax 300

 and sshd_config on the server contains:

 TCPKeepAlive yes
 ClientAliveInterval 60
 ClientAliveCountMax 300

 Have I missed any setting needed to prevent these random
 disconnects?
 I don't think there is anything wrong with the network card, the
 driver, or the cable, since if I am on a VPN connection via
 another
 server, the VPN and any ssh connection stay up indefinitely.

 Thanks.
>>> Another poster has provided some possible reasons for the
>>> disconnections. Whatever the cause, autossh (from the epel repo) is
>>> a
>>> good workaround.
>>>
>> Not that this happens while I do large scp file transfers that may
>> take more than half an hour, simply restarting an ssh session is not
>> going to help since I will lose the file transfer.
>>
> I don't know if this would help but I had a similar issue and it turned
> out that there was a custom script in /etc/profile.d/ that contain
> TMOUT 900.
>
> You can also check in /etc/profile, usually, the security logs has
> something about the disconnects, of you can use wireshark or a similar
> tool to capture and analyse the packets.
>
>
>
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos

I think that I forgot to mention that the problem is not the common "write 
failed: broken pipe" but "packet_write_wait: broken pipe".

I don't think the problem is router related since another computer, albeit 
running CentOS 6, does not have similar issues.

Does the information above suggest any other reason for my problem?



signature.asc
Description: OpenPGP digital signature
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[Kay Diederichs]

On 02/13/2018 11:54 AM, Liam O'Toole wrote:
> On 2018-02-13, H  wrote:
>> On 02/12/2018 07:24 PM, Liam O'Toole wrote:
>>> On 2018-02-12, H  wrote:
 Running CentOS 7 on workstation and having a problem with ssh
 disconnects. My ssh_config contains:

 Host *
 TCPKeepAlive yes
 ServerAliveInterval 30
 ServerAliveCountMax 300

 and sshd_config on the server contains:

 TCPKeepAlive yes
 ClientAliveInterval 60
 ClientAliveCountMax 300

 Have I missed any setting needed to prevent these random disconnects?
 I don't think there is anything wrong with the network card, the
 driver, or the cable, since if I am on a VPN connection via another
 server, the VPN and any ssh connection stay up indefinitely.

 Thanks.
>>> Another poster has provided some possible reasons for the
>>> disconnections. Whatever the cause, autossh (from the epel repo) is a
>>> good workaround.
>>>
>> Not that this happens while I do large scp file transfers that may
>> take more than half an hour, simply restarting an ssh session is not
>> going to help since I will lose the file transfer.
> 
> In that case sftp with the '-a' option would be a better choice.
> 

or rsync --partial

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[Liam O'Toole]

On 2018-02-13, H  wrote:
> On 02/12/2018 07:24 PM, Liam O'Toole wrote:
>> On 2018-02-12, H  wrote:
>>> Running CentOS 7 on workstation and having a problem with ssh
>>> disconnects. My ssh_config contains:
>>>
>>> Host *
>>> TCPKeepAlive yes
>>> ServerAliveInterval 30
>>> ServerAliveCountMax 300
>>>
>>> and sshd_config on the server contains:
>>>
>>> TCPKeepAlive yes
>>> ClientAliveInterval 60
>>> ClientAliveCountMax 300
>>>
>>> Have I missed any setting needed to prevent these random disconnects?
>>> I don't think there is anything wrong with the network card, the
>>> driver, or the cable, since if I am on a VPN connection via another
>>> server, the VPN and any ssh connection stay up indefinitely.
>>>
>>> Thanks.
>> Another poster has provided some possible reasons for the
>> disconnections. Whatever the cause, autossh (from the epel repo) is a
>> good workaround.
>>
> Not that this happens while I do large scp file transfers that may
> take more than half an hour, simply restarting an ssh session is not
> going to help since I will lose the file transfer.

In that case sftp with the '-a' option would be a better choice.

-- 

Liam

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[Earl Ramirez]

On Mon, 2018-02-12 at 20:13 -0500, H wrote:
> On 02/12/2018 07:24 PM, Liam O'Toole wrote:
> > On 2018-02-12, H  wrote:
> > > Running CentOS 7 on workstation and having a problem with ssh
> > > disconnects. My ssh_config contains:
> > > 
> > > Host *
> > > TCPKeepAlive yes
> > > ServerAliveInterval 30
> > > ServerAliveCountMax 300
> > > 
> > > and sshd_config on the server contains:
> > > 
> > > TCPKeepAlive yes
> > > ClientAliveInterval 60
> > > ClientAliveCountMax 300
> > > 
> > > Have I missed any setting needed to prevent these random
> > > disconnects?
> > > I don't think there is anything wrong with the network card, the
> > > driver, or the cable, since if I am on a VPN connection via
> > > another
> > > server, the VPN and any ssh connection stay up indefinitely.
> > > 
> > > Thanks.
> > 
> > Another poster has provided some possible reasons for the
> > disconnections. Whatever the cause, autossh (from the epel repo) is
> > a
> > good workaround.
> > 
> 
> Not that this happens while I do large scp file transfers that may
> take more than half an hour, simply restarting an ssh session is not
> going to help since I will lose the file transfer.
> 

I don't know if this would help but I had a similar issue and it turned
out that there was a custom script in /etc/profile.d/ that contain
TMOUT 900.

You can also check in /etc/profile, usually, the security logs has
something about the disconnects, of you can use wireshark or a similar
tool to capture and analyse the packets.

-- 
Earl Ramirez 

signature.asc
Description: This is a digitally signed message part
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[H]

On 02/12/2018 07:24 PM, Liam O'Toole wrote:
> On 2018-02-12, H  wrote:
>> Running CentOS 7 on workstation and having a problem with ssh
>> disconnects. My ssh_config contains:
>>
>> Host *
>> TCPKeepAlive yes
>> ServerAliveInterval 30
>> ServerAliveCountMax 300
>>
>> and sshd_config on the server contains:
>>
>> TCPKeepAlive yes
>> ClientAliveInterval 60
>> ClientAliveCountMax 300
>>
>> Have I missed any setting needed to prevent these random disconnects?
>> I don't think there is anything wrong with the network card, the
>> driver, or the cable, since if I am on a VPN connection via another
>> server, the VPN and any ssh connection stay up indefinitely.
>>
>> Thanks.
> Another poster has provided some possible reasons for the
> disconnections. Whatever the cause, autossh (from the epel repo) is a
> good workaround.
>
Not that this happens while I do large scp file transfers that may take more 
than half an hour, simply restarting an ssh session is not going to help since 
I will lose the file transfer.

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[Liam O'Toole]

On 2018-02-12, H  wrote:
> Running CentOS 7 on workstation and having a problem with ssh
> disconnects. My ssh_config contains:
>
> Host *
> TCPKeepAlive yes
> ServerAliveInterval 30
> ServerAliveCountMax 300
>
> and sshd_config on the server contains:
>
> TCPKeepAlive yes
> ClientAliveInterval 60
> ClientAliveCountMax 300
>
> Have I missed any setting needed to prevent these random disconnects?
> I don't think there is anything wrong with the network card, the
> driver, or the cable, since if I am on a VPN connection via another
> server, the VPN and any ssh connection stay up indefinitely.
>
> Thanks.

Another poster has provided some possible reasons for the
disconnections. Whatever the cause, autossh (from the epel repo) is a
good workaround.

-- 

Liam

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[H]

On 02/12/2018 06:34 PM, Stephen John Smoogen wrote:
> On Mon, Feb 12, 2018 at 6:25 PM H  wrote:
>
>> Running CentOS 7 on workstation and having a problem with ssh disconnects.
>> My ssh_config contains:
>>
>> Host *
>> TCPKeepAlive yes
>> ServerAliveInterval 30
>> ServerAliveCountMax 300
>>
>> and sshd_config on the server contains:
>>
>> TCPKeepAlive yes
>> ClientAliveInterval 60
>> ClientAliveCountMax 300
>>
>> Have I missed any setting needed to prevent these random disconnects? I
>> don't think there is anything wrong with the network card, the driver, or
>> the cable, since if I am on a VPN connection via another server, the VPN
>> and any ssh connection stay up indefinitely.
>>
>> Thanks.
>
>
> There are usually 2 different reasons for this:
> 1. The VPN is UDP and times out/drops keeps alives so that they no longer
> function properly. [The UDP connection will make it look like you have a
> new SSH connection which of course the system will drop because that would
> allow for security problems.]
>
> 2. A firewall in the chain of things (system you are on, the system you are
> going to, or somewhere in between) has session flushing issues. If you have
> the firewall set up to only accept NEW port 22 connections and then just
> looks to see if the ESTABLISHED, RELATED tables are accepted elsewhere then
> if the session somehow ages out or is flushed due to usage, the ssh
> connection can get dropped.
>
> The solution to one is to see if a TCP VPN fixes the problem. The second
> one is to either make the iptables kernel tables larger or to have all port
> 22 accepted even if it is not ESTABLISHED.
>
> These aren’t the only ways the problem you see can occur but they are some
> of the most common I have run into.
>
>
>
>>
>> ___
>> CentOS mailing list
>> CentOS@centos.org
>> https://lists.centos.org/mailman/listinfo/centos
>>
Not sure if I am reading your reply correctly but I should clarify that I have 
problems when running naked ssh to the server, when I run ssh to the same 
server but over the VPN connection (that goes via third server) everything is 
flawless.

I should also explain that:

- I am on a workstation (located in the US), ssh-ing into server 1 (located in 
the US).

- From server 1 I use scp to transfer large files from server 2 (located in 
Europe) to server 1 (in the US).

The above randomly disconnects.

However, when:

- I use a VPN connection to server 3 (also located in Europe).

- From the same workstation as above, do exactly as above, connections are 
rock-solid.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Problem with ssh disconnecting

[Stephen John Smoogen]

On Mon, Feb 12, 2018 at 6:25 PM H  wrote:

> Running CentOS 7 on workstation and having a problem with ssh disconnects.
> My ssh_config contains:
>
> Host *
> TCPKeepAlive yes
> ServerAliveInterval 30
> ServerAliveCountMax 300
>
> and sshd_config on the server contains:
>
> TCPKeepAlive yes
> ClientAliveInterval 60
> ClientAliveCountMax 300
>
> Have I missed any setting needed to prevent these random disconnects? I
> don't think there is anything wrong with the network card, the driver, or
> the cable, since if I am on a VPN connection via another server, the VPN
> and any ssh connection stay up indefinitely.
>
> Thanks.



There are usually 2 different reasons for this:
1. The VPN is UDP and times out/drops keeps alives so that they no longer
function properly. [The UDP connection will make it look like you have a
new SSH connection which of course the system will drop because that would
allow for security problems.]

2. A firewall in the chain of things (system you are on, the system you are
going to, or somewhere in between) has session flushing issues. If you have
the firewall set up to only accept NEW port 22 connections and then just
looks to see if the ESTABLISHED, RELATED tables are accepted elsewhere then
if the session somehow ages out or is flushed due to usage, the ssh
connection can get dropped.

The solution to one is to see if a TCP VPN fixes the problem. The second
one is to either make the iptables kernel tables larger or to have all port
22 accepted even if it is not ESTABLISHED.

These aren’t the only ways the problem you see can occur but they are some
of the most common I have run into.



>
>
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>
-- 
Stephen J Smoogen.
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

[CentOS] Problem with ssh disconnecting

[H]

Running CentOS 7 on workstation and having a problem with ssh disconnects. My 
ssh_config contains:

Host *
TCPKeepAlive yes
ServerAliveInterval 30
ServerAliveCountMax 300

and sshd_config on the server contains:

TCPKeepAlive yes
ClientAliveInterval 60
ClientAliveCountMax 300

Have I missed any setting needed to prevent these random disconnects? I don't 
think there is anything wrong with the network card, the driver, or the cable, 
since if I am on a VPN connection via another server, the VPN and any ssh 
connection stay up indefinitely.

Thanks.


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos