Re: the little ssh that (sometimes) couldn't

2012-10-29 Thread Mike O'Connor
:
:corruption!
:
:
:http://mina.naguib.ca/blog/2012/10/22/the-little-ssh-that-sometimes-couldnt.html

I ran into a similar issue with a customer just a few days ago!  The
customer's theory was that there was something badly wrong with their
dorky gateway/switch (which we sold and support ).  ssh was
timing out, with a SSH2_MSG_KEX_DH_GEX_GROUP hang/failure during the
ssh protocol exchange.  Based on that, some wireshark captures, and
and stray Google droppings, I advised them to ratchet down the MTU to
make things work.  Through bisectional MTU settings and pinging, we
arrived at an MTU of 850.  And I initially started cursing at the
switch (because that helps move packets, really :) ).

Turns out -- the ssh server in question was running RHEL 5.x Linux,
and that was the key.  Even though "ip route show cache" looked sane,
"ip route flush cache" (which I had them run, just on a lark) made 
the problem go away.  So it probably wasn't my switch (unless it had
done something untoward in the distant past that induced some weird
Linux stack bug).

I'm mostly posting this because I was wondering if anyone else had
run into an MTU of 850 before.  Is that a "magic number" that rings
any bells (or perhaps has seen the Linux route cache behavior I did).

-Mike

-- 
 Michael J. O'Connor  m...@dojo.mi.org
 =--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--=
"It is now the age of now."-Non Campus Mentis


pgpco3nCOlAoW.pgp
Description: PGP signature


Re: the little ssh that (sometimes) couldn't

2012-10-29 Thread Robert Drake



On 10/29/2012 02:54 PM, Jon Lewis wrote:


Bush league.  I debugged a similar issue on Sprint's network about 15
years ago, also nailing it down to which router/router hop had the problem


When I was working for Sprint about 12 years ago, we had a circuit where 
the customer complained that we were blocking executable downloads.


We essentially dismissed his complaints because they were ridiculous. 
We would test his T1 and it would show everything fine.  I was willing 
to entertain his concern because it sounded weird and he had a UNIX box 
I could login to.


Running wget I saw the same issues.  If I zipped a file I could download 
it without issue, anything that was an exe would not.


We narrowed it down to 2-4 bytes of the exe header that the circuit just 
wouldn't pass.  Called the local telco and had them test the circuit 
from the customer prem, they found errors on the reverse.


We fixed it and he could download executables again.  I got an award for 
persistence and the customer canceled his account.





Re: the little ssh that (sometimes) couldn't

2012-10-29 Thread Jon Lewis

On Mon, 29 Oct 2012 bmann...@vacation.karoshi.com wrote:


corruption!

http://mina.naguib.ca/blog/2012/10/22/the-little-ssh-that-sometimes-couldnt.html


Bush league.  I debugged a similar issue on Sprint's network about 15 
years ago, also nailing it down to which router/router hop had the problem 
(a misconfigured interface that couldn't pass certain bit patterns and was 
causing a particular file we were hosting for a customer to be 
non-downloadable by any client who's packets used the bad path), also 
using ping, but with a pattern much more interesting than large packets of 
nulls...and I had to figure out the problematic pattern before I could do 
the ping tests.


But if you want really bizzare, this one never got solved to my 
satisfaction. 
http://mailman.nanog.org/pipermail/nanog/2008-August/002788.html


--
 Jon Lewis, MCP :)   |  I route
 Senior Network Engineer |  therefore you are
 Atlantic Net|
_ http://www.lewis.org/~jlewis/pgp for PGP public key_



Re: the little ssh that (sometimes) couldn't

2012-10-29 Thread George Herbert
On Mon, Oct 29, 2012 at 10:07 AM,   wrote:
>
> corruption!
>
>
> http://mina.naguib.ca/blog/2012/10/22/the-little-ssh-that-sometimes-couldnt.html
>
>
> /bill

This is an excellent full-stack debugging war story.

Thanks for posting it, Bill.

-- 
-george william herbert
george.herb...@gmail.com



the little ssh that (sometimes) couldn't

2012-10-29 Thread bmanning

corruption!


http://mina.naguib.ca/blog/2012/10/22/the-little-ssh-that-sometimes-couldnt.html


/bill