Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Marko Cupać]

On Fri, 22 Dec 2017 13:43:35 +0100
Florian Obser  wrote:

> Yes, quite a lot of effort and money (think travel cost to hackathons)
> was spent by developers between 5.9 and 6.2 releases.
> You are welcome.

Somehow I have the impression that most of the OpenBSD code wasn't
written in fancy guest facilities where priesthood arrives and departs
by means of business class flights to churn out some lines of code. The
time when it wasn't all about MONEY.

But yeah, we have to embrace modern times and not hold on to the past.



Still, OpenBSD is the best :)
-- 
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Florian Obser]

On Thu, Dec 21, 2017 at 09:20:16PM +, Maxim Bourmistrov wrote:
> 
> I had to bypass relayd to roll prod stable.
> Down to apache. Taking care of http and https.
> By redirect.
> Now this setup (if I can call it) is stable.
> 
> .
> 
> P.S.
> Looks like we have to move forward from here.

Buy an appliance and get off my lawn.

> 
> > 21 dec. 2017 kl. 21:58 skrev Maxim Bourmistrov :
> > 
> > 
> > Sorry, but I have to say
> > Releases after 5.9 are NOT production stable.
> > (Until all bugs are smashed within stack changes and SMP unlock).
> > After 5.9 - cost money and effort.
> > MONEY.


Yes, quite a lot of effort and money (think travel cost to hackathons)
was spent by developers between 5.9 and 6.2 releases.
You are welcome.

-- 
I'm not entirely sure you are real.

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Antoine Jacoutot]

On Thu, Dec 21, 2017 at 10:33:23PM +, Maxim Bourmistrov wrote:
> 
> 6.2-stable is NOT STABLE.
> Backport, backport,backport.
> 
> 6.2-stable is a beta release. 
> This is what its IS.
> 
> 5.9 vs. 6.2 - last one is a major downwards.
> I know a lot of stuff done in tcp/ip stack and this is a good job (abt time 
> to ack SMP), but
> Keep those changes in beta, don’t tell ”we have rel and stable here. Eat it”.

Can you keep your rant on the kindergarten mailing list and not tech@. There's a
reason misc@ exists, it's so that people like you can vomit whatever / wherever
while keeping us out of the loop.

-- 
Antoine

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Stuart Henderson]

On 2017/12/21 23:20, Maxim Bourmistrov wrote:
> 
> Fixed in HEAD?! - my ass. Whom puts HEAD into prod?! Not me any more, that's 
> for sure.
> IS LIKE DROPPING A TURBO ENGINE INTO A CAR WITH NO WHEELS.
> 
> I can dig into this as much as I want/like ON MY OWN TIME.
> But if MONEY are on the table…….

If money is on the table, perhaps you could contract someone to look at
your problems. Though unless you give a bit more information than in your
mails you'll be wasting money while someone figures out what you're
talking about.

> I think I’ll revert to 5.9 all of it.

Your call. Seems hell of a lot easier to take relayd/relayctl up to
-current for a test than reinstall to 5.9 though.

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

6.2-stable is NOT STABLE.
Backport, backport,backport.

6.2-stable is a beta release. 
This is what its IS.

5.9 vs. 6.2 - last one is a major downwards.
I know a lot of stuff done in tcp/ip stack and this is a good job (abt time to 
ack SMP), but
Keep those changes in beta, don’t tell ”we have rel and stable here. Eat it”.

//mxb

> 21 dec. 2017 kl. 23:19 skrev Maxim Bourmistrov :
> 
> 
> Fixed in HEAD?! - my ass. Whom puts HEAD into prod?! Not me any more, that's 
> for sure.
> IS LIKE DROPPING A TURBO ENGINE INTO CAR WITH NO WHEELS.
> 
> I can dig into this as much as I want/like ON MY OWN TIME.
> But if MONEY are on the table…….
> 
> I think I’ll revert to 5.9 all of it.
> 
> //mxb
> 
>> 21 dec. 2017 kl. 23:07 skrev Maxim Bourmistrov :
>> 
>> Solved?1
>> 
>> What abt OPTIONS in relay_http.c ?
>> Solved?
>> Maybe in HEAD.(?)
>> I have to hand-rolle this in src for 6.2 to have it working.
>> —> toread=0;
>> You know.
>> 
>> //mxb
>> 
>>> 21 dec. 2017 kl. 22:40 skrev Janne Johansson >> >:
>>> 
>>> 2017-12-21 21:58 GMT+01:00 Maxim Bourmistrov >> >:
>>> 
>>> Sorry, but I have to say
>>> Releases after 5.9 are NOT production stable.
>>> (Until all bugs are smashed within stack changes and SMP unlock).
>>> After 5.9 - cost money and effort.
>>> MONEY.
>>> 
>>> As long as they get quality reports like this, it would soon be solved.
>>>  
>> 
> 

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

Fixed in HEAD?! - my ass. Whom puts HEAD into prod?! Not me any more, that's 
for sure.
IS LIKE DROPPING A TURBO ENGINE INTO A CAR WITH NO WHEELS.

I can dig into this as much as I want/like ON MY OWN TIME.
But if MONEY are on the table…….

I think I’ll revert to 5.9 all of it.

//mxb

> 21 dec. 2017 kl. 23:07 skrev Maxim Bourmistrov :
> 
> Solved?1
> 
> What abt OPTIONS in relay_http.c ?
> Solved?
> Maybe in HEAD.(?)
> I have to hand-rolle this in src for 6.2 to have it working.
> —> toread=0;
> You know.
> 
> //mxb
> 
>> 21 dec. 2017 kl. 22:40 skrev Janne Johansson > >:
>> 
>> 2017-12-21 21:58 GMT+01:00 Maxim Bourmistrov > >:
>> 
>> Sorry, but I have to say
>> Releases after 5.9 are NOT production stable.
>> (Until all bugs are smashed within stack changes and SMP unlock).
>> After 5.9 - cost money and effort.
>> MONEY.
>> 
>> As long as they get quality reports like this, it would soon be solved.
>>  
> 

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

Solved?1

What abt OPTIONS in relay_http.c ?
Solved?
Maybe in HEAD.(?)
I have to hand-rolle this in src for 6.2 to have it working.
—> toread=0;
You know.

//mxb

> 21 dec. 2017 kl. 22:40 skrev Janne Johansson :
> 
> 2017-12-21 21:58 GMT+01:00 Maxim Bourmistrov  >:
> 
> Sorry, but I have to say
> Releases after 5.9 are NOT production stable.
> (Until all bugs are smashed within stack changes and SMP unlock).
> After 5.9 - cost money and effort.
> MONEY.
> 
> As long as they get quality reports like this, it would soon be solved.
>  

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

I had to bypass relayd to roll prod stable.
Down to apache. Taking care of http and https.
By redirect.
Now this setup (if I can call it) is stable.

.

P.S.
Looks like we have to move forward from here.

> 21 dec. 2017 kl. 21:58 skrev Maxim Bourmistrov :
> 
> 
> Sorry, but I have to say
> Releases after 5.9 are NOT production stable.
> (Until all bugs are smashed within stack changes and SMP unlock).
> After 5.9 - cost money and effort.
> MONEY.
> 
> //mxb
> 
>> 21 dec. 2017 kl. 20:29 skrev Maxim Bourmistrov :
>> 
>> Hey,
>> After upgrading from 6.0-stable to 6.2-stable (syspatch) existing setup 
>> started to hang.
>> As of burst of emails from me following is known:
>> 
>> Relayd is a main process to take CPU.
>> Also running ospfd and bgpd (for blocklist distrib)
>> 
>> With 6.0, relayd used to have two or more procs with high CPU usage (99,9% - 
>> 100%, per core).
>> Now, setup is just adjusted to conform new reality with TLS protocol, all 
>> just hangs.
>> No core (even it is enabled in sysctl), no panic - nothing.
>> Syslog sending to remote shows nothing.
>> 
>> I know, it is now enough for any debug and troubleshooting at all.
>> Just FYI.
>> 
>> Any ideas are welcome.
>> 
>> //mxb
> 

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

Sorry, but I have to say
Releases after 5.9 are NOT production stable.
(Until all bugs are smashed within stack changes and SMP unlock).
After 5.9 - cost money and effort.
MONEY.

//mxb

> 21 dec. 2017 kl. 20:29 skrev Maxim Bourmistrov :
> 
> Hey,
> After upgrading from 6.0-stable to 6.2-stable (syspatch) existing setup 
> started to hang.
> As of burst of emails from me following is known:
> 
> Relayd is a main process to take CPU.
> Also running ospfd and bgpd (for blocklist distrib)
> 
> With 6.0, relayd used to have two or more procs with high CPU usage (99,9% - 
> 100%, per core).
> Now, setup is just adjusted to conform new reality with TLS protocol, all 
> just hangs.
> No core (even it is enabled in sysctl), no panic - nothing.
> Syslog sending to remote shows nothing.
> 
> I know, it is now enough for any debug and troubleshooting at all.
> Just FYI.
> 
> Any ideas are welcome.
> 
> //mxb

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

The first node out of two started to respond as soon as second one got reset.
I don’t see any saturation on the network stack.
"Netstat -m” shows 0 for mem denied.
Sysctl -a|grep ifq
or
Sysctl -a|grep drop 
Shows 0 as well.

Whole thing looked like a tcp/ip stack hang.

//mxb

> 21 dec. 2017 kl. 20:29 skrev Maxim Bourmistrov :
> 
> Hey,
> After upgrading from 6.0-stable to 6.2-stable (syspatch) existing setup 
> started to hang.
> As of burst of emails from me following is known:
> 
> Relayd is a main process to take CPU.
> Also running ospfd and bgpd (for blocklist distrib)
> 
> With 6.0, relayd used to have two or more procs with high CPU usage (99,9% - 
> 100%, per core).
> Now, setup is just adjusted to conform new reality with TLS protocol, all 
> just hangs.
> No core (even it is enabled in sysctl), no panic - nothing.
> Syslog sending to remote shows nothing.
> 
> I know, it is now enough for any debug and troubleshooting at all.
> Just FYI.
> 
> Any ideas are welcome.
> 
> //mxb

OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

Hey,
After upgrading from 6.0-stable to 6.2-stable (syspatch) existing setup started 
to hang.
As of burst of emails from me following is known:

Relayd is a main process to take CPU.
Also running ospfd and bgpd (for blocklist distrib)

With 6.0, relayd used to have two or more procs with high CPU usage (99,9% - 
100%, per core).
Now, setup is just adjusted to conform new reality with TLS protocol, all just 
hangs.
No core (even it is enabled in sysctl), no panic - nothing.
Syslog sending to remote shows nothing.

I know, it is now enough for any debug and troubleshooting at all.
Just FYI.

Any ideas are welcome.

//mxb