Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

6.2-stable is NOT STABLE.
Backport, backport,backport.

6.2-stable is a beta release. 
This is what its IS.

5.9 vs. 6.2 - last one is a major downwards.
I know a lot of stuff done in tcp/ip stack and this is a good job (abt time to 
ack SMP), but
Keep those changes in beta, don’t tell ”we have rel and stable here. Eat it”.

//mxb

> 21 dec. 2017 kl. 23:19 skrev Maxim Bourmistrov <m...@alumni.chalmers.se>:
> 
> 
> Fixed in HEAD?! - my ass. Whom puts HEAD into prod?! Not me any more, that's 
> for sure.
> IS LIKE DROPPING A TURBO ENGINE INTO CAR WITH NO WHEELS.
> 
> I can dig into this as much as I want/like ON MY OWN TIME.
> But if MONEY are on the table…….
> 
> I think I’ll revert to 5.9 all of it.
> 
> //mxb
> 
>> 21 dec. 2017 kl. 23:07 skrev Maxim Bourmistrov <m...@alumni.chalmers.se>:
>> 
>> Solved?1
>> 
>> What abt OPTIONS in relay_http.c ?
>> Solved?
>> Maybe in HEAD.(?)
>> I have to hand-rolle this in src for 6.2 to have it working.
>> —> toread=0;
>> You know.
>> 
>> //mxb
>> 
>>> 21 dec. 2017 kl. 22:40 skrev Janne Johansson <icepic...@gmail.com 
>>> <mailto:icepic...@gmail.com>>:
>>> 
>>> 2017-12-21 21:58 GMT+01:00 Maxim Bourmistrov <m...@alumni.chalmers.se 
>>> <mailto:m...@alumni.chalmers.se>>:
>>> 
>>> Sorry, but I have to say
>>> Releases after 5.9 are NOT production stable.
>>> (Until all bugs are smashed within stack changes and SMP unlock).
>>> After 5.9 - cost money and effort.
>>> MONEY.
>>> 
>>> As long as they get quality reports like this, it would soon be solved.
>>>  
>> 
> 

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

Fixed in HEAD?! - my ass. Whom puts HEAD into prod?! Not me any more, that's 
for sure.
IS LIKE DROPPING A TURBO ENGINE INTO A CAR WITH NO WHEELS.

I can dig into this as much as I want/like ON MY OWN TIME.
But if MONEY are on the table…….

I think I’ll revert to 5.9 all of it.

//mxb

> 21 dec. 2017 kl. 23:07 skrev Maxim Bourmistrov <m...@alumni.chalmers.se>:
> 
> Solved?1
> 
> What abt OPTIONS in relay_http.c ?
> Solved?
> Maybe in HEAD.(?)
> I have to hand-rolle this in src for 6.2 to have it working.
> —> toread=0;
> You know.
> 
> //mxb
> 
>> 21 dec. 2017 kl. 22:40 skrev Janne Johansson <icepic...@gmail.com 
>> <mailto:icepic...@gmail.com>>:
>> 
>> 2017-12-21 21:58 GMT+01:00 Maxim Bourmistrov <m...@alumni.chalmers.se 
>> <mailto:m...@alumni.chalmers.se>>:
>> 
>> Sorry, but I have to say
>> Releases after 5.9 are NOT production stable.
>> (Until all bugs are smashed within stack changes and SMP unlock).
>> After 5.9 - cost money and effort.
>> MONEY.
>> 
>> As long as they get quality reports like this, it would soon be solved.
>>  
> 

Re: OpenBSD 6.2 (up2date with syspatch) - HANGING

[Maxim Bourmistrov]

Solved?1

What abt OPTIONS in relay_http.c ?
Solved?
Maybe in HEAD.(?)
I have to hand-rolle this in src for 6.2 to have it working.
—> toread=0;
You know.

//mxb

> 21 dec. 2017 kl. 22:40 skrev Janne Johansson <icepic...@gmail.com>:
> 
> 2017-12-21 21:58 GMT+01:00 Maxim Bourmistrov <m...@alumni.chalmers.se 
> <mailto:m...@alumni.chalmers.se>>:
> 
> Sorry, but I have to say
> Releases after 5.9 are NOT production stable.
> (Until all bugs are smashed within stack changes and SMP unlock).
> After 5.9 - cost money and effort.
> MONEY.
> 
> As long as they get quality reports like this, it would soon be solved.
>  

Re: relayd/ctl alternative control socket

[Maxim Bourmistrov]

But what about people not running relay on rdomain, but rather just want to run 
separate 
instances of relayd ?

> 28 nov. 2017 kl. 16:06 skrev Sebastian benoit 
> :
> 
> Hi,
> 
> your diff looks good, but i would rather do it the way bgpd/bgpctl do it:
> 
> there the default is  /var/run/bgpd.sock. where  is the 
> routing domain bgpctl is running in.  To administer bgpd(8) in a different 
> routing domain, run bgpctl in said routing domain.
> 
> i.e. it detects the rdomain at startup, bgpctl does the same.
> 
> Can you do that in relayd? It was commited there in sometime in summer.
> 
> /Benno
> 
> 
> On 11/28/17 11:54, Kapetanakis Giannis wrote:
>> Hi,
>> On June I've posted a patch about using alternative control socket for 
>> relayd and relayctl.
>> There was a comment from David Gwynne which was evaluated.
>> Is it OK to get this is in order to be able to control multiple relayd 
>> daemons on different rdomains?
>> thanks
>> Giannis
>> Index: config.c
>> ===
>> RCS file: /cvs/src/usr.sbin/relayd/config.c,v
>> retrieving revision 1.35
>> diff -u -p -r1.35 config.c
>> --- config.c 27 Nov 2017 23:21:16 -  1.35
>> +++ config.c 28 Nov 2017 10:43:37 -
>> @@ -44,6 +44,7 @@ config_init(struct relayd *env)
>>  env->sc_conf.interval.tv_usec = 0;
>>  env->sc_conf.prefork_relay = RELAY_NUMPROC;
>>  env->sc_conf.statinterval.tv_sec = RELAY_STATINTERVAL;
>> +env->sc_ps->ps_csock.cs_name = RELAYD_SOCKET;
>>  }
>>  ps->ps_what[PROC_PARENT] = CONFIG_ALL;
>> Index: parse.y
>> ===
>> RCS file: /cvs/src/usr.sbin/relayd/parse.y,v
>> retrieving revision 1.220
>> diff -u -p -r1.220 parse.y
>> --- parse.y  27 Nov 2017 23:21:16 -  1.220
>> +++ parse.y  28 Nov 2017 10:43:38 -
>> @@ -418,6 +418,9 @@ main : INTERVAL NUMBER   {
>>  AGENTX_SOCKET,
>>  sizeof(conf->sc_conf.snmp_path));
>>  }
>> +| SOCKET STRING {
>> +conf->sc_ps->ps_csock.cs_name = $2;
>> +}
>>  ;
>>trap  : /* nothing */ { $$ = 0; }
>> Index: relayd.c
>> ===
>> RCS file: /cvs/src/usr.sbin/relayd/relayd.c,v
>> retrieving revision 1.170
>> diff -u -p -r1.170 relayd.c
>> --- relayd.c 27 Nov 2017 21:06:26 -  1.170
>> +++ relayd.c 28 Nov 2017 10:43:38 -
>> @@ -199,9 +199,6 @@ main(int argc, char *argv[])
>>  if ((ps->ps_pw =  getpwnam(RELAYD_USER)) == NULL)
>>  errx(1, "unknown user %s", RELAYD_USER);
>>  -   /* Configure the control socket */
>> -ps->ps_csock.cs_name = RELAYD_SOCKET;
>> -
>>  log_init(debug, LOG_DAEMON);
>>  log_setverbose(verbose);
>>  Index: relayd.conf.5
>> ===
>> RCS file: /cvs/src/usr.sbin/relayd/relayd.conf.5,v
>> retrieving revision 1.180
>> diff -u -p -r1.180 relayd.conf.5
>> --- relayd.conf.527 Nov 2017 23:21:16 -  1.180
>> +++ relayd.conf.528 Nov 2017 10:43:38 -
>> @@ -163,6 +163,12 @@ will be used.
>>  See
>>  .Xr snmpd.conf 5
>>  for more information about SNMP configuration.
>> +.It Ic socket Qo Ar path Qc
>> +Create a control socket at
>> +.Ar path .
>> +By default
>> +.Pa /var/run/relayd.sock
>> +is created and no other sockets are created.
>>  .It Ic timeout Ar number
>>  Set the global timeout in milliseconds for checks.
>>  This can be overridden by the timeout value in the table definitions.
>> Index: relayctl.8
>> ===
>> RCS file: /cvs/src/usr.sbin/relayctl/relayctl.8,v
>> retrieving revision 1.32
>> diff -u -p -r1.32 relayctl.8
>> --- relayctl.8   28 Nov 2015 01:22:44 -  1.32
>> +++ relayctl.8   28 Nov 2017 10:43:22 -
>> @@ -23,6 +23,7 @@
>>  .Nd control the relay daemon
>>  .Sh SYNOPSIS
>>  .Nm
>> +.Op Fl s Ar socket
>>  .Ar command
>>  .Op Ar argument ...
>>  .Sh DESCRIPTION
>> @@ -31,6 +32,17 @@ The
>>  program controls the
>>  .Xr relayd 8
>>  daemon.
>> +.Pp
>> +The following options are available:
>> +.Bl -tag -width Ds
>> +.It Fl s Ar socket
>> +Use
>> +.Ar socket
>> +instead of the default
>> +.Pa /var/run/relayd.sock
>> +to communicate with
>> +.Xr relayd 8 .
>> +.El
>>  .Pp
>>  The following commands are available:
>>  .Bl -tag -width Ds
>> Index: relayctl.c
>> ===
>> RCS file: /cvs/src/usr.sbin/relayctl/relayctl.c,v
>> retrieving revision 1.57
>> diff -u -p -r1.57 relayctl.c
>> --- relayctl.c   3 Sep 2016 14:44:21 -   1.57
>> +++ relayctl.c   28 Nov 2017 10:43:22 -
>> @@ -88,7 +88,8 @@ usage(void)
>>  {
>>  extern char *__progname;
>>  -   fprintf(stderr, 

Re: relayd: 6.1-stable and relay_http.c rev 1.58

[Maxim Bourmistrov]

Below is a DEBUG dump from failing OPTIONS+GET+GET, e.g. header 
X-Forwarded-Port is not set.

accept_reserve: inflight incremented, now 1
relay_read_http: session 1: size 87, to read -2
relay_read_http: session 1: header 'OPTIONS: /options.php HTTP/1.1'
relay_read_http: session 1: header 'Host: test.jesper.office.se.domain.com'
relay_read_http: session 1: header 'Accept: */*'
relay_test: session 1: matched rule 0
relay_test:1767: next rule
relay_test: session 1, res 0
relay_test: session 1: matched rule 1
relay_test:1767: next rule
relay_test: session 1, res 0
relay_test: session 1: matched rule 2
relay_test:1767: next rule
relay_test: session 1, res 0
relay_test: session 1: matched rule 3
relay_test:1767: next rule
relay_test: session 1, res 0
relay_test:1747: next rule
relay_test: session 1, res 0
relay_test: session 1: action 1
relay_writeheader_kv: Accept: */*
relay_writeheader_kv: Host: test.jesper.office.se.domain.com
relay_writeheader_kv: Keep-Alive: 600
relay_writeheader_kv: X-Forwarded-By: 172.16.1.101:80
relay_writeheader_kv: X-Forwarded-For: 172.17.2.21
relay_writeheader_kv: X-Forwarded-Port: 80
relay_from_table: session 1: table jesper:80 host 172.16.1.30, p 
0x1f1046c6927e3ff3, idx 0, cnt 0, max 1
relay_connect: inflight decremented, now 0
relay_connected: session 1: successful
relay_splice: session 1: splice dir 2, nothing to read -2
relay_splice: session 1: splice dir 1, maximum -1, successful
relay_read_http: session 1: size 182, to read -2
relay_read_http: session 1: header 'HTTP/1.1: 200 OK'
http_version HTTP/1.1 http_rescode 200 http_resmesg OK
relay_read_http: session 1: header 'Date: Mon, 27 Nov 2017 19:01:55 GMT'
relay_read_http: session 1: header 'Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 
OpenSSL/1.0.1e-fips'
relay_read_http: session 1: header 'Content-Length: 2'
relay_read_http: session 1: header 'Content-Type: text/html; charset=UTF-8'
relay_test:1729: skip 1 rules
relay_test: session 1: action 1
version: HTTP/1.1 rescode: 200 resmsg: OK
relay_writeheader_kv: Content-Length: 2
relay_writeheader_kv: Content-Type: text/html; charset=UTF-8
relay_writeheader_kv: Date: Mon, 27 Nov 2017 19:01:55 GMT
relay_writeheader_kv: Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 
OpenSSL/1.0.1e-fips
relay_splice: session 1: splice dir 2, dirty buffer
relay_read_httpcontent: session 1: size 2, to read 2
relay_read_httpcontent: done, size 2, to read 0
relay_read_http: session 1: size 0, to read -2
relay_splice: session 1: splice dir 2, nothing to read -2
relay_read_http: session 1: size 195, to read -2
relay_read_http: session 1: header 'HTTP/1.1: 403 Forbidden'
http_version HTTP/1.1 http_rescode 403 http_resmesg Forbidden
relay_read_http: session 1: header 'Date: Mon, 27 Nov 2017 19:01:55 GMT'
relay_read_http: session 1: header 'Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 
OpenSSL/1.0.1e-fips'
relay_read_http: session 1: header 'Content-Length: 8'
relay_read_http: session 1: header 'Content-Type: text/html; charset=UTF-8'
relay_test:1729: skip 1 rules
relay_test: session 1: action 1
version: HTTP/1.1 rescode: 403 resmsg: Forbidden
relay_writeheader_kv: Content-Length: 8
relay_writeheader_kv: Content-Type: text/html; charset=UTF-8
relay_writeheader_kv: Date: Mon, 27 Nov 2017 19:01:55 GMT
relay_writeheader_kv: Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 
OpenSSL/1.0.1e-fips
relay_splice: session 1: splice dir 2, dirty buffer
relay_read_httpcontent: session 1: size 8, to read 8
relay_read_httpcontent: done, size 8, to read 0
relay_read_http: session 1: size 0, to read -2
relay_splice: session 1: splice dir 2, nothing to read -2
relay_read_http: session 1: size 195, to read -2
relay_read_http: session 1: header 'HTTP/1.1: 403 Forbidden'
http_version HTTP/1.1 http_rescode 403 http_resmesg Forbidden
relay_read_http: session 1: header 'Date: Mon, 27 Nov 2017 19:01:55 GMT'
relay_read_http: session 1: header 'Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 
OpenSSL/1.0.1e-fips'
relay_read_http: session 1: header 'Content-Length: 8'
relay_read_http: session 1: header 'Content-Type: text/html; charset=UTF-8'
relay_test:1729: skip 1 rules
relay_test: session 1: action 1
version: HTTP/1.1 rescode: 403 resmsg: Forbidden
relay_writeheader_kv: Content-Length: 8
relay_writeheader_kv: Content-Type: text/html; charset=UTF-8
relay_writeheader_kv: Date: Mon, 27 Nov 2017 19:01:55 GMT
relay_writeheader_kv: Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 
OpenSSL/1.0.1e-fips
relay_splice: session 1: splice dir 2, dirty buffer
relay_read_httpcontent: session 1: size 8, to read 8
relay_read_httpcontent: done, size 8, to read 0
relay_read_http: session 1: size 0, to read -2
relay_splice: session 1: splice dir 2, nothing to read -2
relay web_test, session 1 (1 active), 0, 172.17.2.21 -> 172.16.1.30:80, done, 
OPTIONS

//mxb

> 27 nov. 2017 kl. 20:20 skrev Maxim Bourmistrov <m...@alumni.chalmers.se>:
> 
> Here is setup which reproduces this problem. Also exists in 6.2.
> 
> Server:
> Apache with mod_ph

Re: relayd: 6.1-stable and relay_http.c rev 1.58

[Maxim Bourmistrov]

Here is setup which reproduces this problem. Also exists in 6.2.

Server:
Apache with mod_php serving following content:

———cut options.php——
 { 1.2.3.4 }
relay web_test {
listen on 5.6.7.8 port 80
protocol http_relay
forward to  port 80 mode loadbalance check tcp
}

Client:
Runs php from CLI
file to run:

http://5.6.7.8/options.php');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "OPTIONS");
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_exec($ch);
echo PHP_EOL;
curl_setopt($ch, CURLOPT_URL, 'http://5.6.7.8/options.php');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_exec($ch);
echo PHP_EOL;
curl_setopt($ch, CURLOPT_URL, 'http://5.6.7.8');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_exec($ch);
echo PHP_EOL; 

With rev. Before 1.58 of relay_http.c following can be observed:
relay web_test, session 1 (1 active), 0, 172.17.2.21 -> 172.16.1.30:80, done, 
OPTIONS GET GET

With the rev. current for 6.2:
relay web_test, session 1 (1 active), 0, 172.17.2.21 -> 172.16.1.30:80, done, 
OPTIONS

//mxb
 
> 22 okt. 2017 kl. 21:02 skrev Maxim Bourmistrov <m...@alumni.chalmers.se>:
> 
> 
>> 22 okt. 2017 kl. 20:16 skrev Maxim Bourmistrov <m...@alumni.chalmers.se>:
>> 
>> Hey,
>> with rev 1.58 OPTIONS in relay_http.c got broken
>> or at least logic inside relay_read_http().
>> Quick fix it to cre->toread=0 and break, but this is probably not what 
>> should be there.
>> 
>> In my test case, from the client side I do an OPTIONS request, followed by a 
>> couple of GET.
>> GET in the middle never gets catched and thus breaks intended usage.
>> 
>> I’m doing a simple printf() debugging here to catch.
>> 
>> cre->toread = strtonum(value, 0, LLONG_MAX, );
>> printf("-- to read %lld\n", cre->toread);
>> 
>> 
>> case HTTP_METHOD_GET:
>>   printf("GOT GET to read: %lld\n", cre->toread);
>> 
>> case HTTP_METHOD_OPTIONS:
>>   printf("GOT OPT to read: %lld\n", cre->toread);
>> 
>> 
>> The output with those in place from ’relayd -d’:
>> 
>> host 10.6.128.38, check http code (1ms,http code ok), state up -> up, 
>> availability 100.00%
>> GOT OPT to read: -2
>> -- to read 0
>> -- to read 214
>> relay test_api_tls, session 1 (1 active), 0, 176.10.170.140 -> 
>> 10.6.128.20:80, done, OPTIONS
>> GOT GET to read: -2
>> -- to read 214
>> relay test_api_tls, session 2 (1 active), 0, 176.10.170.140 -> 
>> 10.6.128.36:80, done, GET
>> ^Chce exiting, pid 96033
>> 
>> //mxb
>> 
> 
> With cre->toread=0  I catch it all
> 
> relay test_api_tls, session 1 (1 active), 0, 176.10.170.140 -> 
> 10.6.128.20:80, done, OPTIONS GET
> relay test_api_tls, session 2 (1 active), 0, 176.10.170.140 -> 
> 10.6.128.36:80, done, GET
> 
> //mxb
> 

Re: relayd: 6.1-stable and relay_http.c rev 1.58

[Maxim Bourmistrov]

> 22 okt. 2017 kl. 20:16 skrev Maxim Bourmistrov <m...@alumni.chalmers.se>:
> 
> Hey,
> with rev 1.58 OPTIONS in relay_http.c got broken
> or at least logic inside relay_read_http().
> Quick fix it to cre->toread=0 and break, but this is probably not what should 
> be there.
> 
> In my test case, from the client side I do an OPTIONS request, followed by a 
> couple of GET.
> GET in the middle never gets catched and thus breaks intended usage.
> 
> I’m doing a simple printf() debugging here to catch.
> 
> cre->toread = strtonum(value, 0, LLONG_MAX, );
> printf("-- to read %lld\n", cre->toread);
> 
> 
> case HTTP_METHOD_GET:
>printf("GOT GET to read: %lld\n", cre->toread);
> 
> case HTTP_METHOD_OPTIONS:
>printf("GOT OPT to read: %lld\n", cre->toread);
> 
> 
> The output with those in place from ’relayd -d’:
> 
> host 10.6.128.38, check http code (1ms,http code ok), state up -> up, 
> availability 100.00%
> GOT OPT to read: -2
> -- to read 0
> -- to read 214
> relay test_api_tls, session 1 (1 active), 0, 176.10.170.140 -> 
> 10.6.128.20:80, done, OPTIONS
> GOT GET to read: -2
> -- to read 214
> relay test_api_tls, session 2 (1 active), 0, 176.10.170.140 -> 
> 10.6.128.36:80, done, GET
> ^Chce exiting, pid 96033
> 
> //mxb
> 

With cre->toread=0  I catch it all

relay test_api_tls, session 1 (1 active), 0, 176.10.170.140 -> 10.6.128.20:80, 
done, OPTIONS GET
relay test_api_tls, session 2 (1 active), 0, 176.10.170.140 -> 10.6.128.36:80, 
done, GET

//mxb

relayd: 6.1-stable and relay_http.c rev 1.58

[Maxim Bourmistrov]

Hey,
with rev 1.58 OPTIONS in relay_http.c got broken
or at least logic inside relay_read_http().
Quick fix it to cre->toread=0 and break, but this is probably not what should 
be there.

In my test case, from the client side I do an OPTIONS request, followed by a 
couple of GET.
GET in the middle never gets catched and thus breaks intended usage.

I’m doing a simple printf() debugging here to catch.

cre->toread = strtonum(value, 0, LLONG_MAX, );
printf("-- to read %lld\n", cre->toread);


case HTTP_METHOD_GET:
printf("GOT GET to read: %lld\n", cre->toread);

case HTTP_METHOD_OPTIONS:
printf("GOT OPT to read: %lld\n", cre->toread);


The output with those in place from ’relayd -d’:

host 10.6.128.38, check http code (1ms,http code ok), state up -> up, 
availability 100.00%
GOT OPT to read: -2
-- to read 0
-- to read 214
relay test_api_tls, session 1 (1 active), 0, 176.10.170.140 -> 10.6.128.20:80, 
done, OPTIONS
GOT GET to read: -2
-- to read 214
relay test_api_tls, session 2 (1 active), 0, 176.10.170.140 -> 10.6.128.36:80, 
done, GET
^Chce exiting, pid 96033

//mxb

relayd - multiple instances

[Maxim Bourmistrov]

Hello,
Are there plans for relayd to run multiple instances?
Eg. dropping socket to a configurable location.

Regards

Re: relayd: incomplete response from a TLS-accelerated apache

[Maxim Bourmistrov]

Compiling relayd with -DDEBUG=3 and watching the output gave me nothing.
No errors what so ever about out of buffers or something else.

However, removing 'socket buffer 65536’ solved my problem.

Br

> 8 maj 2017 kl. 13:27 skrev Maxim Bourmistrov <m...@alumni.chalmers.se>:
> 
> Hey,
> I investigate a problem were TLS-asselerated machine response is incomplete.
> I was able to reproduce this on OpenBSD 5.9, 6.0 and 6.1. Test on 5.8 is 
> about to be.
> 
> Following env I have:
> 
> relay1: relayd machine
> web1: apache 2.2.31 serving the request
> client1: requester
> 
> relay1 is configured following way (relevant lines):
> 
> http protocol http_relay {
>tcp { nodelay, sack, socket buffer 65536, backlog 1024 }
>match header append "X-Forwarded-For" value "$REMOTE_ADDR"
>match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
>match header set "Keep-Alive" value "$TIMEOUT"
>match request header remove "Proxy"
> }
> 
> http protocol tls_accel {
>tcp { nodelay, sack, socket buffer 65536, backlog 1024 }
>match header append "X-Forwarded-For" value "$REMOTE_ADDR"
>match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
>match header set "X-Forwarded-Proto" value "https"
>match header set "X-Forwarded-Port" value "443"
>match header set "Keep-Alive" value "$TIMEOUT"
>match request header remove "Proxy"
> 
>tls { tlsv1, \
>ciphers "AES:!AES256:!aNULL" \
>  }
> }
> 
> table  { 172.16.1.111 }
> 
> relay int_test_tls {
>listen on 172.16.1.99 port 443 tls
>protocol tls_accel
>forward to  port 80 mode roundrobin check http "/" code 200
> }
> 
> relay int_test_http {
>listen on 172.16.1.99 port 80
>protocol http_relay
>forward to  port 80 mode roundrobin check http "/" code 200
> }
> 
> web1 is a std Apache 2.2.31 with enabled deflate for the following
> 
> AddOutputFilterByType DEFLATE application/json
> AddOutputFilterByType DEFLATE text/html
> AddOutputFilterByType DEFLATE text/plain
> AddOutputFilterByType DEFLATE text/xml
> AddOutputFilterByType DEFLATE text/css
> AddOutputFilterByType DEFLATE application/x-javascript
> AddOutputFilterByType DEFLATE application/javascript
> 
> and serving a JS file.
> 
> client1 is running PHP code from CLI to reproduce this problem.
> 
> 
> Following is observed:
> 
> 1. Client1 requests web1 directly on port 80 and gets full response
> 
> shell$ php client3.php
> Expected length: 547204
> Received length: 547204
> 
> [Response Headers]
> HTTP/1.1 200 OK
> Date: Mon, 08 May 2017 11:08:27 GMT
> Server: Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1e-fips
> Last-Modified: Mon, 08 May 2017 07:22:43 GMT
> ETag: "60319-85984-54efe1ae42be3"
> Accept-Ranges: bytes
> Content-Length: 547204
> Vary: Accept-Encoding
> Connection: close
> Content-Type: application/javascript 
> 
> 2. Client1 requests web1 directly on port 80 WITH GZIP enabled and gets full 
> response back
> I see gzipped stream on the screen and then it gets decoded to a complete 
> file. File I get is not cut.
> 
> Expected length: Content-Length not recieved
> Received length: 165454
> 
> [Response Headers]
> HTTP/1.1 200 OK
> Date: Mon, 08 May 2017 11:10:18 GMT
> Server: Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1e-fips
> Last-Modified: Mon, 08 May 2017 07:22:43 GMT
> ETag: "60319-85984-54efe1ae42be3"
> Accept-Ranges: bytes
> Vary: Accept-Encoding
> Content-Encoding: gzip
> Connection: close
> Content-Type: application/javascript
> 
> 3. and 4. Clien1 requests relay1 on port 80 (with and without GZIP) and gets 
> complete response
> 
> 5. Client1 requests relay1 on port 443 without GZIP - response is incomplete
> 
> Expected length: 547204
> Received length: 396424
> 
> [Response Headers]
> HTTP/1.1 200 OK
> Accept-Ranges: bytes
> Connection: close
> Content-Length: 547204
> Content-Type: application/javascript
> Date: Mon, 08 May 2017 11:14:59 GMT
> ETag: "60319-85984-54efe1ae42be3"
> Last-Modified: Mon, 08 May 2017 07:22:43 GMT
> Server: Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1e-fips
> Vary: Accept-Encoding
> 
> 6. Client1 requests relay1 on port 443 with GZIP - response is complete.
> 
> 
> So non-gzipped response from behind the relay1 is incomplete while doing TLS 
> termination.
> Files server.js and client.php can be provided upon request.
> 
> Any ideas?
> 
> Br
> 
> 
> 

relayd: incomplete response from a TLS-accelerated apache

[Maxim Bourmistrov]

Hey,
I investigate a problem were TLS-asselerated machine response is incomplete.
I was able to reproduce this on OpenBSD 5.9, 6.0 and 6.1. Test on 5.8 is about 
to be.

Following env I have:

relay1: relayd machine
web1: apache 2.2.31 serving the request
client1: requester

relay1 is configured following way (relevant lines):

http protocol http_relay {
tcp { nodelay, sack, socket buffer 65536, backlog 1024 }
match header append "X-Forwarded-For" value "$REMOTE_ADDR"
match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
match header set "Keep-Alive" value "$TIMEOUT"
match request header remove "Proxy"
}

http protocol tls_accel {
tcp { nodelay, sack, socket buffer 65536, backlog 1024 }
match header append "X-Forwarded-For" value "$REMOTE_ADDR"
match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
match header set "X-Forwarded-Proto" value "https"
match header set "X-Forwarded-Port" value "443"
match header set "Keep-Alive" value "$TIMEOUT"
match request header remove "Proxy"

tls { tlsv1, \
ciphers "AES:!AES256:!aNULL" \
  }
}

table  { 172.16.1.111 }

relay int_test_tls {
listen on 172.16.1.99 port 443 tls
protocol tls_accel
forward to  port 80 mode roundrobin check http "/" code 200
}

relay int_test_http {
listen on 172.16.1.99 port 80
protocol http_relay
forward to  port 80 mode roundrobin check http "/" code 200
}

web1 is a std Apache 2.2.31 with enabled deflate for the following

AddOutputFilterByType DEFLATE application/json
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/javascript

and serving a JS file.

client1 is running PHP code from CLI to reproduce this problem.


Following is observed:

1. Client1 requests web1 directly on port 80 and gets full response

shell$ php client3.php
Expected length: 547204
Received length: 547204

[Response Headers]
HTTP/1.1 200 OK
Date: Mon, 08 May 2017 11:08:27 GMT
Server: Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1e-fips
Last-Modified: Mon, 08 May 2017 07:22:43 GMT
ETag: "60319-85984-54efe1ae42be3"
Accept-Ranges: bytes
Content-Length: 547204
Vary: Accept-Encoding
Connection: close
Content-Type: application/javascript 

2. Client1 requests web1 directly on port 80 WITH GZIP enabled and gets full 
response back
I see gzipped stream on the screen and then it gets decoded to a complete file. 
File I get is not cut.

Expected length: Content-Length not recieved
Received length: 165454

[Response Headers]
HTTP/1.1 200 OK
Date: Mon, 08 May 2017 11:10:18 GMT
Server: Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1e-fips
Last-Modified: Mon, 08 May 2017 07:22:43 GMT
ETag: "60319-85984-54efe1ae42be3"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: close
Content-Type: application/javascript

3. and 4. Clien1 requests relay1 on port 80 (with and without GZIP) and gets 
complete response

5. Client1 requests relay1 on port 443 without GZIP - response is incomplete

Expected length: 547204
Received length: 396424

[Response Headers]
HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: close
Content-Length: 547204
Content-Type: application/javascript
Date: Mon, 08 May 2017 11:14:59 GMT
ETag: "60319-85984-54efe1ae42be3"
Last-Modified: Mon, 08 May 2017 07:22:43 GMT
Server: Apache/2.2.31 (Unix) mod_ssl/2.2.31 OpenSSL/1.0.1e-fips
Vary: Accept-Encoding

6. Client1 requests relay1 on port 443 with GZIP - response is complete.


So non-gzipped response from behind the relay1 is incomplete while doing TLS 
termination.
Files server.js and client.php can be provided upon request.

Any ideas?

Br

Re: OpenBSD 6.1: relayd does not start more than 3 processes

[Maxim Bourmistrov]

> 5 maj 2017 kl. 15:55 skrev Maxim Bourmistrov <m...@alumni.chalmers.se>:
> 
> 
>> 5 maj 2017 kl. 14:41 skrev Hiltjo Posthuma <hil...@codemadness.org>:
>> 
>> On Fri, May 05, 2017 at 12:30:56PM +0200, Maxim Bourmistrov wrote:
>>> 
>>> Hey,
>>> on OpenBSD 6.0-stable I have following configuration for relayd:
>>> 
>>> snip———
>>> interval 10
>>> timeout 1200
>>> prefork 15
>>> log all
>>> ——
>>> 
>>> Respective login.conf to spawn more relayd procs:
>>> 
>>> relayd:\
>>>   :maxproc-max=31:\
>>>   :maxproc-cur=15:\
>>>   :openfiles=65536:\
>>>   :tc=daemon:
>>> 
>>> 
>>> With config options above moved to a 6.1 creates following:
>>> 
>>> relayd starts but brings up no more that 3 relay-processes.
>>> Also after start up it refuses to do any checks configured (in my simple 
>>> test I used check tcp) 
>>> 
>>> [mxb-test]-[12:21:41]# relayctl sh su
>>> Id  TypeNameAvlblty Status
>>> 1   relay   rabbitmqactive
>>> 1   table   rabbitmqpool:5672   empty
>>> 1   host10.5.96.8   unknown
>>> 2   table   rabbitmqfallback:5672   empty
>>> 2   host10.5.96.9   unknown
>>> 
>>> 
>>> Changing ’prefork’ from 15 to 3 makes it work.
>>> 
>>> Is this a bug?
>>> 
>>> Br
>>> 
>>> 
>>> 
>> 
>> Hey,
>> 
>> This is a random guess since you haven't posted the whole config, but I think
>> it has bitten me too sometime:
>> 
>> Do you have the global options such as prefork defined before your
>> relays and routes or not?
>> 
>> The order of the global options matter. If the global options are set after
>> the table they are not initialized on the tables and can actually crash 
>> relayd.
>> This is because the health checking uses a different prefork value and checks
>> the "wrong" amount.
>> 
>> I'm not sure, but I think it is not a bug: it is documented in 
>> relayd.conf(5).
>> 
>> Thinking about it: would it be acceptable if `relayd -n` shows a warning if
>> global options are defined in the wrong order? I can write the patch for it
>> if it makes sense.
>> 
>> I hope this helps you in some way,
>> 
>> -- 
>> Kind regards,
>> Hiltjo
> 
> The whole config is like this:
> 
> include "/etc/pj/nz/akl1/shared/pf/int/networks"
> include "/etc/pj/nz/akl1/shared/pf/int/common_tables"
> 
> interval 10
> timeout 1000
> prefork 3 #15
> log all
> 
> #include "/etc/pj/shared/relayd/protocols"
> 
> tcp protocol tcp_proto {
>tcp { nodelay, sack, socket buffer 65536, backlog 128 }
> }
> 
> relay rabbitmq {
>listen on 10.5.128.16 port 5674
> #protocol tcp_proto
> #   session timeout 10800
>forward to  port 5672 mode roundrobin check tcp
>forward to  port 5672 mode roundrobin check tcp
> }
> 
> Tables are shared with PF so I don’t have to re-define those.
> ”common_tables” in config contains:
> 
> table  { $RMQ1_VLAN302 }
> table  { $RMQ2_VLAN302 }
> 
> I seems not be able to find a place in manage were it states that global 
> options need to go before
> table definitions. 
> 
> Note, config layout exactly the same which runs already on 6.0-stable.
> 
> My original question is why I can’t fork more than 3 procs any more
> and why relayd starts then prefork > 3 and does not do a health check.
> 
> Br

Hm, I tried this out - re-ordering the layout of the config.
You are, indeed, correct here.

Strange that this runs on 6.0.

Case closed.
Sorry for the noise.

Br

Re: OpenBSD 6.1: relayd does not start more than 3 processes

[Maxim Bourmistrov]

> 5 maj 2017 kl. 14:41 skrev Hiltjo Posthuma <hil...@codemadness.org>:
> 
> On Fri, May 05, 2017 at 12:30:56PM +0200, Maxim Bourmistrov wrote:
>> 
>> Hey,
>> on OpenBSD 6.0-stable I have following configuration for relayd:
>> 
>> snip———
>> interval 10
>> timeout 1200
>> prefork 15
>> log all
>> ——
>> 
>> Respective login.conf to spawn more relayd procs:
>> 
>> relayd:\
>>:maxproc-max=31:\
>>:maxproc-cur=15:\
>>:openfiles=65536:\
>>:tc=daemon:
>> 
>> 
>> With config options above moved to a 6.1 creates following:
>> 
>> relayd starts but brings up no more that 3 relay-processes.
>> Also after start up it refuses to do any checks configured (in my simple 
>> test I used check tcp) 
>> 
>> [mxb-test]-[12:21:41]# relayctl sh su
>> Id  TypeNameAvlblty Status
>> 1   relay   rabbitmqactive
>> 1   table   rabbitmqpool:5672   empty
>> 1   host10.5.96.8   unknown
>> 2   table   rabbitmqfallback:5672   empty
>> 2   host10.5.96.9   unknown
>> 
>> 
>> Changing ’prefork’ from 15 to 3 makes it work.
>> 
>> Is this a bug?
>> 
>> Br
>> 
>> 
>> 
> 
> Hey,
> 
> This is a random guess since you haven't posted the whole config, but I think
> it has bitten me too sometime:
> 
> Do you have the global options such as prefork defined before your
> relays and routes or not?
> 
> The order of the global options matter. If the global options are set after
> the table they are not initialized on the tables and can actually crash 
> relayd.
> This is because the health checking uses a different prefork value and checks
> the "wrong" amount.
> 
> I'm not sure, but I think it is not a bug: it is documented in relayd.conf(5).
> 
> Thinking about it: would it be acceptable if `relayd -n` shows a warning if
> global options are defined in the wrong order? I can write the patch for it
> if it makes sense.
> 
> I hope this helps you in some way,
> 
> -- 
> Kind regards,
> Hiltjo

The whole config is like this:

include "/etc/pj/nz/akl1/shared/pf/int/networks"
include "/etc/pj/nz/akl1/shared/pf/int/common_tables"

interval 10
timeout 1000
prefork 3 #15
log all

#include "/etc/pj/shared/relayd/protocols"

tcp protocol tcp_proto {
tcp { nodelay, sack, socket buffer 65536, backlog 128 }
}

relay rabbitmq {
listen on 10.5.128.16 port 5674
#protocol tcp_proto
#   session timeout 10800
forward to  port 5672 mode roundrobin check tcp
forward to  port 5672 mode roundrobin check tcp
}

Tables are shared with PF so I don’t have to re-define those.
”common_tables” in config contains:

table  { $RMQ1_VLAN302 }
table  { $RMQ2_VLAN302 }

I seems not be able to find a place in manage were it states that global 
options need to go before
table definitions. 

Note, config layout exactly the same which runs already on 6.0-stable.

My original question is why I can’t fork more than 3 procs any more
and why relayd starts then prefork > 3 and does not do a health check.

Br
 

OpenBSD 6.1: relayd does not start more than 3 processes

[Maxim Bourmistrov]

Hey,
on OpenBSD 6.0-stable I have following configuration for relayd:

snip———
interval 10
timeout 1200
prefork 15
log all
——
 
Respective login.conf to spawn more relayd procs:

relayd:\
:maxproc-max=31:\
:maxproc-cur=15:\
:openfiles=65536:\
:tc=daemon:


With config options above moved to a 6.1 creates following:

relayd starts but brings up no more that 3 relay-processes.
Also after start up it refuses to do any checks configured (in my simple test I 
used check tcp) 

[mxb-test]-[12:21:41]# relayctl sh su
Id  TypeNameAvlblty Status
1   relay   rabbitmqactive
1   table   rabbitmqpool:5672   empty
1   host10.5.96.8   unknown
2   table   rabbitmqfallback:5672   empty
2   host10.5.96.9   unknown


Changing ’prefork’ from 15 to 3 makes it work.

Is this a bug?

Br

OSPFd stucks in EXCHG/EXSTA

[Maxim Bourmistrov]

Hey,

ospfd on 6.0-stable stucks in EXCHG/EXSTA while neighboring with Dell N3048 
switch.
According to some documentation around, this is due to MTU mismatch.

This is not in my case.

N3048:
system jumbo mtu 1512

obsd:
trunk1: flags=8943 mtu 1500
lladdr 00:25:90:78:62:b6
description: HW_INTERNAL
index 12 priority 0 llprio 3
trunk: trunkproto lacp
trunk id: [(8000,00:25:90:78:62:b6,4064,,),
 (0001,f8:b1:56:61:a1:e4,02AE,,)]
trunkport bnx1 active,collecting,distributing
trunkport em1 active,collecting,distributing
groups: trunk
media: Ethernet autoselect
status: active
inet 10.4.255.27 netmask 0xffe0 broadcast 10.4.255.31

ping with diff size of pkts and tcpdump reveals that there is no MTU mismatch.

Restart of ospfd does not helps, only REBOOT.

I decided to dig into this and found that changing MTU size on trunk1 can 
reproduce this 100%.
Actually value does not changes, but problem with ospfd can be triggered this 
way:

# ifconfig trunk1 mtu 1500
# rcctl restart ospfd

and now ospfd will be stuck in EXCHG/EXSTA. Reboot helps always.

Then I tried to put mtu for each face involved in trunk1. Result is then same - 
triggered with ’ifconfig trunk1 mtu 1500’.

# cat /etc/hostname.bnx1
up mtu 1500

# cat /etc/hostname.em1
up mtu 1500

Any ideas?

Br
mxb

Re: traceroute TOS

[Maxim Bourmistrov]

Thanks, useful(at least for me).

On Apr 5, 2011, at 5:57 PM, Stuart Henderson wrote:

 On 2011/04/05 16:51, Stuart Henderson wrote:
 if -t is used, display a notice when the TOS changes en-route.
 ok?

 oh, it's better with a (contrived) example:

 $ traceroute -nt 7 naiad
 traceroute to naiad.spacehopper.org (195.95.187.35), 64 hops max, 40 byte
packets
 1  85.158.44.145  0.441 ms (TOS=33!)  0.328 ms  0.396 ms
 2  193.178.223.245  14.637 ms  15.213 ms  15.466 ms
 3  194.39.143.145  16.600 ms  15.468 ms  16.206 ms
 4  194.39.143.205  16.483 ms  16.210 ms  15.843 ms
 5  193.203.5.182  17.475 ms  16.708 ms  16.986 ms
 6  195.95.187.248  17.211 ms (TOS=4!)  16.460 ms  16.468 ms
 7  195.95.187.35  27.99 ms  16.207 ms  17.467 ms



 Index: traceroute.8
 ===
 RCS file: /cvs/src/usr.sbin/traceroute/traceroute.8,v
 retrieving revision 1.44
 diff -u -p -r1.44 traceroute.8
 --- traceroute.8 8 Jul 2010 20:23:03 -   1.44
 +++ traceroute.8 5 Apr 2011 15:50:38 -
 @@ -177,6 +177,8 @@ in probe packets to the following value
 The value must be a decimal integer in the range 0 to 255.
 This option can be used to
 see if different types-of-service result in different paths.
 +If this option is used, changes to the type-of-service in the
 +returned packets are displayed.
 (If you are not running a
 .Bx 4.3 tahoe
 or later system, this may be academic since the normal network
 @@ -384,6 +386,8 @@ ever occur and the associated gateway is
 (destination network or host unreachable for TOS),
 .Sy !code
 (other ICMP unreachable code).
 +.Sy TOS=xxx
 +(TOS bit in returned packet differs from last hop).
 If almost all the probes result in some kind of unreachable,
 .Nm
 will give up and exit.
 Index: traceroute.c
 ===
 RCS file: /cvs/src/usr.sbin/traceroute/traceroute.c,v
 retrieving revision 1.74
 diff -u -p -r1.74 traceroute.c
 --- traceroute.c 22 Mar 2011 10:16:23 -  1.74
 +++ traceroute.c 5 Apr 2011 15:50:38 -
 @@ -293,11 +293,13 @@ main(int argc, char *argv[])
  int mib[4] = { CTL_NET, PF_INET, IPPROTO_IP, IPCTL_DEFTTL };
  int ttl_flag = 0, incflag = 1, protoset = 0, sump = 0;
  int ch, i, lsrr = 0, on = 1, probe, seq = 0, tos = 0;
 +int last_tos, tos_returned;
  size_t size = sizeof(max_ttl);
  struct sockaddr_in from, to;
  struct hostent *hp;
  u_int32_t tmprnd;
 -struct ip *ip;
 +struct ip *ip, *inner_ip;
 +struct icmp *icp;
  u_int8_t ttl;
  char *ep;
  const char *errstr;
 @@ -427,7 +429,7 @@ main(int argc, char *argv[])
  l = strtol(optarg, ep, 10);
  if (errno || !*optarg || *ep || l  0 || l  255)
  errx(1, tos must be 0 to 255.);
 -tos = (int)l;
 +last_tos = tos = (int)l;
  break;
  case 'v':
  verbose++;
 @@ -636,9 +638,21 @@ main(int argc, char *argv[])
  ++got_there;
  break;
  }
 +
 +icp = (struct icmp *) (((u_char 
 *)ip)+(ip-ip_hl2));
 +inner_ip = (struct ip *) (((u_char *)icp)+8);
 +
 +tos_returned = inner_ip-ip_tos;
 +
 +if (tos_returned != last_tos)
 +printf ( (TOS=%d!), tos_returned);
 +
 +last_tos = tos_returned;
 +
  /* time exceeded in transit */
  if (i == -1)
  break;
 +
  code = i - 1;
  switch (code) {
  case ICMP_UNREACH_PORT:

Re: Allegations regarding OpenBSD IPSEC

[Maxim Bourmistrov]

Theo,
this thread is DEAD. Drop it.

No one believes in backdoors planted into OpenBSD.

I se commits - you dig all over the place.
If backdoor existed, then it is gone cause of this digging.

Without proof its just a plain BS.

P.S.
I lost my interest for a while ago now.


On Dec 17, 2010, at 7:23 PM, Theo de Raadt wrote:

 On Fri, Dec 17, 2010 at 7:59 AM, Theo de Raadt dera...@cvs.openbsd.org
wr=
 ote:

 [skipped]

 I have to say that Perry here is credited with one thing he actually di=
 d not
 do -- publish this to the world. There has been talk of alterior motive=
 s here,
 but for any of these motives, Perry had to know or pretty damn well gue=
 ssed
 that =C2=A0the second thing Theo (hi, Theo) would do to his email was t=
 o publish it.
 Would you plan anything based on a predicted behavior of a person you
 haven't communicated with in 10 years?

 This is not to point finger at Theo for creating all this commotion, of=
 course;
 this commotion can, however, be, an unintended accident, but the fact t=
 hat
 it came from Theo gave it a lot of credibility.

 Whoa, wait a second here. =C2=A0If you think I gave it credibility, you
 need to go back and read my words again. =C2=A0I called it an allegation,
 and I stick with that. =C2=A0I was extremely careful with my words, and
y=
 ou
 are wrong to interpret them as you do.

 Look, if somebody like me posted something like this here, it would be
just
 plain dismissed.

 If that is the case -- that people would dismiss it automatically --
 then the community is really stupid.  You are almost arguing that that
 is the way it should be.

 Allegation of not, code should always be checked, and re-checked, and
 re-checked.

 What I am seeing is that we have a ridiculously upside-down trust
 model -- Trust the developers.

 We never asked for people to trust us.  We might have earned some in
 some people's eyes, but if so it has always been false, even before
 this.  People should trust what they test, but the world has become
 incredibly lazy.

 We build this stuff by trusting each other as friends, and that is
 done on an international level.  If anything, the layers and volume of
 trust involved in software development should decrease trust. Oh
 right, let's hear some of that many eyes crap again.  My favorite
 part of the many eyes argument is how few bugs were found by the two
 eyes of Eric (the originator of the statement).  All the many eyes are
 apparently attached to a lot of hands that type lots of words about
 many eyes, and never actually audit code.

 If anything, the collaborative model we use should _decrease_ trust,
 except, well, unless you compare it to the other model -- corporate
 software -- where they don't even start from any position of trust.
 There you are trusting the money, here you are trusting people I've
 never met.

 If Perry posted his email here, he'd just be under fire to
 show some or any proof.

 OK, so I post it, and then noone asks him for proof, now it suddenly
 has more strength?  I am so bloody dissapointed in the community that
 uses our stuff.

 The reason this was so widely picked up
 and generated so much flame and buzz, is because you posted it here.

 How dismal.

 It's an unfortunate consequence of a right action, really. I'm not even
 remotely saying that you intended to give it weight, or that you
 should've swept it under the rug.

 What a dismal world view.