Re: Feature Request for log stdout ...

[Aleksandar Lazic]

Hi.

Am 18-02-2016 15:22, schrieb Willy Tarreau:

Hi Aleks,

On Thu, Feb 18, 2016 at 02:53:29PM +0100, Aleksandar Lazic wrote:


[snipp]


For openshift I will try to use 2 container in 1 pod.

If there any interests I can write here if this works ;-)


Sure, please report anyway.


You can find my solution at this repo

https://github.com/git001/haproxy

I use socklog ( http://smarden.org/socklog/ ) instead of socat.
socklog is in debian but not in centos/rhel so I just build it in the 
docker file.


BR Aleks

Re: Feature Request for log stdout ...

[Aleksandar Lazic]

Hi Bryan.

Am 18-02-2016 21:18, schrieb Bryan Talbot:

Sorry I'm a bit late to this party but when running in a container it's
also easy to configure haproxy to log to a unix socket and bind mount
that socket to the host.

in haproxy.cnf


log /dev/log local2


Then when launching the container an option like "-v /var/log:/var/log"
works quite well to get container syslogs to the host.


Well this way is not possible in openshift due to the fact that the pods 
are not running as root!



-Bryan

On Thu, Feb 18, 2016 at 6:22 AM, Willy Tarreau  wrote:


Hi Aleks,

On Thu, Feb 18, 2016 at 02:53:29PM +0100, Aleksandar Lazic wrote:

But this moves just the stdout handling to other tools and does not
solve the problem with blocking handling of std*, as far as I have
understood right.


Yes it does because if the logging daemon blocks, logs are simply lost
on the UDP socket between haproxy and the daemon without blocking
haproxy.


It also 'violates' the best practice of docker.





https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container


Well it's written "in almost all cases". Otherwise you would not even
be allowed to use nbproc or the systemd wrapper. If you consider your
deamon as the log-dedicated process, it's OK :-)


Okay this could be solved with the linking as described in the link.

For openshift I will try to use 2 container in 1 pod.

If there any interests I can write here if this works ;-)


Sure, please report anyway.

Cheers,
Willy

Re: Feature Request for log stdout ...

[Bryan Talbot]

Sorry I'm a bit late to this party but when running in a container it's
also easy to configure haproxy to log to a unix socket and bind mount that
socket to the host.

in haproxy.cnf

log /dev/log local2


Then when launching the container an option like "-v /var/log:/var/log"
works quite well to get container syslogs to the host.

-Bryan



On Thu, Feb 18, 2016 at 6:22 AM, Willy Tarreau  wrote:

> Hi Aleks,
>
> On Thu, Feb 18, 2016 at 02:53:29PM +0100, Aleksandar Lazic wrote:
> > But this moves just the stdout handling to other tools and does not
> > solve the problem with blocking handling of std*, as far as I have
> > understood right.
>
> Yes it does because if the logging daemon blocks, logs are simply lost
> on the UDP socket between haproxy and the daemon without blocking
> haproxy.
>
> > It also 'violates' the best practice of docker.
> >
> >
> https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container
>
> Well it's written "in almost all cases". Otherwise you would not even
> be allowed to use nbproc or the systemd wrapper. If you consider your
> deamon as the log-dedicated process, it's OK :-)
>
> > Okay this could be solved with the linking as described in the link.
> >
> > For openshift I will try to use 2 container in 1 pod.
> >
> > If there any interests I can write here if this works ;-)
>
> Sure, please report anyway.
>
> Cheers,
> Willy
>
>
>

Re: Feature Request for log stdout ...

[Willy Tarreau]

Hi Aleks,

On Thu, Feb 18, 2016 at 02:53:29PM +0100, Aleksandar Lazic wrote:
> But this moves just the stdout handling to other tools and does not 
> solve the problem with blocking handling of std*, as far as I have 
> understood right.

Yes it does because if the logging daemon blocks, logs are simply lost
on the UDP socket between haproxy and the daemon without blocking
haproxy.

> It also 'violates' the best practice of docker.
> 
> https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container

Well it's written "in almost all cases". Otherwise you would not even
be allowed to use nbproc or the systemd wrapper. If you consider your
deamon as the log-dedicated process, it's OK :-)

> Okay this could be solved with the linking as described in the link.
> 
> For openshift I will try to use 2 container in 1 pod.
> 
> If there any interests I can write here if this works ;-)

Sure, please report anyway.

Cheers,
Willy

Re: Feature Request for log stdout ...

[Alberto]

On 02/18/2016 07:53 AM, Aleksandar Lazic wrote:


Thanks for answers and suggestions.

But this moves just the stdout handling to other tools and does not 
solve the problem with blocking handling of std*, as far as I have 
understood right.


haproxy will not block. If your logging system hangs it will
drop the UDP packets.

I've seen it and had discussions before when people use TCP
for logging systems. A hiccup in the logging systems brings
the whole infrastructure down.


It also 'violates' the best practice of docker.


Your application should be more important than docker's best
practices. These guides are suggestions not mandates.

Having said that, you should run your logging system in its own
container.

Process in this context is a micro-service, not a single kernel
process (ie. pid)

Hope this helps,

Alberto

Re: Feature Request for log stdout ...

[Aleksandar Lazic]

Hi.

Am 18-02-2016 11:47, schrieb Conrad Hoffmann:

Two more cents from my side:

socklog [1] also works pretty well...

[1] http://smarden.org/socklog/

Conrad

On 02/18/2016 11:28 AM, Baptiste wrote:

On Thu, Feb 18, 2016 at 10:57 AM, Willy Tarreau  wrote:

Hi Aleks,

On Wed, Feb 17, 2016 at 04:30:06PM +0100, Aleksandar Lazic wrote:

Hi.

how difficult is it to be able to add "log stdout;" to haproxy?


[snipp]


It's been discussed a few times in the past. The response is "no".
It's totally insane to emit logs to a blocking destination. Your
whole haproxy process will run at the speed of the logs consumer
and the log processing will incure its latency to the process.



[snipp]



My 2 cents: Some tools may be used for this purpose:

Configure HAProxy to send logs to port 2000, then use:

- socat:
socat -u UDP-RECV:2000 -


[snipp]


- netcat:
netcat -l -k -u 2000


[snipp]

Thanks for answers and suggestions.

But this moves just the stdout handling to other tools and does not 
solve the problem with blocking handling of std*, as far as I have 
understood right.


It also 'violates' the best practice of docker.

https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container

Okay this could be solved with the linking as described in the link.

For openshift I will try to use 2 container in 1 pod.

If there any interests I can write here if this works ;-)

BR Aleks

Re: Feature Request for log stdout ...

[Willy Tarreau]

On Thu, Feb 18, 2016 at 11:47:03AM +0100, Conrad Hoffmann wrote:
> Two more cents from my side:
> 
> socklog [1] also works pretty well...
> 
> [1] http://smarden.org/socklog/

Thanks for the links guys. These should probably be added to the
management doc so that people find them more easily.

willy

Re: Feature Request for log stdout ...

[Conrad Hoffmann]

Two more cents from my side:

socklog [1] also works pretty well...

[1] http://smarden.org/socklog/

Conrad

On 02/18/2016 11:28 AM, Baptiste wrote:
> On Thu, Feb 18, 2016 at 10:57 AM, Willy Tarreau  wrote:
>> Hi Aleks,
>>
>> On Wed, Feb 17, 2016 at 04:30:06PM +0100, Aleksandar Lazic wrote:
>>> Hi.
>>>
>>> how difficult is it to be able to add "log stdout;" to haproxy?
>>>
>>> I ask because in some PaaS environment is it difficult to setup a
>>> dedicated user yust for haproxy.
>>>
>>> It fits also a little bit better to http://12factor.net/logs
>>
>> It's been discussed a few times in the past. The response is "no".
>> It's totally insane to emit logs to a blocking destination. Your
>> whole haproxy process will run at the speed of the logs consumer
>> and the log processing will incure its latency to the process.
>>
>> If one day we implement an synchronous stream logging task, this
>> could change, but for now we send immediate logs as datagrams in
>> order never to block.
>>
>> To get an idea about what it can look like with blocking logs,
>> simply run "haproxy -d 2>&1 | more" and don't press any key.
>> You'll quickly see that the system continues to accept new
>> connections and that they will randomly freeze at various steps.
>>
>> Regards,
>> Willy
>>
>>
> 
> My 2 cents: Some tools may be used for this purpose:
> 
> Configure HAProxy to send logs to port 2000, then use:
> 
> - socat:
> socat -u UDP-RECV:2000 -
> <133>Feb 18 11:27:02 haproxy[4134]: Proxy f started.
> <133>Feb 18 11:27:02 haproxy[4134]: Proxy b started.
> <133>Feb 18 11:27:02 haproxy[4134]: Proxy stats started.
> <129>Feb 18 11:27:02 haproxy[4134]: Server b/s is DOWN, reason: Layer4
> connection problem, info: "Connection refused", check duration: 0ms. 0
> active and 0 backup servers left. 0 sessions active, 0 requeued, 0
> remaining in queue.
> <128>Feb 18 11:27:02 haproxy[4134]: backend b has no server available!
> 
> - netcat:
> netcat -l -k -u 2000
> <133>Feb 18 11:28:17 haproxy[4303]: Proxy f started.
> <133>Feb 18 11:28:17 haproxy[4303]: Proxy b started.
> <133>Feb 18 11:28:17 haproxy[4303]: Proxy stats started.
> <129>Feb 18 11:28:17 haproxy[4303]: Server b/s is DOWN, reason: Layer4
> connection problem, info: "Connection refused", check duration: 0ms. 0
> active and 0 backup servers left. 0 sessions active, 0 requeued, 0
> remaining in queue.
> <128>Feb 18 11:28:17 haproxy[4303]: backend b has no server available!
> 
> 
> 
> Baptiste
> 

-- 
Conrad Hoffmann
Traffic Engineer

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany

Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

Re: Feature Request for log stdout ...

[Baptiste]

On Thu, Feb 18, 2016 at 10:57 AM, Willy Tarreau  wrote:
> Hi Aleks,
>
> On Wed, Feb 17, 2016 at 04:30:06PM +0100, Aleksandar Lazic wrote:
>> Hi.
>>
>> how difficult is it to be able to add "log stdout;" to haproxy?
>>
>> I ask because in some PaaS environment is it difficult to setup a
>> dedicated user yust for haproxy.
>>
>> It fits also a little bit better to http://12factor.net/logs
>
> It's been discussed a few times in the past. The response is "no".
> It's totally insane to emit logs to a blocking destination. Your
> whole haproxy process will run at the speed of the logs consumer
> and the log processing will incure its latency to the process.
>
> If one day we implement an synchronous stream logging task, this
> could change, but for now we send immediate logs as datagrams in
> order never to block.
>
> To get an idea about what it can look like with blocking logs,
> simply run "haproxy -d 2>&1 | more" and don't press any key.
> You'll quickly see that the system continues to accept new
> connections and that they will randomly freeze at various steps.
>
> Regards,
> Willy
>
>

My 2 cents: Some tools may be used for this purpose:

Configure HAProxy to send logs to port 2000, then use:

- socat:
socat -u UDP-RECV:2000 -
<133>Feb 18 11:27:02 haproxy[4134]: Proxy f started.
<133>Feb 18 11:27:02 haproxy[4134]: Proxy b started.
<133>Feb 18 11:27:02 haproxy[4134]: Proxy stats started.
<129>Feb 18 11:27:02 haproxy[4134]: Server b/s is DOWN, reason: Layer4
connection problem, info: "Connection refused", check duration: 0ms. 0
active and 0 backup servers left. 0 sessions active, 0 requeued, 0
remaining in queue.
<128>Feb 18 11:27:02 haproxy[4134]: backend b has no server available!

- netcat:
netcat -l -k -u 2000
<133>Feb 18 11:28:17 haproxy[4303]: Proxy f started.
<133>Feb 18 11:28:17 haproxy[4303]: Proxy b started.
<133>Feb 18 11:28:17 haproxy[4303]: Proxy stats started.
<129>Feb 18 11:28:17 haproxy[4303]: Server b/s is DOWN, reason: Layer4
connection problem, info: "Connection refused", check duration: 0ms. 0
active and 0 backup servers left. 0 sessions active, 0 requeued, 0
remaining in queue.
<128>Feb 18 11:28:17 haproxy[4303]: backend b has no server available!



Baptiste

Re: Feature Request for log stdout ...

[Willy Tarreau]

Hi Aleks,

On Wed, Feb 17, 2016 at 04:30:06PM +0100, Aleksandar Lazic wrote:
> Hi.
> 
> how difficult is it to be able to add "log stdout;" to haproxy?
> 
> I ask because in some PaaS environment is it difficult to setup a 
> dedicated user yust for haproxy.
> 
> It fits also a little bit better to http://12factor.net/logs

It's been discussed a few times in the past. The response is "no".
It's totally insane to emit logs to a blocking destination. Your
whole haproxy process will run at the speed of the logs consumer
and the log processing will incure its latency to the process.

If one day we implement an synchronous stream logging task, this
could change, but for now we send immediate logs as datagrams in
order never to block.

To get an idea about what it can look like with blocking logs,
simply run "haproxy -d 2>&1 | more" and don't press any key.
You'll quickly see that the system continues to accept new
connections and that they will randomly freeze at various steps.

Regards,
Willy

Feature Request for log stdout ...

[Aleksandar Lazic]

Hi.

how difficult is it to be able to add "log stdout;" to haproxy?

I ask because in some PaaS environment is it difficult to setup a 
dedicated user yust for haproxy.


It fits also a little bit better to http://12factor.net/logs

BR Aleks