[PATCH] MINOR: systemd: remove comment about HAPROXY_STATS_SOCKET

[Vincent Bernat]

From: Vincent Bernat 

This variable was used by the wrapper which was removed in
a6cfa9098e5a. The correct way to do seamless reload is now to enable
"expose-fd listeners" on the stat socket.
---
 contrib/systemd/haproxy.service.in | 2 --
 1 file changed, 2 deletions(-)

diff --git a/contrib/systemd/haproxy.service.in 
b/contrib/systemd/haproxy.service.in
index edbd4c292dd5..804be3583c1a 100644
--- a/contrib/systemd/haproxy.service.in
+++ b/contrib/systemd/haproxy.service.in
@@ -3,8 +3,6 @@ Description=HAProxy Load Balancer
 After=network.target
 
 [Service]
-# You can point the environment variable HAPROXY_STATS_SOCKET to a stats
-# socket if you want seamless reloads.
 Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
 ExecStartPre=@SBINDIR@/haproxy -f $CONFIG -c -q
 ExecStart=@SBINDIR@/haproxy -Ws -f $CONFIG -p $PIDFILE
-- 
2.15.1

Incredible performance bump between 1.5.12 and 1.8.1 ?

[Christopher Lane]

It is plausible/expected that my version upgrade performance goes like
(about 4K connections, long lived and short lived mix, TCP only, no
HTTP:

1.5.12 (nproc 1, old connections causing about 100 old -sf processes
to linger) uses 100% CPU almost all the time, frequently with >1
process.  (100%, 75%, 48%, ... ).  Highest CPU user has 2-3K
connections.

1.8.1 (nbproc 4, with hard-stop-after 600s) using like 1%, 3%, 3%, 8%
CPU  Also 2-3K connections.

Connect latencies much more stable.

I started reading thru all the change logs, but extrapolating from
that to our (all TCP-only, no HTTP) performance is taxing, thought I'd
ask here first.

Thanks,

--Chris

choosing the best scenario for load balancer and nignx reverse proxy

[Ricky Gutierrez]

hi list, I need some tips to build this solution, I currently have
nginx as a proxy reverse and load balancer with http2 and ssl/tls ,
but I still do not like the load balancer part of nginx and I want to
put haproxy to do the job, but I'm not sure if I put in front haproxy
and then nginx as a reverse proxy something like this:

www > haproxy (port 80) + nginx (port 8080) + web server (IIS /
Apache / nginx)

haproxy and nginx in the same machine.

What do you think?

-- 
rickygm

http://gnuforever.homelinux.com

Re: lua memory usage on haproxy 1.8.1

[Thierry Fournier]

Hi Laurent,I think that I catch the bug. Could you test the attached patch ?Just for information, this patch is not ready to be published, but it fix the bug.Thanks,thierry

0001-BUGFIX.patch
Description: Binary data
On 9 Dec 2017, at 00:37, Thierry Fournier  wrote:Hi Laurent,I reproduce the bug with version 1.81 without thread support. I will check for this.ThierryOn 8 Dec 2017, at 13:05, Laurent Penot  wrote:Hi list,Having added lua to haproxy is really very very helpful, thanks a lot for this. I’m using it heavily.I need to open a socket to get the content of a web page from my local network, and I do that every second using a task.It’s in production use since lua is available in haproxy, and it has always worked fine (for now, production servers are using haproxy 1.7.9 and lua 5.3.4).This task is running well on haproxy 1.7.9 and lua 5.3.4 but seems to not release memory when running on haproxy 1.8.1 with lua (either 5.3.3 or 5.3.4, result is the same)It’s running on a centox 7 box, kernel 3.10.Please find below the task, cfg, and result of haproxy –vv.Do you know if I need to change something in this task to release memory with haproxy 1.8.1 ?I removed all the unnecessary code, so here is the lua task :function updateTest()    local s    local msg    local infos    while true do        s = core.tcp()        s:connect("ipv4@172.16.1.1:10011")        s:send("GET / HTTP/1.0\r\n\r\n")        msg = s:receive("*a")--        if (msg ~= nil) then--            core.Info('msg:' .. tostring(msg))--        end        msg = nil        -- close conection        s:close()        s = nil        infos = core.get_info()        core.Info('PoolUsed_MB:' .. tostring(infos['PoolUsed_MB']))        infos = nil        core.sleep(1)    endendcore.register_task(updateTest)Here is the haproxy infos:HA-Proxy version 1.8.1 2017/12/03Copyright 2000-2017 Willy Tarreau Build options :  TARGET  = linux2628  CPU     = generic  CC      = gcc  CFLAGS  = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=yes USE_DEVICEATLAS=1 USE_SYSTEMD=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1Default settings :  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200Built with OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017Running on OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017OpenSSL library supports TLS extensions : yesOpenSSL library supports SNI : yesOpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2Built with Lua version : Lua 5.3.3Built with DeviceAtlas support.Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBINDEncrypted password support via crypt(3): yesBuilt with PCRE version : 8.41 2017-07-05Running on PCRE version : 8.41 2017-07-05PCRE library supports JIT : yesBuilt with zlib version : 1.2.7Running on zlib version : 1.2.7Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")Built with network namespace support.Available polling systems:      epoll : pref=300,  test result OK       poll : pref=200,  test result OK     select : pref=150,  test result OKTotal: 3 (3 usable), will use epoll.Available filters :	[SPOE] spoe	[COMP] compression	[TRACE] traceHere is a part of the config file (defaults, timeout, … removed):global    chroot      /var/run/haproxy    pidfile     /var/run/haproxy.pid    user        haproxy    group       haproxy    master-worker    lua-load /etc/haproxy/lua/test.luabest regardsLaurent

Re: lua memory usage on haproxy 1.8.1

[Thierry Fournier]

Hi Laurent,

I reproduce the bug with version 1.81 without thread support. I will check for 
this.

Thierry

> On 8 Dec 2017, at 13:05, Laurent Penot  wrote:
> 
> Hi list,
> 
> Having added lua to haproxy is really very very helpful, thanks a lot for 
> this. I’m using it heavily.
> 
> I need to open a socket to get the content of a web page from my local 
> network, and I do that every second using a task.
> It’s in production use since lua is available in haproxy, and it has always 
> worked fine (for now, production servers are using haproxy 1.7.9 and lua 
> 5.3.4).
> This task is running well on haproxy 1.7.9 and lua 5.3.4 but seems to not 
> release memory when running on haproxy 1.8.1 with lua (either 5.3.3 or 5.3.4, 
> result is the same)
> It’s running on a centox 7 box, kernel 3.10.
> Please find below the task, cfg, and result of haproxy –vv.
> Do you know if I need to change something in this task to release memory with 
> haproxy 1.8.1 ?
> 
> 
> I removed all the unnecessary code, so here is the lua task :
> function updateTest()
> 
> local s
> local msg
> local infos
> 
> while true do
> s = core.tcp()
> s:connect("ipv4@172.16.1.1 :10011")
> s:send("GET / HTTP/1.0\r\n\r\n")
> 
> msg = s:receive("*a")
> --if (msg ~= nil) then
> --core.Info('msg:' .. tostring(msg))
> --end
> msg = nil
> 
> -- close conection
> s:close()
> s = nil
> 
> infos = core.get_info()
> core.Info('PoolUsed_MB:' .. tostring(infos['PoolUsed_MB']))
> infos = nil
> 
> core.sleep(1)
> end
> 
> end
> core.register_task(updateTest)
> 
> 
> Here is the haproxy infos:
> HA-Proxy version 1.8.1 2017/12/03
> Copyright 2000-2017 Willy Tarreau  >
> 
> Build options :
>   TARGET  = linux2628
>   CPU = generic
>   CC  = gcc
>   CFLAGS  = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing 
> -Wdeclaration-after-statement -fwrapv -Wno-unused-label
>   OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=yes 
> USE_DEVICEATLAS=1 USE_SYSTEMD=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1
> 
> Default settings :
>   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> 
> Built with OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
> Running on OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
> Built with Lua version : Lua 5.3.3
> Built with DeviceAtlas support.
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
> IP_FREEBIND
> Encrypted password support via crypt(3): yes
> Built with PCRE version : 8.41 2017-07-05
> Running on PCRE version : 8.41 2017-07-05
> PCRE library supports JIT : yes
> Built with zlib version : 1.2.7
> Running on zlib version : 1.2.7
> Compression algorithms supported : identity("identity"), deflate("deflate"), 
> raw-deflate("deflate"), gzip("gzip")
> Built with network namespace support.
> 
> Available polling systems:
>   epoll : pref=300,  test result OK
>poll : pref=200,  test result OK
>  select : pref=150,  test result OK
> Total: 3 (3 usable), will use epoll.
> 
> Available filters :
>   [SPOE] spoe
>   [COMP] compression
>   [TRACE] trace
> 
> 
> 
> Here is a part of the config file (defaults, timeout, … removed):
> global
> chroot  /var/run/haproxy
> pidfile /var/run/haproxy.pid
> userhaproxy
> group   haproxy
> master-worker
> lua-load /etc/haproxy/lua/test.lua
> 
> 
> best regards
> Laurent
> 

Re: Testing master-worker reloads on HAProxy 1.8

[William Lallemand]

> On Wed, Dec 06, 2017 at 10:48:23PM +, Anthony Via wrote:
> > I am testing seamless reloads on HAProxy 1.8.0 using the master-worker
> > model and am running into the following when sending SIGUSR2 to the master
> > process

Sorry I misread, my explanation in the previous post is not correct for the
seamless reload (-x + expose-fd), but only for a "classic" reload.

During a seamless reload, the new process try to get the FDs of the listeners
using the unix socket.

Did you try the seamless reload using -x without the master-worker?

-- 
William Lallemand

Re: Testing master-worker reloads on HAProxy 1.8

[William Lallemand]

Hello Anthony,

On Wed, Dec 06, 2017 at 10:48:23PM +, Anthony Via wrote:
> ?Hello,
> 
> 
> I am testing seamless reloads on HAProxy 1.8.0 using the master-worker model 
> and am running into the following when sending SIGUSR2 to the master process:
> 
> 
> [ALERT] 339/222907 (61399) : Starting frontend internal_http: cannot bind 
> socket [0.0.0.0:80]
> [WARNING] 339/222907 (61399) : Reexecuting Master process in waitpid mode
> [WARNING] 339/222907 (61399) : Reexecuting Master process
> 
> From my understanding, after the master process receives the SIGUSR2 signal
> it should be sending the worker process(es) the SIGUSR1 signal, which does
> not appear to be happening.

When the master receive the SIGUSR2 signal, it should reexec itself with the
-sf argument followed by the PIDs of the current workers.

It follows the execution of a normal haproxy process with -sf:

- It parses the configuration
- Try to bind with SO_REUSEPORT if supported by your system (I don't think your 
system support it)
- If it cannot bind it sends the SIGTTOU signal to the old processes
- the old processes receive the SIGTTOU and disable the bind
- the new process try to bind
- the new process send the SIGUSR1 to the old ones
- the new process fork the children

I just tested the master-worker with the -dR option to validate that the
SIGTTOU system is still working, it seems to work on my linux.

> I have manually sent worker processes the SIGUSR1 signal, and they do shut
> down cleanly as expected. I thought maybe the worker wasn't shutting down
> quick enough, so I played around with "hard-stop-after?", but that did not
> help.
> 

Did you try launching a new haproxy process with the -sf option, without using
the master-worker?

> This is on a Solaris based operating system. I did have success on Ubuntu
> 16.04, so I'm wondering if this is an issue with my OS.
> 

What is your operating system and version exactly?

> Any ideas for narrowing the problem down?
> 
> Thanks,

I think the old processes did not receive the SIGTTOU for an unknown reason,
or did not unbind once it received the signal.

Maybe you could try to compare what's happening on your solaris-like system and
your ubuntu with the -dR option, using strace on linux and truss on solaris.

Regards,

-- 
William Lallemand

subscribe

[Beluc]

lua memory usage on haproxy 1.8.1

[Laurent Penot]

Hi list,

Having added lua to haproxy is really very very helpful, thanks a lot for this. 
I’m using it heavily.

I need to open a socket to get the content of a web page from my local network, 
and I do that every second using a task.
It’s in production use since lua is available in haproxy, and it has always 
worked fine (for now, production servers are using haproxy 1.7.9 and lua 5.3.4).
This task is running well on haproxy 1.7.9 and lua 5.3.4 but seems to not 
release memory when running on haproxy 1.8.1 with lua (either 5.3.3 or 5.3.4, 
result is the same)
It’s running on a centox 7 box, kernel 3.10.
Please find below the task, cfg, and result of haproxy –vv.
Do you know if I need to change something in this task to release memory with 
haproxy 1.8.1 ?


I removed all the unnecessary code, so here is the lua task :
function updateTest()

local s
local msg
local infos

while true do
s = core.tcp()
s:connect("ipv4@172.16.1.1:10011")
s:send("GET / HTTP/1.0\r\n\r\n")

msg = s:receive("*a")
--if (msg ~= nil) then
--core.Info('msg:' .. tostring(msg))
--end
msg = nil

-- close conection
s:close()
s = nil

infos = core.get_info()
core.Info('PoolUsed_MB:' .. tostring(infos['PoolUsed_MB']))
infos = nil

core.sleep(1)
end

end
core.register_task(updateTest)


Here is the haproxy infos:
HA-Proxy version 1.8.1 2017/12/03
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing 
-Wdeclaration-after-statement -fwrapv -Wno-unused-label
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=yes 
USE_DEVICEATLAS=1 USE_SYSTEMD=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with DeviceAtlas support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with PCRE version : 8.41 2017-07-05
Running on PCRE version : 8.41 2017-07-05
PCRE library supports JIT : yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems:
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace



Here is a part of the config file (defaults, timeout, … removed):
global
chroot  /var/run/haproxy
pidfile /var/run/haproxy.pid
userhaproxy
group   haproxy
master-worker
lua-load /etc/haproxy/lua/test.lua


best regards
Laurent

Re: 1.8.1-fe66fd doesn't finish startup and doesn't listen to sockets

[Pavlos Parissis]

On 08/12/2017 10:38 πμ, Pavlos Parissis wrote:
> On 08/12/2017 10:21 πμ, Christopher Faulet wrote:
>> Le 08/12/2017 à 05:52, Willy Tarreau a écrit :
>>>
 pparissis at poseidonas in ~/repo/haproxy-1.8 on (master u=)
 sudo gdb ./haproxy
 GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
 Copyright (C) 2016 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later 
 
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 and "show warranty" for details.
 This GDB was configured as "x86_64-linux-gnu".
 Type "show configuration" for configuration details.
 For bug reporting instructions, please see:
 .
 Find the GDB manual and other documentation resources online at:
 .
 For help, type "help".
 Type "apropos word" to search for commands related to "word"...
 Reading symbols from ./haproxy...done.
 (gdb) run -f /etc/haproxy/haproxy-ams4-dc.cfg
 Starting program: /home/pparissis/repo/haproxy-1.8/haproxy -f
 /etc/haproxy/haproxy-ams4-dc.cfg
 [Thread debugging using libthread_db enabled]
 Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
 [WARNING] 340/221611 (13628) : parsing 
 [/etc/haproxy/haproxy-ams4-dc.cfg:103] : a
 'http-request' rule placed after a 'use_backend' rule will still be 
 processed before.
 ^C
 Program received signal SIGINT, Interrupt.
 0x5562e302 in register_name (name=0x558ee294 
 "selected_dc_backup",
 len=18, scope=scope@entry=0x7fffd83c,
  alloc=alloc@entry=1, err=0x7fffdb28) at src/vars.c:215
 215    HA_RWLOCK_WRLOCK(VARS_LOCK, _names_rwlock);
>>>
>>> Ah, this looks like a double-lock :-)
>>> CCing Christopher who knows better than me how to track them.
>>>
>> Hi,
>>
>> The problem is not a double-lock but an unlock on a free lock, leaving it in 
>> a bad state. This is
>> not obvious on this trace, but compiling HAProxy with the debug on threads 
>> it is easier to observe it.
>>
>> This patch is joined. It should fix the bug. Thanks Pavlos!
> 
> Applied to 1.8 repo and solved the issue as I can start haproxy.
> 

I meant to my local cloned repo:-)

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: 1.8.1-fe66fd doesn't finish startup and doesn't listen to sockets

[Willy Tarreau]

On Fri, Dec 08, 2017 at 10:21:45AM +0100, Christopher Faulet wrote:
> The problem is not a double-lock but an unlock on a free lock, leaving it in
> a bad state. This is not obvious on this trace, but compiling HAProxy with
> the debug on threads it is easier to observe it.
> 
> This patch is joined. It should fix the bug. Thanks Pavlos!

thanks, applied now.

Willy

Re: 1.8.1-fe66fd doesn't finish startup and doesn't listen to sockets

[Christopher Faulet]

Le 08/12/2017 à 05:52, Willy Tarreau a écrit :



pparissis at poseidonas in ~/repo/haproxy-1.8 on (master u=)
sudo gdb ./haproxy
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./haproxy...done.
(gdb) run -f /etc/haproxy/haproxy-ams4-dc.cfg
Starting program: /home/pparissis/repo/haproxy-1.8/haproxy -f
/etc/haproxy/haproxy-ams4-dc.cfg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[WARNING] 340/221611 (13628) : parsing [/etc/haproxy/haproxy-ams4-dc.cfg:103] : 
a
'http-request' rule placed after a 'use_backend' rule will still be processed 
before.
^C
Program received signal SIGINT, Interrupt.
0x5562e302 in register_name (name=0x558ee294 "selected_dc_backup",
len=18, scope=scope@entry=0x7fffd83c,
 alloc=alloc@entry=1, err=0x7fffdb28) at src/vars.c:215
215 HA_RWLOCK_WRLOCK(VARS_LOCK, _names_rwlock);


Ah, this looks like a double-lock :-)
CCing Christopher who knows better than me how to track them.


Hi,

The problem is not a double-lock but an unlock on a free lock, leaving 
it in a bad state. This is not obvious on this trace, but compiling 
HAProxy with the debug on threads it is easier to observe it.


This patch is joined. It should fix the bug. Thanks Pavlos!
--
Christopher Faulet
>From 023f995f5eba5e4c18cbbde0acc2225d3846bbc7 Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Fri, 8 Dec 2017 09:17:39 +0100
Subject: [PATCH] BUG/MEDIUM: threads/vars: Fix deadlock in register_name

In register_name, before locking the var_names array, we check the variable name
validity. So if we try to register an invalid or empty name, we need to return
without unlocking it (because it was never locked).

This patch must be backported in 1.8.
---
 src/vars.c | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/src/vars.c b/src/vars.c
index c8aa5ac89..566ead6ec 100644
--- a/src/vars.c
+++ b/src/vars.c
@@ -174,8 +174,7 @@ static char *register_name(const char *name, int len, enum vars_scope *scope,
 	/* Check length. */
 	if (len == 0) {
 		memprintf(err, "Empty variable name cannot be accepted");
-		res = NULL;
-		goto end;
+		return res;
 	}
 
 	/* Check scope. */
@@ -207,8 +206,7 @@ static char *register_name(const char *name, int len, enum vars_scope *scope,
 	else {
 		memprintf(err, "invalid variable name '%s'. A variable name must be start by its scope. "
 		   "The scope can be 'proc', 'sess', 'txn', 'req' or 'res'", name);
-		res = NULL;
-		goto end;
+		return res;
 	}
 
 	if (alloc)
-- 
2.13.6

Re: Cron Weekly: a weekly newsletter for Linux and Open Source sysadmins

[Peter Hawthorne]

Hi there,

 

I reached out last week but haven't heard back so I wanted to try one last 
time. Is there an opportunity to sponsor a post on Cron Weekly: a weekly 
newsletter for Linux and Open Source sysadmins?

 

Please see my initial email below.

 

Best wishes,

Peter

 

 

On Sat, Dec 2, 2017 at 9:00 AM, Peter Hawthorne 
 wrote:

Hi there,

 

I reached out last week but haven't heard back. I wanted to see if there was an 
opportunity to sponsor a post on Cron Weekly: a weekly newsletter for Linux and 
Open Source sysadmins.

 

Please see my initial email below.

 

Best wishes,

Peter

On Sun, Nov 26, 2017 at 9:16 AM, Peter Hawthorne 
 wrote:

Hi there,
 
I just read cron.weekly issue #84: Hadoop, UUID, Evil.sh, GDB, rl, Monica, 
SFTP, Containers & more, I've shown it to some colleagues and we think a 
collaboration between us could work well. 
 

I represent a digital marketing agency currently working with a leading 
technology company who operate in the same marketplace as Android. We would 
like to feature a unique piece of content on your site on behalf of our client. 
For the privilege, we'd be happy to pay you somewhere in the region of $30.
 

Let me know your thoughts.
 

Best wishes,

Peter