Re: TCP Proxy for database connections
Hi, On Thu, 2020-10-29 at 10:21 +0200, Jonathan Matthews wrote: > I don’t think haproxy is what you’re looking for. You’re looking for more > than a TCP proxy: you need a DB-specific-protocol-proxy. Haproxy can > listen for HTTP, above the TCP layer, but not any specific DB protocols. > > I think you need to look for a proxy that’s designed to work with the > specific DB you’re wanting to expose. > > For mysql, “mysql-proxy” and “mysql-router” come to mind. -proxy never > went GA, and I’ve not used -router. For mysql there are MaxScale and ProxySQL. But I don't think you'll find a proxy that has all the features you'll need especially if you need to support multiple DB protocols (mysql, postgresql, oracle, mssql). -Jarno -- Jarno Huuskonen
Re: TCP Proxy for database connections
On Thu, 29 Oct 2020 at 03:41, Anand Rao wrote: > Hi, > > I'm looking for a TCP proxy that can proxy the connection between a > database client and the database server. I want to be able to look at the > traffic and log the queries etc for mining later. I also want to use the > proxy to remove human knowledge of passwords. The users will point their > client to the TCP Port proxy is listening on and will specify a username > which will be a pointer to a vault account (like cyberark or beyondtrust). > The proxy upon receiving this information will then connect to this vault > and get the password and plug the password in for the connection to the > database server. After the connection is established - all traffic should > be proxied through and logged. > > Would HAProxy be a product that can achieve this? If not, I'd like to ask > this knowledgeable community if they can recommend any other projects that > might be closer to achieve the above. I understand this is a very niche > requirement. Any TCP proxy with the ability to script/transform the packets > on the way to the destination would be helpful. I'm trying to find > something in the open source community that I can use for my needs than > having to write one myself. > Hey Anand, I don’t think haproxy is what you’re looking for. You’re looking for more than a TCP proxy: you need a DB-specific-protocol-proxy. Haproxy can listen for HTTP, above the TCP layer, but not any specific DB protocols. I think you need to look for a proxy that’s designed to work with the specific DB you’re wanting to expose. For mysql, “mysql-proxy” and “mysql-router” come to mind. -proxy never went GA, and I’ve not used -router. Given your requirement for the proxy to dynamically fetch credentials, out of band from the connection, I think you’ll find your options to be limited. I know mysql-proxy had Lua embedded (I don’t know about mysql-router) but I’m not sure if it exposed enough Lua libraries to achieve what you’re looking for. For postgres, I’m afraid I’m only aware of “pgbouncer”. If none of these tools does 100% of what you want, you might be able to combine them with haproxy to achieve something closer to what you need. Your “everything is logged” requirement, depending on the level to which you need things logged, will likely be a sticking point. Best of luck, Jonathan > -- Jonathan Matthews https://jpluscplusm.com
Re: TCP proxy capabilities
On Mon, Mar 14, 2011 at 05:49:05PM -0700, g...@desgames.com wrote: That's something I considered, but the ultimate problem is that the backend service we're running (gearman) sometimes gets backed up with requests from our web servers. In this case, the server still looks 'up' (I *think*) but requests from PHP scripts are held up waiting for the gearman server to process the request and close the connection. So having the backup server option in this case wouldn't work, I don't think, *unless* 'backup' servers in haproxy will cut in if the maxconn setting is exceeded. If that's the case, then I suppose we could set the maxconn limit to a low setting and have the queue option. Do you know if this behaviour exists in haproxy for backup servers? Guy, from what I understand, you'd like haproxy to serve as sort of a connection buffer. It would invent fake responses for your client and try to deliver its request to the servers once they're up, or maybe dump them on the floor. I see a way of queueing requests for a very long time when servers are not available : use a huge retries parameter, but the risk as you stated it is that you displace the bottleneck from the front servers to haproxy. Alternatively, you can use ACLs to detect whether some servers are available or not (or better, use a backup server which will be triggered when the other servers fail to accept the connection). That backup server (or the ACL) would loop back to a frontend that points to nowhere and returns a custom 503 page which basically just contains the data you want to be sent. That's probably as ugly as what you're looking for :-) The request will not be drained though, but it will be dropped as soon as the response is sent. The risk is that the system might send a TCP reset to the client if the client still pushes data after haproxy has closed. I must say I really don't know if that can match your needs, maybe it's something worth trying. Regards, Willy
Re: TCP proxy capabilities
Actually, I should clarify something. We don't really actually want the 'black hole' situation I described - instead, what we want is for haproxy to accept and queue the messages that come in from the requesting server, but to still deliver them when a backend server becomes available. In this way, the requesting process can continue executing and not having to wait for a response. Obviously, we intend to do this only for async type calls. On Mon, Mar 14, 2011 at 4:52 PM, g...@desgames.com g...@desgames.com wrote: Hi, We have a tcp service we'd like to proxy requests to, and we were investigating haproxy as a possible solution for our requirements. So far, it doesn't seem like haproxy is suitable but I thought I'd run it by the community to confirm what I understand to be the case. What we want is a proxy that will accept tcp connections on a specific port and always send a tcp 'ok' response to the requesting process whether or not there is a backend server available. If a backend server (we only plan on having one at the moment) is available then we want the request passed transparently through to the backend server. If the backend server is *not* available, then we want haproxy to operate as a kind of 'black hole', and just accept whatever is sent to it, dumping it to the equivalent of /dev/null. Basically, acting as an equivalent of the backend server. Is the above possible with haproxy? Based on what I've read in the configuration documentation, the answer is no. However, there are a *lot* of options in there so I thought perhaps there's some obscure setting which would allow this to work. Alternatively, does anyone know of a utility that could do what we want? Thanks, Guy -- Guy Knights Systems Administrator DES Games www.desgames.com g...@desgames.com
Re: TCP proxy capabilities
Hi Guy, If you only want HAProxy to queue connections and not send that immediate ok any longer, check out how these folks are doing it to queue to MySQL. http://flavio.tordini.org/a-more-stable-mysql-with-haproxy/comment-page-1 On Mon, Mar 14, 2011 at 5:15 PM, g...@desgames.com g...@desgames.com wrote: Actually, I should clarify something. We don't really actually want the 'black hole' situation I described - instead, what we want is for haproxy to accept and queue the messages that come in from the requesting server, but to still deliver them when a backend server becomes available. In this way, the requesting process can continue executing and not having to wait for a response. Obviously, we intend to do this only for async type calls. On Mon, Mar 14, 2011 at 4:52 PM, g...@desgames.com g...@desgames.comwrote: Hi, We have a tcp service we'd like to proxy requests to, and we were investigating haproxy as a possible solution for our requirements. So far, it doesn't seem like haproxy is suitable but I thought I'd run it by the community to confirm what I understand to be the case. What we want is a proxy that will accept tcp connections on a specific port and always send a tcp 'ok' response to the requesting process whether or not there is a backend server available. If a backend server (we only plan on having one at the moment) is available then we want the request passed transparently through to the backend server. If the backend server is *not* available, then we want haproxy to operate as a kind of 'black hole', and just accept whatever is sent to it, dumping it to the equivalent of /dev/null. Basically, acting as an equivalent of the backend server. Is the above possible with haproxy? Based on what I've read in the configuration documentation, the answer is no. However, there are a *lot* of options in there so I thought perhaps there's some obscure setting which would allow this to work. Alternatively, does anyone know of a utility that could do what we want? Thanks, Guy -- Guy Knights Systems Administrator DES Games www.desgames.com g...@desgames.com
Re: TCP proxy capabilities
On Mon, Mar 14, 2011 at 8:15 PM, g...@desgames.com g...@desgames.com wrote:
Actually, I should clarify something. We don't really actually want the
'black hole' situation I described - instead, what we want is for haproxy to
accept and queue the messages that come in from the requesting server, but
to still deliver them when a backend server becomes available. In this way,
the requesting process can continue executing and not having to wait for a
response. Obviously, we intend to do this only for async type calls.
super hacky, but how about setting up an xinetd service as a backup
server. haproxy switches connections over only when your service is
unavailable.
the xinetd service could capture each request and play them back
later..this service could be as simple as a script:
#!/bin/sh
REPLAY_LOG_DIR=/var/replay_logs
cat ${REPLAY_LOG_DIR}/$(date +%Y_%m_%d)
On Mon, Mar 14, 2011 at 4:52 PM, g...@desgames.com g...@desgames.com wrote:
Hi,
We have a tcp service we'd like to proxy requests to, and we were
investigating haproxy as a possible solution for our requirements. So far,
it doesn't seem like haproxy is suitable but I thought I'd run it by the
community to confirm what I understand to be the case.
What we want is a proxy that will accept tcp connections on a specific
port and always send a tcp 'ok' response to the requesting process whether
or not there is a backend server available. If a backend server (we only
plan on having one at the moment) is available then we want the request
passed transparently through to the backend server. If the backend server is
*not* available, then we want haproxy to operate as a kind of 'black hole',
and just accept whatever is sent to it, dumping it to the equivalent of
/dev/null. Basically, acting as an equivalent of the backend server.
Is the above possible with haproxy? Based on what I've read in the
configuration documentation, the answer is no. However, there are a *lot* of
options in there so I thought perhaps there's some obscure setting which
would allow this to work.
Alternatively, does anyone know of a utility that could do what we want?
Thanks,
Guy
--
Guy Knights
Systems Administrator
DES Games
www.desgames.com
g...@desgames.com
Re: TCP proxy capabilities
Thanks for the quick reply Carlo, but actually sending the immediate ok *is* what we want. We just want haproxy to continue queuing the messages and sending them after it's returned an 'ok' to the requesting server. The people who wrote that page are basically limiting the number of connections to the mysql backend server, but (AFAICT) this just pushes the bottleneck back up to haproxy. Applying this setup to our situation, the requesting server (PHP script) is still going to maintain a connection to haproxy and will be forced to wait until haproxy can send the request to the backend server and then return the ok response back to the script, which will only then be able to continue executing. At least that's my understanding of how it would work, please set me straight if I'm wrong though! Kind regards, Guy On Mon, Mar 14, 2011 at 5:22 PM, carlo flores ca...@petalphile.com wrote: Hi Guy, If you only want HAProxy to queue connections and not send that immediate ok any longer, check out how these folks are doing it to queue to MySQL. http://flavio.tordini.org/a-more-stable-mysql-with-haproxy/comment-page-1 On Mon, Mar 14, 2011 at 5:15 PM, g...@desgames.com g...@desgames.comwrote: Actually, I should clarify something. We don't really actually want the 'black hole' situation I described - instead, what we want is for haproxy to accept and queue the messages that come in from the requesting server, but to still deliver them when a backend server becomes available. In this way, the requesting process can continue executing and not having to wait for a response. Obviously, we intend to do this only for async type calls. On Mon, Mar 14, 2011 at 4:52 PM, g...@desgames.com g...@desgames.comwrote: Hi, We have a tcp service we'd like to proxy requests to, and we were investigating haproxy as a possible solution for our requirements. So far, it doesn't seem like haproxy is suitable but I thought I'd run it by the community to confirm what I understand to be the case. What we want is a proxy that will accept tcp connections on a specific port and always send a tcp 'ok' response to the requesting process whether or not there is a backend server available. If a backend server (we only plan on having one at the moment) is available then we want the request passed transparently through to the backend server. If the backend server is *not* available, then we want haproxy to operate as a kind of 'black hole', and just accept whatever is sent to it, dumping it to the equivalent of /dev/null. Basically, acting as an equivalent of the backend server. Is the above possible with haproxy? Based on what I've read in the configuration documentation, the answer is no. However, there are a *lot* of options in there so I thought perhaps there's some obscure setting which would allow this to work. Alternatively, does anyone know of a utility that could do what we want? Thanks, Guy -- Guy Knights Systems Administrator DES Games www.desgames.com g...@desgames.com -- Guy Knights Systems Administrator DES Games www.desgames.com g...@desgames.com
Re: TCP proxy capabilities
That's something I considered, but the ultimate problem is that the backend
service we're running (gearman) sometimes gets backed up with requests from
our web servers. In this case, the server still looks 'up' (I *think*) but
requests from PHP scripts are held up waiting for the gearman server to
process the request and close the connection. So having the backup server
option in this case wouldn't work, I don't think, *unless* 'backup' servers
in haproxy will cut in if the maxconn setting is exceeded. If that's the
case, then I suppose we could set the maxconn limit to a low setting and
have the queue option. Do you know if this behaviour exists in haproxy for
backup servers?
Thanks,
Guy
On Mon, Mar 14, 2011 at 5:34 PM, David Birdsong david.birds...@gmail.comwrote:
On Mon, Mar 14, 2011 at 8:15 PM, g...@desgames.com g...@desgames.com
wrote:
Actually, I should clarify something. We don't really actually want the
'black hole' situation I described - instead, what we want is for haproxy
to
accept and queue the messages that come in from the requesting server,
but
to still deliver them when a backend server becomes available. In this
way,
the requesting process can continue executing and not having to wait for
a
response. Obviously, we intend to do this only for async type calls.
super hacky, but how about setting up an xinetd service as a backup
server. haproxy switches connections over only when your service is
unavailable.
the xinetd service could capture each request and play them back
later..this service could be as simple as a script:
#!/bin/sh
REPLAY_LOG_DIR=/var/replay_logs
cat ${REPLAY_LOG_DIR}/$(date +%Y_%m_%d)
On Mon, Mar 14, 2011 at 4:52 PM, g...@desgames.com g...@desgames.com
wrote:
Hi,
We have a tcp service we'd like to proxy requests to, and we were
investigating haproxy as a possible solution for our requirements. So
far,
it doesn't seem like haproxy is suitable but I thought I'd run it by the
community to confirm what I understand to be the case.
What we want is a proxy that will accept tcp connections on a specific
port and always send a tcp 'ok' response to the requesting process
whether
or not there is a backend server available. If a backend server (we only
plan on having one at the moment) is available then we want the request
passed transparently through to the backend server. If the backend
server is
*not* available, then we want haproxy to operate as a kind of 'black
hole',
and just accept whatever is sent to it, dumping it to the equivalent of
/dev/null. Basically, acting as an equivalent of the backend server.
Is the above possible with haproxy? Based on what I've read in the
configuration documentation, the answer is no. However, there are a
*lot* of
options in there so I thought perhaps there's some obscure setting which
would allow this to work.
Alternatively, does anyone know of a utility that could do what we want?
Thanks,
Guy
--
Guy Knights
Systems Administrator
DES Games
www.desgames.com
g...@desgames.com
--
Guy Knights
Systems Administrator
DES Games
www.desgames.com
g...@desgames.com
Re: TCP proxy capabilities
On Mon, Mar 14, 2011 at 8:49 PM, g...@desgames.com g...@desgames.com wrote:
That's something I considered, but the ultimate problem is that the backend
service we're running (gearman) sometimes gets backed up with requests from
our web servers. In this case, the server still looks 'up' (I *think*) but
requests from PHP scripts are held up waiting for the gearman server to
process the request and close the connection. So having the backup server
option in this case wouldn't work, I don't think, *unless* 'backup' servers
in haproxy will cut in if the maxconn setting is exceeded. If that's the
case, then I suppose we could set the maxconn limit to a low setting and
have the queue option. Do you know if this behaviour exists in haproxy for
backup servers?
check out acl's.
you can create acl's off almost any ruleset. in your frontend(or
listen) stanza, if one acl is true, use your normal backend. if
another is true, use your backup.
there might be better ways to do this in just the backend itself also.
Thanks,
Guy
On Mon, Mar 14, 2011 at 5:34 PM, David Birdsong david.birds...@gmail.com
wrote:
On Mon, Mar 14, 2011 at 8:15 PM, g...@desgames.com g...@desgames.com
wrote:
Actually, I should clarify something. We don't really actually want the
'black hole' situation I described - instead, what we want is for
haproxy to
accept and queue the messages that come in from the requesting server,
but
to still deliver them when a backend server becomes available. In this
way,
the requesting process can continue executing and not having to wait for
a
response. Obviously, we intend to do this only for async type calls.
super hacky, but how about setting up an xinetd service as a backup
server. haproxy switches connections over only when your service is
unavailable.
the xinetd service could capture each request and play them back
later..this service could be as simple as a script:
#!/bin/sh
REPLAY_LOG_DIR=/var/replay_logs
cat ${REPLAY_LOG_DIR}/$(date +%Y_%m_%d)
On Mon, Mar 14, 2011 at 4:52 PM, g...@desgames.com g...@desgames.com
wrote:
Hi,
We have a tcp service we'd like to proxy requests to, and we were
investigating haproxy as a possible solution for our requirements. So
far,
it doesn't seem like haproxy is suitable but I thought I'd run it by
the
community to confirm what I understand to be the case.
What we want is a proxy that will accept tcp connections on a specific
port and always send a tcp 'ok' response to the requesting process
whether
or not there is a backend server available. If a backend server (we
only
plan on having one at the moment) is available then we want the request
passed transparently through to the backend server. If the backend
server is
*not* available, then we want haproxy to operate as a kind of 'black
hole',
and just accept whatever is sent to it, dumping it to the equivalent of
/dev/null. Basically, acting as an equivalent of the backend server.
Is the above possible with haproxy? Based on what I've read in the
configuration documentation, the answer is no. However, there are a
*lot* of
options in there so I thought perhaps there's some obscure setting
which
would allow this to work.
Alternatively, does anyone know of a utility that could do what we
want?
Thanks,
Guy
--
Guy Knights
Systems Administrator
DES Games
www.desgames.com
g...@desgames.com
--
Guy Knights
Systems Administrator
DES Games
www.desgames.com
g...@desgames.com
Re: TCP Proxy, dual-redundant partitions and least connection balancing
Hi, On Wed, 11 Nov 2009 22:16:38 -0800, Jacques whs...@gmail.com wrote: Hello, There is some complexity here that isn't warranted at 4 servers but the redundancy model allows us to do a number of useful things. Also, while the example has each service existing the same number of times, in reality the number of copies of a service would vary depending on other factors (load, response curve, etc). Is this possible out of the box with just a configuration file? thanks for any guidance, Jacques AFAIK you would have to use dynamic weigths (ability to change server weigth thru unix socket ) in haproxy which is in dev tree. So basically script looking on things like cpu/disk/mem load on each node and then other script setting weighs according to those metrics. Or, in simpler version, script getting connection stats from haproxy stats and then setting weigths. So answer is no, nothing out of the box, but yes, it should be possible :) Regards Mariusz -- Mariusz Gronczewski (XANi) xani...@gmail.com GnuPG: 0xEA8ACE64 http://devrandom.pl signature.asc Description: PGP signature
Re: tcp proxy
Hi Willy, It seems right to implement it. I'll review this with the team and let you know once we have an available patch. regards, n. On 04/04/2009, at 03:49, Willy Tarreau wrote: Hi Nicolas, On Fri, Apr 03, 2009 at 10:29:32PM -0300, Nicolas Cohen wrote: hi, i want to use haproxy to load balance a virtual world app we are developing in java, the app server benefits if most connected users in one particular machine are in the same regions of the virtual world (less objects need to be referrenced and caching strategies are more efficient) i thought of making the login process map the region of the world where the user's avatar is placed to one of a set of subdomains (for instance: region1.domain.com, regions2.domain.com). i would then make the DNS assign a different ip for each subdomain. all ips would be configured for the same haproxy machine, which would try to put all connections to the same ip in the same machine, and then fallback to leastconn if a server is too loaded. this is basically the same algorithm than *source* but for destination. is there any way to achieve this in haproxy? It depends on how many regions you have. If you only have a small set of regions, you can already achieve this using ACLs on the destination address, using which you will select a backend. But this will quickly become very boring when you have more than 4-5 regions. Another possibility is to implement the balance dstip algo which should be pretty trivial. If you're deploying a new site, I think it makes a lot of sense to adapt the tools to do the job right. Do you feel OK with adding this feature yourself ? I could then merge it back into next release, as I think it will not be an invasive change at all. Regards, Willy
Re: tcp proxy
On Sat, Apr 04, 2009 at 11:43:38AM -0300, Nicolas Cohen wrote: Hi Willy, It seems right to implement it. I'll review this with the team and let you know once we have an available patch. Nice, thanks! Willy
