Re: [HACKERS] libpq compression

2012-08-30 Thread Bruce Momjian
On Sun, Jun 17, 2012 at 11:45:54PM +0800, Magnus Hagander wrote: On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need one?  

Re: [HACKERS] libpq compression

2012-07-13 Thread Magnus Hagander
On Mon, Jun 25, 2012 at 2:26 PM, Magnus Hagander mag...@hagander.net wrote: On Mon, Jun 25, 2012 at 4:04 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira eu...@timbira.com wrote: On 20-06-2012 17:40, Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05

Re: [HACKERS] libpq compression

2012-06-26 Thread Robert Haas
On Mon, Jun 25, 2012 at 4:25 PM, k...@rice.edu k...@rice.edu wrote: On Mon, Jun 25, 2012 at 09:45:26PM +0200, Florian Pflug wrote: On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote: Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're

Re: [HACKERS] libpq compression

2012-06-26 Thread Euler Taveira
On 26-06-2012 12:23, Robert Haas wrote: At the risk of making everyone laugh at me, has anyone tested pglz? I observe that if the compression ration and performance are good, we might consider using it for this purpose, too, which would avoid having to add dependencies. Conversely, if they

Re: [HACKERS] libpq compression

2012-06-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Jun 25, 2012 at 4:25 PM, k...@rice.edu k...@rice.edu wrote: Here is the benchmark list from the Google lz4 page: NameRatio C.speed D.speed LZ4 (r59) 2.084 330 915 LZO 2.05 1x_1 2.038 311 480 QuickLZ 1.5

Re: [HACKERS] libpq compression

2012-06-25 Thread Magnus Hagander
On Mon, Jun 25, 2012 at 4:04 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira eu...@timbira.com wrote: On 20-06-2012 17:40, Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on

Re: [HACKERS] libpq compression

2012-06-25 Thread Florian Pflug
On Jun25, 2012, at 04:04 , Robert Haas wrote: If, for example, someone can demonstrate that an awesomebsdlz compresses 10x as fast as OpenSSL... that'd be pretty compelling. That, actually, is demonstrably the case for at least Google's snappy. (and LZO, but that's not an option since its

Re: [HACKERS] libpq compression

2012-06-25 Thread k...@rice.edu
On Mon, Jun 25, 2012 at 03:12:46PM +0200, Florian Pflug wrote: On Jun25, 2012, at 04:04 , Robert Haas wrote: If, for example, someone can demonstrate that an awesomebsdlz compresses 10x as fast as OpenSSL... that'd be pretty compelling. That, actually, is demonstrably the case for at

Re: [HACKERS] libpq compression

2012-06-25 Thread Euler Taveira
On 24-06-2012 23:04, Robert Haas wrote: So I think we really need someone to try this both ways and compare. Right now it seems like we're mostly speculating on how well either approach would work, which is not as good as having some experimental results. Not a problem. That's what I'm

Re: [HACKERS] libpq compression

2012-06-25 Thread Greg Jaskiewicz
Wasn't this more of an issue in de-coupling compression from encryption ? On 25 Jun 2012, at 16:36, Euler Taveira wrote: On 24-06-2012 23:04, Robert Haas wrote: So I think we really need someone to try this both ways and compare. Right now it seems like we're mostly speculating on how

Re: [HACKERS] libpq compression

2012-06-25 Thread Dimitri Fontaine
Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're talking about some LZO things such as snappy from google, and that would be another run time dependency IIUC. I think it's time to talk about fastlz: http://fastlz.org/

Re: [HACKERS] libpq compression

2012-06-25 Thread Florian Pflug
On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote: Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're talking about some LZO things such as snappy from google, and that would be another run time dependency IIUC. I think it's time to

Re: [HACKERS] libpq compression

2012-06-25 Thread Phil Sorber
On Mon, Jun 25, 2012 at 3:45 PM, Florian Pflug f...@phlo.org wrote: On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote: Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're talking about some LZO things such as snappy from google, and that

Re: [HACKERS] libpq compression

2012-06-25 Thread k...@rice.edu
On Mon, Jun 25, 2012 at 09:45:26PM +0200, Florian Pflug wrote: On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote: Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're talking about some LZO things such as snappy from google, and that

Re: [HACKERS] libpq compression

2012-06-25 Thread Euler Taveira
On 25-06-2012 16:45, Florian Pflug wrote: Agreed. If we extend the protocol to support compression, and not rely on SSL, then let's pick one of these LZ77-style compressors, and let's integrate it into our tree. If we have an option to have it out of our tree, good; if not, let's integrate

Re: [HACKERS] libpq compression

2012-06-25 Thread Euler Taveira
On 25-06-2012 14:30, Greg Jaskiewicz wrote: Wasn't this more of an issue in de-coupling compression from encryption ? Let me give a try to take some conclusion. If we decide for an independent compression code instead of an SSL-based one, that is a possibility to be tested: SSL + SSL-based

Re: [HACKERS] libpq compression

2012-06-25 Thread Merlin Moncure
On Mon, Jun 25, 2012 at 3:38 PM, Euler Taveira eu...@timbira.com wrote: On 25-06-2012 16:45, Florian Pflug wrote: Agreed. If we extend the protocol to support compression, and not rely on SSL, then let's pick one of these LZ77-style compressors, and let's integrate it into our tree. If we

Re: [HACKERS] libpq compression

2012-06-24 Thread Robert Haas
On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira eu...@timbira.com wrote: On 20-06-2012 17:40, Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a

Re: [HACKERS] libpq compression

2012-06-22 Thread Euler Taveira
On 20-06-2012 17:40, Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly

Re: [HACKERS] libpq compression

2012-06-20 Thread Florian Pflug
On Jun19, 2012, at 17:36 , Robert Haas wrote: On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout klep...@svana.org wrote: On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of these ideas is that the configure list is not a list of exact cipher

Re: [HACKERS] libpq compression

2012-06-20 Thread Alvaro Herrera
Excerpts from Florian Pflug's message of mié jun 20 06:35:29 -0400 2012: On Jun19, 2012, at 17:36 , Robert Haas wrote: On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout klep...@svana.org wrote: On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with

Re: [HACKERS] libpq compression

2012-06-20 Thread Tom Lane
Florian Pflug f...@phlo.org writes: I wonder though if shouldn't restrict the allowed ciphers list to being a simple list of supported ciphers. If our goal is to support multiple SSL libraries transparently then surely having openssl-specific syntax in the config file isn't exactly great

Re: [HACKERS] libpq compression

2012-06-20 Thread Florian Pflug
On Jun20, 2012, at 17:34 , Tom Lane wrote: Florian Pflug f...@phlo.org writes: I wonder though if shouldn't restrict the allowed ciphers list to being a simple list of supported ciphers. If our goal is to support multiple SSL libraries transparently then surely having openssl-specific syntax

Re: [HACKERS] libpq compression

2012-06-20 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain the list of ciphers starting from the stringified configuration parameter and iterate on them. Do you mean that it will produce an expansion of the set of ciphers meeting

Re: [HACKERS] libpq compression

2012-06-20 Thread Alvaro Herrera
Excerpts from Tom Lane's message of mié jun 20 11:49:51 -0400 2012: Alvaro Herrera alvhe...@commandprompt.com writes: I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain the list of ciphers starting from the stringified configuration parameter and iterate on them.

Re: [HACKERS] libpq compression

2012-06-20 Thread Magnus Hagander
On Wed, Jun 20, 2012 at 12:35 PM, Florian Pflug f...@phlo.org wrote: On Jun19, 2012, at 17:36 , Robert Haas wrote: On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout klep...@svana.org wrote: On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of

Re: [HACKERS] libpq compression

2012-06-20 Thread Florian Pflug
On Jun20, 2012, at 18:42 , Magnus Hagander wrote: That is a very good point. Before we design *another* feature that relies on it, we should verify if the syntax is compatible in the other libraries that would be interesting (gnutls and NSS primarily), and if it's not that at least the

Re: [HACKERS] libpq compression

2012-06-20 Thread Marko Kreen
On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly never was intended to be used... Maybe,

Re: [HACKERS] libpq compression

2012-06-20 Thread Florian Pflug
On Jun20, 2012, at 22:40 , Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite

Re: [HACKERS] libpq compression

2012-06-19 Thread Robert Haas
On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout klep...@svana.org wrote: On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of these ideas is that the configure list is not a list of exact cipher names, as per Magnus' comment that the current

Re: [HACKERS] libpq compression

2012-06-18 Thread Martijn van Oosterhout
On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of these ideas is that the configure list is not a list of exact cipher names, as per Magnus' comment that the current default includes tests like !aNULL. I am not sure that we know how to evaluate such

Re: [HACKERS] libpq compression

2012-06-17 Thread Magnus Hagander
On Sat, Jun 16, 2012 at 11:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted

Re: [HACKERS] libpq compression

2012-06-17 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need one? I am not sure what the cipher negotiation process looks like or which side has the freedom to choose. That, or just have DEFAULT as being

Re: [HACKERS] libpq compression

2012-06-17 Thread Magnus Hagander
On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need one?  I am not sure what the cipher negotiation process looks like or which

Re: [HACKERS] libpq compression

2012-06-17 Thread Dave Page
On Sun, Jun 17, 2012 at 4:45 PM, Magnus Hagander mag...@hagander.net wrote: On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need

Re: [HACKERS] libpq compression

2012-06-17 Thread Florian Pflug
On Jun16, 2012, at 17:15 , Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted SSL connection probably

Re: [HACKERS] libpq compression

2012-06-17 Thread Tom Lane
Florian Pflug f...@phlo.org writes: Would we still tell openssl to only negotiate ciphers in the configured list of available ciphers + NULL? If not, what happens if a connection happens to use a cipher that is actually stronger than any cipher on the list of acceptable ciphers list? The DBA

Re: [HACKERS] libpq compression

2012-06-17 Thread Euler Taveira
On 17-06-2012 12:42, Tom Lane wrote: If our default isn't the same as the underlying default, I have to question why not. But are you sure this ! notation will work with all openssl versions? What is all for you? It seems we don't claim support for an specific version or later in docs or

Re: [HACKERS] libpq compression

2012-06-17 Thread Euler Taveira
On 17-06-2012 12:45, Magnus Hagander wrote: Uh. We have the ! notation in our default *now*. What openssl also supports is the text DEFAULT, which is currently the equivalent of ALL!aNULL!eNULL. The question, which is valid of course, should be if DEFAULT works with all openssl versions.

Re: [HACKERS] libpq compression

2012-06-17 Thread Euler Taveira
On 17-06-2012 12:45, Magnus Hagander wrote: On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need one? I am not sure what the

Re: [HACKERS] libpq compression

2012-06-16 Thread Magnus Hagander
On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Yes, but there's also a lot of such awkward logic we need to add if we *do* go with the SSL library doing the compression: For example, we can no longer trust the SSL library to

Re: [HACKERS] libpq compression

2012-06-16 Thread Magnus Hagander
On Sat, Jun 16, 2012 at 12:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Marko Kreen mark...@gmail.com writes: On Sat, Jun 16, 2012 at 6:39 AM, Magnus Hagander mag...@hagander.net wrote: Fair enough if we decide that - but we should make that decision knowing that we're leaving the JDBC and .Net

Re: [HACKERS] libpq compression

2012-06-16 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted SSL connection probably shouldn't be treated as matching a hostssl

Re: [HACKERS] libpq compression

2012-06-16 Thread k...@rice.edu
On Sat, Jun 16, 2012 at 11:15:30AM -0400, Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted SSL

Re: [HACKERS] libpq compression

2012-06-15 Thread Florian Pflug
On Jun15, 2012, at 07:50 , Magnus Hagander wrote: So I've got very little patience with the idea of let's put in some hooks and then great things will happen. It would be far better all around if we supported exactly one, well-chosen, method. But really I still don't see a reason not to let

Re: [HACKERS] libpq compression

2012-06-15 Thread Magnus Hagander
On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus Hagander wrote: So I've got very little patience with the idea of let's put in some hooks and then great things will happen.  It would be far better all around if we supported exactly one,

Re: [HACKERS] libpq compression

2012-06-15 Thread Florian Pflug
On Jun15, 2012, at 12:09 , Magnus Hagander wrote: On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus Hagander wrote: Second, we also have things like the JDBC driver and the .Net driver that don't use libpq. the JDBC driver uses the native

Re: [HACKERS] libpq compression

2012-06-15 Thread Merlin Moncure
On Fri, Jun 15, 2012 at 5:48 AM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 12:09 , Magnus Hagander wrote: On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus Hagander wrote: Second, we also have things like the JDBC driver and the

Re: [HACKERS] libpq compression

2012-06-15 Thread k...@rice.edu
On Fri, Jun 15, 2012 at 07:18:34AM -0500, Merlin Moncure wrote: On Fri, Jun 15, 2012 at 5:48 AM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 12:09 , Magnus Hagander wrote: On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus

Re: [HACKERS] libpq compression

2012-06-15 Thread Magnus Hagander
On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 12:09 , Magnus Hagander wrote: On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus Hagander wrote: Second, we also have things like the JDBC driver and the

Re: [HACKERS] libpq compression

2012-06-15 Thread Heikki Linnakangas
On 15.06.2012 17:39, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflugf...@phlo.org wrote: The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports

Re: [HACKERS] libpq compression

2012-06-15 Thread Magnus Hagander
On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:39, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflugf...@phlo.org  wrote: The way I see it, if we use SSL-based compression then non-libpq clients there's at

Re: [HACKERS] libpq compression

2012-06-15 Thread Heikki Linnakangas
On 15.06.2012 17:58, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:39, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflugf...@phlo.orgwrote: The way I see it, if we use

Re: [HACKERS] libpq compression

2012-06-15 Thread Magnus Hagander
On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:58, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: On 15.06.2012 17:39, Magnus Hagander wrote: On Fri,

Re: [HACKERS] libpq compression

2012-06-15 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Umm, then you use a real SSL libray, not the dummy one? But (in this scenario, and so far nobody has proven it to be wrong) there exists no real SSL

Re: [HACKERS] libpq compression

2012-06-15 Thread Ryan Kelly
On Fri, Jun 15, 2012 at 11:28:48PM +0800, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:58, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas

Re: [HACKERS] libpq compression

2012-06-15 Thread Heikki Linnakangas
On 15.06.2012 18:28, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:58, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.comwrote: You

Re: [HACKERS] libpq compression

2012-06-15 Thread Euler Taveira
On 15-06-2012 11:39, Magnus Hagander wrote: As long as a free implementation exists, it can be ported to Java/.Net. Sure, it takes more work, but it *can be done*. Good point. IMHO, if there isn't a solution that cover all PostgreSQL (it seems it is not), we should pick the most appropriate

Re: [HACKERS] libpq compression

2012-06-15 Thread Euler Taveira
On 15-06-2012 11:10, k...@rice.edu wrote: I agree and think that the SSL-based compression is an excellent default compression scheme. The plugable compression approach allows for the choice of the most appropriate compression implementation based on the application needs. It really addresses

Re: [HACKERS] libpq compression

2012-06-15 Thread Tom Lane
Euler Taveira eu...@timbira.com writes: I see the point in not adding another dependencies or reinventing the wheel but I see more drawbacks than benefits in adopting a SSL-based compression. In the end, judging this tradeoff is a matter of opinion, but I come to the opposite conclusion.

Re: [HACKERS] libpq compression

2012-06-15 Thread Tom Lane
I wrote: Euler Taveira eu...@timbira.com writes: I see the point in not adding another dependencies or reinventing the wheel but I see more drawbacks than benefits in adopting a SSL-based compression. In the end, judging this tradeoff is a matter of opinion, but I come to the opposite

Re: [HACKERS] libpq compression

2012-06-15 Thread Bruce Momjian
On Fri, Jun 15, 2012 at 12:48:24PM +0200, Florian Pflug wrote: Yeah, but that alone is IMO a rather big blocker for claiming that this is the only way to do it :( And I think the fact that that wikipedia page doesn't list any other ones, is a sign that there might not be a lot of other

Re: [HACKERS] libpq compression

2012-06-15 Thread Magnus Hagander
On Sat, Jun 16, 2012 at 12:03 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 18:28, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: On 15.06.2012 17:58, Magnus Hagander wrote: On Fri,

Re: [HACKERS] libpq compression

2012-06-15 Thread Magnus Hagander
On Sat, Jun 16, 2012 at 4:04 AM, Euler Taveira eu...@timbira.com wrote: On 15-06-2012 11:39, Magnus Hagander wrote: As long as a free implementation exists, it can be ported to Java/.Net. Sure, it takes more work, but it *can be done*. Good point. IMHO, if there isn't a solution that cover

Re: [HACKERS] libpq compression

2012-06-15 Thread Magnus Hagander
On Sat, Jun 16, 2012 at 6:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Euler Taveira eu...@timbira.com writes: I see the point in not adding another dependencies or reinventing the wheel but I see more drawbacks than benefits in adopting a SSL-based compression. In the end, judging

Re: [HACKERS] libpq compression

2012-06-15 Thread Euler Taveira
On 16-06-2012 00:43, Magnus Hagander wrote: For example, we can no longer trust the SSL library to always do encryption, since we specifically want to support null encryption. Meaning we need to teach pg_hba to treat a connection with null encryption as hostnossl, even if it's an

Re: [HACKERS] libpq compression

2012-06-15 Thread Marko Kreen
On Sat, Jun 16, 2012 at 6:39 AM, Magnus Hagander mag...@hagander.net wrote: On Sat, Jun 16, 2012 at 4:04 AM, Euler Taveira eu...@timbira.com wrote: On 15-06-2012 11:39, Magnus Hagander wrote: As long as a free implementation exists, it can be ported to Java/.Net. Sure, it takes more work, but

Re: [HACKERS] libpq compression

2012-06-15 Thread Tom Lane
Marko Kreen mark...@gmail.com writes: On Sat, Jun 16, 2012 at 6:39 AM, Magnus Hagander mag...@hagander.net wrote: Fair enough if we decide that - but we should make that decision knowing that we're leaving the JDBC and .Net people in a bad position where they are not likely to be able to

Re: [HACKERS] libpq compression

2012-06-15 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: Yes, but there's also a lot of such awkward logic we need to add if we *do* go with the SSL library doing the compression: For example, we can no longer trust the SSL library to always do encryption, since we specifically want to support null

Re: [HACKERS] libpq compression

2012-06-14 Thread Albe Laurenz
Euler Taveira wrote: There was already some discussion about compressing libpq data [1][2][3]. Recently, I faced a scenario that would become less problematic if we have had compression support. The scenario is frequent data load (aka COPY) over slow/unstable links. It should be executed in a

Re: [HACKERS] libpq compression

2012-06-14 Thread Euler Taveira
On 14-06-2012 02:19, Tom Lane wrote: I still think that pushing this off to openssl (not an ssh tunnel, but the underlying transport library) would be an adequate solution. If you are shoving data over a connection that is long enough to need compression, the odds that every bit of it is

Re: [HACKERS] libpq compression

2012-06-14 Thread Florian Pflug
On Jun14, 2012, at 15:28 , Euler Taveira wrote: On 14-06-2012 02:19, Tom Lane wrote: I still think that pushing this off to openssl (not an ssh tunnel, but the underlying transport library) would be an adequate solution. If you are shoving data over a connection that is long enough to need

Re: [HACKERS] libpq compression

2012-06-14 Thread Phil Sorber
On Thu, Jun 14, 2012 at 10:14 AM, Florian Pflug f...@phlo.org wrote: On Jun14, 2012, at 15:28 , Euler Taveira wrote: On 14-06-2012 02:19, Tom Lane wrote: I still think that pushing this off to openssl (not an ssh tunnel, but the underlying transport library) would be an adequate solution. If

Re: [HACKERS] libpq compression

2012-06-14 Thread Merlin Moncure
On Thu, Jun 14, 2012 at 9:57 AM, Phil Sorber p...@omniti.com wrote: It doesn't sound like there is a lot of support for this idea, but I think it would be nice to get something like lz4 (http://code.google.com/p/lz4/) or snappy (http://code.google.com/p/snappy/) support. Both are BSD-ish

Re: [HACKERS] libpq compression

2012-06-14 Thread Tom Lane
Euler Taveira eu...@timbira.com writes: On 14-06-2012 02:19, Tom Lane wrote: ... I especially think that importing bzip2 is a pretty bad idea --- it's not only a new dependency, but bzip2's compression versus speed tradeoff is entirely inappropriate for this use-case. I see, the idea is

Re: [HACKERS] libpq compression

2012-06-14 Thread Merlin Moncure
On Thu, Jun 14, 2012 at 1:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: So I've got very little patience with the idea of let's put in some hooks and then great things will happen.  It would be far better all around if we supported exactly one, well-chosen, method.  But really I still don't see a

Re: [HACKERS] libpq compression

2012-06-14 Thread k...@rice.edu
On Thu, Jun 14, 2012 at 02:38:02PM -0500, Merlin Moncure wrote: On Thu, Jun 14, 2012 at 1:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: So I've got very little patience with the idea of let's put in some hooks and then great things will happen.  It would be far better all around if we supported

Re: [HACKERS] libpq compression

2012-06-14 Thread Bruce Momjian
On Thu, Jun 14, 2012 at 02:43:04PM -0400, Tom Lane wrote: Euler Taveira eu...@timbira.com writes: On 14-06-2012 02:19, Tom Lane wrote: ... I especially think that importing bzip2 is a pretty bad idea --- it's not only a new dependency, but bzip2's compression versus speed tradeoff is

Re: [HACKERS] libpq compression

2012-06-14 Thread Magnus Hagander
On Fri, Jun 15, 2012 at 10:19 AM, Bruce Momjian br...@momjian.us wrote: On Thu, Jun 14, 2012 at 02:43:04PM -0400, Tom Lane wrote: Euler Taveira eu...@timbira.com writes: On 14-06-2012 02:19, Tom Lane wrote: ...  I especially think that importing bzip2 is a pretty bad idea --- it's not only

Re: [HACKERS] libpq compression

2012-06-13 Thread Tom Lane
Euler Taveira eu...@timbira.com writes: There was already some discussion about compressing libpq data [1][2][3]. Recently, I faced a scenario that would become less problematic if we have had compression support. The scenario is frequent data load (aka COPY) over slow/unstable links. It