Re: [HACKERS] libpq compression
On Sun, Jun 17, 2012 at 11:45:54PM +0800, Magnus Hagander wrote: On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need one? I am not sure what the cipher negotiation process looks like or which side has the freedom to choose. I haven't looked into the details, but it seems reasonable that *either* side should be able to at least define a list of ciphers it *doens't* want to talk with. Do we need it - well, it makes sense for the client to be able to say I won't trust 56-bit encryption before it sends over the password, imo.. That, or just have DEFAULT as being the default (which in current openssl means ALL:!aNULL:!eNULL. If our default isn't the same as the underlying default, I have to question why not. Yeah, that's exaclty what I'm questioning here.. But are you sure this ! notation will work with all openssl versions? Uh. We have the ! notation in our default *now*. What openssl also supports is the text DEFAULT, which is currently the equivalent of ALL!aNULL!eNULL. The question, which is valid of course, should be if DEFAULT works with all openssl versions. It would seem reasonable it does, but I haven't investigated. Do we want to change our ssl_ciphers default to 'DEFAULT'? Currently it is 'ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH'. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Mon, Jun 25, 2012 at 2:26 PM, Magnus Hagander mag...@hagander.net wrote: On Mon, Jun 25, 2012 at 4:04 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira eu...@timbira.com wrote: On 20-06-2012 17:40, Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly never was intended to be used... Maybe, but what is the argument that we should avoid on encryption+compression at the same time? AES is quite lightweight compared to compression, so should be no problem in situations where you care about compression. If we could solve compression problem without AES that will turn things easier. Compression-only via encryption is a weird manner to solve the problem in the user's POV. RSA is noticeable, but only for short connections. Thus easily solvable with connection pooling. RSA overhead is not the main problem. SSL/TLS setup is. And for really special compression needs you can always create a UDF that does custom compression for you. You have to own the code to modify it; it is not always an option. So what exactly is the situation we need to solve with postgres-specific protocol compression? Compression only support. Why do I need to set up SSL/TLS just for compression? IMHO SSL/TLS use is no different from relying in another library to handle compression for the protocol and more it is compression-specific. That way, we could implement another algorithms in such library without needing to modify libpq code. Using SSL/TLS you are bounded by what SSL/TLS software products decide to use as compression algorithms. I'll be happy to maintain the code iif it is postgres-specific or even as close as possible to core. I guess my feeling on this is that, so far as I can see, supporting compression via OpenSSL involves work and trade-offs, and supporting it without depending on OpenSSL also involves work, and trade-offs. Nice summary :) So it's not real evident to me that we should prefer one to the other on general principle. It seems to me that a lot might come down to performance. If someone can demonstrate that using an external library involves gets significantly better compression, chews up significantly less CPU time, and/or is significantly less code than supporting this via OpenSSL, then maybe we ought to consider it. I think we should, yes. But as you say, we need to know first. It's also a question of if one of these compression schemes are trivial enough that we could embed the code rather than rely on it externally - I have no idea if that's even remotely possibe, but that would move the goalposts a bit too. A followup on this thread. I was researching something else, and stumbled across http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html, which says: The TLS standard (or SSLv3) allows the integration of compression methods into the communication. The TLS RFC does however not specify compression methods or their corresponding identifiers, so there is currently no compatible way to integrate compression with unknown peers. It is therefore currently not recommended to integrate compression into applications. Applications for non-public use may agree on certain compression methods. Using different compression methods with the same identifier will lead to connection failure. Which I think is another reason to not go down that path as our official way to do compression. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Mon, Jun 25, 2012 at 4:25 PM, k...@rice.edu k...@rice.edu wrote: On Mon, Jun 25, 2012 at 09:45:26PM +0200, Florian Pflug wrote: On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote: Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're talking about some LZO things such as snappy from google, and that would be another run time dependency IIUC. I think it's time to talk about fastlz: http://fastlz.org/ http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c 551 lines of C code under MIT license, works also under windows I guess it would be easy (and safe) enough to embed in our tree should we decide going this way. Agreed. If we extend the protocol to support compression, and not rely on SSL, then let's pick one of these LZ77-style compressors, and let's integrate it into our tree. We should then also make it possible to enable compression only for the server - client direction. Since those types of compressions are usually pretty easy to decompress, that reduces the amount to work non-libpq clients have to put in to take advantage of compression. Here is the benchmark list from the Google lz4 page: Name Ratio C.speed D.speed LZ4 (r59) 2.084 330 915 LZO 2.05 1x_1 2.038 311 480 QuickLZ 1.5 -1 2.233 257 277 Snappy 1.0.5 2.024 227 729 LZF 2.076 197 465 FastLZ 2.030 190 420 zlib 1.2.5 -1 2.728 39 195 LZ4 HC (r66) 2.712 18 1020 zlib 1.2.5 -6 3.095 14 210 lz4 absolutely dominates on compression/decompression speed. While fastlz is faster than zlib(-1) on compression, lz4 is almost 2X faster still. At the risk of making everyone laugh at me, has anyone tested pglz? I observe that if the compression ration and performance are good, we might consider using it for this purpose, too, which would avoid having to add dependencies. Conversely, if they are bad, and we decide to support another algorithm, we might consider also using that other algorithm, at least optionally, for the purposes for which we now use pglz. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 26-06-2012 12:23, Robert Haas wrote: At the risk of making everyone laugh at me, has anyone tested pglz? I observe that if the compression ration and performance are good, we might consider using it for this purpose, too, which would avoid having to add dependencies. Conversely, if they are bad, and we decide to support another algorithm, we might consider also using that other algorithm, at least optionally, for the purposes for which we now use pglz. I'll remember to test it too. -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Robert Haas robertmh...@gmail.com writes: On Mon, Jun 25, 2012 at 4:25 PM, k...@rice.edu k...@rice.edu wrote: Here is the benchmark list from the Google lz4 page: NameRatio C.speed D.speed LZ4 (r59) 2.084 330 915 LZO 2.05 1x_1 2.038 311 480 QuickLZ 1.5 -1 2.233 257 277 Snappy 1.0.52.024 227 729 LZF 2.076 197 465 FastLZ 2.030 190 420 zlib 1.2.5 -1 2.72839 195 LZ4 HC (r66)2.71218 1020 zlib 1.2.5 -6 3.09514 210 lz4 absolutely dominates on compression/decompression speed. While fastlz is faster than zlib(-1) on compression, lz4 is almost 2X faster still. At the risk of making everyone laugh at me, has anyone tested pglz? Another point here is that those Google numbers (assuming that they apply to our use-cases, a point not in evidence) utterly fail to make the argument that zlib is not the thing to use. zlib is beating all the others on compression ratio, often by 50%. Before making any comparisons, you have to make some assumptions about what the network speed is ... and unless it's pretty damn fast relative to your CPU speed getting the better compression ratio is going to be more attractive than saving some cycles. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Mon, Jun 25, 2012 at 4:04 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira eu...@timbira.com wrote: On 20-06-2012 17:40, Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly never was intended to be used... Maybe, but what is the argument that we should avoid on encryption+compression at the same time? AES is quite lightweight compared to compression, so should be no problem in situations where you care about compression. If we could solve compression problem without AES that will turn things easier. Compression-only via encryption is a weird manner to solve the problem in the user's POV. RSA is noticeable, but only for short connections. Thus easily solvable with connection pooling. RSA overhead is not the main problem. SSL/TLS setup is. And for really special compression needs you can always create a UDF that does custom compression for you. You have to own the code to modify it; it is not always an option. So what exactly is the situation we need to solve with postgres-specific protocol compression? Compression only support. Why do I need to set up SSL/TLS just for compression? IMHO SSL/TLS use is no different from relying in another library to handle compression for the protocol and more it is compression-specific. That way, we could implement another algorithms in such library without needing to modify libpq code. Using SSL/TLS you are bounded by what SSL/TLS software products decide to use as compression algorithms. I'll be happy to maintain the code iif it is postgres-specific or even as close as possible to core. I guess my feeling on this is that, so far as I can see, supporting compression via OpenSSL involves work and trade-offs, and supporting it without depending on OpenSSL also involves work, and trade-offs. Nice summary :) So it's not real evident to me that we should prefer one to the other on general principle. It seems to me that a lot might come down to performance. If someone can demonstrate that using an external library involves gets significantly better compression, chews up significantly less CPU time, and/or is significantly less code than supporting this via OpenSSL, then maybe we ought to consider it. I think we should, yes. But as you say, we need to know first. It's also a question of if one of these compression schemes are trivial enough that we could embed the code rather than rely on it externally - I have no idea if that's even remotely possibe, but that would move the goalposts a bit too. OpenSSL is kind of an ugly piece of code, and all things being equal depending on it more heavily would not be my first choice. Indeed. But we should really stop saying rely on openssl and start saying rely on the ssl library. There are other SSL libraries which are not quite so ugly, which we should eventually support. That said, it's *still* a bit ugly to rely on the SSL library for this, IMO. On the other hand, this does not seem to me to be a situation where we should accept a patch to use an external library just because someone takes the time to write one, because there is also a non-trivial cost for the entire project to depending on more things; or if the compression code gets put into the backend, then there's a cost to us to maintain that code inside our source base. So I think we really need someone to try this both ways and compare. Right now it seems like we're mostly speculating on how well either approach would work, which is not as good as having some experimental results. If, for example, someone can demonstrate that an awesomebsdlz compresses 10x as fast as OpenSSL... that'd be pretty compelling. Or that it takes less code/generates cleaner code... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun25, 2012, at 04:04 , Robert Haas wrote: If, for example, someone can demonstrate that an awesomebsdlz compresses 10x as fast as OpenSSL... that'd be pretty compelling. That, actually, is demonstrably the case for at least Google's snappy. (and LZO, but that's not an option since its license is GPL) They state in their documentation that In our tests, Snappy usually is faster than algorithms in the same class (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression ratios. The only widely supported compression method for SSL seems to be DEFLATE, which is also what gzip/zlib uses. I've benchmarked LZO against gzip/zlib a few months ago, and LZO outperformed zlib in fast mode (i.e. gzip -1) by an order of magnitude. The compression ratio achieved by DEFLATE/gzip/zlib is much better, though. The snappy documentation states Typical compression ratios (based on the benchmark suite) are about 1.5-1.7x for plain text, about 2-4x for HTML, and of course 1.0x for JPEGs, PNGs and other already-compressed data. Similar numbers for zlib in its fastest mode are 2.6-2.8x, 3-7x and 1.0x, respectively. Here are a few numbers for LZO vs. gzip. Snappy should be comparable to LZO - I tested LZO because I still had the command-line compressor lzop lying around on my machine, whereas I'd have needed to download and compile snappy first. $ dd if=/dev/random of=data bs=1m count=128 $ time gzip -1 data data.gz real0m6.189s user0m5.947s sys 0m0.224s $ time lzop data data.lzo real0m2.697s user0m0.295s sys 0m0.224s $ ls -lh data* -rw-r--r-- 1 fgp staff 128M Jun 25 14:43 data -rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.gz -rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.lzo $ dd if=/dev/zero of=zeros bs=1m count=128 $ time gzip -1 zeros zeros.gz real0m1.083s user0m1.019s sys 0m0.052s $ time lzop zeros zeros.lzo real0m0.186s user0m0.123s sys 0m0.053s $ ls -lh zeros* -rw-r--r-- 1 fgp staff 128M Jun 25 14:47 zeros -rw-r--r-- 1 fgp staff 572K Jun 25 14:47 zeros.gz -rw-r--r-- 1 fgp staff 598K Jun 25 14:47 zeros.lzo To summarize, on my 2.66 Ghz Core2 Duo Macbook Pro, LZO compresses about 350MB/s if the data is purely random, and about 800MB/s if the data compresses extremely well. (Numbers based on user time since that indicates the CPU time used, and ignores the IO overhead, which is substantial) IMHO, the only compelling argument (and a very compelling one) to use SSL compression was that it requires very little code on our side. We've since discovered that it's not actually that simple, at least if we want to support compression without authentication or encryption, and don't want to restrict ourselves to using OpenSSL forever. So unless we give up at least one of those requirements, the arguments for using SSL-compression are rather thin, I think. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Mon, Jun 25, 2012 at 03:12:46PM +0200, Florian Pflug wrote: On Jun25, 2012, at 04:04 , Robert Haas wrote: If, for example, someone can demonstrate that an awesomebsdlz compresses 10x as fast as OpenSSL... that'd be pretty compelling. That, actually, is demonstrably the case for at least Google's snappy. (and LZO, but that's not an option since its license is GPL) They state in their documentation that In our tests, Snappy usually is faster than algorithms in the same class (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression ratios. The only widely supported compression method for SSL seems to be DEFLATE, which is also what gzip/zlib uses. I've benchmarked LZO against gzip/zlib a few months ago, and LZO outperformed zlib in fast mode (i.e. gzip -1) by an order of magnitude. The compression ratio achieved by DEFLATE/gzip/zlib is much better, though. The snappy documentation states Typical compression ratios (based on the benchmark suite) are about 1.5-1.7x for plain text, about 2-4x for HTML, and of course 1.0x for JPEGs, PNGs and other already-compressed data. Similar numbers for zlib in its fastest mode are 2.6-2.8x, 3-7x and 1.0x, respectively. Here are a few numbers for LZO vs. gzip. Snappy should be comparable to LZO - I tested LZO because I still had the command-line compressor lzop lying around on my machine, whereas I'd have needed to download and compile snappy first. $ dd if=/dev/random of=data bs=1m count=128 $ time gzip -1 data data.gz real 0m6.189s user 0m5.947s sys 0m0.224s $ time lzop data data.lzo real 0m2.697s user 0m0.295s sys 0m0.224s $ ls -lh data* -rw-r--r-- 1 fgp staff 128M Jun 25 14:43 data -rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.gz -rw-r--r-- 1 fgp staff 128M Jun 25 14:44 data.lzo $ dd if=/dev/zero of=zeros bs=1m count=128 $ time gzip -1 zeros zeros.gz real 0m1.083s user 0m1.019s sys 0m0.052s $ time lzop zeros zeros.lzo real 0m0.186s user 0m0.123s sys 0m0.053s $ ls -lh zeros* -rw-r--r-- 1 fgp staff 128M Jun 25 14:47 zeros -rw-r--r-- 1 fgp staff 572K Jun 25 14:47 zeros.gz -rw-r--r-- 1 fgp staff 598K Jun 25 14:47 zeros.lzo To summarize, on my 2.66 Ghz Core2 Duo Macbook Pro, LZO compresses about 350MB/s if the data is purely random, and about 800MB/s if the data compresses extremely well. (Numbers based on user time since that indicates the CPU time used, and ignores the IO overhead, which is substantial) IMHO, the only compelling argument (and a very compelling one) to use SSL compression was that it requires very little code on our side. We've since discovered that it's not actually that simple, at least if we want to support compression without authentication or encryption, and don't want to restrict ourselves to using OpenSSL forever. So unless we give up at least one of those requirements, the arguments for using SSL-compression are rather thin, I think. best regards, Florian Pflug +1 for http://code.google.com/p/lz4/ support. It has a BSD license too. Using SSL libraries give all the complexity without any real benefit. Regards, Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 24-06-2012 23:04, Robert Haas wrote: So I think we really need someone to try this both ways and compare. Right now it seems like we're mostly speculating on how well either approach would work, which is not as good as having some experimental results. Not a problem. That's what I'm thinking too but I would like to make sure that others don't object to general idea. Let me give it a try in both ideas... -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Wasn't this more of an issue in de-coupling compression from encryption ? On 25 Jun 2012, at 16:36, Euler Taveira wrote: On 24-06-2012 23:04, Robert Haas wrote: So I think we really need someone to try this both ways and compare. Right now it seems like we're mostly speculating on how well either approach would work, which is not as good as having some experimental results. Not a problem. That's what I'm thinking too but I would like to make sure that others don't object to general idea. Let me give it a try in both ideas... -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're talking about some LZO things such as snappy from google, and that would be another run time dependency IIUC. I think it's time to talk about fastlz: http://fastlz.org/ http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c 551 lines of C code under MIT license, works also under windows I guess it would be easy (and safe) enough to embed in our tree should we decide going this way. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote: Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're talking about some LZO things such as snappy from google, and that would be another run time dependency IIUC. I think it's time to talk about fastlz: http://fastlz.org/ http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c 551 lines of C code under MIT license, works also under windows I guess it would be easy (and safe) enough to embed in our tree should we decide going this way. Agreed. If we extend the protocol to support compression, and not rely on SSL, then let's pick one of these LZ77-style compressors, and let's integrate it into our tree. We should then also make it possible to enable compression only for the server - client direction. Since those types of compressions are usually pretty easy to decompress, that reduces the amount to work non-libpq clients have to put in to take advantage of compression. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Mon, Jun 25, 2012 at 3:45 PM, Florian Pflug f...@phlo.org wrote: On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote: Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're talking about some LZO things such as snappy from google, and that would be another run time dependency IIUC. I think it's time to talk about fastlz: http://fastlz.org/ http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c 551 lines of C code under MIT license, works also under windows I guess it would be easy (and safe) enough to embed in our tree should we decide going this way. Agreed. If we extend the protocol to support compression, and not rely on SSL, then let's pick one of these LZ77-style compressors, and let's integrate it into our tree. We should then also make it possible to enable compression only for the server - client direction. Since those types of compressions are usually pretty easy to decompress, that reduces the amount to work non-libpq clients have to put in to take advantage of compression. +1 best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Mon, Jun 25, 2012 at 09:45:26PM +0200, Florian Pflug wrote: On Jun25, 2012, at 21:21 , Dimitri Fontaine wrote: Magnus Hagander mag...@hagander.net writes: Or that it takes less code/generates cleaner code... So we're talking about some LZO things such as snappy from google, and that would be another run time dependency IIUC. I think it's time to talk about fastlz: http://fastlz.org/ http://code.google.com/p/fastlz/source/browse/trunk/fastlz.c 551 lines of C code under MIT license, works also under windows I guess it would be easy (and safe) enough to embed in our tree should we decide going this way. Agreed. If we extend the protocol to support compression, and not rely on SSL, then let's pick one of these LZ77-style compressors, and let's integrate it into our tree. We should then also make it possible to enable compression only for the server - client direction. Since those types of compressions are usually pretty easy to decompress, that reduces the amount to work non-libpq clients have to put in to take advantage of compression. best regards, Florian Pflug Here is the benchmark list from the Google lz4 page: NameRatio C.speed D.speed LZ4 (r59) 2.084 330 915 LZO 2.05 1x_1 2.038 311 480 QuickLZ 1.5 -1 2.233 257 277 Snappy 1.0.52.024 227 729 LZF 2.076 197 465 FastLZ 2.030 190 420 zlib 1.2.5 -1 2.72839 195 LZ4 HC (r66)2.71218 1020 zlib 1.2.5 -6 3.09514 210 lz4 absolutely dominates on compression/decompression speed. While fastlz is faster than zlib(-1) on compression, lz4 is almost 2X faster still. Regards, Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 25-06-2012 16:45, Florian Pflug wrote: Agreed. If we extend the protocol to support compression, and not rely on SSL, then let's pick one of these LZ77-style compressors, and let's integrate it into our tree. If we have an option to have it out of our tree, good; if not, let's integrate it. I, particularly, don't see a compelling reason to integrate compression code in our tree, I mean, if we want to support more than one algorithm, it is clear that the overhead for maintain the compression code is too high (a lot of my-new-compression-algorithms). We should then also make it possible to enable compression only for the server - client direction. Since those types of compressions are usually pretty easy to decompress, that reduces the amount to work non-libpq clients have to put in to take advantage of compression. I don't buy this idea. My use case (data load) will not be covered if we only enable server - client compression. I figure that there are use cases for server - client compression (replication, for example) but also there are important use cases for client - server (data load, for example). If you implement decompression, why not code compression code too? -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 25-06-2012 14:30, Greg Jaskiewicz wrote: Wasn't this more of an issue in de-coupling compression from encryption ? Let me give a try to take some conclusion. If we decide for an independent compression code instead of an SSL-based one, that is a possibility to be tested: SSL + SSL-based compression x SSL + our compression code. If the latter is faster then we could discuss the possibility to disable compression in our SSL code and put in documentation that it is recommended to enable compression in SSL connections. -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Mon, Jun 25, 2012 at 3:38 PM, Euler Taveira eu...@timbira.com wrote: On 25-06-2012 16:45, Florian Pflug wrote: Agreed. If we extend the protocol to support compression, and not rely on SSL, then let's pick one of these LZ77-style compressors, and let's integrate it into our tree. If we have an option to have it out of our tree, good; if not, let's integrate it. I, particularly, don't see a compelling reason to integrate compression code in our tree, I mean, if we want to support more than one algorithm, it is clear that the overhead for maintain the compression code is too high (a lot of my-new-compression-algorithms). We should then also make it possible to enable compression only for the server - client direction. Since those types of compressions are usually pretty easy to decompress, that reduces the amount to work non-libpq clients have to put in to take advantage of compression. I don't buy this idea. My use case (data load) will not be covered if we only enable server - client compression. I figure that there are use cases for server - client compression (replication, for example) but also there are important use cases for client - server (data load, for example). If you implement decompression, why not code compression code too? +1. lz4, which is looking like a strong candidate, has c#, java, etc. which are the main languages that are going to lag behind in terms of protocol support. I don't think you're saving a lot by going only one way (although you could make a case for the client to signal interest in compression separately from decompression?) another point: It's been obvious for years now that zlib is somewhat of a dog in terms of cpu usage for what it gives you. however, raw performance #s are not the whole story -- it would be interesting to compress real world protocol messages to/from the server in various scenarios to see how compression would work, in particular with OLTP type queries -- for example pgbench runs, etc. That would speak a lot more to the benefits than canned benchmarks. merlin merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira eu...@timbira.com wrote: On 20-06-2012 17:40, Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly never was intended to be used... Maybe, but what is the argument that we should avoid on encryption+compression at the same time? AES is quite lightweight compared to compression, so should be no problem in situations where you care about compression. If we could solve compression problem without AES that will turn things easier. Compression-only via encryption is a weird manner to solve the problem in the user's POV. RSA is noticeable, but only for short connections. Thus easily solvable with connection pooling. RSA overhead is not the main problem. SSL/TLS setup is. And for really special compression needs you can always create a UDF that does custom compression for you. You have to own the code to modify it; it is not always an option. So what exactly is the situation we need to solve with postgres-specific protocol compression? Compression only support. Why do I need to set up SSL/TLS just for compression? IMHO SSL/TLS use is no different from relying in another library to handle compression for the protocol and more it is compression-specific. That way, we could implement another algorithms in such library without needing to modify libpq code. Using SSL/TLS you are bounded by what SSL/TLS software products decide to use as compression algorithms. I'll be happy to maintain the code iif it is postgres-specific or even as close as possible to core. I guess my feeling on this is that, so far as I can see, supporting compression via OpenSSL involves work and trade-offs, and supporting it without depending on OpenSSL also involves work, and trade-offs. So it's not real evident to me that we should prefer one to the other on general principle. It seems to me that a lot might come down to performance. If someone can demonstrate that using an external library involves gets significantly better compression, chews up significantly less CPU time, and/or is significantly less code than supporting this via OpenSSL, then maybe we ought to consider it. OpenSSL is kind of an ugly piece of code, and all things being equal depending on it more heavily would not be my first choice. On the other hand, this does not seem to me to be a situation where we should accept a patch to use an external library just because someone takes the time to write one, because there is also a non-trivial cost for the entire project to depending on more things; or if the compression code gets put into the backend, then there's a cost to us to maintain that code inside our source base. So I think we really need someone to try this both ways and compare. Right now it seems like we're mostly speculating on how well either approach would work, which is not as good as having some experimental results. If, for example, someone can demonstrate that an awesomebsdlz compresses 10x as fast as OpenSSL... that'd be pretty compelling. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 20-06-2012 17:40, Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly never was intended to be used... Maybe, but what is the argument that we should avoid on encryption+compression at the same time? AES is quite lightweight compared to compression, so should be no problem in situations where you care about compression. If we could solve compression problem without AES that will turn things easier. Compression-only via encryption is a weird manner to solve the problem in the user's POV. RSA is noticeable, but only for short connections. Thus easily solvable with connection pooling. RSA overhead is not the main problem. SSL/TLS setup is. And for really special compression needs you can always create a UDF that does custom compression for you. You have to own the code to modify it; it is not always an option. So what exactly is the situation we need to solve with postgres-specific protocol compression? Compression only support. Why do I need to set up SSL/TLS just for compression? IMHO SSL/TLS use is no different from relying in another library to handle compression for the protocol and more it is compression-specific. That way, we could implement another algorithms in such library without needing to modify libpq code. Using SSL/TLS you are bounded by what SSL/TLS software products decide to use as compression algorithms. I'll be happy to maintain the code iif it is postgres-specific or even as close as possible to core. -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun19, 2012, at 17:36 , Robert Haas wrote: On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout klep...@svana.org wrote: On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of these ideas is that the configure list is not a list of exact cipher names, as per Magnus' comment that the current default includes tests like !aNULL. I am not sure that we know how to evaluate such conditions if we are applying an after-the-fact check on the selected cipher. Does OpenSSL expose any API for evaluating whether a selected cipher meets such a test? I'm not sure whether there's an API for it, but you can certainly check manually with openssl ciphers -v, for example: $ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP' NULL-SHASSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1 NULL-MD5SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5 ...etc... So unless the openssl includes the code twice there must be a way to extract the list from the library. There doubtless is, but I'd being willing to wager that you won't be able to figure out the exact method without reading the source code for 'opennssl ciphers' to see how it was done there, and most likely you'll find that at least one of the functions they use has no man page. Documentation isn't their strong point. Yes, unfortunately. I wonder though if shouldn't restrict the allowed ciphers list to being a simple list of supported ciphers. If our goal is to support multiple SSL libraries transparently then surely having openssl-specific syntax in the config file isn't exactly great anyway... best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Excerpts from Florian Pflug's message of mié jun 20 06:35:29 -0400 2012: On Jun19, 2012, at 17:36 , Robert Haas wrote: On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout klep...@svana.org wrote: On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of these ideas is that the configure list is not a list of exact cipher names, as per Magnus' comment that the current default includes tests like !aNULL. I am not sure that we know how to evaluate such conditions if we are applying an after-the-fact check on the selected cipher. Does OpenSSL expose any API for evaluating whether a selected cipher meets such a test? I'm not sure whether there's an API for it, but you can certainly check manually with openssl ciphers -v, for example: $ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP' NULL-SHASSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1 NULL-MD5SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5 ...etc... So unless the openssl includes the code twice there must be a way to extract the list from the library. There doubtless is, but I'd being willing to wager that you won't be able to figure out the exact method without reading the source code for 'opennssl ciphers' to see how it was done there, and most likely you'll find that at least one of the functions they use has no man page. Documentation isn't their strong point. Yes, unfortunately. I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain the list of ciphers starting from the stringified configuration parameter and iterate on them. The problem is figuring out whether any given cipher meets some criteria; all the stuff that the command prints after the cipher name comes from a get cipher description API call and it doesn't look like there's any simple way of getting the individual bits in some better form (assuming we don't want to parse the description string). Now if the cipher name is enough for whatever it is that we want, then that looks easy. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Florian Pflug f...@phlo.org writes: I wonder though if shouldn't restrict the allowed ciphers list to being a simple list of supported ciphers. If our goal is to support multiple SSL libraries transparently then surely having openssl-specific syntax in the config file isn't exactly great anyway... No, we don't want to go there, because then we'd have to worry about keeping the default list in sync with what's supported by the particular version of the particular library we chance to be using. That's about as far from transparent as you can get. A notation like DEFAULT is really quite ideal for our purposes in that respect. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun20, 2012, at 17:34 , Tom Lane wrote: Florian Pflug f...@phlo.org writes: I wonder though if shouldn't restrict the allowed ciphers list to being a simple list of supported ciphers. If our goal is to support multiple SSL libraries transparently then surely having openssl-specific syntax in the config file isn't exactly great anyway... No, we don't want to go there, because then we'd have to worry about keeping the default list in sync with what's supported by the particular version of the particular library we chance to be using. That's about as far from transparent as you can get. A notation like DEFAULT is really quite ideal for our purposes in that respect. No argument with that, but does that mean we have to allow the full syntax supported by OpenSSL (i.e., those +,-,! prefixes)? Maybe we could map an empty list to DEFAULT and otherwise interpret it as a list of ciphers? It'd make the whole NULL-cipher business easy, because once we know that the cipher specified doesn't contain !NULL (which removes NULL *permanently*), we can simply append NULL to allow all these ciphers plus NULL. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Alvaro Herrera alvhe...@commandprompt.com writes: I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain the list of ciphers starting from the stringified configuration parameter and iterate on them. Do you mean that it will produce an expansion of the set of ciphers meeting criteria like !aNULL? If so, I think we are set; we can easily check to see if the active cipher is in that list, no? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Excerpts from Tom Lane's message of mié jun 20 11:49:51 -0400 2012:
Alvaro Herrera alvhe...@commandprompt.com writes:
I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain
the list of ciphers starting from the stringified configuration
parameter and iterate on them.
Do you mean that it will produce an expansion of the set of ciphers
meeting criteria like !aNULL?
Attached is a simple program that does that. You pass 'ALL:!aNULL' as
its first arg and it produces such a list.
If so, I think we are set; we can
easily check to see if the active cipher is in that list, no?
Great.
--
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
#include stdio.h
#include stdlib.h
#include string.h
#include openssl/err.h
#include openssl/ssl.h
int main(int argc, char *argv[])
{
const SSL_METHOD *method = TLSv1_client_method();
SSL_CTX*ctx;
SSL*ssl = NULL;
char *ciphers;
int i;
if (argc 2)
{
fprintf(stderr, ciphers not specified\n);
exit(1);
}
ciphers = argv[1];
SSL_library_init();
ctx = SSL_CTX_new(method);
if (!ctx)
{
fprintf(stderr, something went wrong\n);
exit(1);
}
if (!SSL_CTX_set_cipher_list(ctx, ciphers))
{
fprintf(stderr, unable to set cipher list\n);
exit(1);
}
ssl = SSL_new(ctx);
if (!ssl)
{
fprintf(stderr, unable to create the SSL object\n);
exit(1);
}
for (i = 0;; i++)
{
const char *cipher;
cipher = SSL_get_cipher_list(ssl, i);
if (cipher == NULL)
{
fprintf(stderr, end of cipher list?\n);
break;
}
printf(cipher: %s\n, cipher);
}
SSL_CTX_free(ctx);
SSL_free(ssl);
return 0;
}
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Wed, Jun 20, 2012 at 12:35 PM, Florian Pflug f...@phlo.org wrote: On Jun19, 2012, at 17:36 , Robert Haas wrote: On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout klep...@svana.org wrote: On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of these ideas is that the configure list is not a list of exact cipher names, as per Magnus' comment that the current default includes tests like !aNULL. I am not sure that we know how to evaluate such conditions if we are applying an after-the-fact check on the selected cipher. Does OpenSSL expose any API for evaluating whether a selected cipher meets such a test? I'm not sure whether there's an API for it, but you can certainly check manually with openssl ciphers -v, for example: $ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP' NULL-SHA SSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1 NULL-MD5 SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5 ...etc... So unless the openssl includes the code twice there must be a way to extract the list from the library. There doubtless is, but I'd being willing to wager that you won't be able to figure out the exact method without reading the source code for 'opennssl ciphers' to see how it was done there, and most likely you'll find that at least one of the functions they use has no man page. Documentation isn't their strong point. Yes, unfortunately. I wonder though if shouldn't restrict the allowed ciphers list to being a simple list of supported ciphers. If our goal is to support multiple SSL libraries transparently then surely having openssl-specific syntax in the config file isn't exactly great anyway... That is a very good point. Before we design *another* feature that relies on it, we should verify if the syntax is compatible in the other libraries that would be interesting (gnutls and NSS primarily), and if it's not that at least the *functionality* exists ina compatible way. So we don't put ourselves in a position where we can't proceed. And yes, that's vapourware so far. But I know at least Claes (added to CC) has said he wants to work on it during this summer, and I've promised to help him out and review as well, if/when he gets that far. But even without that, we should try to keep the door to these other library implementations as open as possible. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun20, 2012, at 18:42 , Magnus Hagander wrote: That is a very good point. Before we design *another* feature that relies on it, we should verify if the syntax is compatible in the other libraries that would be interesting (gnutls and NSS primarily), and if it's not that at least the *functionality* exists ina compatible way. So we don't put ourselves in a position where we can't proceed. Hm, here's another problem with relying on SSL/TLS for compression. RFC2246, which defines TLS 1.0, explicitly states that TLS_NULL_WITH_NULL_NULL is specified and is the initial state of a TLS connection during the first handshake on that channel, but MUST NOT be negotiated, as it provides no more protection than an unsecured connection. [RFC2246, A.5. The Cipher Suite] and that paragraph is still present in RFC5246 (TLS 1.2). The other cipher suits without actual encryption seem to be TLS_RSA_WITH_NULL_MD5 TLS_RSA_WITH_NULL_SHA TLS_RSA_WITH_NULL_SHA256 (TLS 1.2) Unless I'm missing something, that leaves us with no way of skipping the initial RSA handshake and also (more importantly) of computing a MD5 or SHA digest of every packet sent. I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly never was intended to be used... best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly never was intended to be used... Maybe, but what is the argument that we should avoid on encryption+compression at the same time? AES is quite lightweight compared to compression, so should be no problem in situations where you care about compression. RSA is noticeable, but only for short connections. Thus easily solvable with connection pooling. And for really special compression needs you can always create a UDF that does custom compression for you. So what exactly is the situation we need to solve with postgres-specific protocol compression? -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun20, 2012, at 22:40 , Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly never was intended to be used... Maybe, but what is the argument that we should avoid on encryption+compression at the same time? Mostly orthogonality, I think. I personally could live supporting compression only together with encryption. We *shouldn't* require people to create an RSA certificate just to use compression, though. I guess we could do that by using one of the cipher suits that use Diffie-Hellman key exchange (TLS_DH_anon_WITH_AES_128_CBC_SHA for example), but that again requires fiddling with the allowed ciphers list. Or we could auto-generate a self-signed RSA certificate if none is available. Which is doable as long as we're going to stick with OpenSSL or GnuTLS, but if we eventually want to support more SSL libraries server-side, this could be a road blocker... AES is quite lightweight compared to compression, so should be no problem in situations where you care about compression. Hm, yeah, that's especially true since DEFLATE (zlib) seems to be the only universally supported compression method for TLS, which is rather slow anyway (At least compared to alternatives such as LZO or Google's Snappy). So what exactly is the situation we need to solve with postgres-specific protocol compression? All we really want IMHO is a way to enable compression which requires no more than specifying compress=on or the like in the connection string. The million dollar question is, what is the easiest way to get there? I initially though that relying on TLS for compression should be relatively straight-forward, but it's starting to look rather messy… best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout klep...@svana.org wrote: On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of these ideas is that the configure list is not a list of exact cipher names, as per Magnus' comment that the current default includes tests like !aNULL. I am not sure that we know how to evaluate such conditions if we are applying an after-the-fact check on the selected cipher. Does OpenSSL expose any API for evaluating whether a selected cipher meets such a test? I'm not sure whether there's an API for it, but you can certainly check manually with openssl ciphers -v, for example: $ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP' NULL-SHA SSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1 NULL-MD5 SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5 ...etc... So unless the openssl includes the code twice there must be a way to extract the list from the library. There doubtless is, but I'd being willing to wager that you won't be able to figure out the exact method without reading the source code for 'opennssl ciphers' to see how it was done there, and most likely you'll find that at least one of the functions they use has no man page. Documentation isn't their strong point. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of these ideas is that the configure list is not a list of exact cipher names, as per Magnus' comment that the current default includes tests like !aNULL. I am not sure that we know how to evaluate such conditions if we are applying an after-the-fact check on the selected cipher. Does OpenSSL expose any API for evaluating whether a selected cipher meets such a test? I'm not sure whether there's an API for it, but you can certainly check manually with openssl ciphers -v, for example: $ openssl ciphers -v 'ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP' NULL-SHASSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1 NULL-MD5SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5 ...etc... So unless the openssl includes the code twice there must be a way to extract the list from the library. Have a nice ay, -- Martijn van Oosterhout klep...@svana.org http://svana.org/kleptog/ He who writes carelessly confesses thereby at the very outset that he does not attach much importance to his own thoughts. -- Arthur Schopenhauer signature.asc Description: Digital signature
Re: [HACKERS] libpq compression
On Sat, Jun 16, 2012 at 11:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted SSL connection probably shouldn't be treated as matching a hostssl line; which is not something that requires any fundamental rearrangements, since it only requires an after-the-fact check of what was selected. Maybe I spelled it out wrong. It does require it insofar that if we want to use this for compression, we must *always* enable openssl on the connection. So the with these encryption method boils down to NULL encryption only or whatever other standards I have for encryption. We don't need the ability to change the whatever other standards per subnet, but we need to control the accept-NULL-encryption on a per subnet basis. After sleeping on it, I wonder if we couldn't redefine the existing list of acceptable ciphers option as the list of ciphers that are considered to provide encrypted transport. So you'd be allowed to connect with SSL using any unapproved cipher (including NULL), the backend just considers it as equivalent to a non-SSL connection for pg_hba purposes. Then no change is needed in any configuration stuff. That seems reasonable. In looking at our current defaults, two things hit me: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? and Shouldn't our default SSL methods include !aNULL, meaning by default we exclude all ciphers that don't provide authentication (which means they can be man-in-the-middle'd). AFACIT, eNULL/NULL is disabled by default unless explicitly enabled, but aNULL isn't.. I don't think it matters from a pure security perspective since we look inside the actual cert anyway (which shouldn't work with these methods, I think), but it seems like a wrong default. That, or just have DEFAULT as being the default (which in current openssl means ALL:!aNULL:!eNULL. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need one? I am not sure what the cipher negotiation process looks like or which side has the freedom to choose. That, or just have DEFAULT as being the default (which in current openssl means ALL:!aNULL:!eNULL. If our default isn't the same as the underlying default, I have to question why not. But are you sure this ! notation will work with all openssl versions? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need one? I am not sure what the cipher negotiation process looks like or which side has the freedom to choose. I haven't looked into the details, but it seems reasonable that *either* side should be able to at least define a list of ciphers it *doens't* want to talk with. Do we need it - well, it makes sense for the client to be able to say I won't trust 56-bit encryption before it sends over the password, imo.. That, or just have DEFAULT as being the default (which in current openssl means ALL:!aNULL:!eNULL. If our default isn't the same as the underlying default, I have to question why not. Yeah, that's exaclty what I'm questioning here.. But are you sure this ! notation will work with all openssl versions? Uh. We have the ! notation in our default *now*. What openssl also supports is the text DEFAULT, which is currently the equivalent of ALL!aNULL!eNULL. The question, which is valid of course, should be if DEFAULT works with all openssl versions. It would seem reasonable it does, but I haven't investigated. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sun, Jun 17, 2012 at 4:45 PM, Magnus Hagander mag...@hagander.net wrote: On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need one? I am not sure what the cipher negotiation process looks like or which side has the freedom to choose. I haven't looked into the details, but it seems reasonable that *either* side should be able to at least define a list of ciphers it *doens't* want to talk with. Do we need it - well, it makes sense for the client to be able to say I won't trust 56-bit encryption before it sends over the password, imo.. I would certainly like to see that. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun16, 2012, at 17:15 , Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted SSL connection probably shouldn't be treated as matching a hostssl line; which is not something that requires any fundamental rearrangements, since it only requires an after-the-fact check of what was selected. Maybe I spelled it out wrong. It does require it insofar that if we want to use this for compression, we must *always* enable openssl on the connection. So the with these encryption method boils down to NULL encryption only or whatever other standards I have for encryption. We don't need the ability to change the whatever other standards per subnet, but we need to control the accept-NULL-encryption on a per subnet basis. After sleeping on it, I wonder if we couldn't redefine the existing list of acceptable ciphers option as the list of ciphers that are considered to provide encrypted transport. So you'd be allowed to connect with SSL using any unapproved cipher (including NULL), the backend just considers it as equivalent to a non-SSL connection for pg_hba purposes. Then no change is needed in any configuration stuff. Would we still tell openssl to only negotiate ciphers in the configured list of available ciphers + NULL? If not, what happens if a connection happens to use a cipher that is actually stronger than any cipher on the list of acceptable ciphers list? The DBA wouldn't necessarily be aware that such a cipher even exists, since it could have been made available by an openssl upgrade… But if we restrict the negotiable ciphers to the configure list + NULL, then we're good I think. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Florian Pflug f...@phlo.org writes: Would we still tell openssl to only negotiate ciphers in the configured list of available ciphers + NULL? If not, what happens if a connection happens to use a cipher that is actually stronger than any cipher on the list of acceptable ciphers list? The DBA wouldn't necessarily be aware that such a cipher even exists, since it could have been made available by an openssl upgrade So? If the DBA has gone so far as to list specific ciphers, who are we to second guess his judgment? It's not for us to decide that cipher X is stronger than the ones he listed. But if we restrict the negotiable ciphers to the configure list + NULL, then we're good I think. The fly in the ointment with any of these ideas is that the configure list is not a list of exact cipher names, as per Magnus' comment that the current default includes tests like !aNULL. I am not sure that we know how to evaluate such conditions if we are applying an after-the-fact check on the selected cipher. Does OpenSSL expose any API for evaluating whether a selected cipher meets such a test? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 17-06-2012 12:42, Tom Lane wrote: If our default isn't the same as the underlying default, I have to question why not. But are you sure this ! notation will work with all openssl versions? What is all for you? It seems we don't claim support for an specific version or later in docs or even configure. But looking at an old version (0.9.7, 2003-12-31), it seems to support ! notation. -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 17-06-2012 12:45, Magnus Hagander wrote: Uh. We have the ! notation in our default *now*. What openssl also supports is the text DEFAULT, which is currently the equivalent of ALL!aNULL!eNULL. The question, which is valid of course, should be if DEFAULT works with all openssl versions. AFAICS, DEFAULT works for ancient openssl versions (~10 years ago). -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 17-06-2012 12:45, Magnus Hagander wrote: On Sun, Jun 17, 2012 at 11:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Is there a reason why we don't have a parameter on the client mirroring ssl_ciphers? Dunno, do we need one? I am not sure what the cipher negotiation process looks like or which side has the freedom to choose. Both. Client sends a cipher list and the server determines which cipher is used getting the first supported cipher in the client list. I haven't looked into the details, but it seems reasonable that *either* side should be able to at least define a list of ciphers it *doens't* want to talk with. +1. -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Yes, but there's also a lot of such awkward logic we need to add if we *do* go with the SSL library doing the compression: For example, we can no longer trust the SSL library to always do encryption, since we specifically want to support null encryption. True, but are you sure we don't need to do that anyway? What happens today, if a non-libpq client connects with SSL and specifies null encryption? openssl rejects the connection unless you have explicitly allowed NULL encryption in ssl_ciphers. Which is the only sensible default. And we currently have no way to specify different encryption options on a per-host basis, which is something we'd have to do (e.g. i want to be able to say that subnet x requires encryption with these encryptions methods and subnet y doesn't require encryption but should do compression. [ shrug... ] Having that sort of control over a homebrew compression solution will *also* require a lot of control logic that does not exist today. The important part isn't really being able to control the compression in this. It's that we're overloading a convenience feature (compression) in the settings of a security feature (encryption). Which leads to both complex processing, and also a fairly high risk of accidentally configuring what you wouldn't want unless we change the interface to make it look like separate things even if they aren't. So there's quite a bit of complexity that needs to be put in there just to deal with the fact that we're using SSL to do compression, if we want to support it in a way that's not hackish. It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted SSL connection probably shouldn't be treated as matching a hostssl line; which is not something that requires any fundamental rearrangements, since it only requires an after-the-fact check of what was selected. Things like subnet x requires encryption with these encryption methods are features that are sensible with our existing feature set. But we don't have that now and nobody has asked for it, so I think you are moving the goalposts rather unfairly by claiming that a compression-related patch needs to add it. Maybe I spelled it out wrong. It does require it insofar that if we want to use this for compression, we must *always* enable openssl on the connection. So the with these encryption method boils down to NULL encryption only or whatever other standards I have for encryption. We don't need the ability to change the whatever other standards per subnet, but we need to control the accept-NULL-encryption on a per subnet basis. It also risks some level of information leak - assuming someone connects with NULL encryption and we don't support it, unless we do something particular about it, the error message will go out in cleartext. Today, you will get a client generated error message and no actual message crosses the wire in cleartext. It's not that we can't deal with those things. It's just that it's going to take some work, and some careful thought about exactly which parts can be exposed over NULL encrypted connections. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sat, Jun 16, 2012 at 12:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Marko Kreen mark...@gmail.com writes: On Sat, Jun 16, 2012 at 6:39 AM, Magnus Hagander mag...@hagander.net wrote: Fair enough if we decide that - but we should make that decision knowing that we're leaving the JDBC and .Net people in a bad position where they are not likely to be able to implement his. The JDBC people have a theoretical chance if the JDK is open. The .Net people are stuck with schannel that doesn't support it at this point. It might well do in the future (since it's in the standard); but they're at the mercy of Microsoft. Both Java and C# are open-source enough that anybody can take existing SSL implementation and add compression to it, then distribute it as improved SSL library. Possibly more to the point: that is work they might have to do, if nobody else steps up to the plate --- and if they do end up doing it, it could benefit other projects too. On the other hand, if we roll-our-own transport compression solution, that is work they *will* have to do, with no chance of sharing the effort with other projects. True - provided said upstream (Oracle in the Java case) are interested in accepting the patches... If they end up having to port one of the compressoin algorithms, let's dake LZ4 as an example, then they can certainly release that as open source under a compatible license, thus making it available to others. Though that's not necessarily that relevant - LZ4 already has a C# implementation for .net, a JNI wrapper for Java. Snappy even has a native Java implementation. So if we went down that road, there wouldn't *be* a need to implement it. Just the protocol parts itself, which are - compared to implementing the actual compression in either scheme - trivial. BTW, as far as the .Net case goes, it took only a moment's googling to find this: http://openssl-net.sourceforge.net/ which is a .Net wrapper around real OpenSSL. It doesn't appear to provide wrappers for the compression selection functions, but surely that's just a lack of round tuits, not that it would take more than five minutes to add them. that would then loose all the advantages that npgsql get from schannel, such as integrated certificate management. So it can be done - but it would AFAICT require a fairly large rearchitecture of how security is handled, it would add a license-incompatible requirement, and it would loose other features. But it can be done. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Magnus Hagander mag...@hagander.net writes: On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted SSL connection probably shouldn't be treated as matching a hostssl line; which is not something that requires any fundamental rearrangements, since it only requires an after-the-fact check of what was selected. Maybe I spelled it out wrong. It does require it insofar that if we want to use this for compression, we must *always* enable openssl on the connection. So the with these encryption method boils down to NULL encryption only or whatever other standards I have for encryption. We don't need the ability to change the whatever other standards per subnet, but we need to control the accept-NULL-encryption on a per subnet basis. After sleeping on it, I wonder if we couldn't redefine the existing list of acceptable ciphers option as the list of ciphers that are considered to provide encrypted transport. So you'd be allowed to connect with SSL using any unapproved cipher (including NULL), the backend just considers it as equivalent to a non-SSL connection for pg_hba purposes. Then no change is needed in any configuration stuff. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sat, Jun 16, 2012 at 11:15:30AM -0400, Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Sat, Jun 16, 2012 at 12:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted SSL connection probably shouldn't be treated as matching a hostssl line; which is not something that requires any fundamental rearrangements, since it only requires an after-the-fact check of what was selected. Maybe I spelled it out wrong. It does require it insofar that if we want to use this for compression, we must *always* enable openssl on the connection. So the with these encryption method boils down to NULL encryption only or whatever other standards I have for encryption. We don't need the ability to change the whatever other standards per subnet, but we need to control the accept-NULL-encryption on a per subnet basis. After sleeping on it, I wonder if we couldn't redefine the existing list of acceptable ciphers option as the list of ciphers that are considered to provide encrypted transport. So you'd be allowed to connect with SSL using any unapproved cipher (including NULL), the backend just considers it as equivalent to a non-SSL connection for pg_hba purposes. Then no change is needed in any configuration stuff. regards, tom lane +1 That is nice and clean. Regards, Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun15, 2012, at 07:50 , Magnus Hagander wrote: So I've got very little patience with the idea of let's put in some hooks and then great things will happen. It would be far better all around if we supported exactly one, well-chosen, method. But really I still don't see a reason not to let openssl do it for us. Do we just need to document SSL's NULL encryption option? Does the SSL NULL encryption+compression thing work if you're not using openssl? The compression support is defined in RFC 3749, and according to http://en.wikipedia.org/wiki/Comparison_of_TLS_Implementations it's supported in openssl and gnutls. gnutls also seems to support a NULL cipher - gnutls-cli on my Ubuntu 10.04 box prints Ciphers: AES-256-CBC, AES-128-CBC, 3DES-CBC, DES-CBC, ARCFOUR-128, ARCFOUR-40, RC2-40, CAMELLIA-256-CBC, CAMELLIA-128-CBC, NULL. For one thing, some of us still hold a hope to support non-openssl libraries in both libpq and server side, so it's something that would need to be supported by the standard and thus available in most libraries not to invalidate that. Well, it's a standard a least, and both openssl and gnutls seem to support it. Are there any other ssl implementations beside gnutls and openssl that we need to worry about? Second, we also have things like the JDBC driver and the .Net driver that don't use libpq. the JDBC driver uses the native java ssl support, AFAIK. Does that one support the compression, and does it support controlling it? Java uses pluggable providers with standardized interfaces for most things related to encryption. SSL support is provided by JSSE (Java Secure Socket Extension). The JSSE implementation included with the oracle JRE doesn't seem to support compression according to the wikipedia page quoted above. But chances are that there exists an alternative implementation which does. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus Hagander wrote: So I've got very little patience with the idea of let's put in some hooks and then great things will happen. It would be far better all around if we supported exactly one, well-chosen, method. But really I still don't see a reason not to let openssl do it for us. Do we just need to document SSL's NULL encryption option? Does the SSL NULL encryption+compression thing work if you're not using openssl? The compression support is defined in RFC 3749, and according to http://en.wikipedia.org/wiki/Comparison_of_TLS_Implementations it's supported in openssl and gnutls. gnutls also seems to support a NULL cipher - gnutls-cli on my Ubuntu 10.04 box prints Ciphers: AES-256-CBC, AES-128-CBC, 3DES-CBC, DES-CBC, ARCFOUR-128, ARCFOUR-40, RC2-40, CAMELLIA-256-CBC, CAMELLIA-128-CBC, NULL. ah, thanks for looking that up for me! The other big one to consider would be GNUTLS - which also has support for compression, I see. I guess a related question is if they all alow us to turn it *off*, which we now do support on openssl :) gnutls does, I didn't look into nss. For one thing, some of us still hold a hope to support non-openssl libraries in both libpq and server side, so it's something that would need to be supported by the standard and thus available in most libraries not to invalidate that. Well, it's a standard a least, and both openssl and gnutls seem to support it. Are there any other ssl implementations beside gnutls and openssl that we need to worry about? NSS would be the big one, an din theory microsoft schannel if we were to go there (that would give us access to easy use of the windows certificate store so ther emight be a reason - but not a very big one, to support that). Second, we also have things like the JDBC driver and the .Net driver that don't use libpq. the JDBC driver uses the native java ssl support, AFAIK. Does that one support the compression, and does it support controlling it? Java uses pluggable providers with standardized interfaces for most things related to encryption. SSL support is provided by JSSE (Java Secure Socket Extension). The JSSE implementation included with the oracle JRE doesn't seem to support compression according to the wikipedia page quoted above. But chances are that there exists an alternative implementation which does. Yeah, but that alone is IMO a rather big blocker for claiming that this is the only way to do it :( And I think the fact that that wikipedia page doesn't list any other ones, is a sign that there might not be a lot of other choices out there in reality - expecially not opensource... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun15, 2012, at 12:09 , Magnus Hagander wrote: On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus Hagander wrote: Second, we also have things like the JDBC driver and the .Net driver that don't use libpq. the JDBC driver uses the native java ssl support, AFAIK. Does that one support the compression, and does it support controlling it? Java uses pluggable providers with standardized interfaces for most things related to encryption. SSL support is provided by JSSE (Java Secure Socket Extension). The JSSE implementation included with the oracle JRE doesn't seem to support compression according to the wikipedia page quoted above. But chances are that there exists an alternative implementation which does. Yeah, but that alone is IMO a rather big blocker for claiming that this is the only way to do it :( And I think the fact that that wikipedia page doesn't list any other ones, is a sign that there might not be a lot of other choices out there in reality - expecially not opensource… Hm, but things get even harder for the JDBC and .NET folks if we go with a third-party compression method. Or would we require that the existence of a free Java (and maybe .NET) implementation of such a method would be an absolute must? The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 15, 2012 at 5:48 AM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 12:09 , Magnus Hagander wrote: On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus Hagander wrote: Second, we also have things like the JDBC driver and the .Net driver that don't use libpq. the JDBC driver uses the native java ssl support, AFAIK. Does that one support the compression, and does it support controlling it? Java uses pluggable providers with standardized interfaces for most things related to encryption. SSL support is provided by JSSE (Java Secure Socket Extension). The JSSE implementation included with the oracle JRE doesn't seem to support compression according to the wikipedia page quoted above. But chances are that there exists an alternative implementation which does. Yeah, but that alone is IMO a rather big blocker for claiming that this is the only way to do it :( And I think the fact that that wikipedia page doesn't list any other ones, is a sign that there might not be a lot of other choices out there in reality - expecially not opensource… Hm, but things get even harder for the JDBC and .NET folks if we go with a third-party compression method. Or would we require that the existence of a free Java (and maybe .NET) implementation of such a method would be an absolute must? The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. hm, that's a really excellent point. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 15, 2012 at 07:18:34AM -0500, Merlin Moncure wrote: On Fri, Jun 15, 2012 at 5:48 AM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 12:09 , Magnus Hagander wrote: On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus Hagander wrote: Second, we also have things like the JDBC driver and the .Net driver that don't use libpq. the JDBC driver uses the native java ssl support, AFAIK. Does that one support the compression, and does it support controlling it? Java uses pluggable providers with standardized interfaces for most things related to encryption. SSL support is provided by JSSE (Java Secure Socket Extension). The JSSE implementation included with the oracle JRE doesn't seem to support compression according to the wikipedia page quoted above. But chances are that there exists an alternative implementation which does. Yeah, but that alone is IMO a rather big blocker for claiming that this is the only way to do it :( And I think the fact that that wikipedia page doesn't list any other ones, is a sign that there might not be a lot of other choices out there in reality - expecially not opensource… Hm, but things get even harder for the JDBC and .NET folks if we go with a third-party compression method. Or would we require that the existence of a free Java (and maybe .NET) implementation of such a method would be an absolute must? The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. hm, that's a really excellent point. merlin I agree and think that the SSL-based compression is an excellent default compression scheme. The plugable compression approach allows for the choice of the most appropriate compression implementation based on the application needs. It really addresses corner cases such as high- performance system. Regards, Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 12:09 , Magnus Hagander wrote: On Fri, Jun 15, 2012 at 5:52 PM, Florian Pflug f...@phlo.org wrote: On Jun15, 2012, at 07:50 , Magnus Hagander wrote: Second, we also have things like the JDBC driver and the .Net driver that don't use libpq. the JDBC driver uses the native java ssl support, AFAIK. Does that one support the compression, and does it support controlling it? Java uses pluggable providers with standardized interfaces for most things related to encryption. SSL support is provided by JSSE (Java Secure Socket Extension). The JSSE implementation included with the oracle JRE doesn't seem to support compression according to the wikipedia page quoted above. But chances are that there exists an alternative implementation which does. Yeah, but that alone is IMO a rather big blocker for claiming that this is the only way to do it :( And I think the fact that that wikipedia page doesn't list any other ones, is a sign that there might not be a lot of other choices out there in reality - expecially not opensource… Hm, but things get even harder for the JDBC and .NET folks if we go with a third-party compression method. Or would we require that the existence of a free Java (and maybe .NET) implementation of such a method would be an absolute must? As long as a free implementation exists, it can be ported to Java/.Net. Sure, it takes more work, but it *can be done*. The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. I only partially agree. If there *is* no third party SSL libary that does support it, then they're stuck reimplementing an *entire SSL library*, which is surely many orders of magnitude more work, and suddenly steps into writing encryption code which is a lot more sensitive. Basically if they have to do that, then they're stuck *never* being able to fix the problem. If we can prove such a third party library *exists*, that makes it different. But from what I can tell so far, I haven't seen a single one - let alone one that supports compression. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 15.06.2012 17:39, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflugf...@phlo.org wrote: The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. I only partially agree. If there *is* no third party SSL libary that does support it, then they're stuck reimplementing an *entire SSL library*, which is surely many orders of magnitude more work, and suddenly steps into writing encryption code which is a lot more sensitive. You could write a dummy SSL implementation that only does compression, not encryption. Ie. only support the 'null' encryption method. That should be about the same amount of work as writing an implementation of compression using whatever protocol we would decide to use for negotiating the compression. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:39, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflugf...@phlo.org wrote: The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. I only partially agree. If there *is* no third party SSL libary that does support it, then they're stuck reimplementing an *entire SSL library*, which is surely many orders of magnitude more work, and suddenly steps into writing encryption code which is a lot more sensitive. You could write a dummy SSL implementation that only does compression, not encryption. Ie. only support the 'null' encryption method. That should be about the same amount of work as writing an implementation of compression using whatever protocol we would decide to use for negotiating the compression. Sure, but then what do you do if you actually want both? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 15.06.2012 17:58, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:39, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflugf...@phlo.orgwrote: The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. I only partially agree. If there *is* no third party SSL libary that does support it, then they're stuck reimplementing an *entire SSL library*, which is surely many orders of magnitude more work, and suddenly steps into writing encryption code which is a lot more sensitive. You could write a dummy SSL implementation that only does compression, not encryption. Ie. only support the 'null' encryption method. That should be about the same amount of work as writing an implementation of compression using whatever protocol we would decide to use for negotiating the compression. Sure, but then what do you do if you actually want both? Umm, then you use a real SSL libray, not the dummy one? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:58, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:39, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflugf...@phlo.org wrote: The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. I only partially agree. If there *is* no third party SSL libary that does support it, then they're stuck reimplementing an *entire SSL library*, which is surely many orders of magnitude more work, and suddenly steps into writing encryption code which is a lot more sensitive. You could write a dummy SSL implementation that only does compression, not encryption. Ie. only support the 'null' encryption method. That should be about the same amount of work as writing an implementation of compression using whatever protocol we would decide to use for negotiating the compression. Sure, but then what do you do if you actually want both? Umm, then you use a real SSL libray, not the dummy one? But (in this scenario, and so far nobody has proven it to be wrong) there exists no real SSL library that does support compression. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Magnus Hagander mag...@hagander.net writes: On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Umm, then you use a real SSL libray, not the dummy one? But (in this scenario, and so far nobody has proven it to be wrong) there exists no real SSL library that does support compression. I do not think it is incumbent on this project to rectify that problem ... especially when nobody has proven that such a library exists (and is not obsolete, nor are its authors busy fixing the lack so as to be interoperable with openssl). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 15, 2012 at 11:28:48PM +0800, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:58, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:39, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 6:48 PM, Florian Pflugf...@phlo.org wrote: The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. I only partially agree. If there *is* no third party SSL libary that does support it, then they're stuck reimplementing an *entire SSL library*, which is surely many orders of magnitude more work, and suddenly steps into writing encryption code which is a lot more sensitive. You could write a dummy SSL implementation that only does compression, not encryption. Ie. only support the 'null' encryption method. That should be about the same amount of work as writing an implementation of compression using whatever protocol we would decide to use for negotiating the compression. Sure, but then what do you do if you actually want both? Umm, then you use a real SSL libray, not the dummy one? But (in this scenario, and so far nobody has proven it to be wrong) there exists no real SSL library that does support compression. gnutls and openssl both support compression: http://www.gnu.org/software/gnutls/manual/html_node/Compression-algorithms-used-in-the-record-layer.html http://www.openssl.org/docs/apps/enc.html -Ryan Kelly -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 15.06.2012 18:28, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:58, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.comwrote: You could write a dummy SSL implementation that only does compression, not encryption. Ie. only support the 'null' encryption method. That should be about the same amount of work as writing an implementation of compression using whatever protocol we would decide to use for negotiating the compression. Sure, but then what do you do if you actually want both? Umm, then you use a real SSL libray, not the dummy one? But (in this scenario, and so far nobody has proven it to be wrong) there exists no real SSL library that does support compression. Oh, I see. Then you're screwed. But I think the right solution to that is to write/extend a Java SSL implementation to support compression, not to invent our own in PostgreSQL. The JDK is open source nowadays. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 15-06-2012 11:39, Magnus Hagander wrote: As long as a free implementation exists, it can be ported to Java/.Net. Sure, it takes more work, but it *can be done*. Good point. IMHO, if there isn't a solution that cover all PostgreSQL (it seems it is not), we should pick the most appropriate one for *libpq* and let other drivers implement it at their time. I only partially agree. If there *is* no third party SSL libary that does support it, then they're stuck reimplementing an *entire SSL library*, which is surely many orders of magnitude more work, and suddenly steps into writing encryption code which is a lot more sensitive. Basically if they have to do that, then they're stuck *never* being able to fix the problem. If we can prove such a third party library *exists*, that makes it different. But from what I can tell so far, I haven't seen a single one - let alone one that supports compression. Using SSL-based compression could be a solution but I would like to emphasize that (i) I'm obligated to use cryptography library to compress data, (ii) I'm paying the price for SSL overhead and (iii) it will confuse people when we said that for compression we need a SSL connection or (iv) even transform the libpq communication code into a spaghetti to support compression using SSL in non-SSL connections). I see the point in not adding another dependencies or reinventing the wheel but I see more drawbacks than benefits in adopting a SSL-based compression. I like the Farina's idea in supporting compression outside libpq but I'm ok with adding a standard algorithm for compression (problem is that in the future others could want to add another interesting compression algorithms). -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 15-06-2012 11:10, k...@rice.edu wrote: I agree and think that the SSL-based compression is an excellent default compression scheme. The plugable compression approach allows for the choice of the most appropriate compression implementation based on the application needs. It really addresses corner cases such as high- performance system. That is my opinion too. I'm free to use to most appropriate algorithm for compression. It is just a matter of coding an interface for my favorite compression algorithm. We could even add some algorithms in a new contrib module. -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Euler Taveira eu...@timbira.com writes: I see the point in not adding another dependencies or reinventing the wheel but I see more drawbacks than benefits in adopting a SSL-based compression. In the end, judging this tradeoff is a matter of opinion, but I come to the opposite conclusion. Transport-level compression is not part of the core competence of this project. As such, if we have an opportunity to farm out that work to other projects (particularly ones that we are already relying on), we should do so. Not expend our limited resources on re-inventing this wheel, which we'd be more likely than not to do so less well than it's already been done. To draw an analogy: on the basis of the arguments that have been made about how some users might not have access to an SSL library implementing feature X, we should drop our use of OpenSSL entirely and re-implement transport encryption from scratch, incompatibly with OpenSSL. Now that's obviously ridiculous, not least because it does nothing at all to ease the pain of people who need a non-C implementation. But arguing that we should not use OpenSSL's compression features because some people might need to use a different SSL implementation doesn't seem to me to be any different from that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
I wrote: Euler Taveira eu...@timbira.com writes: I see the point in not adding another dependencies or reinventing the wheel but I see more drawbacks than benefits in adopting a SSL-based compression. In the end, judging this tradeoff is a matter of opinion, but I come to the opposite conclusion. BTW, there is an additional technical argument that I don't think has been made yet. Assume that we implement our own transport compression, and then somebody runs an SSL connection using a recent OpenSSL version (in which, IIRC, SSL-level compression is enabled by default). Now, OpenSSL is trying to compress already-compressed data. That's not merely a waste of cycles but is very likely to be counterproductive, ie recompressed data usually gets larger not smaller. We could possibly address this by adding control logic to tell OpenSSL not to compress ... but that's almost exactly the code you don't want to write, just making a different option selection. And I wonder whether SSL implementations that don't support compression will accept a set-the-compression-option call at all. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 15, 2012 at 12:48:24PM +0200, Florian Pflug wrote: Yeah, but that alone is IMO a rather big blocker for claiming that this is the only way to do it :( And I think the fact that that wikipedia page doesn't list any other ones, is a sign that there might not be a lot of other choices out there in reality - expecially not opensource… Hm, but things get even harder for the JDBC and .NET folks if we go with a third-party compression method. Or would we require that the existence of a free Java (and maybe .NET) implementation of such a method would be an absolute must? The way I see it, if we use SSL-based compression then non-libpq clients there's at least a chance of those clients being able to use it easily (if their SSL implementation supports it). If we go with a third-party compression method, they *all* need to add yet another dependency, or may even need to re-implement the compression method in their implementation language of choice. Does OpenSSL use hardware acceleration for its compression? I know it often does for encryption --- that would be a big reason to do compression at the SSL layer. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sat, Jun 16, 2012 at 12:03 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 18:28, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 11:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.06.2012 17:58, Magnus Hagander wrote: On Fri, Jun 15, 2012 at 10:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: You could write a dummy SSL implementation that only does compression, not encryption. Ie. only support the 'null' encryption method. That should be about the same amount of work as writing an implementation of compression using whatever protocol we would decide to use for negotiating the compression. Sure, but then what do you do if you actually want both? Umm, then you use a real SSL libray, not the dummy one? But (in this scenario, and so far nobody has proven it to be wrong) there exists no real SSL library that does support compression. Oh, I see. Then you're screwed. But I think the right solution to that is to write/extend a Java SSL implementation to support compression, not to invent our own in PostgreSQL. The JDK is open source nowadays. I don't have any personal experience with it, but it's my understanding that it's only opensource in the published opensource product sense. Meaning it's not really something that solicits (or even accepts? ast least not easily...) contributions from the outside. And forgive me for being negative, but I think you're going to have an even harder time to get Oracle to accept a contribution if the motivation for having it is to make the PostgreSQL driver work better... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sat, Jun 16, 2012 at 4:04 AM, Euler Taveira eu...@timbira.com wrote: On 15-06-2012 11:39, Magnus Hagander wrote: As long as a free implementation exists, it can be ported to Java/.Net. Sure, it takes more work, but it *can be done*. Good point. IMHO, if there isn't a solution that cover all PostgreSQL (it seems it is not), we should pick the most appropriate one for *libpq* and let other drivers implement it at their time. Fair enough if we decide that - but we should make that decision knowing that we're leaving the JDBC and .Net people in a bad position where they are not likely to be able to implement his. The JDBC people have a theoretical chance if the JDK is open. The .Net people are stuck with schannel that doesn't support it at this point. It might well do in the future (since it's in the standard); but they're at the mercy of Microsoft. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sat, Jun 16, 2012 at 6:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Euler Taveira eu...@timbira.com writes: I see the point in not adding another dependencies or reinventing the wheel but I see more drawbacks than benefits in adopting a SSL-based compression. In the end, judging this tradeoff is a matter of opinion, but I come to the opposite conclusion. BTW, there is an additional technical argument that I don't think has been made yet. Assume that we implement our own transport compression, and then somebody runs an SSL connection using a recent OpenSSL version (in which, IIRC, SSL-level compression is enabled by default). Now, OpenSSL is trying to compress already-compressed data. That's not merely a waste of cycles but is very likely to be counterproductive, ie recompressed data usually gets larger not smaller. We could possibly address this by adding control logic to tell OpenSSL not to compress ... but that's almost exactly the code you don't want to write, just making a different option selection. And I wonder whether SSL implementations that don't support compression will accept a set-the-compression-option call at all. Yes, but there's also a lot of such awkward logic we need to add if we *do* go with the SSL library doing the compression: For example, we can no longer trust the SSL library to always do encryption, since we specifically want to support null encryption. Meaning we need to teach pg_hba to treat a connection with null encryption as hostnossl, even if it's an openssl-backed connection, and mirrored. And in libpq, we have to make sure that a requiressl connection *does* fail even if we have ssl, when we're using null encryption. And we currently have no way to specify different encryption options on a per-host basis, which is something we'd have to do (e.g. i want to be able to say that subnet x requires encryption with these encryptions methods and subnet y doesn't require encryption but should do compression. Which in the easiest first look would require ssl_ciphers to be controllable from pg_hba.conf - but that doesn't work since we don't get to pg_hba.conf until after we've negotiated the SSL mode... So there's quite a bit of complexity that needs to be put in there just to deal with the fact that we're using SSL to do compression, if we want to support it in a way that's not hackish. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 16-06-2012 00:43, Magnus Hagander wrote: For example, we can no longer trust the SSL library to always do encryption, since we specifically want to support null encryption. Meaning we need to teach pg_hba to treat a connection with null encryption as hostnossl, even if it's an openssl-backed connection, and mirrored. And in libpq, we have to make sure that a requiressl connection *does* fail even if we have ssl, when we're using null encryption. And we currently have no way to specify different encryption options on a per-host basis, which is something we'd have to do (e.g. i want to be able to say that subnet x requires encryption with these encryptions methods and subnet y doesn't require encryption but should do compression. Which in the easiest first look would require ssl_ciphers to be controllable from pg_hba.conf - but that doesn't work since we don't get to pg_hba.conf until after we've negotiated the SSL mode... So there's quite a bit of complexity that needs to be put in there just to deal with the fact that we're using SSL to do compression, if we want to support it in a way that's not hackish. That's exactly the complexity I wouldn't add to the code. I'm in favor of experimenting an standard algorithm (zlib, for example -- let's say, it is the easiest way to implement it) or even hooks (libpq and backend -- that seems to be complex but less than openssl-backed connection just for compression). -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Sat, Jun 16, 2012 at 6:39 AM, Magnus Hagander mag...@hagander.net wrote: On Sat, Jun 16, 2012 at 4:04 AM, Euler Taveira eu...@timbira.com wrote: On 15-06-2012 11:39, Magnus Hagander wrote: As long as a free implementation exists, it can be ported to Java/.Net. Sure, it takes more work, but it *can be done*. Good point. IMHO, if there isn't a solution that cover all PostgreSQL (it seems it is not), we should pick the most appropriate one for *libpq* and let other drivers implement it at their time. Fair enough if we decide that - but we should make that decision knowing that we're leaving the JDBC and .Net people in a bad position where they are not likely to be able to implement his. The JDBC people have a theoretical chance if the JDK is open. The .Net people are stuck with schannel that doesn't support it at this point. It might well do in the future (since it's in the standard); but they're at the mercy of Microsoft. Both Java and C# are open-source enough that anybody can take existing SSL implementation and add compression to it, then distribute it as improved SSL library. -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Marko Kreen mark...@gmail.com writes: On Sat, Jun 16, 2012 at 6:39 AM, Magnus Hagander mag...@hagander.net wrote: Fair enough if we decide that - but we should make that decision knowing that we're leaving the JDBC and .Net people in a bad position where they are not likely to be able to implement his. The JDBC people have a theoretical chance if the JDK is open. The .Net people are stuck with schannel that doesn't support it at this point. It might well do in the future (since it's in the standard); but they're at the mercy of Microsoft. Both Java and C# are open-source enough that anybody can take existing SSL implementation and add compression to it, then distribute it as improved SSL library. Possibly more to the point: that is work they might have to do, if nobody else steps up to the plate --- and if they do end up doing it, it could benefit other projects too. On the other hand, if we roll-our-own transport compression solution, that is work they *will* have to do, with no chance of sharing the effort with other projects. BTW, as far as the .Net case goes, it took only a moment's googling to find this: http://openssl-net.sourceforge.net/ which is a .Net wrapper around real OpenSSL. It doesn't appear to provide wrappers for the compression selection functions, but surely that's just a lack of round tuits, not that it would take more than five minutes to add them. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Magnus Hagander mag...@hagander.net writes: Yes, but there's also a lot of such awkward logic we need to add if we *do* go with the SSL library doing the compression: For example, we can no longer trust the SSL library to always do encryption, since we specifically want to support null encryption. True, but are you sure we don't need to do that anyway? What happens today, if a non-libpq client connects with SSL and specifies null encryption? And we currently have no way to specify different encryption options on a per-host basis, which is something we'd have to do (e.g. i want to be able to say that subnet x requires encryption with these encryptions methods and subnet y doesn't require encryption but should do compression. [ shrug... ] Having that sort of control over a homebrew compression solution will *also* require a lot of control logic that does not exist today. So there's quite a bit of complexity that needs to be put in there just to deal with the fact that we're using SSL to do compression, if we want to support it in a way that's not hackish. It's not obvious to me that we actually *need* anything except the ability to recognize that a null-encrypted SSL connection probably shouldn't be treated as matching a hostssl line; which is not something that requires any fundamental rearrangements, since it only requires an after-the-fact check of what was selected. Things like subnet x requires encryption with these encryption methods are features that are sensible with our existing feature set. But we don't have that now and nobody has asked for it, so I think you are moving the goalposts rather unfairly by claiming that a compression-related patch needs to add it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Euler Taveira wrote: There was already some discussion about compressing libpq data [1][2][3]. Recently, I faced a scenario that would become less problematic if we have had compression support. The scenario is frequent data load (aka COPY) over slow/unstable links. It should be executed in a few hundreds of PostgreSQL servers all over Brazil. Someone could argue that I could use ssh tunnel to solve the problem but (i) it is complex because it involves a different port in the firewall and (ii) it's an opportunity to improve other scenarios like reducing bandwidth consumption during replication or normal operation over slow/unstable links. Maybe I'm missing something obvious, but shouldn't a regular SSL connection (sslmode=require) do what you are asking for? At least from OpenSSL 0.9.8 on, data get compressed by default. You don't need an extra port in the firewall for that. Yours, Laurenz Albe -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On 14-06-2012 02:19, Tom Lane wrote: I still think that pushing this off to openssl (not an ssh tunnel, but the underlying transport library) would be an adequate solution. If you are shoving data over a connection that is long enough to need compression, the odds that every bit of it is trustworthy seem pretty small, so you need encryption too. I don't want to pay the SSL connection overhead. Also I just want compression, encryption is not required. OpenSSL give us encryption with/without compression; we need an option to obtain compression in non-SSL connections. We do need the ability to tell openssl to use compression. We don't need to implement it ourselves, nor to bring a bunch of new library dependencies into our builds. I especially think that importing bzip2 is a pretty bad idea --- it's not only a new dependency, but bzip2's compression versus speed tradeoff is entirely inappropriate for this use-case. I see, the idea is that bzip2 would be a compiled-in option (not enabled by default) just to give another compression option. I don't have a strong opinion about including it as another dependency. We already depend on zlib and implementing compression using it won't add another dependency. What do you think about adding a hook at libpq to load an extension that does the compression? That way we don't add another dependency at libpq and also a lot of extensions could be coded to cover a variety of algorithms without putting us in trouble because of patent infringement. -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Jun14, 2012, at 15:28 , Euler Taveira wrote: On 14-06-2012 02:19, Tom Lane wrote: I still think that pushing this off to openssl (not an ssh tunnel, but the underlying transport library) would be an adequate solution. If you are shoving data over a connection that is long enough to need compression, the odds that every bit of it is trustworthy seem pretty small, so you need encryption too. I don't want to pay the SSL connection overhead. Also I just want compression, encryption is not required. OpenSSL give us encryption with/without compression; we need an option to obtain compression in non-SSL connections. AFAIR, openssl supports a NULL cipher which doesn't do any encryption. We could have a connection parameter, say compress=on, which selects that cipher (unless sslmode is set to prefer or higher, of course). SSL NULL-chipher connections would be treated like unencrypted connections when matching against pg_hba.conf. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Thu, Jun 14, 2012 at 10:14 AM, Florian Pflug f...@phlo.org wrote: On Jun14, 2012, at 15:28 , Euler Taveira wrote: On 14-06-2012 02:19, Tom Lane wrote: I still think that pushing this off to openssl (not an ssh tunnel, but the underlying transport library) would be an adequate solution. If you are shoving data over a connection that is long enough to need compression, the odds that every bit of it is trustworthy seem pretty small, so you need encryption too. I don't want to pay the SSL connection overhead. Also I just want compression, encryption is not required. OpenSSL give us encryption with/without compression; we need an option to obtain compression in non-SSL connections. AFAIR, openssl supports a NULL cipher which doesn't do any encryption. We could have a connection parameter, say compress=on, which selects that cipher (unless sslmode is set to prefer or higher, of course). SSL NULL-chipher connections would be treated like unencrypted connections when matching against pg_hba.conf. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers It doesn't sound like there is a lot of support for this idea, but I think it would be nice to get something like lz4 (http://code.google.com/p/lz4/) or snappy (http://code.google.com/p/snappy/) support. Both are BSD-ish licensed. It could be useful for streaming replication as well. A hook (as Euler mentioned) might be a nice compromise. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Thu, Jun 14, 2012 at 9:57 AM, Phil Sorber p...@omniti.com wrote: It doesn't sound like there is a lot of support for this idea, but I think it would be nice to get something like lz4 (http://code.google.com/p/lz4/) or snappy (http://code.google.com/p/snappy/) support. Both are BSD-ish licensed. It could be useful for streaming replication as well. A hook (as Euler mentioned) might be a nice compromise. There is a lot of support for the idea: it's one of the more requested features. I think a well thought out framework that bypassed the dependency issues via plugging might get some serious traction. Emphasis 'on well thought out' :-). merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Euler Taveira eu...@timbira.com writes: On 14-06-2012 02:19, Tom Lane wrote: ... I especially think that importing bzip2 is a pretty bad idea --- it's not only a new dependency, but bzip2's compression versus speed tradeoff is entirely inappropriate for this use-case. I see, the idea is that bzip2 would be a compiled-in option (not enabled by default) just to give another compression option. I'm not particularly thrilled with let's have more compression options just to have options. Each such option you add is another source of fail-to-connect incompatibilities (when either the client or the server doesn't have it). Moreover, while there are a lot of compression algorithms out there, a lot of them are completely unsuited for this use-case. If memory serves, bzip2 for example requires fairly large data blocks in order to get decent compression, which means you are either going to get terrible compression or suffer very bad latency when trying to apply it to a connection data stream. So I've got very little patience with the idea of let's put in some hooks and then great things will happen. It would be far better all around if we supported exactly one, well-chosen, method. But really I still don't see a reason not to let openssl do it for us. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Thu, Jun 14, 2012 at 1:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: So I've got very little patience with the idea of let's put in some hooks and then great things will happen. It would be far better all around if we supported exactly one, well-chosen, method. But really I still don't see a reason not to let openssl do it for us. Well, for toast compression the right choice is definitely one of the lz based algorithms (not libz!). For transport compression you have the case of sending large data over very slow and/or expensive links in which case you want to use bzip type methods. But in the majority of cases I'd probably be using lz there too. So if I had to pick just one, there you go. But which one? the lz algorithm with arguably the best pedigree (lzo) is gnu but there are many other decent candidates, some of which have really tiny implementations. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Thu, Jun 14, 2012 at 02:38:02PM -0500, Merlin Moncure wrote: On Thu, Jun 14, 2012 at 1:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: So I've got very little patience with the idea of let's put in some hooks and then great things will happen. It would be far better all around if we supported exactly one, well-chosen, method. But really I still don't see a reason not to let openssl do it for us. Well, for toast compression the right choice is definitely one of the lz based algorithms (not libz!). For transport compression you have the case of sending large data over very slow and/or expensive links in which case you want to use bzip type methods. But in the majority of cases I'd probably be using lz there too. So if I had to pick just one, there you go. But which one? the lz algorithm with arguably the best pedigree (lzo) is gnu but there are many other decent candidates, some of which have really tiny implementations. merlin +1 for a very fast compressor/de-compressor. lz4 from Google has a BSD license and at 8.5X faster compression than zlib(-1) and 5X faster de-compression the zlib (-1), 2X faster than LZO even would be my pick. Regards, Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Thu, Jun 14, 2012 at 02:43:04PM -0400, Tom Lane wrote: Euler Taveira eu...@timbira.com writes: On 14-06-2012 02:19, Tom Lane wrote: ... I especially think that importing bzip2 is a pretty bad idea --- it's not only a new dependency, but bzip2's compression versus speed tradeoff is entirely inappropriate for this use-case. I see, the idea is that bzip2 would be a compiled-in option (not enabled by default) just to give another compression option. I'm not particularly thrilled with let's have more compression options just to have options. Each such option you add is another source of fail-to-connect incompatibilities (when either the client or the server doesn't have it). Moreover, while there are a lot of compression algorithms out there, a lot of them are completely unsuited for this use-case. If memory serves, bzip2 for example requires fairly large data blocks in order to get decent compression, which means you are either going to get terrible compression or suffer very bad latency when trying to apply it to a connection data stream. So I've got very little patience with the idea of let's put in some hooks and then great things will happen. It would be far better all around if we supported exactly one, well-chosen, method. But really I still don't see a reason not to let openssl do it for us. Do we just need to document SSL's NULL encryption option? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 15, 2012 at 10:19 AM, Bruce Momjian br...@momjian.us wrote: On Thu, Jun 14, 2012 at 02:43:04PM -0400, Tom Lane wrote: Euler Taveira eu...@timbira.com writes: On 14-06-2012 02:19, Tom Lane wrote: ... I especially think that importing bzip2 is a pretty bad idea --- it's not only a new dependency, but bzip2's compression versus speed tradeoff is entirely inappropriate for this use-case. I see, the idea is that bzip2 would be a compiled-in option (not enabled by default) just to give another compression option. I'm not particularly thrilled with let's have more compression options just to have options. Each such option you add is another source of fail-to-connect incompatibilities (when either the client or the server doesn't have it). Moreover, while there are a lot of compression algorithms out there, a lot of them are completely unsuited for this use-case. If memory serves, bzip2 for example requires fairly large data blocks in order to get decent compression, which means you are either going to get terrible compression or suffer very bad latency when trying to apply it to a connection data stream. Agreed. I think there's probably arguments to be had for supporting compression without openssl (see below), but I don't think we need to have a whole set of potentially incompatible ways of doing it. Picking one that's good for the common case and not completely crap for the corner cases would be a better choice (meaning bzip2 is probably a very bad choice). So I've got very little patience with the idea of let's put in some hooks and then great things will happen. It would be far better all around if we supported exactly one, well-chosen, method. But really I still don't see a reason not to let openssl do it for us. Do we just need to document SSL's NULL encryption option? Does the SSL NULL encryption+compression thing work if you're not using openssl? For one thing, some of us still hold a hope to support non-openssl libraries in both libpq and server side, so it's something that would need to be supported by the standard and thus available in most libraries not to invalidate that. Second, we also have things like the JDBC driver and the .Net driver that don't use libpq. the JDBC driver uses the native java ssl support, AFAIK. Does that one support the compression, and does it support controlling it? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] libpq compression
Hi, There was already some discussion about compressing libpq data [1][2][3]. Recently, I faced a scenario that would become less problematic if we have had compression support. The scenario is frequent data load (aka COPY) over slow/unstable links. It should be executed in a few hundreds of PostgreSQL servers all over Brazil. Someone could argue that I could use ssh tunnel to solve the problem but (i) it is complex because it involves a different port in the firewall and (ii) it's an opportunity to improve other scenarios like reducing bandwidth consumption during replication or normal operation over slow/unstable links. AFAICS there aren't objections about implementing compression in libpq. The problem is what algorithm use for compression. I mean, there is a lot of patents in this area. As others spotted at [4], we should not implement algorithms that possibly infringe patents in core. Derivated products are free to plug whatever algorithms they want. There will be an API to do it. This work will be sponsored by a company that is interested in this feature. === Design === - algorithm: zlib, bzip2, (another patent free and bsd licensed?) - compiled-in option: --with-bzip2 - PGCOMPRESSMODE env * disable: only try non-compressed connection (default) * prefer: try compressed connection; if that fails, try a non-compressed connection * require: only try compressed connection - PGCOMPRESSALGO env * zlib * bzip2 - compressmode and compressalgo string connection - compress all data - compress before send() and decompress after recv() I am all ears for improving this design. Some of my choices are based on my research in compression at protocols and PostgreSQL internals. Keep in mind that I prefer compressing all data instead of a selected set of messages because (i) every new data message could be coded with compression support and (ii) avoid that the protocol code turns into a spaghetti. I'll try to post a patch soon with the ideas discussed at this thread. [1] http://archives.postgresql.org/pgsql-hackers/2012-03/msg00929.php [2] http://archives.postgresql.org/pgsql-hackers/2011-01/msg00337.php [3] http://archives.postgresql.org/pgsql-hackers/2002-03/msg00664.php [4] http://archives.postgresql.org/pgsql-performance/2009-08/msg00053.php -- Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
Euler Taveira eu...@timbira.com writes: There was already some discussion about compressing libpq data [1][2][3]. Recently, I faced a scenario that would become less problematic if we have had compression support. The scenario is frequent data load (aka COPY) over slow/unstable links. It should be executed in a few hundreds of PostgreSQL servers all over Brazil. Someone could argue that I could use ssh tunnel to solve the problem but (i) it is complex because it involves a different port in the firewall and (ii) it's an opportunity to improve other scenarios like reducing bandwidth consumption during replication or normal operation over slow/unstable links. I still think that pushing this off to openssl (not an ssh tunnel, but the underlying transport library) would be an adequate solution. If you are shoving data over a connection that is long enough to need compression, the odds that every bit of it is trustworthy seem pretty small, so you need encryption too. We do need the ability to tell openssl to use compression. We don't need to implement it ourselves, nor to bring a bunch of new library dependencies into our builds. I especially think that importing bzip2 is a pretty bad idea --- it's not only a new dependency, but bzip2's compression versus speed tradeoff is entirely inappropriate for this use-case. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
