Re: [HACKERS] EvalPlanQual behaves oddly for FDW queries involving system columns

2015-04-14 Thread Kyotaro HORIGUCHI
Hello, At Tue, 14 Apr 2015 12:10:35 +0900, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote in 552c852b.2050...@lab.ntt.co.jp On 2015/04/13 23:25, Jim Nasby wrote: On 4/13/15 4:58 AM, Etsuro Fujita wrote: On 2015/04/10 21:40, Etsuro Fujita wrote: On 2015/04/09 12:07, Etsuro Fujita wrote:

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-04-14 Thread Shigeru HANADA
KaiGai-san, 2015/04/14 14:04、Kouhei Kaigai kai...@ak.jp.nec.com のメール: * Fix typos Please review the v11 patch, and mark it as “ready for committer” if it’s ok. It's OK for me, and wants to be reviewed by other people to get it committed. Thanks! In addition to essential features, I

[HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Michael Paquier
Hi all, I noticed that src/bin/initdb/t/001_initdb.pl uses directly rm via a system() call like that: system_or_bail rm -rf '$tempdir'/*; This way of doing is not portable, particularly on platforms that do not have rm like... Windows where the equivalent is del. And we could actually use

Re: [HACKERS] inherit support for foreign tables

2015-04-14 Thread Etsuro Fujita
On 2015/03/23 2:57, Tom Lane wrote: Etsuro Fujita fujita.ets...@lab.ntt.co.jp writes: [ fdw-inh-8.patch ] I've committed this with some substantial rearrangements, notably: * I thought that if we were doing this at all, we should go all the way and allow foreign tables to be both

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Magnus Hagander
On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 04/14/2015 05:42 AM, Robert Haas wrote: On Sun, Apr 12, 2015 at 8:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: As to RLS - yeah, that's where I think a lot of the possible covert channel attacks are. But it

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Stephen Frost
* Magnus Hagander (mag...@hagander.net) wrote: On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas hlinn...@iki.fi wrote: I'm not a big fan of locking down WAL position information either. If we have to treat the current WAL position is security-sensitive information, we're doing something

Re: [HACKERS] [COMMITTERS] pgsql: Use Intel SSE 4.2 CRC instructions where available.

2015-04-14 Thread Simon Riggs
On 14 April 2015 at 11:34, Heikki Linnakangas hlinn...@iki.fi wrote: Or we can punt and always build the version with the runtime check, unless overridden manually at configure command line. That seems safer as the default, which is what the original patch did. We can optimise that for known

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread David E. Wheeler
On Apr 14, 2015, at 9:05 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: http://perldoc.perl.org/File/Path.html With this formulation: remove_tree($tempdir, {keep_root = 1}); Does Perl 5.8 have this? Yes, it does. http://cpansearch.perl.org/src/NWCLARK/perl-5.8.9/lib/File/Path.pm

Re: [HACKERS] deparsing utility commands

2015-04-14 Thread David Steele
On 4/9/15 12:14 PM, Alvaro Herrera wrote: Alvaro Herrera wrote: Executive summary: There is now a CommandDeparse_hook; deparse_utility_command is provided as an extension, intended for 9.6; rest of patch would be pushed to 9.5. Actually here's another approach I like better: use a new

Re: [HACKERS] BRIN range operator class

2015-04-14 Thread Emre Hasegeli
Judging from a quick look, I think patches 1 and 5 can be committed quickly; they imply no changes to other parts of BRIN. (Not sure why 1 and 5 are separate. Any reason for this?) Also patch 2. Not much reason except that 1 includes only functions, but 5 includes operators. Patch 4 looks

[HACKERS] Re: [COMMITTERS] pgsql: Use Intel SSE 4.2 CRC instructions where available.

2015-04-14 Thread Heikki Linnakangas
On 04/14/2015 07:01 PM, Simon Riggs wrote: On 14 April 2015 at 11:34, Heikki Linnakangas hlinn...@iki.fi wrote: Or we can punt and always build the version with the runtime check, unless overridden manually at configure command line. That seems safer as the default, which is what the

Re: [HACKERS] inherit support for foreign tables

2015-04-14 Thread Jim Nasby
On 4/14/15 5:49 AM, Etsuro Fujita wrote: postgres=# create foreign table ft1 (c1 int) server myserver options (table_name 't1'); CREATE FOREIGN TABLE postgres=# create foreign table ft2 (c1 int) server myserver options (table_name 't2'); CREATE FOREIGN TABLE postgres=# alter foreign table

Re: [HACKERS] EvalPlanQual behaves oddly for FDW queries involving system columns

2015-04-14 Thread Jim Nasby
On 4/14/15 1:05 AM, Kyotaro HORIGUCHI wrote: As an example, the following operations cause an unexpected result. Ex. 1 Session A=# create table t (a int primary key, b int); Session A=# insert into t (select a, 1 from generate_series(0, 99) a); Session A=# begin; Session A=# select * from t

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Alvaro Herrera
Michael Paquier wrote: Hi all, I noticed that src/bin/initdb/t/001_initdb.pl uses directly rm via a system() call like that: system_or_bail rm -rf '$tempdir'/*; This way of doing is not portable, particularly on platforms that do not have rm like... Windows where the equivalent is del.

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Heikki Linnakangas
On 04/14/2015 05:42 AM, Robert Haas wrote: On Sun, Apr 12, 2015 at 8:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: Care to name some? This is certainly quite cumbersome to exploit, but it's doable. We've talked a lot about covert channels and timing attacks on RLS, but this makes me more

[HACKERS] Re: [COMMITTERS] pgsql: Use Intel SSE 4.2 CRC instructions where available.

2015-04-14 Thread Heikki Linnakangas
On 04/14/2015 06:28 PM, Simon Riggs wrote: On 14 April 2015 at 10:09, Heikki Linnakangas heikki.linnakan...@iki.fi wrote: Abhijit Menon-Sen, heavily modified by me, reviewed by Andres Freund. Did the heavy modifications have any affect on the patch behaviour, or was this just related to

Re: [HACKERS] Auditing extension for PostgreSQL (Take 2)

2015-04-14 Thread David Steele
On 4/14/15 7:13 PM, Tatsuo Ishii wrote: This patch does not apply cleanly due to the moving of pgbench (patch to filelist.sgml failed). Thank you for pointing that out! Ironic that it was the commit directly after the one I was testing with that broke the patch. It appears the end of the last

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Michael Paquier
On Wed, Apr 15, 2015 at 5:29 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: David E. Wheeler wrote: On Apr 14, 2015, at 1:21 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Castoroides has 5.8.4. Oops. WUT. Yeah, eh? Anyway I don't think it matters much: just don't enable TAP

Re: [HACKERS] inherit support for foreign tables

2015-04-14 Thread Kyotaro HORIGUCHI
Hi, Before suppressing the symptom, I doubt the necessity and/or validity of giving foreign tables an ability to be a parent. Is there any reasonable usage for the ability? I think we should choose to inhibit foreign tables from becoming a parent rather than leaving it allowed then taking

Re: [HACKERS] Auditing extension for PostgreSQL (Take 2)

2015-04-14 Thread Tatsuo Ishii
Thank you for pointing that out! Ironic that it was the commit directly after the one I was testing with that broke the patch. It appears the end of the last CF is a very bad time to be behind HEAD. Fixed in attached v8 patch. Thank you for your quick response. BTW, in my understanding

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Michael Paquier
On Wed, Apr 15, 2015 at 11:10 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Apr 15, 2015 at 12:00 AM, Magnus Hagander mag...@hagander.net wrote: On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 04/14/2015 05:42 AM, Robert Haas wrote: On Sun, Apr 12, 2015

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Michael Paquier
On Wed, Apr 15, 2015 at 12:15 AM, Stephen Frost wrote: We need a proper hardening guide anyway, this would just need to be included in that documentation. +1. I am sure that many users would like a hardening manual in the official documentation. -- Michael -- Sent via pgsql-hackers mailing

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Fujii Masao
On Wed, Apr 15, 2015 at 12:00 AM, Magnus Hagander mag...@hagander.net wrote: On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 04/14/2015 05:42 AM, Robert Haas wrote: On Sun, Apr 12, 2015 at 8:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: As to RLS - yeah,

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Michael Paquier
On Wed, Apr 15, 2015 at 11:55 AM, Michael Paquier wrote: OK. I am fine to implement anything required here if needed, meaning the following: 1) Doc patch to mention that it is possible that compression can give hints to attackers when working on sensible fields that have a non-fixed size. 2)

Re: [HACKERS] Auditing extension for PostgreSQL (Take 2)

2015-04-14 Thread Sawada Masahiko
On Wed, Apr 15, 2015 at 8:57 AM, David Steele da...@pgmasters.net wrote: On 4/14/15 7:13 PM, Tatsuo Ishii wrote: This patch does not apply cleanly due to the moving of pgbench (patch to filelist.sgml failed). Thank you for pointing that out! Ironic that it was the commit directly after the

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-14 Thread Robert Haas
On Tue, Apr 14, 2015 at 6:07 PM, Simon Riggs si...@2ndquadrant.com wrote: On 11 March 2015 at 20:55, Peter Eisentraut pete...@gmx.net wrote: I don't know how to move forward. We could give users a knob: This might make your queries faster or not -- good luck. But of course nobody will like

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Robert Haas
On Tue, Apr 14, 2015 at 6:22 PM, Peter Geoghegan p...@heroku.com wrote: Why is that good? We did discuss this before. I've recapped some of what I believe to be the most salient points below. I think that people were all too quick to dismiss the idea of a wall time interval playing some role

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Amit Kapila
On Wed, Apr 15, 2015 at 2:55 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Apr 16, 2014 at 2:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: Merlin Moncure mmonc...@gmail.com writes: Anyways, I'm still curious if you can post similar numbers basing the throttling on gross allocation

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Noah Misch
On Tue, Apr 14, 2015 at 05:29:36PM -0300, Alvaro Herrera wrote: David E. Wheeler wrote: On Apr 14, 2015, at 1:21 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Castoroides has 5.8.4. Oops. WUT. Yeah, eh? Anyway I don't think it matters much: just don't enable TAP tests

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Robert Haas
On Tue, Apr 14, 2015 at 7:10 PM, Greg Stark st...@mit.edu wrote: The way the clock sweep algorithm is meant to be thought about is that it's an approximate lru. Each usage count corresponds to an ntile of the lru. So we don't know which buffer is least recently used but it must be in the set

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Fujii Masao
On Wed, Apr 15, 2015 at 11:55 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Apr 15, 2015 at 11:10 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Apr 15, 2015 at 12:00 AM, Magnus Hagander mag...@hagander.net wrote: On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Robert Haas
On Wed, Apr 15, 2015 at 12:15 AM, Amit Kapila amit.kapil...@gmail.com wrote: IIUC, this will allow us to increase usage count only when the buffer is touched by clocksweep to decrement the usage count. Yes. I think such a solution will be good for the cases when many evictions needs to be

Re: [HACKERS] What exactly is our CRC algorithm?

2015-04-14 Thread Heikki Linnakangas
On 04/03/2015 05:28 AM, Abhijit Menon-Sen wrote: At 2015-04-03 00:33:10 +0300, hlinn...@iki.fi wrote: I came up with the attached. I like it very much. src/port/Makefile has (note src/srv): +# pg_crc32c_sse42.o and its _src.o version need CFLAGS_SSE42 +pg_crc32c_sse42.o:

Re: [HACKERS] logical column ordering

2015-04-14 Thread Robert Haas
On Tue, Apr 14, 2015 at 2:38 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: As I said, I'm still writing the first pieces of this so I'm not sure what other ramifications it will have. If there are any thoughts, I would appreciate them. (Particularly useful input on whether it is

Re: [HACKERS] logical column ordering

2015-04-14 Thread Alvaro Herrera
I've been looking at this again. It has become apparent to me that what we're doing in parse analysis is wrong, and the underlying node representation is wrong too. Here's a different approach, which I hope will give better fruits. I'm still working on implementing the ideas here (and figuring

Re: [HACKERS] inherit support for foreign tables

2015-04-14 Thread Alvaro Herrera
Jim Nasby wrote: On 4/14/15 5:49 AM, Etsuro Fujita wrote: postgres=# create foreign table ft1 (c1 int) server myserver options (table_name 't1'); CREATE FOREIGN TABLE postgres=# create foreign table ft2 (c1 int) server myserver options (table_name 't2'); CREATE FOREIGN TABLE

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread David Fetter
On Tue, Apr 14, 2015 at 09:25:33AM -0700, David E. Wheeler wrote: On Apr 14, 2015, at 9:05 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: http://perldoc.perl.org/File/Path.html With this formulation: remove_tree($tempdir, {keep_root = 1}); Does Perl 5.8 have this? Yes, it

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Alvaro Herrera
David E. Wheeler wrote: On Apr 14, 2015, at 9:05 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: http://perldoc.perl.org/File/Path.html With this formulation: remove_tree($tempdir, {keep_root = 1}); Does Perl 5.8 have this? Yes, it does.

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Alvaro Herrera
David E. Wheeler wrote: On Apr 14, 2015, at 1:21 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Castoroides has 5.8.4. Oops. WUT. Yeah, eh? Anyway I don't think it matters much: just don't enable TAP tests on machines with obsolete Perl. I think this is fine since 5.8's latest

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread David E. Wheeler
On Apr 14, 2015, at 1:21 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Castoroides has 5.8.4. Oops. WUT. smime.p7s Description: S/MIME cryptographic signature

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Jim Nasby
On 4/14/15 5:22 PM, Peter Geoghegan wrote: As long as we're doing random brainstorming, I'd suggest looking at making clocksweep actually approximate LRU-K/LRU-2 (which, again, to be clear, my prototype did not do). The clocksweep could maintain statistics about the recency of the second-to-last

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-14 Thread Simon Riggs
On 11 March 2015 at 20:55, Peter Eisentraut pete...@gmx.net wrote: I don't know how to move forward. We could give users a knob: This might make your queries faster or not -- good luck. But of course nobody will like that either. What is clear is that large SELECT queries are doing the work

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Peter Geoghegan
On Tue, Apr 14, 2015 at 2:25 PM, Robert Haas robertmh...@gmail.com wrote: So, I was thinking about this a little bit more today, prodded by my coworker John Gorman. I'm wondering if we could drive this off of the clock sweep; that is, every time the clock sweep touches a buffer, its usage

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Robert Haas
On Wed, Apr 16, 2014 at 2:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: Merlin Moncure mmonc...@gmail.com writes: Anyways, I'm still curious if you can post similar numbers basing the throttling on gross allocation counts instead of time. Meaning: some number of buffer allocations has to have

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Greg Stark
I've been meaning to write this since PGConf and now isn't a great time since I'm on my phone but I think it's time. The way the clock sweep algorithm is meant to be thought about is that it's an approximate lru. Each usage count corresponds to an ntile of the lru. So we don't know which buffer

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-14 Thread David Steele
On 4/14/15 6:07 PM, Simon Riggs wrote: On 11 March 2015 at 20:55, Peter Eisentraut pete...@gmx.net wrote: I don't know how to move forward. We could give users a knob: This might make your queries faster or not -- good luck. But of course nobody will like that either. What is clear is

Re: [HACKERS] Auditing extension for PostgreSQL (Take 2)

2015-04-14 Thread Tatsuo Ishii
This patch does not apply cleanly due to the moving of pgbench (patch to filelist.sgml failed). Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese:http://www.sraoss.co.jp Attached is the v7 pg_audit patch. I've tried to address Peter's