Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Antonin Houska
ace, we expect this value is enough * for most usecase. */ #define KMGR_KEYRING_SIZE 128 There's no guarantee that the number of tablespaces won't exceed any (reasonably low) constant value. The KMGR module should be able to allocate additional memory dynamically. [

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-19 Thread Antonin Houska
Tomas Vondra wrote: > On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote: > >Tomas Vondra wrote: > > > >> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: > >> >On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-19 Thread Antonin Houska
new fork would be that contents of its buffer (the MAC values) is not determined until the corresponding buffers of the MAIN fork get encrypted. However encryption is performed by the storage layer (md.c), which is not expected to lock other buffers (such as those of the "MAC fork"), read their pages from disk or insert their WAL records. This is different from the FSM or VM forks whose buffers are only updated above the storage layer. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-19 Thread Antonin Houska
ably make the database crash soon.) We can guarantee integrity and authenticity of backup, but that's a separate feature: someone may need this although it's o.k. for him to run the cluster unencrypted. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: [HACKERS] WIP: Aggregation push-down

2019-07-17 Thread Antonin Houska
grouped_rel_list, rel); > - add_rel_info(root->agg_info_list, rel); > + add_rel_info(root->agg_info_list, agg_info); > } > Thanks! Yes, this is what I had to fix. -- Antonin Houska Web: https://www.cybertec-postgresql.com >From b2672f2b06f521fdc11f7e7293ab669ca58206

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Antonin Houska
x27;s competition between us but now that I've checked your patch set I see that you already use some parts of https://commitfest.postgresql.org/23/2104/ although not the latest version. I'm supposed to work on the encryption now, so thinking what to do next. I think we should coordinate the effort, possibly off-list. The earlier we have a single patch set the more efficient the work should be. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Antonin Houska
sqrt(2^b) and thus n ~ 2^(b/2) So if the number of safely encrypted blocks was derived this way, I agree that IV was not taken into consideration: if there is an IV, then identical blocks of ciphertext are not a problem because they represent different blocks of plaintext. [1] https://en.wikipedia.org/wiki/Birthday_problem#Square_approximation -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-11 Thread Antonin Houska
her because it complies to ... standard". -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: pg_log_fatal vs pg_log_error

2019-07-10 Thread Antonin Houska
Peter Eisentraut wrote: > On 2019-06-17 14:19, Antonin Houska wrote: > > Can anyone please give me a hint (and possibly add some comments to the > > code) > > when pg_log_fatal() should be used in frontend code and when it's > > appropriate > > to call pg

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Antonin Houska
nk that works, because that'd mean we're encrypting the same > page with the same nonce over and over, which means reusing the reuse > (even if you hash/encrypt it). Or did I miss something? I found out that it's wrong to use the same key (or (key, IV) pair) to encrypt different plain texts [1], however this is about *stream cipher*. There should be some evidence that *block cipher* has similar weakness before we accept another restriction on the IV setup. [1] https://en.wikipedia.org/wiki/Stream_cipher_attacks#Reused_key_attack -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Antonin Houska
not sure if this kind of attack can reveal more information than something about the first block of the page (the page header), since each of the following blocks uses ciphertext of the previous block as the IV. * "Altering the IV Before Decryption" I don't think this attack needs

Re: [HACKERS] WIP: Aggregation push-down

2019-07-09 Thread Antonin Houska
Richard Guo wrote: > Another rebase is needed for the patches. Done. -- Antonin Houska Web: https://www.cybertec-postgresql.com >From f656bd8d46afb9cb0a331cf3d34b9eed39f5e360 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Tue, 9 Jul 2019 15:30:13 +0200 Subject: [PATCH 1/3] Int

Re: Attempt to consolidate reading of XLOG page

2019-07-09 Thread Antonin Houska
Thomas Munro wrote: > On Tue, May 21, 2019 at 9:12 PM Antonin Houska wrote: > > Robert Haas wrote: > > > It seems to me that it's better to unwind the stack i.e. have the > > > function return the error information to the caller and let the caller > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-17 Thread Antonin Houska
Antonin Houska wrote: > Masahiko Sawada wrote: > > > The cluster-wide TDE eventually encrypts SLRU data and all WAL > > including non-user data related WAL while table/tablespace TDE doesn't > > unless we develop such functionality. In addition, the cluster-wide

Re: pg_log_fatal vs pg_log_error

2019-06-17 Thread Antonin Houska
Michael Paquier wrote: > On Mon, Jun 17, 2019 at 02:19:30PM +0200, Antonin Houska wrote: > > I'd expect that the pg_log_fatal() should be called when the error is > > serious > > enough to cause premature exit, but I can see cases where even > > pg_log_e

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-17 Thread Antonin Houska
s a good > choice for users. IMO it does not matter which approach (cluster vs table/tablespace) is implemented first. What matters is to design the user interface so that addition of the other of the two features does not make users confused. -- Antonin Houska Web: https://www.cybertec-postgresql.com

pg_log_fatal vs pg_log_error

2019-06-17 Thread Antonin Houska
r does not fully control (things that happen during the actual execution). But this is rather a guess. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-17 Thread Antonin Houska
I'm thinking how to teach postmaster to accept FEBE protocol connections temporarily, just to receive the key. The user applications like pg_ctl, initdb or pg_upgrade would retrieve the key / password from the DBA, then start postmaster and send it the key. Perhaps the message format should be a

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-05 Thread Antonin Houska
Robert Haas wrote: > On Fri, May 31, 2019 at 2:59 AM Antonin Houska wrote: > > > Sounds good. I'm not quite sure how this is going to work. Ideally > > > you'd like the encryption key command to fetch the key from something > > > like ssh-agent, or mayb

Comment typo in tableam.h

2019-05-31 Thread Antonin Houska
Please see the diff attached. -- Antonin Houska Web: https://www.cybertec-postgresql.com diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index bf3a472018..e3619b8e7e 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -1331,7 +1331,7

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-05-31 Thread Antonin Houska
Robert Haas wrote: > On Tue, May 28, 2019 at 11:27 AM Antonin Houska wrote: > > We thought that it's more convenient for administrator to enter password. To > > achieve that, we can instead tell the admin how to use the key derivation > > tool > > (pg_keysetup)

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-05-28 Thread Antonin Houska
Thanks of another round of review. Robert Haas wrote: > On Wed, Apr 24, 2019 at 10:48 AM Antonin Houska wrote: > > Attached is the next version. It tries to address various problems pointed > > out > > upthread, including documentation. > > It's not immediatel

Re: Remove page-read callback from XLogReaderState.

2019-05-28 Thread Antonin Houska
Kyotaro HORIGUCHI wrote: > v3-0001 : Changed macrosas suggested. This attachment is missing, please send it too. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Converting NOT IN to anti-joins during planning

2019-05-27 Thread Antonin Houska
Antonin Houska wrote: > One problem I see is that SubLink can be in the JOIN/ON clause and thus it's > not necessarily at the top of the join tree. Consider this example: > > CREATE TABLE a(i int); > CREATE TABLE b(j int); > CREATE TABLE c(k int NOT NULL); > CREATE TAB

Re: Converting NOT IN to anti-joins during planning

2019-05-27 Thread Antonin Houska
n condition between "a" and "d" is not strict in this case, the ANTI JOIN will receive the NULL values of a.i. It seems tricky, I've got no idea of an alternative approach right now. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Remove page-read callback from XLogReaderState.

2019-05-22 Thread Antonin Houska
goto *((void *) (XLR_STATE).j); \ XLR_CASE(XLR_INIT_STATE); \ } while (0) I think it's better to set the label always to (&&XLR_INIT_STATE) so that XLR_SWITCH can perform the jump unconditionally. Attached is also an (unrelated) comment fix proposal

Re: A few more opportunities to use TupleTableSlotOps fields

2019-05-21 Thread Antonin Houska
Robert Haas wrote: > On Tue, May 21, 2019 at 8:02 AM Antonin Houska wrote: > > Perhaps it's just a matter of taste, but I think the TupleTableSlotOps > > structure, once initialized, should be used wherever possible. At least for > > me > > personally, when

A few more opportunities to use TupleTableSlotOps fields

2019-05-21 Thread Antonin Houska
Perhaps it's just a matter of taste, but I think the TupleTableSlotOps structure, once initialized, should be used wherever possible. At least for me personally, when I read the code, the particular callback function name is a bit disturbing wherever it's not necessary. -- Antonin

Re: Attempt to consolidate reading of XLOG page

2019-05-21 Thread Antonin Houska
Robert Haas wrote: > It seems to me that it's better to unwind the stack i.e. have the > function return the error information to the caller and let the caller > do as it likes. Thanks for a hint. The next version tries to do that. -- Antonin Houska Web: https://www.cybertec-

Inaccurate header comment of issue_xlog_fsync_comment

2019-05-20 Thread Antonin Houska
Someone probably forgot to update the comment when changing the arguments. -- Antonin Houska Web: https://www.cybertec-postgresql.com diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index c00b63c751..527522f165 100644 --- a/src/backend/access/transam/xlog.c

Re: Attempt to consolidate reading of XLOG page

2019-05-02 Thread Antonin Houska
Antonin Houska wrote: > Alvaro Herrera wrote: > > > Not sure about the walsize; maybe it can be a member in XLogReadPos, and > > given to XLogReadInitPos()? (Maybe rename XLogReadPos as > > XLogReadContext or something like that, indicating it's not just th

Re: Remove page-read callback from XLogReaderState.

2019-04-25 Thread Antonin Houska
g with ->reqLen. Maybe also ->loadPagePtr should be renamed to ->targetPagePtr. * trailing whitespace: xlogreader.h:130, xlogreader.c:1058 * The 2nd argument of SimpleXLogPageRead() is "private", which seems too generic given that the function is no longer used as a callb

Accidental setting of XLogReaderState.private_data ?

2019-04-15 Thread Antonin Houska
ied. Thus I assume that assignment is a thinko, isn't it? -- Antonin Houska Web: https://www.cybertec-postgresql.com diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index 424fe86a1b..fbca486b59 100644 --- a/src/backend/replication/logical/log

Re: Attempt to consolidate reading of XLOG page

2019-04-15 Thread Antonin Houska
ad position.) As pointed out by others, XLogReadPos is not necessary. So if XLogRead() receives XLogReaderState instead, it can get the segment size from there. Thanks. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Attempt to consolidate reading of XLOG page

2019-04-15 Thread Antonin Houska
register it to the next CF if you want feedback. ok, will do. (A link to mailing list is needed for the CF entry, so I had to post something anyway :-) Since I don't introduce any kind of "cool new feature" here, I believe it did not disturb much.) > Here are some basic thoughts a

Re: Attempt to consolidate reading of XLOG page

2019-04-15 Thread Antonin Houska
Kyotaro HORIGUCHI wrote: > Hello. > > At Thu, 11 Apr 2019 18:05:42 +0200, Antonin Houska wrote > in <14984.1554998...@spoje.net> > > While working on the instance encryption I found it annoying to apply > > decyption of XLOG page to three different functions.

Attempt to consolidate reading of XLOG page

2019-04-11 Thread Antonin Houska
hanks. -- Antonin Houska Web: https://www.cybertec-postgresql.com diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c index f9a4960f8a..444b5bf910 100644 --- a/src/backend/access/transam/twophase.c +++ b/src/backend/access/transam/twophase.c @@ -1369,7 +1

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-06 Thread Antonin Houska
Robert Haas wrote: > On Fri, Apr 5, 2019 at 11:22 AM Antonin Houska wrote: > > > Wouldn't Tom's proposal to use a stream cipher fix all this? > > > > Yes it would make the extra alignment unnecessary, but our solution tries to > > meet specific requir

Re: Question on alignment

2019-04-05 Thread Antonin Houska
Tom Lane wrote: > Antonin Houska writes: > > Antonin Houska wrote: > >> Since palloc() only ensures MAXIMUM_ALIGNOF, that wouldn't help here > >> anyway. > > > After some more search I'm not sure about that. The following comment > > in

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-05 Thread Antonin Houska
Robert Haas wrote: > On Thu, Mar 21, 2019 at 7:46 AM Antonin Houska wrote: > > Nevertheless, with the current version of our patch, PG should be resistant > > against such a partial write anyway because we chose to align XLOG records > > to > > 16 bytes (as long a

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread Antonin Houska
seem likely that in this situation I'll be assigned many other tasks of higher priority. So yes, I expect to have quite some time for this patch. Thanks. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread Antonin Houska
Robert Haas wrote: > On Thu, Mar 14, 2019 at 9:26 AM Antonin Houska wrote: > > > Hint bits also rely on this principle. I thought there might be some > > > interaction between this work and wal_log_hints for this reason, but I see > > > nothing of that sort in t

Re: Question on alignment

2019-04-01 Thread Antonin Houska
Antonin Houska wrote: > Since palloc() only ensures MAXIMUM_ALIGNOF, that wouldn't help here anyway. After some more search I'm not sure about that. The following comment indicates that MAXALIGN helps too: /* * Use this, not "char buf[BLCKSZ]", to declare a field or lo

Re: Question on alignment

2019-04-01 Thread Antonin Houska
Heikki Linnakangas wrote: > On 01/04/2019 11:01, Antonin Houska wrote: > > In copydir.c:copy_file() I read > > > > /* Use palloc to ensure we get a maxaligned buffer */ > > buffer = palloc(COPY_BUF_SIZE); > > > > No data type wider than a single

Question on alignment

2019-04-01 Thread Antonin Houska
please explain why alignment matters here? -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-21 Thread Antonin Houska
to deal with encryption. This design decision can be changed, but there's one more thing to consider: if the XLOG stream is decrypted, the decryption cannot be disabled unless the XLOG records are aligned to 16 bytes (and in turn, the XLOG alignment cannot be enabled w/o initdb). -- Antonin Houska https://www.cybertec-postgresql.com

Re: Suggestions on message transfer among backends

2019-03-14 Thread Antonin Houska
Robert Haas wrote: > On Tue, Mar 12, 2019 at 4:34 AM Antonin Houska wrote: > > Andy Fan wrote: > > > I just don't know why shm_mq is designed to single-reader & single-writer. > > > > shm_mq was implemented as a part of infrastructure for parallel q

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-14 Thread Antonin Houska
er it should just refuse to start unless they are set properly, but I > think it probably needs to do one or the other. Maybe it's more polite to refuse to start so that user knows what's going on. I'm not sure if PG ever changes any configuration variable forcibly. > Hope this

Re: Suggestions on message transfer among backends

2019-03-12 Thread Antonin Houska
One queue is used to send request (query plan) to the worker, one queue is there to receive data from it, and I think there's one more queue to receive error messages. -- Antonin Houska https://www.cybertec-postgresql.com

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-03-08 Thread Antonin Houska
Antonin Houska wrote: > Masahiko Sawada wrote: > > > Agreed. > > > > For the WAL encryption, I wonder if we can have a encryption key > > dedicated for WAL. Regardless of keys of tables and indexes all WAL > > are encrypted with the WAL key. During

Re: Problems with plan estimates in postgres_fdw

2019-03-08 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/03/01 20:16), Antonin Houska wrote: > > Etsuro Fujita wrote: > > >> Conversely, it appears that add_foreign_ordered_paths() added by the > >> patchset > >> would generate such pre-sorted paths *redundantly* when the inpu

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-03-07 Thread Antonin Houska
pplies it, and then the table data will be > encrypted with its table key when flushing. So we just control the > scope of encryption object: WAL of tables and indexes etc or > everything. My point of view is that different key usually means different user. The user who can decrypt WAL can effectively see all the data, even though another user put them (encrypted with another key) into tables. So in this case, different keys don't really separate users in terms of data access. -- Antonin Houska https://www.cybertec-postgresql.com

Re: openLogOff is not needed anymore

2019-03-06 Thread Antonin Houska
Robert Haas wrote: > On Wed, Mar 6, 2019 at 6:10 AM Antonin Houska wrote: > > ... obviously due to commit c24dcd0. The following patch removes it. > > Committed, after a short struggle to get the patch out of the email > body in a usable form. It was just convenient for

openLogOff is not needed anymore

2019-03-06 Thread Antonin Houska
XLByteToPrevSeg(LogwrtResult.Write, openLogSegNo, wal_segment_size); openLogFile = XLogFileOpen(openLogSegNo); - openLogOff = 0; } issue_xlog_fsync(openLogFile, openLogSegNo); -- Antonin Houska https://www.cybertec-postgresql.com

Re: Ordered Partitioned Table Scans

2019-03-04 Thread Antonin Houska
David Rowley wrote: > On Mon, 5 Nov 2018 at 10:46, David Rowley > wrote: > > On 1 November 2018 at 22:05, Antonin Houska wrote: > > > I think these conditions are too restrictive: > > > > > > /* > > > * Determine if these pathk

Re: Problems with plan estimates in postgres_fdw

2019-03-03 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/03/01 20:00), Antonin Houska wrote: > > Etsuro Fujita wrote: > > It's still unclear to me why add_foreign_ordered_paths() passes the input > > relation (input_rel) to estimate_path_cost_size(). If it passed the output > > rel &g

Re: Problems with plan estimates in postgres_fdw

2019-03-01 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/03/01 20:00), Antonin Houska wrote: > > Etsuro Fujita wrote: > > I used gdb to help me understand, however the condition > > > > if (fpextra&& !IS_UPPER_REL(foreignrel)) > > > > never evaluated to true with the

Re: Problems with plan estimates in postgres_fdw

2019-03-01 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/23 0:21), Antonin Houska wrote: > > Etsuro Fujita wrote: > > >>> (2019/02/08 2:04), Antonin Houska wrote: > >>>> * regression tests: I think test(s) should be added for queries that have > >>>> ORDER BY claus

Re: Problems with plan estimates in postgres_fdw

2019-03-01 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/22 22:54), Antonin Houska wrote: > > Etsuro Fujita wrote: > >> On the other hand, the code bit added by > >> 0001-postgres_fdw-Perform-UPPERREL_ORDERED-step-remotely-v3.patch handles > >> the > >> case where a post-

Re: [HACKERS] WIP: Aggregation push-down

2019-02-28 Thread Antonin Houska
Tom Lane wrote: > Antonin Houska writes: > > Michael Paquier wrote: > >> Latest patch set fails to apply, so moved to next CF, waiting on > >> author. > > > Rebased. > > This is in need of rebasing again :-(. I went ahead and pushed the 001 > p

Re: Referential Integrity Checks with Statement-level Triggers

2019-02-25 Thread Antonin Houska
rate queue would be needed for the constraint-specific events. In general, the (after) triggers and constraints have too much in common, so separation of these w/o seeing code changes is beyond my imagination. -- Antonin Houska https://www.cybertec-postgresql.com

Re: Unnecessary checks for new rows by some RI trigger functions?

2019-02-24 Thread Antonin Houska
crosscheck just as UPDATE / DELETE would do. If they don't, there's > a hazard of mis-enforcing the FK constraint in the face of > concurrent updates. Maybe I missed something. When I searched through the code I saw the crosscheck_snapshot passed only to heap_update() and heap_delete(). -- Antonin Houska https://www.cybertec-postgresql.com

Re: Problems with plan estimates in postgres_fdw

2019-02-22 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/08 21:35), Etsuro Fujita wrote: > > (2019/02/08 2:04), Antonin Houska wrote: > >> * regression tests: I think test(s) should be added for queries that have > >> ORDER BY clause but do not have GROUP BY (and also no LIMIT / OFFSET) >

Re: postgres_fdw: another oddity in costing aggregate pushdown paths

2019-02-22 Thread Antonin Houska
arget as * the pathtarget for the resulting path. It seems like a good idea to do * it unconditionally. */ rel->reltarget = llast_node(PathTarget, scanjoin_targets); -- Antonin Houska https://www.cybertec-postgresql.com

Re: Problems with plan estimates in postgres_fdw

2019-02-22 Thread Antonin Houska
urs me now that no paths should be added to UPPERREL_ORDERED relation if the query needs WindowAgg node, because postgres_fdw currently cannot handle UPPERREL_WINDOW relation. Or rather in general, the FDW should not skip any stage just because it doesn't know how to build the paths. -- Antonin Houska https://www.cybertec-postgresql.com

Re: Unnecessary checks for new rows by some RI trigger functions?

2019-02-22 Thread Antonin Houska
Tom Lane wrote: > Robert Haas writes: > > On Wed, Feb 20, 2019 at 9:27 AM Antonin Houska wrote: > >> However I find it confusing that the trigger functions pass > >> detectNewRows=true > >> even if they only execute SELECT statement. > > > I don

Inappropriate scope of local variable

2019-02-21 Thread Antonin Houska
if (!TRIGGER_TYPE_MATCHES(trigger->tgtype, tgtype_level, -- Antonin Houska https://www.cybertec-postgresql.com

Unnecessary checks for new rows by some RI trigger functions?

2019-02-20 Thread Antonin Houska
* must detect new rows */ + false, SPI_OK_SELECT); if (SPI_finish() != SPI_OK_FINISH) [1] https://commitfest.postgresql.org/22/1975/ -- Antonin Houska https://www.cybertec-postgresql.com

Re: Referential Integrity Checks with Statement-level Triggers

2019-02-20 Thread Antonin Houska
the FK. Have you considered bulk processing of individual rows by row-level trigger? For IMMEDIATE constraints we'd have to ensure that the trigger is notified that the current row is the last one from the current query, but that might not be difficult. -- Antonin Houska https://www.cybertec-postgresql.com

Re: Problems with plan estimates in postgres_fdw

2019-02-18 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/15 21:46), Antonin Houska wrote: > > ok, I understand now. I assume that the patch > > > > https://www.postgresql.org/message-id/5C66A056.60007%40lab.ntt.co.jp > > > > obsoletes the code snippet we discussed above. > > Sor

Re: Problems with plan estimates in postgres_fdw

2019-02-15 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/12 18:03), Antonin Houska wrote: > > Etsuro Fujita wrote: > >> (2019/02/08 2:04), Antonin Houska wrote: > >>> > >>> And maybe related problem: why should FDW care about the effect of > >>> appl

Re: Incorrect visibility test function assigned to snapshot

2019-02-15 Thread Antonin Houska
Michael Paquier wrote: > On Fri, Feb 08, 2019 at 11:59:05AM +0100, Antonin Houska wrote: > > Sorry, I forgot. Patch is below and I'm going to add an entry to the > > next CF. > > > @@ -615,6 +615,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder) > > >

Re: Problems with plan estimates in postgres_fdw

2019-02-12 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/08 21:35), Etsuro Fujita wrote: > > (2019/02/08 2:04), Antonin Houska wrote: > >> * add_foreign_ordered_paths() > >> > >> Exprssion root->upper_targets[UPPERREL_FINAL] is used to access the > >> target. > >>

Re: Problems with plan estimates in postgres_fdw

2019-02-12 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/08 2:04), Antonin Houska wrote: > > > First, I wonder if the information on LIMIT needs to be passed to the > > FDW. Cannot the FDW routines use root->tuple_fraction? > > I think it's OK for the FDW to use the tupl

Re: Incorrect visibility test function assigned to snapshot

2019-02-08 Thread Antonin Houska
Bruce Momjian wrote: > On Wed, May 30, 2018 at 09:28:54AM -0400, Alvaro Herrera wrote: > > On 2018-May-30, Antonin Houska wrote: > > > > > In the header comment, SnapBuildInitialSnapshot() claims to set > > > snapshot->satisfies to the HeapTupleSatisfiesMVCC

Handling of ORDER BY by postgres_fdw

2019-02-07 Thread Antonin Houska
andle grouping too, so some refactoring would be needed. Do I misunderstand anything? [1] https://commitfest.postgresql.org/22/1950/ -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web: https://www.cybertec-postgresql.com

Re: Problems with plan estimates in postgres_fdw

2019-02-07 Thread Antonin Houska
nrel" means "input_rel" or "output_rel" from the perspective of postgresGetForeignUpperPaths(). [1] https://www.postgresql.org/message-id/87pnz1aby9@news-spur.riddles.org.uk -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web

Re: Unused parameters & co in code

2019-01-31 Thread Antonin Houska
e other two are still there. -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web: https://www.cybertec-postgresql.com

Re: [HACKERS] WIP: Aggregation push-down

2018-12-17 Thread Antonin Houska
Tom Lane wrote: > Antonin Houska writes: > > [ agg_pushdown_v8.tgz ] > > I spent a few hours looking at this today. Thanks! > It seems to me that at no point has there been a clear explanation of what > the patch is trying to accomplish, so let me see whether I'v

Re: Optimze usage of immutable functions as relation

2018-11-08 Thread Antonin Houska
not necessary: foreach(lc, functions_list) Also, there seems to be a lot of copy & paste from pull_up_simple_values(), so some refactoring would make sense. -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web: https://www.cybertec-postgresql.com

Re: Optimze usage of immutable functions as relation

2018-11-08 Thread Antonin Houska
up_simple_values() does not have to care because pull_up_simple_subquery() handles such cases when it's processing the owning subquery). -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web: https://www.cybertec-postgresql.com

Re: Doubts about pushing LIMIT to MergeAppendPath

2018-11-02 Thread Antonin Houska
Tomas Vondra wrote: > On 11/02/2018 08:16 AM, Antonin Houska wrote: > > Tomas Vondra wrote: > > > >> OK, so the reason is that when building child paths, we don't keep > >> the pathkeys unless it matches the "interesting" pathkeys. > >&g

Re: Doubts about pushing LIMIT to MergeAppendPath

2018-11-02 Thread Antonin Houska
Y we need. I don't agree that IndexPath will necessarily have pathkeys set to NIL in such a case. Even if the index ordering does not match ORDER BY clause of the query, the path can still be useful, typically for merge join. -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgass

Re: Doubts about pushing LIMIT to MergeAppendPath

2018-11-01 Thread Antonin Houska
Tomas Vondra wrote: > On 11/01/2018 12:48 PM, Antonin Houska wrote: > > Review of [1] made me think of this optimization, currently used only in > > create_merge_append_path(): > > > > /* > > * Apply query-wide LIMIT if known and path is for sole ba

Doubts about pushing LIMIT to MergeAppendPath

2018-11-01 Thread Antonin Houska
ust don't know how to check when it's correct. Does anyone have an idea? [1] https://commitfest.postgresql.org/20/1850/ [2] https://commitfest.postgresql.org/20/1124/ -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web: https://www.cybertec-postgresql.com

Re: Ordered Partitioned Table Scans

2018-11-01 Thread Antonin Houska
David Rowley wrote: > On 1 November 2018 at 04:01, Antonin Houska wrote: > > * As for the logic, I found generate_mergeappend_paths() to be the most > > interesting part: > > > > Imagine table partitioned by "i", so "partition_pathkeys&q

Re: Ordered Partitioned Table Scans

2018-10-31 Thread Antonin Houska
() of subpaths just like create_merge_append_path() does? And if so, then create_append_path() and create_merge_append_path() might eventually have some common code (at least for the subpath processing) to be put into a separate function. * Likewise, create_append_plan() / create_merge_append_plan()

Re: shared-memory based stats collector

2018-10-29 Thread Antonin Houska
nditionally and see if you find some entries. And if you really need to count the entries, I suggest that you use the per-partition counts (dshash_partition.count) instead of scanning individual entries. [1] https://www.postgresql.org/message-id/ca+tgmobqvbz4k_+rsmim9herkpy3vs5xnbkl95gsen

Re: Comment fix and question about dshash.c

2018-10-27 Thread Antonin Houska
Antonin Houska wrote: > Thomas Munro wrote: > > > On Sat, Oct 27, 2018 at 12:30 AM Antonin Houska wrote: > > > > Are you saying there is a bug in this logic (which is nbuckets * 0.75 > > expressed as integer maths), or saying that 0.75 is not a good maximum >

Re: Comment fix and question about dshash.c

2018-10-27 Thread Antonin Houska
Thomas Munro wrote: > On Sat, Oct 27, 2018 at 12:30 AM Antonin Houska wrote: > > Thanks, will fix. > > > 2. Can anyone please explain this macro? > > > > /* Max entries before we need to grow. Half + quarter = 75% load factor. */ > > #defin

Comment fix and question about dshash.c

2018-10-26 Thread Antonin Houska
the number of buckets in that partition. The fact that MAX_COUNT_PER_PARTITION refers to entries and not buckets is obvious from this condition in dshash_find_or_insert() /* Check if we are getting too full. */ if (partition->count > MAX_COUNT_PER_PARTITION(hash_table)) -

Re: shared-memory based stats collector

2018-09-20 Thread Antonin Houska
quot; -> "oneshot" ** snapshot_statentry_all(): "Returns a local hash contains ..." -> "Returns a local hash containing ..." [1] https://www.postgresql.org/message-id/20180711000605.sqjik3vqe5opq...@alap3.anarazel.de -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de, http://www.cybertec.at

Re: [HACKERS] Removing LEFT JOINs in more cases

2018-09-18 Thread Antonin Houska
the "rel" argument. Besides removing it you probably should rename the function. -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de, http://www.cybertec.at

Re: remove duplicated words in comments .. across lines

2018-09-09 Thread Antonin Houska
# Characters used in ASCII charts are not duplicate words. $2 != "|" && $2 != "}") # Found a duplicate. printf("%s:%s, duplicate token: %s\n", FILENAME, FNR, $2); } if (NF > 1 && ($1 == "*" || $1 == "/*")) prev_line_last_token = $NF; else { # Empty line or not a comment line. Start a new search. prev_line_last_token = NULL; } } -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web: https://www.cybertec-postgresql.com

Re: [HACKERS] WIP: Aggregation push-down

2018-08-31 Thread Antonin Houska
Antonin Houska wrote: > I've reworked the patch so that separate RelOptInfo is used for grouped > relation. The attached patch is only the core part. Neither postgres_fdw > changes nor the part that tries to avoid the final aggregation is included > here. I'll add these w

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2018-08-12 Thread Antonin Houska
#9 FAILED at 2464. > > 1 out of 7 hunks FAILED -- saving rejects to file > src/bin/pg_rewind/pg_rewind.c.rej > > 1 out of 6 hunks FAILED -- saving rejects to file > src/bin/pg_upgrade/controldata.c.rej The patch should be applicable to the master branch. -- Antonin Houska Cyber

Re: "WIP: Data at rest encryption" patch and, 2 phase commit.

2018-07-25 Thread Antonin Houska
rg/message-id/11678.1532519255%40localhost fixes an issue (unitialized pointer) that caused failure here, but it was SEGFAULT rather than ERROR. And the scope of the bug was broader than just CHECKPOINT. Can you please test it again with the new version of the patch? Anyway, thanks for your reports

Re: [HACKERS] WIP: Aggregation push-down

2018-07-06 Thread Antonin Houska
Robert Haas wrote: > On Fri, Feb 23, 2018 at 11:08 AM, Antonin Houska wrote: > > I spent some more time thinking about this. What about adding a new strategy > > number for hash index operator classes, e.g. HTBinaryEqualStrategyNumber? > > For > > most types bot

Re: Push down Aggregates below joins

2018-06-22 Thread Antonin Houska
version. > And if we punt on the partial aggregation, and only allow pushing down the > whole Aggregate, how much smaller would your patch get? I can't tell now, need to spend some time looking at the code. > (I'll be on vacation for the next two weeks, but I'll be f

Re: Push down Aggregates below joins

2018-06-20 Thread Antonin Houska
low a join. I've been > > hacking on a patch to allow that. > > > > There was a patch [1] from Antonin Houska aiming to achieve something > similar. IIRC it aimed to push the aggregate down in more cases, > leveraging the partial aggregation stuff. Yes, I interrupted the

<    1   2   3   4   >