Re: Temporary file access API

2022-03-08 Thread Antonin Houska
Stephen Frost wrote: > * Antonin Houska (a...@cybertec.at) wrote: > > Here I'm starting a new thread to discuss a topic that's related to the > > Transparent Data Encryption (TDE), but could be useful even without that. > > The > > problem has been addre

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-06 Thread Antonin Houska
or even in create_modifytable_path(), and ensure that it gets propagated to the Gather node (generate_gather_pahs currently uses rel->reltarget), then you should no longer need to tweak setrefs.c. Moreover, ExecInitModifyTable() would no longer need to set the targetlist. However I don't guarantee that this is the best approach - some planner expert should speak up. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-07 Thread Antonin Houska
Amit Kapila wrote: > On Wed, Jan 6, 2021 at 2:09 PM Antonin Houska wrote: > > > > Greg Nancarrow wrote: > > > > > Posting an updated set of patches to address recent feedback: > > > > Following is my review. > > > > v11-0001

pg_config_h.in not up-to-date

2021-02-18 Thread Antonin Houska
When I run "autoreconf" on the master branch, git generates the diff below. Shouldn't it just be applied? I suppose someone changed configure.ac and forgot to update the generated file. -- Antonin Houska Web: https://www.cybertec-postgresql.com diff --git a/src/include/pg_co

Re: POC: Cleaning up orphaned files using undo logs

2021-03-10 Thread Antonin Houska
tsunakawa.ta...@fujitsu.com wrote: > I'm crawling like a snail to read the patch set. Below are my first set of > review comments, which are all minor. Thanks. I will check your comments when I'll be preparing the next version of the patch. -- Antonin Houska Web: http

Re: WIP: Aggregation push-down

2021-03-11 Thread Antonin Houska
rogress for several years, I don't believe another rebase(s) is anything but wasted effort. So I've withdrawn the patch, also to save CFMs from examining the history again and again uselessly. The code is free so anyone can continue if he thinks it makes sense. If it finally finds its way

Re: POC: Cleaning up orphaned files using undo logs

2020-11-13 Thread Antonin Houska
Thomas Munro wrote: > On Thu, Nov 12, 2020 at 10:15 PM Antonin Houska wrote: > I saw that -- great news! -- and have been meaning to write for a > while. I think I am nearly ready to talk about it again. I'm looking forward to it :-) > 100% that it's worth tryin

Re: POC: Cleaning up orphaned files using undo logs

2020-11-13 Thread Antonin Houska
Amit Kapila wrote: > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote: > > > > > > No background undo > > -- > > > > Reduced complexity of the patch seems to be the priority at the moment. Amit > > suggested that cleanup of an

Re: POC: Cleaning up orphaned files using undo logs

2020-11-18 Thread Antonin Houska
Amit Kapila wrote: > On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote: > > > > Amit Kapila wrote: > > > > > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote: > > > > > > > > > > > > No background undo > > > &g

Re: POC: Cleaning up orphaned files using undo logs

2020-11-25 Thread Antonin Houska
Antonin Houska wrote: > Amit Kapila wrote: > > I think we also need to maintain oldestXidHavingUndo for CLOG truncation and > > transaction-wraparound. We can't allow CLOG truncation for the transaction > > whose undo is not discarded as that could be required by

Re: POC: Cleaning up orphaned files using undo logs

2020-11-25 Thread Antonin Houska
Amit Kapila wrote: > On Wed, Nov 18, 2020 at 4:03 PM Antonin Houska wrote: > > > > Amit Kapila wrote: > > > > > On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote: > > > > > > > > Amit Kapila wrote: > > > > > >

Re: POC: Cleaning up orphaned files using undo logs

2020-11-26 Thread Antonin Houska
Amit Kapila wrote: > On Wed, Nov 25, 2020 at 7:47 PM Antonin Houska wrote: > > > > Antonin Houska wrote: > > > > > Amit Kapila wrote: > > > > > > I think we also need to maintain oldestXidHavingUndo for CLOG > > > > trunca

Re: POC: Cleaning up orphaned files using undo logs

2020-11-26 Thread Antonin Houska
Amit Kapila wrote: > On Wed, Nov 25, 2020 at 8:00 PM Antonin Houska wrote: > > I meant that AbortOutOfAnyTransaction should PANIC itself if it sees that > > there is unapplied undo, so nothing changes for its callers. Do I still miss > > something? > > > >

Re: POC: Cleaning up orphaned files using undo logs

2021-01-18 Thread Antonin Houska
f the patch (along with some other small changes) this week. > There are also here and there messages about not found undo files: > > ERROR: cannot open undo segment file 'base/undo/08.02': No > such file or directory > WARNING: failed to undo transaction I don't see this one in the log so far, will try again. Thanks for the report! -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: POC: Cleaning up orphaned files using undo logs

2021-02-03 Thread Antonin Houska
Bruce Momjian wrote: > On Fri, Jan 29, 2021 at 06:30:15PM +0100, Antonin Houska wrote: > > Antonin Houska wrote: > > > Well, on repeated run of the test I could also hit the first one. I could > > > fix > > > it and will post a new version of th

xl_heap_header alignment?

2020-07-21 Thread Antonin Houska
e any alignment, and therefore it uses memcpy() to copy the structure into a local variable before accessing its fields. For example, heap_xlog_insert(). -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: xl_heap_header alignment?

2020-07-21 Thread Antonin Houska
Tom Lane wrote: > I don't particularly want to remove the field, but we ought to > change or remove the comment. I'm not concerned about the existence of the field as well. The comment just made me worried that I might be missing some fundamental concept. Thanks for your opini

Re: xl_heap_header alignment?

2020-08-22 Thread Antonin Houska
Bruce Momjian wrote: > On Wed, Jul 22, 2020 at 06:58:33AM +0200, Antonin Houska wrote: > > Tom Lane wrote: > > > > > I don't particularly want to remove the field, but we ought to > > > change or remove the comment. > > > > I'm not

Re: xl_heap_header alignment?

2020-08-22 Thread Antonin Houska
Antonin Houska wrote: > If the comment tells that t_hoff can be computed (i.e. it's no necessary to > include it in the structure), I think the comment should tell why it's yet > included. Maybe something about "historical reasons"? Perhaps we can say that > th

Re: storing an explicit nonce

2021-11-01 Thread Antonin Houska
ard cipher function to an unique value. Can you please try to explain once again what you consider a requirement (uniqueness, randomness, etc.) on the IV for the XTS mode? Thanks. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: [PATCH] Full support for index LP_DEAD hint bits on standby

2021-11-03 Thread Antonin Houska
;all_dead". The same for "*page_lsn". * I can see no test for the INDEX_LP_DEAD_OK_MIN_LSN value of the IndexLpDeadAllowedResult enumeration. Shouldn't there be only two values, e.g. INDEX_LP_DEAD_OK and INDEX_LP_DEAD_MAYBE_OK ? Or a boolean variable (in index_fetch_heap())

Re: [PATCH] Full support for index LP_DEAD hint bits on standby

2021-11-09 Thread Antonin Houska
lp_dead_maybe_allowed()? > > Yes, this way it is looks better. Done. Also, I have added some checks > for “maybe” LSN-related logic to the test. Attached is a proposal for a minor addition that would make sense to me, add it if you think it's appropriate. I think I'v

Re: Reuse of State value in Aggregates

2021-11-09 Thread Antonin Houska
likely be different. Maybe you need to post an example where you think such reusing would be useful. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: XTS cipher mode for cluster file encryption

2021-11-28 Thread Antonin Houska
and BufFileReadTransient() replace OpenTransientFile(), CloseTransientFile(), write()/fwrite() and read()/fread() respectively in reorderbuffer.c and in pgstat.c. These changes become a little bit less invasive in TDE 1.1 than they were in 1.0, see [1], see the diffs attached. (I expect that [2] will get

Re: [PATCH] buffile: ensure start offset is aligned with BLCKSZ

2021-11-29 Thread Antonin Houska
the initialization vector (IV) is BLCKSZ bytes. Another problem is that you might want to store the IV somewhere in between the data. In short, the encryption makes the buffered IO rather different and the specific code should be kept aside, although the same API is used to invoke it. -- Antonin Ho

Re: WIP: Data at rest encryption

2019-09-03 Thread Antonin Houska
.jp ? This thread started later than our effort but important design questions are being discussed there. So far there seems to be no consensus whether full-instance encryption should be implemented first, so any effort spent on this patch might get wasted. When/if there will be an agreement on

Re: [HACKERS] WIP: Aggregation push-down

2019-09-04 Thread Antonin Houska
about 2), so this problem might be resolved soon: https://www.postgresql.org/message-id/CAH2-Wzn3Ee49Gmxb7V1VJ3-AC8fWn-Fr8pfWQebHe8rYRxt5OQ%40mail.gmail.com But 1) still remains. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Attempt to consolidate reading of XLOG page

2019-09-09 Thread Antonin Houska
Alvaro Herrera wrote: > Hi Antonin, could you please rebase again? Attached. -- Antonin Houska Web: https://www.cybertec-postgresql.com >From 7b15d7bbdae13c743ddeae10b8ff79e9b02d8243 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Mon, 9 Sep 2019 11:53:54 +0200 Subject: [PAT

Re: Pulling up direct-correlated ANY_SUBLINK

2019-09-10 Thread Antonin Houska
e not used in unique keys of the inner path and so the SEMI JOIN didn't work well. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Pulling up direct-correlated ANY_SUBLINK

2019-09-11 Thread Antonin Houska
imposed by LATERAL might be less significant. Nevertheless, I don't know how to overcome the problems that I mentioned upthread. -- Antonin Houska Web: https://www.cybertec-postgresql.com

JSON parser discards value of string token

2019-09-12 Thread Antonin Houska
something that recalls the "de-escaping", e.g. "noesc"? -- Antonin Houska Web: https://www.cybertec-postgresql.com diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 26d293709a..2ef16fb089 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/

Re: Pulling up direct-correlated ANY_SUBLINK

2019-09-12 Thread Antonin Houska
Richard Guo wrote: > On Wed, Sep 11, 2019 at 3:25 PM Antonin Houska > wrote: > > > Nevertheless, I don't know how to overcome the problems that I > mentioned > upthread. > > > Do you mean the problem "the WHERE clause of the subque

Re: Attempt to consolidate reading of XLOG page

2019-09-23 Thread Antonin Houska
open WAL segment is now in the callbacks, the only thing that XLogRead() can ereport is that read() failed. BTW, I introduced one more structure, XLogReadError, in this patch version. I think it's better than adding error-specific fields to the WALOpenSegment structure. -- Antonin Houska Web: http

Re: Attempt to consolidate reading of XLOG page

2019-09-24 Thread Antonin Houska
emains on top. No objections here. > 2. Not a fan of the InvalidTimeLineID stuff offhand. Maybe it's okay ... > not convinced yet either way. Well, it seems that the individual callbacks only use this constant in Assert() statements. I'll consider if we really need it. The argument value should not determine whether the callback derives the TLI or not. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Attempt to consolidate reading of XLOG page

2019-09-26 Thread Antonin Houska
Alvaro Herrera wrote: > On 2019-Sep-24, Antonin Houska wrote: > > > Alvaro Herrera wrote: > > > > If you don't have any strong dislikes for these changes, I'll push this > > > part and let you rebase the remains on top. > > > > No obj

Re: Attempt to consolidate reading of XLOG page

2019-09-27 Thread Antonin Houska
Alvaro Herrera wrote: > On 2019-Sep-26, Antonin Houska wrote: > You placed the errinfo in XLogRead's stack rather than its callers' ... > I don't think that works, because as soon as XLogRead returns that > memory is no longer guaranteed to exist. I was aware o

Re: Attempt to consolidate reading of XLOG page

2019-09-28 Thread Antonin Houska
because some extensions might already have their implementation of XLogPageReadCB w/o XLogRead, and this change would break them. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Attempt to consolidate reading of XLOG page

2019-09-28 Thread Antonin Houska
Tom Lane wrote: > Alvaro Herrera writes: > > On 2019-Sep-27, Antonin Houska wrote: > >>> You placed the errinfo in XLogRead's stack rather than its callers' ... > >>> I don't think that works, because as soon as XLogRead returns that > >>

Re: Attempt to consolidate reading of XLOG page

2019-09-28 Thread Antonin Houska
Antonin Houska wrote: > Alvaro Herrera wrote: > > > BTW that tli_p business to the openSegment callback is horribly > > inconsistent. Some callers accept a NULL tli_p, others will outright > > crash, even though the API docs say that the callback must determine the >

Re: Building infrastructure for B-Tree deduplication that recognizes when opclass equality is also equivalence

2019-09-30 Thread Antonin Houska
incorrectly. > It is set to false for numeric and float4, float8. Are you sure about these? -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Attempt to consolidate reading of XLOG page

2019-09-30 Thread Antonin Houska
Kyotaro Horiguchi wrote: > At Sat, 28 Sep 2019 15:00:35 +0200, Antonin Houska wrote > in <9236.1569675635@antos> > > Antonin Houska wrote: > > > > > Alvaro Herrera wrote: > > > > > > > BTW that tli_p business to the openSegment ca

Re: Attempt to consolidate reading of XLOG page

2019-10-02 Thread Antonin Houska
Kyotaro Horiguchi wrote: > At Tue, 01 Oct 2019 08:28:03 +0200, Antonin Houska wrote > in <2188.1569911283@antos> > > Kyotaro Horiguchi wrote: > > > > The problem of walsender.c is that its implementation of XLogRead() > > > > does not > > >

Re: Attempt to consolidate reading of XLOG page

2019-10-04 Thread Antonin Houska
This is the next version. -- Antonin Houska Web: https://www.cybertec-postgresql.com >From 01f5cc8b0c1e6133e16242ec99957a78551058a7 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 4 Oct 2019 12:07:22 +0200 Subject: [PATCH 1/2] Use only xlogreader.c:XLogRead() The implementations

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-07 Thread Antonin Houska
7;ll be glad to help. [1] https://commitfest.postgresql.org/25/2104/ [2] https://wiki.postgresql.org/wiki/Transparent_Data_Encryption [3] https://commitfest.postgresql.org/25/1708/ -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-08 Thread Antonin Houska
n encryption block of 16 bytes at a time. Thus if only a part of the block gets written (a torn page write), decryption of the block results in garbage. Unlike relations, there's nothing like full-page write for SLRU pages, so there's no way to recover from this problem. However, if the curr

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-08 Thread Antonin Houska
Robert Haas wrote: > On Mon, Oct 7, 2019 at 3:01 PM Antonin Houska wrote: > > However the design doesn't seem to be stable enough at the > > moment for coding to make sense. > > Well, I think the question is whether working further on your patch > could produce so

Re: pg_init

2019-10-08 Thread Antonin Houska
n by > pg_init, for example. And it's probably a good idea to explain what > you're trying to achieve. This question was familiar to me so I searched the archives. It seems related to https://www.postgresql.org/message-id/17058.1570166272%40sss.pgh.pa.us -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-08 Thread Antonin Houska
Moon, Insung wrote: > Hello. > > On Tue, Oct 8, 2019 at 8:52 PM Antonin Houska wrote: > > > > Robert Haas wrote: > > > > > On Mon, Oct 7, 2019 at 3:01 PM Antonin Houska wrote: > > > > However the design doesn't seem to be stable

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-08 Thread Antonin Houska
Moon, Insung wrote: > On Wed, Oct 9, 2019 at 2:42 PM Antonin Houska wrote: > > > > Moon, Insung wrote: > > > > > I also tried to generate IV using PID (32bit) + tempCounter (64bit) at > > > first, but in the worst-case PID and tempCounter are used in the

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-13 Thread Antonin Houska
Masahiko Sawada wrote: > On Wed, Oct 9, 2019 at 3:57 PM Antonin Houska wrote: > > > > Moon, Insung wrote: > > > > v04-0011-Make-buffile.c-aware-of-encryption.patch in [1] changes buffile.c > > so > > that data is read and written in 8kB blocks if encr

Re: [HACKERS] WIP: Aggregation push-down

2019-02-28 Thread Antonin Houska
Tom Lane wrote: > Antonin Houska writes: > > Michael Paquier wrote: > >> Latest patch set fails to apply, so moved to next CF, waiting on > >> author. > > > Rebased. > > This is in need of rebasing again :-(. I went ahead and pushed the 001 > p

Re: Problems with plan estimates in postgres_fdw

2019-03-01 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/22 22:54), Antonin Houska wrote: > > Etsuro Fujita wrote: > >> On the other hand, the code bit added by > >> 0001-postgres_fdw-Perform-UPPERREL_ORDERED-step-remotely-v3.patch handles > >> the > >> case where a post-

Re: Problems with plan estimates in postgres_fdw

2019-03-01 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/23 0:21), Antonin Houska wrote: > > Etsuro Fujita wrote: > > >>> (2019/02/08 2:04), Antonin Houska wrote: > >>>> * regression tests: I think test(s) should be added for queries that have > >>>> ORDER BY claus

Re: Problems with plan estimates in postgres_fdw

2019-03-01 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/03/01 20:00), Antonin Houska wrote: > > Etsuro Fujita wrote: > > I used gdb to help me understand, however the condition > > > > if (fpextra&& !IS_UPPER_REL(foreignrel)) > > > > never evaluated to true with the

Re: Problems with plan estimates in postgres_fdw

2019-03-03 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/03/01 20:00), Antonin Houska wrote: > > Etsuro Fujita wrote: > > It's still unclear to me why add_foreign_ordered_paths() passes the input > > relation (input_rel) to estimate_path_cost_size(). If it passed the output > > rel &g

Re: Ordered Partitioned Table Scans

2019-03-04 Thread Antonin Houska
David Rowley wrote: > On Mon, 5 Nov 2018 at 10:46, David Rowley > wrote: > > On 1 November 2018 at 22:05, Antonin Houska wrote: > > > I think these conditions are too restrictive: > > > > > > /* > > > * Determine if these pathk

openLogOff is not needed anymore

2019-03-06 Thread Antonin Houska
XLByteToPrevSeg(LogwrtResult.Write, openLogSegNo, wal_segment_size); openLogFile = XLogFileOpen(openLogSegNo); - openLogOff = 0; } issue_xlog_fsync(openLogFile, openLogSegNo); -- Antonin Houska https://www.cybertec-postgresql.com

Re: openLogOff is not needed anymore

2019-03-06 Thread Antonin Houska
Robert Haas wrote: > On Wed, Mar 6, 2019 at 6:10 AM Antonin Houska wrote: > > ... obviously due to commit c24dcd0. The following patch removes it. > > Committed, after a short struggle to get the patch out of the email > body in a usable form. It was just convenient for

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-03-07 Thread Antonin Houska
pplies it, and then the table data will be > encrypted with its table key when flushing. So we just control the > scope of encryption object: WAL of tables and indexes etc or > everything. My point of view is that different key usually means different user. The user who can decrypt WAL can effectively see all the data, even though another user put them (encrypted with another key) into tables. So in this case, different keys don't really separate users in terms of data access. -- Antonin Houska https://www.cybertec-postgresql.com

Re: Problems with plan estimates in postgres_fdw

2019-03-08 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/03/01 20:16), Antonin Houska wrote: > > Etsuro Fujita wrote: > > >> Conversely, it appears that add_foreign_ordered_paths() added by the > >> patchset > >> would generate such pre-sorted paths *redundantly* when the inpu

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-03-08 Thread Antonin Houska
Antonin Houska wrote: > Masahiko Sawada wrote: > > > Agreed. > > > > For the WAL encryption, I wonder if we can have a encryption key > > dedicated for WAL. Regardless of keys of tables and indexes all WAL > > are encrypted with the WAL key. During

Re: Suggestions on message transfer among backends

2019-03-12 Thread Antonin Houska
One queue is used to send request (query plan) to the worker, one queue is there to receive data from it, and I think there's one more queue to receive error messages. -- Antonin Houska https://www.cybertec-postgresql.com

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-14 Thread Antonin Houska
er it should just refuse to start unless they are set properly, but I > think it probably needs to do one or the other. Maybe it's more polite to refuse to start so that user knows what's going on. I'm not sure if PG ever changes any configuration variable forcibly. > Hope this

Re: Suggestions on message transfer among backends

2019-03-14 Thread Antonin Houska
Robert Haas wrote: > On Tue, Mar 12, 2019 at 4:34 AM Antonin Houska wrote: > > Andy Fan wrote: > > > I just don't know why shm_mq is designed to single-reader & single-writer. > > > > shm_mq was implemented as a part of infrastructure for parallel q

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-21 Thread Antonin Houska
to deal with encryption. This design decision can be changed, but there's one more thing to consider: if the XLOG stream is decrypted, the decryption cannot be disabled unless the XLOG records are aligned to 16 bytes (and in turn, the XLOG alignment cannot be enabled w/o initdb). -- Antonin Houska https://www.cybertec-postgresql.com

Question on alignment

2019-04-01 Thread Antonin Houska
please explain why alignment matters here? -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Question on alignment

2019-04-01 Thread Antonin Houska
Heikki Linnakangas wrote: > On 01/04/2019 11:01, Antonin Houska wrote: > > In copydir.c:copy_file() I read > > > > /* Use palloc to ensure we get a maxaligned buffer */ > > buffer = palloc(COPY_BUF_SIZE); > > > > No data type wider than a single

Re: Question on alignment

2019-04-01 Thread Antonin Houska
Antonin Houska wrote: > Since palloc() only ensures MAXIMUM_ALIGNOF, that wouldn't help here anyway. After some more search I'm not sure about that. The following comment indicates that MAXALIGN helps too: /* * Use this, not "char buf[BLCKSZ]", to declare a field or lo

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Antonin Houska
ace, we expect this value is enough * for most usecase. */ #define KMGR_KEYRING_SIZE 128 There's no guarantee that the number of tablespaces won't exceed any (reasonably low) constant value. The KMGR module should be able to allocate additional memory dynamically. [

Re: Converting NOT IN to anti-joins during planning

2019-08-13 Thread Antonin Houska
David Rowley wrote: > On Mon, 27 May 2019 at 20:43, Antonin Houska wrote: > > I've spent some time looking into this. > > Thank you for having a look at this. > > > One problem I see is that SubLink can be in the JOIN/ON clause and thus it's > > not

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-14 Thread Antonin Houska
mail.gmail.com [3] https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#TODO_for_Full-Cluster_Encryption -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-15 Thread Antonin Houska
Bruce Momjian wrote: > On Wed, Aug 14, 2019 at 04:36:35PM +0200, Antonin Houska wrote: > > I can work on it right away but don't know where to start. > > I think the big open question is whether there will be acceptance of an > all-cluster encyption feature. I guess if

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Antonin Houska
need at least an outline of the approaches proposed because I also missed some parts of the thread. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Antonin Houska
ay need to send some sort of credentials to the KMS. If it runs a separate process to execute the command, it needs to pass those credentials to it. Whether it does so via parameters or environment variables, both can be seen by other users. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Building infrastructure for B-Tree deduplication that recognizes when opclass equality is also equivalence

2019-08-26 Thread Antonin Houska
to consider the nondeterministic collations too, I missed that so far.) [1] https://commitfest.postgresql.org/24/1247/ [2] https://www.postgresql.org/message-id/10529.1547561178%40localhost -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: [HACKERS] [PATCH] Incremental sort

2018-01-08 Thread Antonin Houska
Alexander Korotkov wrote: > Antonin Houska wrote: > > Shouldn't the test contain *both* cases? > Thank you for pointing that. Sure, both cases are better. I've added second > case as well as comments. Patch is attached. I'm fine with the tests now but have a

Re: [HACKERS] Secondary index access optimizations

2018-01-09 Thread Antonin Houska
00) > Filter: (k <= 2) > (6 rows) > > It is because operator_predicate_proof is not able to understand that k < > 20001 and k <= 2 are equivalent for integer type. > If I rewrite query as (k >= 1 and k < 20001) then plan is optimal: > > postgres=# exp

Re: [HACKERS] Secondary index access optimizations

2018-01-11 Thread Antonin Houska
Konstantin Knizhnik wrote: > On 09.01.2018 19:48, Antonin Houska wrote: > >> Have you considered using the range types (internally in >> operator_predicate_proof()) instead of hard-coding operator OIDs? The range >> types do have the knowledge that k < 20001 and k

Unnecessary static variable?

2018-01-17 Thread Antonin Houska
ort_wait_start(WAIT_EVENT_WAL_READ); ! if (read(readFile, readBuf, readLen) != readLen) { charfname[MAXFNAMELEN]; -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26, A-2700 Wiener Neustadt Web: https://www.cybertec-postgresql.com

Re: Unnecessary static variable?

2018-01-17 Thread Antonin Houska
Tom Lane wrote: > Antonin Houska writes: > but your patch also does this: Yes, that should have been a separate diff. > > *** retry: > > *** 11648,11654 > > } > > > > pgstat_report_wait_start(WAIT_EVENT_WAL_READ); > > !

Re: [HACKERS] Possible gaps/garbage in the output of XLOG reader

2018-01-18 Thread Antonin Houska
Michael Paquier wrote: > On Thu, Apr 9, 2015 at 7:05 PM, Antonin Houska wrote: > > While playing with xlogreader, I was lucky enough to see one of the many > > record validations to fail. After having some fun with gdb, I found out that > > in some cases the reader does no

Re: Unnecessary static variable?

2018-01-18 Thread Antonin Houska
Tom Lane wrote: > Antonin Houska writes: > > Tom Lane wrote: > >> At one time, I think, readLen told how much data in readBuf was > >> actually valid. It seems not to be used for that anymore, but > >> I don't much like the idea that readBuf is on

Re: [HACKERS] Secondary index access optimizations

2018-01-19 Thread Antonin Houska
Konstantin Knizhnik wrote: > On 11.01.2018 12:34, Antonin Houska wrote: > > Konstantin Knizhnik wrote: > > I haven't thought that much about details, so just one comment: you > > shouldn't > > need the conversion to text and back to binary form

Re: [HACKERS] WIP: Aggregation push-down

2018-01-26 Thread Antonin Houska
Robert Haas wrote: > On Fri, Dec 22, 2017 at 10:43 AM, Antonin Houska wrote: > > Michael Paquier wrote: > >> On Sat, Nov 4, 2017 at 12:33 AM, Antonin Houska wrote: > >> > I'm not about to add any other features now. Implementation of the > >> >

Re: [HACKERS] WIP: Aggregation push-down

2018-01-29 Thread Antonin Houska
Robert Haas wrote: > On Fri, Jan 26, 2018 at 8:04 AM, Antonin Houska wrote: > > So one problem is that the grouping expression can be inappropriate for > > partial aggregation even if there's no type change during the > > translation. What I consider typical for this

MarkBufferDirtyHint() and LSN update

2019-10-30 Thread Antonin Houska
the next call of FlushBuffer(). However page LSN is hasn't been updated so the requirement that WAL must be flushed first is not met. I think that PageSetLSN() should be called regardless BM_DIRTY. Do I miss any subtle detail? -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: MarkBufferDirtyHint() and LSN update

2019-10-31 Thread Antonin Houska
Tomas Vondra wrote: > On Wed, Oct 30, 2019 at 02:44:18PM +0100, Antonin Houska wrote: > >Please consider this scenario (race conditions): > > > >1. FlushBuffer() has written the buffer but hasn't yet managed to clear the > >BM_DIRTY flag (however BM_JUST

Re: MarkBufferDirtyHint() and LSN update

2019-11-01 Thread Antonin Houska
Robert Haas wrote: > On Wed, Oct 30, 2019 at 9:43 AM Antonin Houska wrote: > > 5. In the first session, FlushBuffer()->TerminateBufferIO() will not clear > > BM_DIRTY because MarkBufferDirtyHint() has eventually set > > BM_JUST_DIRTIED. Thus the hint bit change itsel

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-02 Thread Antonin Houska
he use of LSN as encryption IV). Do you mean any other case? If LSN does not get changed, then the related full-page image WAL record is not guaranteed to be on disk during crash recovery. Thus if page checksum is invalid due to torn-page write, there's now WAL record to fix the page. --

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-02 Thread Antonin Houska
his in his patch) I avoided repeated encryption of different data using the same key+IV by omitting the unused part of the WAL page from encryption. Already written records can be encrypted repeatedly because they do not change. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: MarkBufferDirtyHint() and LSN update

2019-11-11 Thread Antonin Houska
Michael Paquier wrote: > Does something like the attached patch make sense? Reviews are > welcome. This looks good to me. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Attempt to consolidate reading of XLOG page

2019-11-11 Thread Antonin Houska
Michael Paquier wrote: > On Fri, Oct 04, 2019 at 12:11:11PM +0200, Antonin Houska wrote: > > This is the next version. > > So... These are the two last bits to look at, reducing a bit the code > size: > 5 files changed, 396 insertions(+), 419 deletions(-) > > And

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-11 Thread Antonin Houska
Robert Haas wrote: > On Sat, Nov 2, 2019 at 8:23 AM Antonin Houska wrote: > > Change to hint bits does not result in LSN change in the case I described > > here > > > > https://www.postgresql.org/message-id/28452.1572443058%40antos > > > > but I cons

Re: Attempt to consolidate reading of XLOG page

2019-11-12 Thread Antonin Houska
Michael Paquier wrote: > On Mon, Nov 11, 2019 at 04:25:56PM +0100, Antonin Houska wrote: > >> On Fri, Oct 04, 2019 at 12:11:11PM +0200, Antonin Houska wrote: > >> Your patch removes all the three optional lseek() calls which can > >> happen in a segment. Am I mi

Re: MarkBufferDirtyHint() and LSN update

2019-11-12 Thread Antonin Houska
Kyotaro Horiguchi wrote: > At Mon, 11 Nov 2019 10:03:14 +0100, Antonin Houska wrote > in > > Michael Paquier wrote: > > > Does something like the attached patch make sense? Reviews are > > > welcome. > > > > This looks good to me. > > I ha

Re: MarkBufferDirtyHint() and LSN update

2019-11-14 Thread Antonin Houska
Michael Paquier wrote: > On Mon, Nov 11, 2019 at 10:03:14AM +0100, Antonin Houska wrote: > > This looks good to me. > > Actually, no, this is not good. I have been studying more the patch, > and after stressing more this code path with a cluster having > checksums enabled

Re: Attempt to consolidate reading of XLOG page

2019-11-20 Thread Antonin Houska
Alvaro Herrera wrote: > On 2019-Nov-12, Antonin Houska wrote: > > > ok, the next version uses explicit lseek(). Maybe the fact that XLOG is > > mostly > > read sequentially (i.e. without frequent seeks) is the reason pread() has't > > been adopted so far. &g

Re: Attempt to consolidate reading of XLOG page

2019-11-21 Thread Antonin Houska
uot;extern" declaration. I'll fix this as well as the other problem reported in [1] as soon as I know whether pg_pread() should be used or not. [1] https://www.postgresql.org/message-id/20191121080550.GG153437%40paquier.xyz -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Attempt to consolidate reading of XLOG page

2019-11-22 Thread Antonin Houska
in pages). Yes, I agree that page numbers would be sufficient. > But then that's not something that we need to change in this patch. -- Antonin Houska Web: https://www.cybertec-postgresql.com

Re: Attempt to consolidate reading of XLOG page

2019-11-25 Thread Antonin Houska
Alvaro Herrera wrote: > On 2019-Nov-22, Antonin Houska wrote: > > > As I pointed out in > > > > https://www.postgresql.org/message-id/88183.1574261429%40antos > > > > seg.ws_off only replaced readOff in XLogReaderState. So we should only > > update &

  1   2   3   4   >