Re: Detecting File Damage & Inconsistencies

2021-07-31 Thread Amit Kapila
On Thu, Jul 22, 2021 at 7:10 PM Simon Riggs wrote: > > On Wed, 14 Jul 2021 at 05:01, Amit Kapila wrote: > > But I do understand the overall request, so I propose adding > * pg_waldump output for wal_sessioninfo data, if it exists > * pg_waldump --user=USERNAME as a filter on username > to

Re: Detecting File Damage & Inconsistencies

2021-07-22 Thread Simon Riggs
On Wed, 14 Jul 2021 at 05:01, Amit Kapila wrote: > The patch is simple but its use doesn't seem to be very clear. You > have mentioned its use for future PITR patches and Craig mentioned > some use cases in logical decoding and it appears to me that to > support the use cases mentioned by Craig,

Re: Detecting File Damage & Inconsistencies

2021-07-13 Thread Amit Kapila
On Tue, Jul 13, 2021 at 8:29 PM Simon Riggs wrote: > > On Tue, Jul 6, 2021 at 4:21 AM Amit Kapila wrote: > > > > > > > > > If you don't think the sorts of use cases I presented are worth the > > > > trouble that's fair enough. I'm not against adding it on the commit > > > > record. It's just

Re: Detecting File Damage & Inconsistencies

2021-07-13 Thread Simon Riggs
On Tue, Jul 6, 2021 at 4:21 AM Amit Kapila wrote: > > On Fri, Jul 2, 2021 at 8:29 PM Simon Riggs > wrote: > > > > On Fri, Jul 2, 2021 at 5:34 AM Craig Ringer > > wrote: > > > > > > > > If you don't think the sorts of use cases I presented are worth the > > > trouble that's fair enough. I'm

Re: Detecting File Damage & Inconsistencies

2021-07-05 Thread Amit Kapila
On Fri, Jul 2, 2021 at 8:29 PM Simon Riggs wrote: > > On Fri, Jul 2, 2021 at 5:34 AM Craig Ringer > wrote: > > > > > If you don't think the sorts of use cases I presented are worth the trouble > > that's fair enough. I'm not against adding it on the commit record. It's > > just that with

Re: Detecting File Damage & Inconsistencies

2021-07-02 Thread Simon Riggs
On Fri, Jul 2, 2021 at 5:34 AM Craig Ringer wrote: > > On Fri, 2 Jul 2021 at 00:19, Simon Riggs wrote: > >> >> > So yeah. I think it'd be better to log the info you want at start-of-txn >> > unless there's a compelling reason not so, and I don't see one yet. >> >> AFAIK, XLOG_XACT_ASSIGNMENT

Re: Detecting File Damage & Inconsistencies

2021-07-01 Thread Craig Ringer
On Fri, 2 Jul 2021 at 00:19, Simon Riggs wrote: > > So yeah. I think it'd be better to log the info you want at start-of-txn > unless there's a compelling reason not so, and I don't see one yet. > > AFAIK, XLOG_XACT_ASSIGNMENT does not occur for normal top-level > transactions, only for

Re: Detecting File Damage & Inconsistencies

2021-07-01 Thread Simon Riggs
On Tue, Jun 22, 2021 at 6:32 AM Craig Ringer wrote: > IIRC the restart_lsn horizon already ensures that we can't miss the > xl_xact_assignment at the start of a txn. We would ensure that the desired > info is available throughout decoding of the txn, including at commit record > processing

Re: Detecting File Damage & Inconsistencies

2021-06-21 Thread Craig Ringer
On Tue, 22 Jun 2021 at 00:24, Simon Riggs wrote: > On Thu, Mar 18, 2021 at 6:20 AM Craig Ringer > wrote: > > > > On Mon, 15 Mar 2021 at 21:01, David Steele wrote: > >> > >> On 11/18/20 5:23 AM, Simon Riggs wrote: > >> > On Wed, 18 Nov 2020 at 06:42, Craig Ringer > >> > wrote: > >> >> > >> >>

Re: Detecting File Damage & Inconsistencies

2021-06-21 Thread Simon Riggs
On Thu, Mar 18, 2021 at 6:20 AM Craig Ringer wrote: > > On Mon, 15 Mar 2021 at 21:01, David Steele wrote: >> >> On 11/18/20 5:23 AM, Simon Riggs wrote: >> > On Wed, 18 Nov 2020 at 06:42, Craig Ringer >> > wrote: >> >> >> >> On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs wrote: >> >>> >> >>> >>

Re: Detecting File Damage & Inconsistencies

2021-03-18 Thread Craig Ringer
On Mon, 15 Mar 2021 at 21:01, David Steele wrote: > On 11/18/20 5:23 AM, Simon Riggs wrote: > > On Wed, 18 Nov 2020 at 06:42, Craig Ringer > > wrote: > >> > >> On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs > wrote: > >>> > >>> > >>> What I'm proposing is an option to add 16 bytes onto each

Re: Detecting File Damage & Inconsistencies

2021-03-15 Thread David Steele
On 11/18/20 5:23 AM, Simon Riggs wrote: On Wed, 18 Nov 2020 at 06:42, Craig Ringer wrote: On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs wrote: What I'm proposing is an option to add 16 bytes onto each COMMIT record Would it make sense to write this at the time we write a topxid

Re: Detecting File Damage & Inconsistencies

2020-12-28 Thread Masahiko Sawada
Hi Simon, On Wed, Nov 18, 2020 at 2:14 AM Simon Riggs wrote: > > On Fri, 13 Nov 2020 at 11:24, Simon Riggs wrote: > > > > On Fri, 13 Nov 2020 at 00:50, tsunakawa.ta...@fujitsu.com > > wrote: > > > > > > From: Simon Riggs > > > > If a rogue user/process is suspected, this would allow you to

Re: Detecting File Damage & Inconsistencies

2020-11-18 Thread Simon Riggs
On Wed, 18 Nov 2020 at 06:42, Craig Ringer wrote: > > On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs wrote: >> >> >> What I'm proposing is an option to add 16 bytes onto each COMMIT >> record > > > Would it make sense to write this at the time we write a topxid assignment to > WAL instead? > >

Re: Detecting File Damage & Inconsistencies

2020-11-17 Thread Craig Ringer
On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs wrote: > > What I'm proposing is an option to add 16 bytes onto each COMMIT > record > Would it make sense to write this at the time we write a topxid assignment to WAL instead? Otherwise it won't be accessible to streaming-mode logical decoding.

Re: Detecting File Damage & Inconsistencies

2020-11-17 Thread Simon Riggs
On Fri, 13 Nov 2020 at 11:24, Simon Riggs wrote: > > On Fri, 13 Nov 2020 at 00:50, tsunakawa.ta...@fujitsu.com > wrote: > > > > From: Simon Riggs > > > If a rogue user/process is suspected, this would allow you to identify > > > more easily the changes made by specific sessions/users. > > > >

Re: Detecting File Damage & Inconsistencies

2020-11-13 Thread Simon Riggs
On Fri, 13 Nov 2020 at 00:50, tsunakawa.ta...@fujitsu.com wrote: > > From: Simon Riggs > > If a rogue user/process is suspected, this would allow you to identify > > more easily the changes made by specific sessions/users. > > Isn't that kind of auditing a job of pgAudit or log_statement = mod?

RE: Detecting File Damage & Inconsistencies

2020-11-12 Thread tsunakawa.ta...@fujitsu.com
From: Simon Riggs > If a rogue user/process is suspected, this would allow you to identify > more easily the changes made by specific sessions/users. Isn't that kind of auditing a job of pgAudit or log_statement = mod? Or, does "more easily" mean that you find pgAudit complex to use and/or

Re: Detecting File Damage & Inconsistencies

2020-11-11 Thread Simon Riggs
On Thu, 12 Nov 2020 at 06:42, tsunakawa.ta...@fujitsu.com wrote: > > From: Simon Riggs > > I would like to propose a few points that will help us detect file > > damage, inconsistencies in files and track actions of users. > > Hello, Simon san. Long time no see. I'm happy to see you be back

RE: Detecting File Damage & Inconsistencies

2020-11-11 Thread tsunakawa.ta...@fujitsu.com
From: Simon Riggs > I would like to propose a few points that will help us detect file > damage, inconsistencies in files and track actions of users. Hello, Simon san. Long time no see. I'm happy to see you be back here recently. What kind of improvement do you expect? What problems would

Re: Detecting File Damage & Inconsistencies

2020-11-11 Thread Jose Luis Tallon
On 11/11/20 21:56, Simon Riggs wrote: [ŝnip] REINDEX VERIFY After the new index is created, but before we drop the old index: Check whether the two indexes match: * checks whether the previous index had pointers to row versions that don't exist * checks whether the heap has rows that were not