Re: Disable WAL logging to speed up data loading

2021-07-05 Thread Stephen Frost
Greetings, * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > On Monday, July 5, 2021 10:32 AM Michael Paquier wrote: > > On Sun, Jul 04, 2021 at 11:02:01AM -0400, Stephen Frost wrote: > > > Rather than RfC, the appropriate status seems like it should be > > > Rejected, as

RE: Disable WAL logging to speed up data loading

2021-07-04 Thread osumi.takami...@fujitsu.com
On Monday, July 5, 2021 10:32 AM Michael Paquier wrote: > On Sun, Jul 04, 2021 at 11:02:01AM -0400, Stephen Frost wrote: > > Rather than RfC, the appropriate status seems like it should be > > Rejected, as otherwise it's just encouraging someone to ultimately > > waste their time rebasing and

Re: Disable WAL logging to speed up data loading

2021-07-04 Thread Michael Paquier
On Sun, Jul 04, 2021 at 11:02:01AM -0400, Stephen Frost wrote: > Rather than RfC, the appropriate status seems like it should be > Rejected, as otherwise it's just encouraging someone to ultimately waste > their time rebasing and updating the patch when it isn't going to ever > actually be

Re: Disable WAL logging to speed up data loading

2021-07-04 Thread Stephen Frost
Greetings, * vignesh C (vignes...@gmail.com) wrote: > On Wed, Apr 7, 2021 at 12:13 PM osumi.takami...@fujitsu.com > wrote: > > Mainly affected by a commit 9de9294, > > I've fixed minor things to rebase the patch. > > All modifications I did are cosmetic changes and > > a little bit of

Re: Disable WAL logging to speed up data loading

2021-07-03 Thread vignesh C
On Wed, Apr 7, 2021 at 12:13 PM osumi.takami...@fujitsu.com wrote: > > Hi > > > Mainly affected by a commit 9de9294, > I've fixed minor things to rebase the patch. > All modifications I did are cosmetic changes and > a little bit of documentation updates. > Please have a look at attached v09. >

RE: Disable WAL logging to speed up data loading

2021-04-07 Thread osumi.takami...@fujitsu.com
Hi Mainly affected by a commit 9de9294, I've fixed minor things to rebase the patch. All modifications I did are cosmetic changes and a little bit of documentation updates. Please have a look at attached v09. Best Regards, Takamichi Osumi disable_WAL_logging_v09.patch Description:

Re: Disable WAL logging to speed up data loading

2021-03-24 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: Stephen Frost > > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > As for data loading tools, surely they support loading data into UNLOGGED > > tables and it's certainly not hard to have

RE: Disable WAL logging to speed up data loading

2021-03-24 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > As Laurenz-san kindly replied, the database server refuses to start with a > clear message. So, it's similarly very clear what happens. The user will > never > unknowingly resume operation with

Re: Disable WAL logging to speed up data loading

2021-03-23 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: Stephen Frost > > First- what are you expecting would actually happen during crash recovery in > > this specific case with your proposed new WAL level? > ... > > I'm not suggesting it's somehow more crash safe-

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Laurenz Albe
On Mon, 2021-03-22 at 13:22 -0400, Stephen Frost wrote: > * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > > On Mon, 2021-03-22 at 11:05 -0400, Stephen Frost wrote: > > > > Perhaps allowing to set unlogged tables to logged ones without writing > > > > WAL > > > > is a more elegant way to do

RE: Disable WAL logging to speed up data loading

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > First- what are you expecting would actually happen during crash recovery in > this specific case with your proposed new WAL level? ... > I'm not suggesting it's somehow more crash safe- but it's at least very clear > what happens in such a case, to wit: the entire table is

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > On Mon, 2021-03-22 at 11:05 -0400, Stephen Frost wrote: > > > Perhaps allowing to set unlogged tables to logged ones without writing WAL > > > is a more elegant way to do that, but I cannot see how that would be any > > > more crash

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Laurenz Albe
On Mon, 2021-03-22 at 11:05 -0400, Stephen Frost wrote: > > Perhaps allowing to set unlogged tables to logged ones without writing WAL > > is a more elegant way to do that, but I cannot see how that would be any > > more crash safe than this patch. Besides, the feature doesn't exist yet. > > I'm

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > On Mon, 2021-03-22 at 09:46 -0400, Stephen Frost wrote: > > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > > From: Stephen Frost > > > > The argument here seems to stem from the idea that issueing a 'TRUNCATE'

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Laurenz Albe
On Mon, 2021-03-22 at 09:46 -0400, Stephen Frost wrote: > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > From: Stephen Frost > > > The argument here seems to stem from the idea that issueing a 'TRUNCATE' > > > inside the transaction before starting the 'COPY' command is

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: Stephen Frost > > The argument here seems to stem from the idea that issueing a 'TRUNCATE' > > inside the transaction before starting the 'COPY' command is 'too hard'. > > > I could be wrong and perhaps

RE: Disable WAL logging to speed up data loading

2021-03-21 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > The argument here seems to stem from the idea that issueing a 'TRUNCATE' > inside the transaction before starting the 'COPY' command is 'too hard'. > I could be wrong and perhaps opinions will change in the future, but it really > doesn't seem like the there's much support

Re: Disable WAL logging to speed up data loading

2021-03-19 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: David Steele > > After reading through the thread (but not reading the patch) I am -1 on > > this proposal. > > > > The feature seems ripe for abuse and misunderstanding, and as has been > > noted in the

RE: Disable WAL logging to speed up data loading

2021-03-19 Thread tsunakawa.ta...@fujitsu.com
From: Laurenz Albe > Now I'm not saying that this feature should not go in (I set it to > "ready for committer", because I see no technical flaw with the > implementation), but it remains debatable if we want the feature or not. Oh, yes, thank you very much for supporting this and other relevant

Re: Disable WAL logging to speed up data loading

2021-03-19 Thread Laurenz Albe
On Fri, 2021-03-19 at 06:53 +, tsunakawa.ta...@fujitsu.com wrote: > From: David Steele > > After reading through the thread (but not reading the patch) I am -1 on > > this proposal. > > > > The feature seems ripe for abuse and misunderstanding, and as has been > > noted in the thread, there

RE: Disable WAL logging to speed up data loading

2021-03-19 Thread tsunakawa.ta...@fujitsu.com
From: David Steele > After reading through the thread (but not reading the patch) I am -1 on > this proposal. > > The feature seems ripe for abuse and misunderstanding, and as has been > noted in the thread, there are a variety of alternatives that can > provide a similar effect. > > It doesn't

Re: Disable WAL logging to speed up data loading

2021-03-18 Thread David Steele
On 1/17/21 12:16 AM, tsunakawa.ta...@fujitsu.com wrote: From: Osumi, Takamichi/大墨 昂道 I updated my patch to take in those feedbacks ! Have a look at the latest patch. Looks good to me (the status remains ready for committer.) After reading through the thread (but not reading the patch) I am

RE: Disable WAL logging to speed up data loading

2021-01-16 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > I updated my patch to take in those feedbacks ! > Have a look at the latest patch. Looks good to me (the status remains ready for committer.) Regards Takayuki Tsunakawa

RE: Disable WAL logging to speed up data loading

2021-01-15 Thread osumi.takami...@fujitsu.com
Hi, Movead Thanks for your comments. > I read the patch and have two points: > > 1. I do basebackup for database then switch wal level from logical to none to > logical and of cause I archive the wal segments. Next I do PITR base on the > basebackup, as a result it success startup with a waring

RE: Disable WAL logging to speed up data loading

2021-01-15 Thread osumi.takami...@fujitsu.com
Hi Thank you everyone On Thursday, January 14, 2021 9:27 AM Tsunakawa, Takayuki/綱川 貴之 wrote: > From: Kyotaro Horiguchi > > > XLogSetRecordFlags(XLOG_MARK_UNIMPORTANT | > > XLOG_MARK_ESSENTIAL); > > > XLogRegisterData((char *) , sizeof(dummy)); > > > > > > (Here's a word play -

RE: Disable WAL logging to speed up data loading

2021-01-13 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > > XLogSetRecordFlags(XLOG_MARK_UNIMPORTANT | > XLOG_MARK_ESSENTIAL); > > XLogRegisterData((char *) , sizeof(dummy)); > > > > (Here's a word play - unimportant but essential, what's that?) > > Hmm. Food may not be important to someone but it is essential for >

Re: Disable WAL logging to speed up data loading

2021-01-13 Thread Kyotaro Horiguchi
At Wed, 13 Jan 2021 06:01:34 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Kyotaro Horiguchi > > XLogBeginInsert(); > > XLogSetRecrodFlags(XLOG_MARK_ESSENTIAL); # new flag value > > XLOGInsert(); > > Oh, sounds like a nice idea. That's more flexible by allowing WAL-emitting >

RE: Disable WAL logging to speed up data loading

2021-01-12 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > XLogBeginInsert(); > XLogSetRecrodFlags(XLOG_MARK_ESSENTIAL); # new flag value > XLOGInsert(); Oh, sounds like a nice idea. That's more flexible by allowing WAL-emitting modules to specify which WAL records are mandatory even when wal_level is none. For example,

Re: Disable WAL logging to speed up data loading

2021-01-12 Thread movead li
I read the patch and have two points: 1. I do basebackup for database then switch wal level from logical to none to logical and of cause I archive the wal segments. Next I do PITR base on the basebackup, as a result it success startup with a waring said maybe data missed. Because the 'none'

Re: Disable WAL logging to speed up data loading

2021-01-12 Thread Kyotaro Horiguchi
At Tue, 12 Jan 2021 07:09:28 +, "osumi.takami...@fujitsu.com" wrote in > On Tuesday, January 12, 2021 12:52 PM Takayuki/綱川 貴之 > wrote: > > From: Osumi, Takamichi/大墨 昂道 > > > I updated the patch following this discussion, and fixed the > > > documentation as well. > > > > > > +

RE: Disable WAL logging to speed up data loading

2021-01-11 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > Oops, sorry for this careless mistake. > Fixed. And again, regression test produces no failure. Now correct. (Remains ready for committer.) Regards Takayuki Tsunakawa

RE: Disable WAL logging to speed up data loading

2021-01-11 Thread osumi.takami...@fujitsu.com
On Tuesday, January 12, 2021 12:52 PM Takayuki/綱川 貴之 wrote: > From: Osumi, Takamichi/大墨 昂道 > > I updated the patch following this discussion, and fixed the > > documentation as well. > > > + (rmid == RM_GIST_ID && info == RM_GIST_ID) || > + (rmid ==

RE: Disable WAL logging to speed up data loading

2021-01-11 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > I updated the patch following this discussion, and fixed the documentation as > well. + (rmid == RM_GIST_ID && info == RM_GIST_ID) || + (rmid == RM_GENERIC_ID))) info is wrong for GiST, and one pair of parenthesis is unnecessary.

RE: Disable WAL logging to speed up data loading

2021-01-11 Thread osumi.takami...@fujitsu.com
Hi On Mon, Jan 11, 2021 9:14 AM Tsunakawa, Takayuki wrote: > From: Masahiko Sawada > > I think it's better to have index AM (and perhaps table AM) control it > > instead of filtering in XLogInsert(). Because otherwise third-party > > access methods that use LSN like gist indexes cannot write

RE: Disable WAL logging to speed up data loading

2021-01-10 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > I think it's better to have index AM (and perhaps table AM) control it > instead of filtering in XLogInsert(). Because otherwise third-party > access methods that use LSN like gist indexes cannot write WAL records > at all when wal_level = none even if they want. Hm,

Re: Disable WAL logging to speed up data loading

2021-01-08 Thread Masahiko Sawada
On Fri, Jan 8, 2021 at 2:12 PM tsunakawa.ta...@fujitsu.com wrote: > > From: Robert Haas > > Were the issues that I mentioned regarding GIST (and maybe other AMs) > > in the last paragraph of > > http://postgr.es/m/CA+TgmoZEZ5RONS49C7mEpjhjndqMQtVrz_LCQUkpRW > > dmrev...@mail.gmail.com > >

Re: Disable WAL logging to speed up data loading

2021-01-07 Thread Kyotaro Horiguchi
At Fri, 8 Jan 2021 05:12:02 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Robert Haas > > Were the issues that I mentioned regarding GIST (and maybe other AMs) > > in the last paragraph of > > http://postgr.es/m/CA+TgmoZEZ5RONS49C7mEpjhjndqMQtVrz_LCQUkpRW > > dmrev...@mail.gmail.com >

RE: Disable WAL logging to speed up data loading

2021-01-07 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > Were the issues that I mentioned regarding GIST (and maybe other AMs) > in the last paragraph of > http://postgr.es/m/CA+TgmoZEZ5RONS49C7mEpjhjndqMQtVrz_LCQUkpRW > dmrev...@mail.gmail.com > addressed in some way? That seems like a pretty hard engineering > problem to me, and I

Re: Disable WAL logging to speed up data loading

2021-01-07 Thread Robert Haas
On Wed, Dec 2, 2020 at 10:53 PM tsunakawa.ta...@fujitsu.com wrote: > The code looks good, and the performance seems to be nice, so I marked this > ready for committer. Were the issues that I mentioned regarding GIST (and maybe other AMs) in the last paragraph of

RE: Disable WAL logging to speed up data loading

2021-01-06 Thread osumi.takami...@fujitsu.com
Hi, On Tuesday, January 5, 2021 5:45 PM Masahiko Sawada wrote: > On Tue, Jan 5, 2021 at 10:54 AM osumi.takami...@fujitsu.com > wrote: > > On Monday, December 28, 2020 7:12 PM Masahiko Sawada > wrote: > > > On Mon, Dec 28, 2020 at 4:29 PM osumi.takami...@fujitsu.com > > > wrote: > > > > On

Re: Disable WAL logging to speed up data loading

2021-01-05 Thread Masahiko Sawada
On Tue, Jan 5, 2021 at 10:54 AM osumi.takami...@fujitsu.com wrote: > > Hi, Sawada-San > > > On Monday, December 28, 2020 7:12 PM Masahiko Sawada > wrote: > > On Mon, Dec 28, 2020 at 4:29 PM osumi.takami...@fujitsu.com > > wrote: > > > On Monday, December 28, 2020 2:29 PM Masahiko Sawada > >

RE: Disable WAL logging to speed up data loading

2021-01-04 Thread osumi.takami...@fujitsu.com
Hi, Sawada-San On Monday, December 28, 2020 7:12 PM Masahiko Sawada wrote: > On Mon, Dec 28, 2020 at 4:29 PM osumi.takami...@fujitsu.com > wrote: > > On Monday, December 28, 2020 2:29 PM Masahiko Sawada > wrote: > > > On Thu, Dec 3, 2020 at 12:14 PM osumi.takami...@fujitsu.com > > > wrote:

RE: Disable WAL logging to speed up data loading

2020-12-31 Thread tsunakawa.ta...@fujitsu.com
From: Simon Riggs > Agreed, it is a footgun. -1 to commit the patch as-is. > > The patch to avoid WAL is simple but it is dangerous for both the user > and the PostgreSQL project. > > In my experience, people will use this option and when it crashes and > they lose their data, they will claim

RE: Disable WAL logging to speed up data loading

2020-12-31 Thread tsunakawa.ta...@fujitsu.com
From: Michael Paquier > Something that has not been mentioned on this thread is that if you could also > put pg_wal/ on a RAM disk. That's similarly unsafe, of course, but it does > not > require any extra upstream patching, and that should be really fast for the > case > of this thread. If

Re: Disable WAL logging to speed up data loading

2020-12-30 Thread Simon Riggs
On Wed, 30 Dec 2020 at 13:04, Michael Paquier wrote: > > Yes, we did think of this feature already and rejected it. > > I don't recall seeing such a discussion. Do you have some references? > Just a matter of curiosity :) Off-list discussions. -- Simon Riggs

Re: Disable WAL logging to speed up data loading

2020-12-30 Thread Michael Paquier
On Mon, Dec 28, 2020 at 11:06:30AM +, Simon Riggs wrote: > Agreed, it is a footgun. -1 to commit the patch as-is. > > The patch to avoid WAL is simple but it is dangerous for both the user > and the PostgreSQL project. Something that has not been mentioned on this thread is that if you could

Re: Disable WAL logging to speed up data loading

2020-12-28 Thread Simon Riggs
On Fri, 25 Dec 2020 at 07:09, Michael Paquier wrote: > > On Thu, Dec 03, 2020 at 03:52:47AM +, tsunakawa.ta...@fujitsu.com wrote: > > The code looks good, and the performance seems to be nice, so I > > marked this ready for committer. > > FWIW, I am extremely afraid of this proposal because

Re: Disable WAL logging to speed up data loading

2020-12-28 Thread Masahiko Sawada
On Mon, Dec 28, 2020 at 4:29 PM osumi.takami...@fujitsu.com wrote: > > Hello Sawada-San > > > On Monday, December 28, 2020 2:29 PM Masahiko Sawada > wrote: > > On Thu, Dec 3, 2020 at 12:14 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > I've made a new patch v05 that took in comments to

RE: Disable WAL logging to speed up data loading

2020-12-27 Thread osumi.takami...@fujitsu.com
Hello Sawada-San On Monday, December 28, 2020 2:29 PM Masahiko Sawada wrote: > On Thu, Dec 3, 2020 at 12:14 PM osumi.takami...@fujitsu.com > wrote: > > > > I've made a new patch v05 that took in comments to filter out WALs > > more strictly and addressed some minor fixes that were discussed >

Re: Disable WAL logging to speed up data loading

2020-12-27 Thread Masahiko Sawada
On Thu, Dec 3, 2020 at 12:14 PM osumi.takami...@fujitsu.com wrote: > > Hello > > > I've made a new patch v05 that took in comments > to filter out WALs more strictly and addressed some minor fixes > that were discussed within past few days. > Also, I changed the documentations, considering those

RE: Disable WAL logging to speed up data loading

2020-12-25 Thread osumi.takami...@fujitsu.com
Hi, Michael Thank you for your attention to this thread. On Friday, December 25, 2020 4:09 PM Michael Paquier wrote: > On Thu, Dec 03, 2020 at 03:52:47AM +, tsunakawa.ta...@fujitsu.com > wrote: > > The code looks good, and the performance seems to be nice, so I marked > > this ready for

Re: Disable WAL logging to speed up data loading

2020-12-24 Thread Michael Paquier
On Thu, Dec 03, 2020 at 03:52:47AM +, tsunakawa.ta...@fujitsu.com wrote: > The code looks good, and the performance seems to be nice, so I > marked this ready for committer. FWIW, I am extremely afraid of this proposal because this is basically a footgun able to corrupt customer instances,

RE: Disable WAL logging to speed up data loading

2020-12-02 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > I've made a new patch v05 that took in comments to filter out WALs more > strictly > and addressed some minor fixes that were discussed within past few days. > Also, I changed the documentations, considering those modifications. The code looks

RE: Disable WAL logging to speed up data loading

2020-12-02 Thread osumi.takami...@fujitsu.com
Hello I've made a new patch v05 that took in comments to filter out WALs more strictly and addressed some minor fixes that were discussed within past few days. Also, I changed the documentations, considering those modifications. Best, Takamichi Osumi disable_WAL_logging_v05.patch

RE: Disable WAL logging to speed up data loading

2020-12-01 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > I executed each wal_level three times and calculated the average time > and found that disabling WAL logging reduced about 73 % of the minimal's > loading speed > in this test. This speed-up came from the difference of generated WAL sizes. So, it's 4x speedup when

RE: Disable WAL logging to speed up data loading

2020-12-01 Thread osumi.takami...@fujitsu.com
Hi Sorry for being late. On Tuesday, December 1, 2020 10:42 AM Tsunakawa, Takayuki wrote: > From: Kyotaro Horiguchi > > Yeah, although it's enough only to restrict non-harmful records > > practically, if we find that only a few kinds of records are needed, > > maybe it's cleaner to allow

RE: Disable WAL logging to speed up data loading

2020-11-30 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > We're emitting only redo logs. So I think theoretically we don't need > anything other than the shutdown checkpoint record because we don't > perform recovery and checkpoint record is required at startup. > > RM_XLOG_ID: > XLOG_FPI_FOR_HINT - not needed? > XLOG_FPI

Re: Disable WAL logging to speed up data loading

2020-11-29 Thread Masahiko Sawada
() On Mon, Nov 30, 2020 at 2:39 PM Kyotaro Horiguchi wrote: > > At Fri, 27 Nov 2020 13:34:49 +, "osumi.takami...@fujitsu.com" > wrote in > > Thank you, Horiguchi-San > > > > > I haven't seen a criteria of whether a record is emitted or not for > > > wal_leve=none. > > > > > > We're

Re: Disable WAL logging to speed up data loading

2020-11-29 Thread Kyotaro Horiguchi
At Fri, 27 Nov 2020 13:34:49 +, "osumi.takami...@fujitsu.com" wrote in > Thank you, Horiguchi-San > > > I haven't seen a criteria of whether a record is emitted or not for > > wal_leve=none. > > > > We're emitting only redo logs. So I think theoretically we don't need > > anything > >

RE: Disable WAL logging to speed up data loading

2020-11-27 Thread osumi.takami...@fujitsu.com
Thank you, Horiguchi-San > I haven't seen a criteria of whether a record is emitted or not for > wal_leve=none. > > We're emitting only redo logs. So I think theoretically we don't need > anything > other than the shutdown checkpoint record because we don't perform > recovery and checkpoint

RE: Disable WAL logging to speed up data loading

2020-11-27 Thread osumi.takami...@fujitsu.com
Hi, Tsunakawa-San > I'm afraid "none" doesn't represent the behavior because RM_XLOG_ID and > RM_XACT_ID WAL records, except for XLOG_FPI_*, are emitted. What's the > good name? IIUC, "minimal" is named after the fact that the minimal > amount of WAL necessary for crash recovery is generated.

RE: Disable WAL logging to speed up data loading

2020-11-27 Thread osumi.takami...@fujitsu.com
Hello, Sawada-San On Friday, November 27, 2020 3:08 PM Masahiko Sawada wrote: > - (errmsg("WAL was generated with wal_level=minimal, > data may be missing"), > + (errmsg("WAL was generated with wal_level<=minimal, > data may be missing"), >

Re: Disable WAL logging to speed up data loading

2020-11-27 Thread Kyotaro Horiguchi
At Fri, 27 Nov 2020 07:01:16 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Masahiko Sawada > > While testing the patch on some workload, I realized that > > XLOG_FPI_FOR_HINT record could still be emitted even when wal_level = > > none. IIUC that WAL record is not necessary during

RE: Disable WAL logging to speed up data loading

2020-11-26 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > While testing the patch on some workload, I realized that > XLOG_FPI_FOR_HINT record could still be emitted even when wal_level = > none. IIUC that WAL record is not necessary during wal_level = none > since the server cannot be the primary server and the server crash >

Re: Disable WAL logging to speed up data loading

2020-11-26 Thread Kyotaro Horiguchi
At Fri, 27 Nov 2020 15:07:34 +0900, Masahiko Sawada wrote in > On Tue, Nov 24, 2020 at 11:19 AM osumi.takami...@fujitsu.com > wrote: > > > > Hi > > > > On Monday, Nov 23, 2020 12:17 PM Tsunakawa, Takayuki > > wrote: > > > PREPARE TRANSACTION is the same as COMMIT in that it persists > > >

Re: Disable WAL logging to speed up data loading

2020-11-26 Thread Masahiko Sawada
On Tue, Nov 24, 2020 at 11:19 AM osumi.takami...@fujitsu.com wrote: > > Hi > > On Monday, Nov 23, 2020 12:17 PM Tsunakawa, Takayuki > wrote: > > PREPARE TRANSACTION is the same as COMMIT in that it persists > > transaction updates. A crash during wal_level = none loses both of them. > > So, I

RE: Disable WAL logging to speed up data loading

2020-11-23 Thread osumi.takami...@fujitsu.com
Hi On Monday, Nov 23, 2020 12:17 PM Tsunakawa, Takayuki wrote: > PREPARE TRANSACTION is the same as COMMIT in that it persists > transaction updates. A crash during wal_level = none loses both of them. > So, I don't think PREPARE TRANSACTION needs special treatment. Yeah, I got it. That makes

RE: Disable WAL logging to speed up data loading

2020-11-22 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > This time, I updated my patch to address comments below only. I forgot to mentionthis. I confirmed all review comments are reflected correctly. Regards Takayuki Tsunakawa

RE: Disable WAL logging to speed up data loading

2020-11-22 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > > case TRANS_STMT_PREPARE: > > + if (wal_level == > > WAL_LEVEL_NONE) > > + ereport(ERROR, > > + > > errmsg("cannot execute PREPARE

Re: Disable WAL logging to speed up data loading

2020-11-20 Thread Stephen Frost
Greetings, * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > On Friday, Nov 20, 2020 9:33 AM Tsunakawa, Takayuki > wrote: > > From: Kyotaro Horiguchi > > > At Thu, 19 Nov 2020 11:04:17 -0500, Stephen Frost > > > > * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > > > > > I

RE: Disable WAL logging to speed up data loading

2020-11-20 Thread osumi.takami...@fujitsu.com
Hi This time, I updated my patch to address comments below only. Please have a look at updated one. On Thursday, Nov 19, 2020 4:51 PM Tsunakawa, Takayuki > (1) > #define RelationNeedsWAL(relation) > \ > + (wal_level != WAL_LEVEL_NONE && >

RE: Disable WAL logging to speed up data loading

2020-11-19 Thread osumi.takami...@fujitsu.com
Hello On Friday, Nov 20, 2020 9:33 AM Tsunakawa, Takayuki wrote: > From: Kyotaro Horiguchi > > At Thu, 19 Nov 2020 11:04:17 -0500, Stephen Frost > > > * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > > > > I missed that this is only a warning when I looked at it before. > > > > Yes, it

RE: Disable WAL logging to speed up data loading

2020-11-19 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > At Thu, 19 Nov 2020 11:04:17 -0500, Stephen Frost > > * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > > > I missed that this is only a warning when I looked at it before. > > > Yes, it should be a fatal error. > > > > Yeah, the more that I think about it, the more

Re: Disable WAL logging to speed up data loading

2020-11-19 Thread Kyotaro Horiguchi
At Thu, 19 Nov 2020 11:04:17 -0500, Stephen Frost wrote in > Greetings, > > * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > > On Thu, 2020-11-19 at 05:24 +, osumi.takami...@fujitsu.com wrote: > > > > > > ereport(WARNING, > > > > > > (errmsg("WAL was generated with

Re: Disable WAL logging to speed up data loading

2020-11-19 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > On Thu, 2020-11-19 at 05:24 +, osumi.takami...@fujitsu.com wrote: > > > > > ereport(WARNING, > > > > > (errmsg("WAL was generated with wal_level=minimal, data may > > > > > be missing"), > > > > > errhint("This

RE: Disable WAL logging to speed up data loading

2020-11-19 Thread osumi.takami...@fujitsu.com
Hello, Laurenz On Thursday, Nov19, 2020 4:50 PM Laurenz Albe wrote > On Thu, 2020-11-19 at 05:24 +, osumi.takami...@fujitsu.com wrote: > > > > > ereport(WARNING, > > > > > (errmsg("WAL was generated with wal_level=minimal, data > > > > > may be missing"), > > > > >

RE: Disable WAL logging to speed up data loading

2020-11-18 Thread tsunakawa.ta...@fujitsu.com
(1) #define RelationNeedsWAL(relation) \ + (wal_level != WAL_LEVEL_NONE && \ ((relation)->rd_rel->relpersistence ==

Re: Disable WAL logging to speed up data loading

2020-11-18 Thread Laurenz Albe
On Thu, 2020-11-19 at 05:24 +, osumi.takami...@fujitsu.com wrote: > > > > ereport(WARNING, > > > > (errmsg("WAL was generated with wal_level=minimal, data may > > > > be missing"), > > > > errhint("This happens if you temporarily set > > > > wal_level=minimal without taking a

RE: Disable WAL logging to speed up data loading

2020-11-18 Thread osumi.takami...@fujitsu.com
Hello On Thursday, Nov 19, 2020 12:45 PM Tsunakawa, Takayuki/綱川 貴之 > > On Tuesday, Nov 3, 2020 3:02 AM Stephen Frost > > wrote: > > > Checking the WAL level certainly seems critical to anything that's > > > reading the WAL. We certainly do this already when running as a > > > replica: > > >

RE: Disable WAL logging to speed up data loading

2020-11-18 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > there were some ideas to trace the change of wal_level, > in other words, *stronger mechanism* to check wal_level. > I agree with the idea to have a new monitoring item > and would like to implement those kind of, or one of those ideas for the next > patch. > But

RE: Disable WAL logging to speed up data loading

2020-11-18 Thread osumi.takami...@fujitsu.com
Hello In the past discussion of wal_level=none, there were some ideas to trace the change of wal_level, in other words, *stronger mechanism* to check wal_level. I agree with the idea to have a new monitoring item and would like to implement those kind of, or one of those ideas for the next

RE: Disable WAL logging to speed up data loading

2020-11-16 Thread tsunakawa.ta...@fujitsu.com
# It'd be helpful if you could send mails in text format, not HTML. From: David G. Johnston > For this case the fundamental feature that would seem to be required is an > ability for a transaction commit to return only after the system has ensured > that all of the new pages added to the

RE: Disable WAL logging to speed up data loading

2020-11-16 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > I'm also concerned about the way that this proposed feature interacts with > incremental backup capabilities that already exist in tools like pgBackRest, > EDB's BART, pg_probackup, and future things we might want to introduce into > core, along the lines of what I have

RE: Disable WAL logging to speed up data loading

2020-11-15 Thread osumi.takami...@fujitsu.com
Hi On Thursday, October 29, 2020 11:42 AM Fujii Masao wrote: > BTW, with the patch, I observed that PREPARE TRANSACTION and COMMIT > PREPARED caused assertion failure in my env, as I pointed upthread. > > How does the patch handle other feature depending on the existence of WAL, > e.g.,

RE: Disable WAL logging to speed up data loading

2020-11-11 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > I'm not sure that I see this as really being much of an issue. Perhaps there > are > some things we can do, as I mentioned before, to make it easier for users to > have tables be created as unlogged from the start, or to be able to ALTER > TABLE a bunch of tables at once

Re: Disable WAL logging to speed up data loading

2020-11-11 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > * ALTER TABLE SET UNLOGGED/LOGGED without data copy > Good: > - Does not require server restart (if this feature can be used in all > wal_level settings). > > Bad: > - The user has to maintain and modify some

RE: Disable WAL logging to speed up data loading

2020-11-11 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > Thanks! Since this feature is different from the feature that is > being proposed in this thhead, I started another thread for this > feature. > > https://www.postgresql.org/message-id/2020.173317.460890039962481 > 381.horikyota@gmail.com Thank you,

Re: Disable WAL logging to speed up data loading

2020-11-11 Thread Kyotaro Horiguchi
At Tue, 10 Nov 2020 09:33:12 -0500, Stephen Frost wrote in > Greetings, > > * Kyotaro Horiguchi (horikyota@gmail.com) wrote: > > For fuel(?) of the discussion, I tried a very-quick PoC for in-place > > ALTER TABLE SET LOGGED/UNLOGGED and resulted as attached. After some > > trials of

RE: Disable WAL logging to speed up data loading

2020-11-10 Thread osumi.takami...@fujitsu.com
Horiguchi-San On Tuesday, November 10, 2020 10:00 AM Kyotaro Horiguchi wrote: > FWIW, the following is that, I think it works not only when wal_level=minimal > for SET UNLOGGED, and only works when minimal for SET LOGGED. > >

Re: Disable WAL logging to speed up data loading

2020-11-10 Thread Stephen Frost
Greetings, * Kyotaro Horiguchi (horikyota@gmail.com) wrote: > For fuel(?) of the discussion, I tried a very-quick PoC for in-place > ALTER TABLE SET LOGGED/UNLOGGED and resulted as attached. After some > trials of several ways, I drifted to the following way after poking > several ways. > >

Re: Disable WAL logging to speed up data loading

2020-11-10 Thread Stephen Frost
Greetings, * Kyotaro Horiguchi (horikyota@gmail.com) wrote: > At Mon, 9 Nov 2020 10:18:08 -0500, Stephen Frost wrote > in > > Greetings, > > > > * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > > > When I consider the use case is the system of data warehouse > > > as

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Kyotaro Horiguchi
At Mon, 9 Nov 2020 10:18:08 -0500, Stephen Frost wrote in > Greetings, > > * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > > When I consider the use case is the system of data warehouse > > as described upthread, the size of each table can be large. > > Thus, changing the

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Stephen Frost
Greetings, * David G. Johnston (david.g.johns...@gmail.com) wrote: > On Mon, Nov 9, 2020 at 10:36 AM Stephen Frost wrote: > > * David G. Johnston (david.g.johns...@gmail.com) wrote: > > > If the commit doesn't complete all of the newly created pages are junk. > > > Otherwise, you have a

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread David G. Johnston
On Mon, Nov 9, 2020 at 10:36 AM Stephen Frost wrote: > * David G. Johnston (david.g.johns...@gmail.com) wrote: > > > If the commit doesn't complete all of the newly created pages are junk. > > Otherwise, you have a crash-recoverable state for those tables as regards > > those specific pages. > >

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Stephen Frost
Greetings, * David G. Johnston (david.g.johns...@gmail.com) wrote: > On Mon, Nov 9, 2020 at 8:18 AM Stephen Frost wrote: > > Presently, my feeling is that we could address this use-case without > > having to introduce a new cluster-wide WAL level, and that's the > > direction I'd want to see

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread David G. Johnston
On Mon, Nov 9, 2020 at 8:18 AM Stephen Frost wrote: > Presently, my feeling is that we could address this use-case without > having to introduce a new cluster-wide WAL level, and that's the > direction I'd want to see this going. Perhaps I'm missing something > about why the approach I've set

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Stephen Frost
Greetings, * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > On Tuesday, Nov 3, 2020 3:02 AM Stephen Frost wrote: > > I'm not sure that wal_level=none is really the right way to address this > > use-case. We already have unlogged tables and that's pretty clean and > > meets

RE: Disable WAL logging to speed up data loading

2020-11-08 Thread osumi.takami...@fujitsu.com
Hello, Stephen On Tuesday, Nov 3, 2020 3:02 AM Stephen Frost wrote: > * Magnus Hagander (mag...@hagander.net) wrote: > > On Mon, Nov 2, 2020 at 4:28 PM Robert Haas > wrote: > > > On Thu, Oct 29, 2020 at 4:00 PM Fujii Masao > wrote: > > > > Yes. What I meant was such a safe guard needs to be

Re: Disable WAL logging to speed up data loading

2020-11-02 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > On Mon, Nov 2, 2020 at 4:28 PM Robert Haas wrote: > > > > On Thu, Oct 29, 2020 at 4:00 PM Fujii Masao > > wrote: > > > Yes. What I meant was such a safe guard needs to be implemented. > > > > > > This may mean that if we want to

Re: Disable WAL logging to speed up data loading

2020-11-02 Thread Magnus Hagander
On Mon, Nov 2, 2020 at 4:28 PM Robert Haas wrote: > > On Thu, Oct 29, 2020 at 4:00 PM Fujii Masao > wrote: > > Yes. What I meant was such a safe guard needs to be implemented. > > > > This may mean that if we want to recover the database from that backup, > > we need to specify the recovery

  1   2   >