RE: Transactions involving multiple postgres foreign servers, take 2

2020-10-02 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > You proposed the first idea > to avoid such a situation that FDW implementor can write the code > while trying to reduce the possibility of errors happening as much as > possible, for example by usingpalloc_extended(MCXT_ALLOC_NO_OOM) and > hash_search(HASH_ENTER_NULL) but

RE: Disable WAL logging to speed up data loading

2020-10-01 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > > The speedup has already been achieved with higher durability by > > wal_level=minimal in that case. > > I was thinking the same, i.e., wal_level=minimal + wal_skip_threshold would > speed up that initial data loading. First of all, thank you Horiguchi-san for trying to

RE: New statistics for tuning WAL buffer size

2020-10-01 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > I think that we can improve that, for example, by storing backend id > into WalSndCtl and making pg_stat_get_wal_senders() directly > get the walsender's LocalPgBackendStatus with the backend id, > rather than joining pg_stat_get_activity() and pg_stat_get_wal_senders().

RE: Should walsernder check correctness of WAL records?

2020-10-01 Thread tsunakawa.ta...@fujitsu.com
From: Michael Paquier > CRC calculation would unlikely be the bottleneck here, no? I would assume > that the extra lseek() calls needed to look after the record data to be more > harmful. Maybe, although I'm not sure lseek() is necessary. I simply thought walsender was designed to just read

RE: Should walsernder check correctness of WAL records?

2020-10-01 Thread tsunakawa.ta...@fujitsu.com
From: Konstantin Knizhnik > Investigating one of customer's support cases I found out that walsender > is not calculating WAL records CRC and send them to replicas without any > checks. > As a result damaged WAL record causes errors on all replicas: IIUC, walsender tries hard to send WAL as fast

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-01 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > I thought that the advantage of this optimization is that we don't > need to visit all buffers? If we need to run a full-scan for any > reason, there's no point in looking-up already-visited buffers > again. That's just wastefull cycles. Am I missing somethig? > > I

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-01 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > For non-recovery path, did you mean by any chance > measuring the cache hit rate for varying shared_buffers? No. You can test the speed of DropRelFileNodeBuffers() during normal operation, i.e. by running TRUNCATE on psql, instead of performing recovery. To

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-30 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > In more detail, if smgrcachednblocks() returned InvalidBlockNumber for > any of the forks, we should give up the optimization at all since we > need to run a full scan anyway. On the other hand, if any of the > forks is smaller than the threshold, we still can use the

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-30 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > I have one idea for performance testing. We can even test this for > non-recovery paths by removing the recovery-related check like only > use it when there are cached blocks. You can do this if testing via > recovery path is difficult because at the end performance should be

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-30 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > Recovery performance measurement results below. > But it seems there are overhead even with large shared buffers. > > | s_b | master | patched | %reg | > |---||-|---| > | 128MB | 36.052 | 39.451 | 8.62% | > | 1GB | 21.731 | 21.73 |

RE: New statistics for tuning WAL buffer size

2020-09-30 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > Another reason that I mildly want to object to subdivided functions is > I was annoyed that a stats view makes many individual calls to > functions that internally share the same statistics entry. That > behavior required me to provide an entry-caching feature to my >

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-30 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > To avoid misunderstanding, I didn't mean to disregard the performance. > I mean especially for the transaction management feature it's > essential to work fine even in failure cases. So I hope we have a > safe, robust, and probably simple design for the first version that

RE: BLOB / CLOB support in PostgreSQL

2020-09-29 Thread tsunakawa.ta...@fujitsu.com
From: Vladimir Sitnikov > Just in case, I'm PgJDBC committer. Thank you very much for your great efforts for the wonderful PgJDBC. I saw you active. # I'd be happy if you send emails in text format so that the reply looks nice. Your email seems to be in HTML. > and apparently, Andrew is

Disable WAL logging to speed up data loading

2020-09-29 Thread tsunakawa.ta...@fujitsu.com
Hello, We'd like to propose a feature to disable WAL to speed up data loading. This was inspired by a feature added in the latest MySQL. I wish you won't fear this feature... BACKGROUND This branches off from [1] as mentioned therein. Briefly

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-28 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > No. Please imagine a case where a user executes PREPARE TRANSACTION on > the transaction that modified data on foreign servers. The backend > process prepares both the local transaction and foreign transactions. > But another client can execute COMMIT PREPARED on the

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-28 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > I also did not remove the duplicate code from smgrnblocks because Amit-san > mentioned that when the caching for non-recovery cases is implemented, we > can use it for non-recovery cases as well. But the extra code is not used now. The code for future usage

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-28 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > Is my understanding above correct? No. I simply meant DropRelFileNodeBuffers() calls the following function, and avoids the optimization if it returns InvalidBlockNumber. BlockNumber smgrcachednblocks(SMgrRelation reln, ForkNumber forknum) {

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-27 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > On Fri, 25 Sep 2020 at 18:21, tsunakawa.ta...@fujitsu.com > wrote: > > Why does the client backend have to create a new connection cache entry > during PREPARE or COMMIT PREPARE? Doesn't the client backend naturally > continue to use connections that

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-27 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > I agree with the above two points. Thank you. I'm relieved to know I didn't misunderstand. > > * Then, add a new function, say, smgrnblocks_cached() that simply returns > the cached block count, and DropRelFileNodeBuffers() uses it instead of > smgrnblocks(). > > > > I am

RE: Global snapshots

2020-09-27 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey san, all, From: tsunakawa.ta...@fujitsu.com > And please wait. As below, the patent holder just says that Clock-SI is not > based on the patent and an independent development. He doesn't say > Clock-SI does not overlap with the patent or implementing Clock-SI does not &

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-25 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > I don't think it's always possible to avoid raising errors in advance. > Considering how postgres_fdw can implement your idea, I think > postgres_fdw would need PG_TRY() and PG_CATCH() for its connection > management. It has a connection cache in the local memory using

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-25 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > [Results] > Recovery/Failover performance (in seconds). 3 trial runs. > > | shared_buffers | master | patch | %reg| > ||||-| > | 128MB | 32.406 | 33.785 | 4.08% | > | 1GB| 36.188 | 32.747 |

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-25 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > No, during recovery also we need to be careful. We need to ensure that > we use cached value during recovery and cached value is always > up-to-date. We can't rely on lseek and I have provided some scenario > up thread [1] where such behavior can cause problem and then see the

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-24 Thread tsunakawa.ta...@fujitsu.com
From: Ashutosh Bapat > The way I am looking at is to put the parallelism in the resolution > worker and not in the FDW. If we use multiple resolution workers, they > can fire commit/abort on multiple foreign servers at a time. From a single session's view, yes. However, the requests from

RE: I'd like to discuss scaleout at PGCon

2020-09-24 Thread tsunakawa.ta...@fujitsu.com
From: MauMau > I intentionally have put little conclusion on our specification and > design. I'd like you to look at recent distributed databases, and > then think about and discuss what we want to aim for together. I feel > it's better to separate a thread per topic or group of topics.

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-24 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > So with your idea, I think we require FDW developers to not call > ereport(ERROR) as much as possible. If they need to use a function > including palloc, lappend etc that could call ereport(ERROR), they > need to use PG_TRY() and PG_CATCH() and return the control along

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-23 Thread tsunakawa.ta...@fujitsu.com
In v15: (1) + for (cur_blk = firstDelBlock[j]; cur_blk < nblocks; cur_blk++) The right side of "cur_blk <" should not be nblocks, because nblocks is not the number of the relation fork anymore. (2) + BlockNumber nblocks; +

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-23 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > The idea is that we can't use this optimization if the value is not > cached because we can't rely on lseek behavior. See all the discussion > between Horiguchi-San and me in the thread above. So, how would you > ensure that if we don't use Kirk-San's proposal? Hmm, buggy

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-22 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > I revised the patch based from my understanding of Horiguchi-san's comment, > but I could be wrong. > Quoting: > > " > + /* Get the number of blocks for the supplied relation's > fork */ > + nblocks = smgrnblocks(smgr_reln,

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-22 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > > > > Don't > > > > we need to guarantee the cache to be valid while recovery? > > > > > > > > > > One possibility could be that we somehow detect that the value we > > > are using is cached one and if so then only do this optimization. > > > > I basically like this direction.

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-22 Thread tsunakawa.ta...@fujitsu.com
I looked at v14. (1) + /* Get the total number of blocks for the supplied relation's fork */ + for (j = 0; j < nforks; j++) + { + BlockNumber block = smgrnblocks(smgr_reln, forkNum[j]); + nblocks

RE: Global snapshots

2020-09-22 Thread tsunakawa.ta...@fujitsu.com
From: Andrey Lepikhov > Thank you for this work! > As I can see, main development difficulties placed in other areas: CSN, > resolver, > global deadlocks, 2PC commit... I'm not lawyer too. But if we get remarks from > the patent holders, we can rewrite our Clock-SI implementation. Yeah, I

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-22 Thread tsunakawa.ta...@fujitsu.com
From: Ashutosh Bapat > parallelism here has both pros and cons. If one of the servers errors > out while preparing for a transaction, there is no point in preparing > the transaction on other servers. In parallel execution we will > prepare on multiple servers before realising that one of them

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-21 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > Yes, but it still seems hard to me that we require for all FDW > implementations to commit/rollback prepared transactions without the > possibility of ERROR. Of course we can't eliminate the possibility of error, because remote servers require network communication.

RE: Global snapshots

2020-09-21 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey-san, all, From: Andrey V. Lepikhov > On 7/27/20 11:22 AM, tsunakawa.ta...@fujitsu.com wrote: > > Could you take a look at this patent? I'm afraid this is the Clock-SI for > > MVCC. > Microsoft holds this until 2031. I couldn't find this with the k

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-15 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > The resolver process has two functionalities: resolving foreign > transactions automatically when the user issues COMMIT (the case you > described in the second paragraph), and resolving foreign transaction > when the corresponding backend no longer exist or when the

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-15 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > > If so, can't we stipulate that the FDW implementor should ensure that the > commit function always returns control to the caller? > > How can the FDW implementor ensure that? Since even palloc could call > ereport(ERROR) I guess it's hard to require that to all FDW >

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-15 Thread tsunakawa.ta...@fujitsu.com
The code doesn't seem to be working correctly. (1) + for (block_num = 0; block_num <= nblocks; block_num++) should be + for (block_num = firstDelBlock[fork_num]; block_num < nblocks; block_num++) because: * You only want to

RE: Implement UNLOGGED clause for COPY FROM

2020-09-11 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > If we can skip the table-copy when ALTER TABLE SET LOGGED on > wal_level=minimal, is your objective achived? I expect so, if we can skip the table copy during ALTER TABLE SET LOGGED/UNLOGGED. On the other hand, both approaches have different pros and cons. It's nice

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-11 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > On Tue, 8 Sep 2020 at 13:00, tsunakawa.ta...@fujitsu.com > wrote: > > 2. 2PC processing is queued and serialized in one background worker. That > severely subdues transaction throughput. Each backend should perform > 2PC. > > Not sure it'

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-11 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > Originally start(), commit() and rollback() are supported as FDW interfaces. > As far as I and Sawada-san discussed this upthread, to support MySQL, > another type of start() would be necessary to issue "XA START id" command. > end() might be also necessary to issue

RE: Implement UNLOGGED clause for COPY FROM

2020-09-10 Thread tsunakawa.ta...@fujitsu.com
From: Peter Smith On Thu, Sep 10, 2020 at 7:16 PM tsunakawa.ta...@fujitsu.com > wrote: > > ALTER TABLE takes long time proportional to the amount of existing data, > while wal_level = none doesn't. > > Right, but if wal_level=none is considered OK for that table with > ex

RE: Implement UNLOGGED clause for COPY FROM

2020-09-10 Thread tsunakawa.ta...@fujitsu.com
From: Peter Smith > Earlier, Osumi-san was rejecting the idea of using ALTER TABLE tbl SET > UNLOGGED on basis that it is too time consuming for large data to > switch the table modes [1]. > Doesn't wal_level=none essentially just behave as if every table was > UNLOGGED; not just the ones we are

RE: Global snapshots

2020-09-10 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > But I'm concerned about that it's really hard to say there is no patent risk > around that. I'm not sure who can judge there is no patent risk, > in the community. Maybe no one? Anyway, I was thinking that Google Spanner, > YugabyteDB, etc use the global transaction approach

RE: SIGQUIT handling, redux

2020-09-09 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > This is straying a bit from the stated topic of this thread, but ... > I did some further looking around to see whether there were any > unsafe signal handlers besides SIGQUIT ones. The situation is not > too awful, but I did find several issues not already mentioned > in this

RE: Global snapshots

2020-09-09 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey san, From: Andrey V. Lepikhov > > From: tsunakawa.ta...@fujitsu.com > >> While Clock-SI seems to be considered the best promising for global > >>> > Could you take a look at this patent? I'm afraid this is the Clock-SI > >>> > for MVCC. &

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-09 Thread tsunakawa.ta...@fujitsu.com
Alexey-san, Sawada-san, cc: Fujii-san, From: Fujii Masao > But if we > implement 2PC as the improvement on FDW independently from PostgreSQL > sharding, I think that it's necessary to support other FDW. And this is our > direction, isn't it? I understand the same way as Fujii san. 2PC FDW is

RE: Inconsistent Japanese name order in v13 contributors list

2020-09-08 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > On 2020/09/09 14:15, Etsuro Fujita wrote: > > Attached is a patch to standardize Japanese names as given-name-first > > in the v13 contributors list as before. > > Using given-name-first order is our consensus? I was thinking we have not > reached that yet and our "vague"

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-08 Thread tsunakawa.ta...@fujitsu.com
From: tsunakawa.ta...@fujitsu.com > (1) > + INIT_BUFFERTAG(newTag, > rnode.node, forkNum[j], firstDelBlock[j]); > > And you need to use i here, too. I remember the books "Code Complete" and/or "Readable Code" suggest to

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-07 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > if (RelFileNodeEquals(bufHdr->tag.rnode, rnode.node) && > + bufHdr->tag.forkNum == forkNum[j] && > + bufHdr->tag.blockNum >= firstDelBlock[j]) > > Here, I think you need to use 'i' not 'j' for forkNum and > firstDelBlock as those are arrays w.r.t forks. That might fix

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > I intend to say that the global-visibility work can impact this in a > major way and we have analyzed that to some extent during a discussion > on the other thread. So, I think without having a complete > design/solution that addresses both the 2PC and global-visibility, it >

RE: New statistics for tuning WAL buffer size

2020-09-03 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > > I changed the view name from pg_stat_walwrites to pg_stat_walwriter. > > I think it is better to match naming scheme with other views like > pg_stat_bgwriter, > > which is for bgwriter statistics but it has the statistics related to > > backend. > > I prefer the view name

RE: On login trigger: take three

2020-09-03 Thread tsunakawa.ta...@fujitsu.com
From: Konstantin Knizhnik > Recently I have asked once again by one of our customers about login trigger > in > postgres. People are migrating to Postgres from Oracle and looking for > Postgres > analog of this Oracle feature. > This topic is not new: > I attached my prototype implementation

RE: Implement UNLOGGED clause for COPY FROM

2020-08-26 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > Sure, but on a daily basis, one requires only incremental WAL to > complete the backup but in this case, it would require the entire > database back up unless we have some form of block-level incremental > backup method. Regarding the backup time, I think users can shorten

RE: Implement UNLOGGED clause for COPY FROM

2020-08-26 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > So you want your users to shutdown and restart the server before Copy > because that would be required if you want to change the wal_level. Yes. They seem to be fine with it, as far as I heard from a person who is involved in the system design. > However, even if we do

RE: Implement UNLOGGED clause for COPY FROM

2020-08-26 Thread tsunakawa.ta...@fujitsu.com
Hello, I think it's worth thinking about a sophisticated feature like Oracle's UNRECOVERABLE data loading (because SQL Server's BCP load utility also has such a feature, but for an empty table), how about an easier approach like MySQL? I expect this won't complicate Postgres code much. The

RE: New statistics for tuning WAL buffer size

2020-08-20 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > Just idea; it may be worth exposing the number of when new WAL file is > created and zero-filled. This initialization may have impact on > the performance of write-heavy workload generating lots of WAL. If this > number is reported high, to reduce the number of this

RE: New statistics for tuning WAL buffer size

2020-08-20 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > I agree to expose the number of WAL write caused by full of WAL buffers. > It's helpful when tuning wal_buffers size. Haribabu separated that number > into two fields in his patch; one is the number of WAL write by backend, > and another is by background processes and workers.

RE: New statistics for tuning WAL buffer size

2020-08-18 Thread tsunakawa.ta...@fujitsu.com
From: Masahiro Ikeda > If my understanding is correct, we have to measure the performance > impact first. > Do you know HariBabu is now trying to solve it? If not, I will try to > modify patches to apply HEAD. No, he's not doing it anymore. It'd be great if you could resume it. However, I

RE: New statistics for tuning WAL buffer size

2020-08-18 Thread tsunakawa.ta...@fujitsu.com
From: Masahiro Ikeda > It's important to provide the metrics for tuning the size of WAL buffers. > For now, it's lack of the statistics how often processes wait to write WAL > because WAL buffer is full. > > If those situation are often occurred, WAL buffer is too small for the > workload. >

RE: Libpq support to connect to standby server as priority

2020-08-13 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > I think it would be better to have read-write and read-only check > trnasaction_read_only, and primary and standby can check the new > thing. There can never be any real advantage in having synonyms for > the same thing, but there can be an advantage to letting users choose >

Autonomous database is coming to Postgres?

2020-08-12 Thread tsunakawa.ta...@fujitsu.com
Hello, I'm not sure if I should have posted this to pgsql-advocacy, but this is being developed so I posted here. Does anyone know if this development come to open source Postgres, or only to the cloud services of Microsoft and Google? (I wonder this will become another reason that Postgres

RE: Can I test Extended Query in core test framework

2020-08-11 Thread tsunakawa.ta...@fujitsu.com
Tatsuo Ishii san, a committer, proposed this to test extended query protocol. Can it be included in Postgres core? A toool to test programs by issuing frontend/backend protocol messages https://github.com/tatsuo-ishii/pgproto Regards Takayuki Tsunakawa

RE: Global snapshots

2020-07-27 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey san, Movead san, From: tsunakawa.ta...@fujitsu.com > While Clock-SI seems to be considered the best promising for global > serializability here, > > * Why does Clock-SI gets so much attention? How did Clock-SI become the > only choice? > > * Clock-SI was

RE: Global snapshots

2020-07-22 Thread tsunakawa.ta...@fujitsu.com
Hello, While I'm thinking of the following issues of the current approach Andrey raised, I'm getting puzzled and can't help asking certain things. Please forgive me if I'm missing some discussions in the past. > 1. Dependency on clocks synchronization > 2. Needs guarantees of monotonically

RE: Transactions involving multiple postgres foreign servers, take 2

2020-07-20 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > I also believe they do. But I'm concerned that some FDW needs to start > a transaction differently when using 2PC. For instance, IIUC MySQL > also supports 2PC but the transaction needs to be started with "XA > START id” when the transaction needs to be prepared. The

RE: Transactions involving multiple postgres foreign servers, take 2

2020-07-17 Thread tsunakawa.ta...@fujitsu.com
From: Laurenz Albe > On Fri, 2020-07-17 at 05:21 +0000, tsunakawa.ta...@fujitsu.com wrote: > > And most importantly, do other major DBMSs, including Oracle, provide the > API for > > preparing a transaction? In other words, will the FDWs other than > postgres_fdw >

RE: Transactions involving multiple postgres foreign servers, take 2

2020-07-16 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada I have briefly checked the only oracle_fdw but in general I think that > if an existing FDW supports transaction begin, commit, and rollback, > these can be ported to new FDW transaction APIs easily. Does oracle_fdw support begin, commit and rollback? And most importantly,

RE: Transactions involving multiple postgres foreign servers, take 2

2020-07-15 Thread tsunakawa.ta...@fujitsu.com
Hi Sawada san, I'm reviewing this patch series, and let me give some initial comments and questions. I'm looking at this with a hope that this will be useful purely as a FDW enhancement for our new use cases, regardless of whether the FDW will be used for Postgres scale-out. I don't think

RE: Postgres is not able to handle more than 4k tables!?

2020-07-09 Thread tsunakawa.ta...@fujitsu.com
From: Konstantin Knizhnik > Unfortunately we have not to wait for decade or two. > Postgres is faced with multiple problems at existed multiprocessor > systems (64, 96,.. cores). > And it is not even necessary to initiate thousands of connections: just > enough to load all this cores and let them

RE: Postgres is not able to handle more than 4k tables!?

2020-07-09 Thread tsunakawa.ta...@fujitsu.com
From: Konstantin Knizhnik > Looks like it is not true (at lest for PG9.6): > > #0 0x7fa6d30da087 in semop () from /lib64/libc.so.6 > #1 0x00682241 in PGSemaphoreLock > (sema=sema@entry=0x7fa66f5655d8) at pg_sema.c:387 > #2 0x006ec6eb in LWLockAcquire >

RE: Implement UNLOGGED clause for COPY FROM

2020-07-09 Thread tsunakawa.ta...@fujitsu.com
From: David G. Johnston > This step seems to invalidate the idea outright. The checkpoint command is > superuser only and isn’t table specific. This seems to require both those > things to be changed. Perhaps FlushRelationBuffers() followed by smgrsync() can be used instead. Or, depending

RE: Postgres is not able to handle more than 4k tables!?

2020-07-08 Thread tsunakawa.ta...@fujitsu.com
From: Konstantin Knizhnik > Autovacuum is periodically processing all this 5k relations (because > them are actively updated). > And as far as most of this tables are small enough autovacuum complete > processing of them almost in the same time. > As a result autovacuum workers produce ~5k

RE: I'd like to discuss scaleout at PGCon

2020-06-17 Thread tsunakawa.ta...@fujitsu.com
Hello, It seems you didn't include pgsql-hackers. From: Sumanta Mukherjee > I saw the presentation and it is great except that it seems to be unclear of > both SD and SN if the storage and the compute are being explicitly > separated. Separation of storage and compute would have some cost

RE: [bug?] Is the format of tables in the documentation broken in PG 13?

2020-06-15 Thread tsunakawa.ta...@fujitsu.com
From: Daniel Gustafsson > Yes, this was a deliberate change made to be able to fit more expansive > descriptions of columns etc. Thanks for your quick response and information. I'm relieved to know that it was not broken. > That's less good. The W3C Web Accessibility Initiative has guidance

[bug?] Is the format of tables in the documentation broken in PG 13?

2020-06-15 Thread tsunakawa.ta...@fujitsu.com
Hello, The tables for pg_stat_ views in the following page, starting from Table 27.3, have only one column in PG 13. They had 3 columns in PG 12 and earlier. https://www.postgresql.org/docs/13/monitoring-stats.html Is this intentional? It has become a bit unfriendly to read for me, a

RE: POC: GUC option for skipping shared buffers in core dumps

2020-02-11 Thread tsunakawa.ta...@fujitsu.com
From: Craig Ringer > Currently my options are "dump all shmem including shared_buffers" or > "dump no shmem". But I usually want "dump all shmem except > shared_buffers". It's tolerable to just dump s_b on a test system with > a small s_b, but if enabling coredumps to track down some >

RE: open-source equivalent of golden-gate

2020-02-11 Thread tsunakawa.ta...@fujitsu.com
From: Chapman Flack > I read the question as perhaps concerning the other direction, whether > there might be an open source foreign data wrapper installable in Oracle > for talking to PostgreSQL (which might, I suppose, also have a name like > "postgres_fdw", which helps explain the number of

RE: Internal key management system

2020-02-09 Thread tsunakawa.ta...@fujitsu.com
From: Andres Freund > Perhaps this has already been discussed (I only briefly looked): I'd > strongly advise against having any new infrastrure depend on > pgcrypto. Its code quality imo is well below our standards and contains > serious red flags like very outdated copies of cryptography

RE: Just for fun: Postgres 20?

2020-02-09 Thread tsunakawa.ta...@fujitsu.com
From: Jose Luis Tallon >     Musing some other date-related things I stumbled upon the thought > that naming the upcoming release PostgreSQL 20 might be preferrable to > the current/expected "PostgreSQL 13". +1 Users can easily know how old/new the release is that they are using. Regards

RE: Make ringbuffer threshold and ringbuffer sizes configurable?

2020-02-05 Thread tsunakawa.ta...@fujitsu.com
From: Andres Freund > While I think there's a number of improvements[2] we could make to the > ringbuffer logic, I think we should also just allow to make them > configurable. I think that'll allow a decent number of systems perform > better (especially on slightly bigger systems the the current

RE: Complete data erasure

2020-02-04 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > Up to now, we've sort of looked the other way with respect to failures > of file unlinks post-commit, reasoning that the worst that will happen > is disk space leakage from no-longer-referenced files that we failed to > unlink. (Which is bad, certainly, but not catastrophic;

RE: Complete data erasure

2020-02-03 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > That's not really what I meant - let me explain. When I said DROP TABLE > should do everything as usual, that includes catalog changes. I.e. after > the commit there would not be any remaining entries in system catalogs > or anything like that. > > The only thing we'd do

RE: SLRU statistics

2020-01-20 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > You're right the users can't really take advantage of this - my primary > motivation was providing a feedback for devs, benchmarking etc. That > might have been done with DEBUG messages or something, but this seems > more convenient. Understood. I'm in favor of adding

RE: SLRU statistics

2020-01-19 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > One of the stats I occasionally wanted to know are stats for the SLRU > stats (we have couple of those - clog, subtrans, ...). So here is a WIP > version of a patch adding that. How can users take advantage of this information? I think we also need the ability to set the

RE: Let people set host(no)ssl settings from initdb

2020-01-07 Thread tsunakawa.ta...@fujitsu.com
From: David Fetter > > But I see two problems with the proposed approach: (1) initdb > > doesn't support setting up SSL, so the only thing you can achieve > > here is to reject all TCP/IP connections, until you have set up SSL. > > I don't believe any special setup is needed to require TLS for

RE: Libpq support to connect to standby server as priority

2020-01-05 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > So, we can know whether server is primary/standby by checking > in_recovery, as opposed to knowing whether read-write which is done by > checking transaction_read_only. So we can keep read-write as a synonym > for "primary", and check in_recovery when used in servers that

RE: Libpq support to connect to standby server as priority

2019-12-26 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > I'm not sure I understand why we end up with "prefer-read" in addition > to "prefer-standby" (and similar seeming redundancy between "primary" > and "read-write"). Do we really need more than one way to identify > hosts' roles? It seems 0001 adds the "prefer-read" modes

RE: Implementing Incremental View Maintenance

2019-12-24 Thread tsunakawa.ta...@fujitsu.com
From: Tatsuo Ishii > AFAIK benefit of ON STATEMENT is the transaction can see the result of > update to the base tables. With ON COMMIT, the transaction does not > see the result until the transaction commits. > > Well, I can see use cases of IVM in both DWH and OLTP. > > For example, a user

RE: Implementing Incremental View Maintenance

2019-12-23 Thread tsunakawa.ta...@fujitsu.com
From: Yugo Nagata > On Mon, 23 Dec 2019 08:08:53 + > "tsunakawa.ta...@fujitsu.com" wrote: > > How about unlogged tables ? I thought the point of using a temp table is to > avoid WAL overhead. > > Hmm... this might be another option. However, if we use

RE: Implementing Incremental View Maintenance

2019-12-23 Thread tsunakawa.ta...@fujitsu.com
From: Tatsuo Ishii > First of all, we do not think that current approach is the final > one. Instead we want to implement IVM feature one by one: i.e. we > start with "immediate update" approach, because it's simple and easier > to implement. Then we will add "deferred update" mode later on. I

RE: Implementing Incremental View Maintenance

2019-12-23 Thread tsunakawa.ta...@fujitsu.com
From: Yugo Nagata > 1. Create a temporary table only once at the first view maintenance in > this session. This is possible if we store names or oid of temporary > tables used for each materialized view in memory. However, users may > access to these temptables whenever during the session. > >

RE: Implementing Incremental View Maintenance

2019-12-22 Thread tsunakawa.ta...@fujitsu.com
From: Tatsuo Ishii > the target data? In the discussion, someone referred to master data with low > update frequency, because the proposed IVM implementation adds triggers on > source tables, which limits the applicability to update-heavy tables. > > But if you want to get always up-to-data you

RE: Implementing Incremental View Maintenance

2019-12-22 Thread tsunakawa.ta...@fujitsu.com
From: legrand legrand > For each insert into a base table there are 3 statements: > - ANALYZE pg_temp_3.pg_temp_81976 > - WITH updt AS ( UPDATE public.mv1 AS mv SET __ivm_count__ = ... > - DROP TABLE pg_temp_3.pg_temp_81976 Does it also include CREATE TEMPORARY TABLE, because there's DROP? I

RE: Implementing Incremental View Maintenance

2019-12-22 Thread tsunakawa.ta...@fujitsu.com
From: Tatsuo Ishii > > The following IVM wiki page returns an error. Does anybody know what's > wrong? > > > > https://wiki.postgresql.org/wiki/Incremental_View_Maintenance > > I don't have any problem with the page. Maybe temporary error? Yeah, I can see it now. I could see it on the

RE: Implementing Incremental View Maintenance

2019-12-19 Thread tsunakawa.ta...@fujitsu.com
Hello, I'm starting to take a closer look at this feature. I've just finished reading the discussion, excluding other referenced materials. The following IVM wiki page returns an error. Does anybody know what's wrong? https://wiki.postgresql.org/wiki/Incremental_View_Maintenance [screen]

RE: Hooks for session start and end, take two

2019-12-19 Thread tsunakawa.ta...@fujitsu.com
From: Michael Paquier > Adding extra custom logging information, or plug that information > elsewhere than Postgres. I have use cases for that when it comes to > store external telemetry data or audit things for events happening > specifically in Postgres. > > Well, hook authors can do a lot of

RE: Libpq support to connect to standby server as priority

2019-12-19 Thread tsunakawa.ta...@fujitsu.com
From: Greg Nancarrow > With the permission of the original patch author, Haribabu Kommi, I’ve > rationalized the existing 8 patches into 3 patches, merging patches > 1-5 and 6-7, and tidying up some documentation and code comments. I > also rebased them to the latest PG12 source code (as of

RE: reducing memory usage by using "proxy" memory contexts?

2019-12-16 Thread tsunakawa.ta...@fujitsu.com
From: Andres Freund > We waste a lot of space due to all these small contexts. Even leaving > aside the overhead of the context and its blocks - not insignificant - > they are mostly between ~1/2 a ~1/4 empty. > > > But what if we had a new type of memory context that did not itself > manage

RE: get_database_name() from background worker

2019-12-11 Thread tsunakawa.ta...@fujitsu.com
From: Koichi Suzuki > I'm not using this. Is this the must to use get_database_name()? I don't think pg_background is a must, but the system catalog access by get_database_name() should require database connection and transaction. See src/test/modules/worker_spi/worker_spi.c for an example

<    1   2   3   4   5   >