[HACKERS] Tee for COPY

Hi, I am trying to create version of COPY command which can scatter/replicate data to different nodes based on some distribution method. There is some master process, having information about data distribution, to which all clients are connected. This master process should receive copied data

Re: [HACKERS] Patch: fix lock contention for HASHHDR.mutex

On 11 December 2015 at 16:14, Aleksander Alekseev wrote: > I see your point, but I would like to clarify a few things. > > 1. Do we consider described measurement method good enough to conclude > that sometimes PostgreSQL really spends 3 ms in a spinlock (like a RTT

Re: [HACKERS] Logical replication and multimaster

On 13 December 2015 at 11:02, Andres Freund wrote: > On December 13, 2015 10:19:07 AM CET, Simon Riggs > wrote: > >I didn't see the patch for this anywhere. Where is the code? > > Where I'd the code for all of pg logical? > Thanks for asking, perhaps

Re: [HACKERS] Logical replication and multimaster

On 2015-12-13 11:39:32 +, Simon Riggs wrote: > On 13 December 2015 at 11:02, Andres Freund wrote: > > On December 13, 2015 10:19:07 AM CET, Simon Riggs > > wrote: > > >I didn't see the patch for this anywhere. Where is the code? > > > > Where is the

Re: [HACKERS] Logical replication and multimaster

On December 13, 2015 10:19:07 AM CET, Simon Riggs wrote: >I didn't see the patch for this anywhere. Where is the code? Where I'd the code for all of pg logical? --- Please excuse brevity and formatting - I am writing this on my mobile phone. -- Sent via pgsql-hackers

Re: [HACKERS] Logical replication and multimaster

On 6 December 2015 at 17:39, Konstantin Knizhnik wrote: > I have integrated pglogical_output in multimaster, using bdr_apply from > BDR as template for implementation of receiver part. > I didn't see the patch for this anywhere. Where is the code? -- Simon Riggs

Re: Using a single standalone-backend run in initdb (was Re: [HACKERS] Bootstrap DATA is a pita)

On 13 December 2015 at 06:31, Tom Lane wrote: > I'm not particularly wedded to this rule. In principle we could go so > far as to import psql's code that parses commands and figures out which > semicolons are command terminators --- but that is a pretty large chunk > of

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

On 2015-12-12 21:15:52 -0500, Robert Haas wrote: > On Sat, Dec 12, 2015 at 1:17 PM, and...@anarazel.de > wrote: > > Here's two patches doing that. The first is an adaption of your > > constants patch, using an enum and also converting xlog.c's locks. The > > second is the

Re: [HACKERS] Disabling an index temporarily

On Sun, Dec 13, 2015 at 1:33 AM, Oleg Bartunov wrote: > > On Sun, Dec 13, 2015 at 1:16 AM, Jaime Casanova < > jaime.casan...@2ndquadrant.com> wrote: > >> indexrelid = 'indexname'::regclass; > > > This works, but might bloat system catalog. > > +1 for the functionality. +1

Re: [HACKERS] Disabling an index temporarily

>> On Sun, Dec 13, 2015 at 1:16 AM, Jaime Casanova < >> jaime.casan...@2ndquadrant.com> wrote: >> >>> indexrelid = 'indexname'::regclass; >> >> >> This works, but might bloat system catalog. >> >> > +1 for the functionality. > +1 for ALTER INDEX foo SET DISABLED -1 for the reason I mentioned in

Re: [HACKERS] Disabling an index temporarily

On Sun, 13 Dec 2015 22:15:31 -0500 Corey Huinker wrote: > ALTER TABLE foo DISABLE [NONUNIQUE] INDEXES > -- same, but joining to pg_class and possibly filtering on indisunique I would think that NONUNIQUE should be the default, and you should have to specify something

Re: [HACKERS] Using quicksort for every external sort run

On Sun, Dec 13, 2015 at 3:40 PM, Peter Geoghegan wrote: > On Sat, Dec 12, 2015 at 4:41 PM, Jeff Janes wrote: > Also, if I am reading this correctly, when we refill a pool from a logical tape we still transform each tuple as it is read from the

Re: [HACKERS] Disabling an index temporarily

On Sun, Dec 13, 2015 at 7:27 PM, Tom Lane wrote: > Corey Huinker writes: >> So, I'd propose we following syntax: >> ALTER INDEX foo SET DISABLED >> -- does the SET indisvalid = false shown earlier. > > This is exactly *not* what Tatsuo-san was after,

Re: [HACKERS] pg_stat_replication log positions vs base backups

On Fri, Nov 27, 2015 at 6:07 AM, Michael Paquier wrote: > On Thu, Nov 26, 2015 at 10:53 PM, Magnus Hagander > wrote: > > On Thu, Nov 26, 2015 at 1:03 PM, Michael Paquier < > michael.paqu...@gmail.com> > > wrote: > >> > >> On Thu, Nov 26, 2015 at

Re: [HACKERS] Logical replication and multimaster

On 12/13/2015 12:19 PM, Simon Riggs wrote: On 6 December 2015 at 17:39, Konstantin Knizhnik > wrote: I have integrated pglogical_output in multimaster, using bdr_apply from BDR as template for implementation of receiver part.

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

On Fri, Dec 11, 2015 at 6:34 PM, Andres Freund wrote: > > On 2015-12-11 15:56:46 +0300, Alexander Korotkov wrote: > > > > Yes, there is a cycle with retries in LWLockAcquire function. The case of > > retry is when waiter is waked up, but someone other steal the lock before > >

Re: Using a single standalone-backend run in initdb (was Re: [HACKERS] Bootstrap DATA is a pita)

Craig Ringer writes: > On 13 December 2015 at 06:31, Tom Lane wrote: >> I'm not particularly wedded to this rule. In principle we could go so >> far as to import psql's code that parses commands and figures out which >> semicolons are command

Re: [HACKERS] Tee for COPY

On Sun, Dec 13, 2015 at 11:29:23AM +0300, Konstantin Knizhnik wrote: > Hi, > > I am trying to create version of COPY command which can scatter/replicate > data to different nodes based on some distribution method. > There is some master process, having information about data distribution, to >

Re: [HACKERS] Logical replication and multimaster

On Sat, Dec 12, 2015 at 06:48:58PM +0800, Craig Ringer wrote: > Being able to access pg_dump and pg_restore's dependency resolution logic, > object dumping routines, etc from regular SQL and from the SPI would be > wonderful. As I understand it, pushing these into a library has been proposed but

Re: [HACKERS] strange CREATE INDEX tab completion cases

On Sat, Dec 12, 2015 at 11:17 AM, Peter Eisentraut wrote: > These two tab completion pieces look strange to me: > > /* If we have CREATE|UNIQUE INDEX CONCURRENTLY, then add "ON" */ > else if ((pg_strcasecmp(prev3_wd, "INDEX") == 0 || >

[HACKERS] Proposal: custom compression methods

Hackers, I'd like to propose a new feature: "Custom compression methods". *Motivation* Currently when datum doesn't fit the page PostgreSQL tries to compress it using PGLZ algorithm. Compression of particular attributes could be turned on/off by tuning storage parameter of column. Also, there

Re: Using a single standalone-backend run in initdb (was Re: [HACKERS] Bootstrap DATA is a pita)

> On Dec 12, 2015, at 9:40 PM, Tom Lane wrote: > > Mark Dilger writes: >>> On Dec 12, 2015, at 3:42 PM, Tom Lane wrote: >>> ... In general, though, I'd rather not try to >>> teach InteractiveBackend() such a large amount about

Re: [HACKERS] Logical replication and multimaster

On 13 December 2015 at 11:53, Andres Freund wrote: > > Thanks for asking, perhaps our plans weren't public enough. pglogical has > > already been announced as open source, under the postgres licence and > that > > it will be a submission to core PostgreSQL, just as BDR was.

Re: [HACKERS] Memory prefetching while sequentially fetching from SortTuple array, tuplestore

On Mon, Nov 30, 2015 at 1:04 PM, Peter Geoghegan wrote: > Perhaps we can consider more selectively applying prefetching in the > context of writing out tuples. It may still be worth selectively applying these techniques to writing out tuples, per my previous remarks (and the

Re: [HACKERS] PATCH: add pg_current_xlog_flush_location function

Hi, On 12/13/2015 06:13 AM, Amit Kapila wrote: > ... > Is there a reason why you can't use existing function GetFlushRecPtr() in xlog.c? No, not really. I think I somehow missed that function when writing the initial version of the patch. Will fix in v2 of the patch. thanks -- Tomas

Re: [HACKERS] Making tab-complete.c easier to maintain

On Sun, Dec 13, 2015 at 1:08 AM, Michael Paquier wrote: > On Wed, Dec 9, 2015 at 8:17 PM, Thomas Munro wrote: >> Thanks for looking at this Michael. It's probably not much fun to >> review! Here is a new version with that bit removed. More responses >> inline below.

Re: [HACKERS] pg_stat_replication log positions vs base backups

On Mon, Dec 14, 2015 at 1:01 AM, Magnus Hagander wrote: > I've applied these two patches now. > > The one that fixes the initialization backpatched to 9.3 which is the oldest > one that has it, and the one that changes the actual 0-vs-NULL output to 9.5 > only as it's a

Re: [HACKERS] Using quicksort for every external sort run

On Sat, Dec 12, 2015 at 4:41 PM, Jeff Janes wrote: > Those usages make sense to me, as they are locally self-contained and > it is clear what they are in contradistinction to. But your usage is > spread throughout (even in function names, not just comments) and > seems to

Re: [HACKERS] Using quicksort for every external sort run

On Tue, Dec 8, 2015 at 6:39 PM, Greg Stark wrote: > On Wed, Dec 9, 2015 at 12:02 AM, Jeff Janes wrote: >> >> >> Then in the next (final) merge, it is has to read in this huge >> fragmented tape run emulation, generating a lot of random IO to read >> it. > >

[HACKERS] Function and view to retrieve WAL receiver status

Hi all, Currently there is no equivalent of pg_stat_get_wal_senders for the WAL receiver on a node, and it seems that it would be useful to have an SQL representation of what is in shared memory should a WAL receiver be active without going through the ps display for example. So, any opinion

Re: [HACKERS] Parallel Aggregate

On Fri, Dec 11, 2015 at 5:42 PM, Haribabu Kommi wrote: > 3. Performance test to observe the effect of parallel aggregate. Here I attached the performance test report of parallel aggregate. Summary of the result is: 1. Parallel aggregate is not giving any improvement or

Re: [HACKERS] Disabling an index temporarily

Corey Huinker writes: > So, I'd propose we following syntax: > ALTER INDEX foo SET DISABLED > -- does the SET indisvalid = false shown earlier. This is exactly *not* what Tatsuo-san was after, though; he was asking for a session-local disable, which I would think would

Re: [HACKERS] Disabling an index temporarily

Bill Moran writes: > I would think that NONUNIQUE should be the default, and you should have > to specify something special to also disable unique indexes. Arguably, > unique indexes are actually an implementation detail of unique > constraints. Disabling a

Re: [HACKERS] Disabling an index temporarily

On Mon, Dec 14, 2015 at 12:27 PM, Tom Lane wrote: > Certainly, there's opportunities to improve the flexibility of the > index-disable specifications in the plug-in Oleg and Teodor did. But > I think that that is the right basic approach: some sort of SET command, > not

Re: [HACKERS] Disabling an index temporarily

Jeff Janes writes: > Not to hijack the thread even further in the wrong direction, but I > think what Corey really wants here is to stop maintaining the index at > retail while preserving the existing definition and existing index > data, and then to do a wholesale fix-up,

Re: [HACKERS] Making tab-complete.c easier to maintain

On Mon, Dec 14, 2015 at 8:10 AM, Thomas Munro wrote: > I've also add (very) primitive negative pattern support and used it in > 5 places. Improvement? Examples: > > /* ALTER TABLE xxx RENAME yyy */ > - else if (TailMatches5("ALTER", "TABLE",

Re: [HACKERS] Using quicksort for every external sort run

On Sun, Dec 13, 2015 at 7:31 PM, Jeff Janes wrote: > The reason for memtuples is to handle random access. Since we are no > longer doing random access, we no longer need it. > > We could free memtuples, re-allocate just enough to form the binary > heap for the N-way merge,

Re: [HACKERS] Proposal: custom compression methods

On 14 December 2015 at 15:27, Chapman Flack wrote: > On 12/14/15 01:50, Craig Ringer wrote: > > > pg_upgrade means you can't just redefine the current toast bits so the > > compressed bit means "data is compressed, check first byte of varlena > data > > for algorithm"

Re: [HACKERS] Function and view to retrieve WAL receiver status

On Dec 13, 2015 9:56 PM, "Michael Paquier" wrote: > > If the node has no WAL receiver active, a tuple with NULL values is > returned instead. IMO, in the absence of a WAL receiver the SRF (and the view) should not return any rows.

Re: [HACKERS] Function and view to retrieve WAL receiver status

On Mon, Dec 14, 2015 at 3:09 PM, Gurjeet Singh wrote: > On Dec 13, 2015 9:56 PM, "Michael Paquier" > wrote: >> If the node has no WAL receiver active, a tuple with NULL values is >> returned instead. > > IMO, in the absence of a WAL receiver

Re: [HACKERS] W-TinyLfu for cache eviction

> a global lock would be good enough for a proof of concept that only evaluates cache hit ratios. I think emulator can be used to check hit ratios. That way we can see how different algorithms affect hit ratio. Is there a set of traces of "buffer load events"? (I did some Google searches like

Re: [HACKERS] [sqlsmith] Failed to generate plan on lateral subqueries

On Mon, Dec 14, 2015 at 03:06:18PM +0900, Michael Paquier wrote: > On Sun, Dec 13, 2015 at 10:14 AM, Andreas Seltenreich > wrote: > >> Did you publish the source already? I haven't been following all > >> along, sorry if these are all answered questions. > > > > It's not had a

Re: [HACKERS] Disabling an index temporarily

2015-12-14 5:34 GMT+02:00 Tom Lane : > Maybe I misunderstood, but I thought what was being discussed here is > preventing the planner from selecting an index for use in queries, while > still requiring all table updates to maintain validity of the index. > The O-ther big DBMS

Re: [HACKERS] [sqlsmith] Failed to generate plan on lateral subqueries

On Sun, Dec 13, 2015 at 10:14 AM, Andreas Seltenreich wrote: >> Did you publish the source already? I haven't been following all >> along, sorry if these are all answered questions. > > It's not had a proper release yet, but the code is available via github > in all its

Re: [HACKERS] Proposal: custom compression methods

On 14 December 2015 at 01:28, Alexander Korotkov wrote: > Hackers, > > I'd like to propose a new feature: "Custom compression methods". > Are you aware of the past work in this area? There's quite a bit of history and I strongly advise you to read the relevant threads

Re: [HACKERS] pgbench stats per script & other stuff

On Sat, Oct 3, 2015 at 3:11 PM, Fabien COELHO wrote: > >> Here is a v10, which is a rebase because of the "--progress-timestamp" >> option addition. > > > Here is a v11, which is a rebase after some recent changes committed to > pgbench. +The provided scriptname

Re: [HACKERS] Declarative partitioning

On 2015/11/24 2:23, Robert Haas wrote: > To me, it seems like there is a pretty obvious approach here: each > table can be either a plain table, or a partition root (which can look > just like an empty inheritance parent). Then multi-level partitioning > falls right out of that design without

Re: [HACKERS] Proposal: custom compression methods

On 12/14/15 01:50, Craig Ringer wrote: > pg_upgrade means you can't just redefine the current toast bits so the > compressed bit means "data is compressed, check first byte of varlena data > for algorithm" because existing data won't have that, the first byte will > be the start of the compressed

[HACKERS] Fdw cleanup

Hi, Hackers, I need some help to understand foreign table error handling. For a query on foreign table, ExecInitForeignScan is called, which in turn calls the BeginForeignScan hook. Inside this hook, I allocated some resource, node->fdw_state = allocate_resource(...); If everything goes