Re: [HACKERS] sequences and pg_upgrade

2016-08-30 Thread Bruce Momjian
On Tue, Aug 30, 2016 at 08:46:48AM -0400, Peter Eisentraut wrote: > I think the other solution mentioned in that thread would also work: > Have pg_upgrade treat sequences more like system catalogs, whose format > changes between major releases, and transferred them via the > dump/restore route.

[HACKERS] pageinspect: Hash index support

2016-08-30 Thread Jesper Pedersen
Hi, Attached is a patch that adds support for hash indexes in pageinspect. The functionality (hash_metap, hash_page_stats and hash_page_items) follows the B-tree functions. This patch will need an update once Amit's and Mithun's work on Concurrent Hash Indexes is committed to account for

Re: [HACKERS] GIN logging GIN_SEGMENT_UNMODIFIED actions?

2016-08-30 Thread Tom Lane
Fujii Masao writes: > I found that pg_xlogdump code for XLOG_GIN_INSERT record with > GIN_INSERT_ISLEAF flag has the same issue, i.e., > "unknown action 0" error is thrown for that record. > The latest patch fixes this. Hmm, comparing gin_desc() to ginRedoInsert() makes me

Re: [HACKERS] standalone backend PANICs during recovery

2016-08-30 Thread Tom Lane
Michael Paquier writes: > Does the attached suit better then? Thinking about it some more ... what we actually need to prevent, AFAICS, is standby_mode becoming true in a standalone backend. If you don't set that, then the process will do PITR recovery, but I'm not

[HACKERS] some requests on auditing

2016-08-30 Thread Pavel Stehule
Hi I am working on pgaudit customization for one my customer. There are few requests: 1. flat format without complex types, without nesting - CSV is ideal. 2. all important attributes should be separated - is not possible to search in original queries: table name, database name, role name,

Re: [HACKERS] New SQL counter statistics view (pg_stat_sql)

2016-08-30 Thread Alvaro Herrera
Haribabu Kommi wrote: > Apart from the above, here are the following list of command tags that > are generated in the code, I took only the first word of the command tag > just to see how many categories present. The number indicates the > subset of operations or number of types it is used. Like

Re: [HACKERS] pageinspect: Hash index support

2016-08-30 Thread Alvaro Herrera
Jesper Pedersen wrote: > Hi, > > Attached is a patch that adds support for hash indexes in pageinspect. > > The functionality (hash_metap, hash_page_stats and hash_page_items) follows > the B-tree functions. I suggest that pageinspect functions are more convenient to use via the get_raw_page

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Tom Lane
Michael Paquier writes: > And with an actual patch things are better. Working through this patch, it suddenly strikes me that we are going about fixing the callers of simple_prompt the wrong way. The existing definition with returning a malloc'd string creates a

Re: [HACKERS] [WIP] Patches to enable extraction state of query execution from external session

2016-08-30 Thread Andres Freund
On 2016-08-30 11:22:43 +0300, Maksim Milyutin wrote: > > Hi, > > > > On 2016-08-29 18:22:56 +0300, maksim wrote: > > > Now I complete extension that provides facility to see the current state > > > of > > > query execution working on external session in form of EXPLAIN ANALYZE > > > output. This

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-08-30 Thread Heikki Linnakangas
On 08/30/2016 02:38 PM, Tom Lane wrote: Heikki Linnakangas writes: While profiling some queries and looking at executor overhead, I realized that we're not making much use of TupleTableSlot's ability to hold a buffer pin. In a SeqScan, the buffer is held pinned by the

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-08-30 Thread Andres Freund
Hi, On 2016-08-30 13:12:41 +0300, Heikki Linnakangas wrote: > While profiling some queries and looking at executor overhead, I realized > that we're not making much use of TupleTableSlot's ability to hold a buffer > pin. FWIW, I came to a similar conclusion, while working on passing around

Re: [HACKERS] sequences and pg_upgrade

2016-08-30 Thread Andres Freund
On 2016-08-30 08:46:48 -0400, Peter Eisentraut wrote: > I was toying with a couple of ideas that would involve changing the > storage of sequences. (Say, for the sake of discussion, removing the > problematic/useless sequence_name field.) I'd be quite interested to know what changes that are...

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-08-30 Thread Andres Freund
On 2016-08-30 21:59:44 +0100, Greg Stark wrote: > On Tue, Aug 30, 2016 at 11:12 AM, Heikki Linnakangas wrote: > > While profiling some queries and looking at executor overhead, I realized > > that we're not making much use of TupleTableSlot's ability to hold a buffer > > pin. In

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-08-30 Thread Greg Stark
On Tue, Aug 30, 2016 at 11:12 AM, Heikki Linnakangas wrote: > While profiling some queries and looking at executor overhead, I realized > that we're not making much use of TupleTableSlot's ability to hold a buffer > pin. In a SeqScan, the buffer is held pinned by the underlying

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-08-30 Thread Heikki Linnakangas
On 08/30/2016 03:26 AM, Andreas Karlsson wrote: On 08/26/2016 11:31 AM, Heikki Linnakangas wrote: On 07/05/2016 04:46 PM, Andreas Karlsson wrote: @@ -280,8 +287,9 @@ px_find_digest(const char *name, PX_MD **res) digest = px_alloc(sizeof(*digest)); digest->algo = md; -

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Michael Paquier
On Tue, Aug 30, 2016 at 2:57 PM, Michael Paquier wrote: > The funny part here is that ProcGlobal->allProcs is actually handled, > but not the two others. Well yes, you are right, we really need to > fail on FATAL for all of them if ShmemAlloc returns NULL as they >

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Michael Paquier
On Mon, Aug 29, 2016 at 11:26 PM, Tom Lane wrote: > Aleksander Alekseev writes: >>> if (prodesc->user_proname == NULL || prodesc->internal_proname == NULL) >>> + { >>> +free(prodesc); > >> I think that prodesc->user_proname and

Re: [HACKERS] multivariate statistics (v19)

2016-08-30 Thread Michael Paquier
On Wed, Aug 24, 2016 at 2:03 AM, Robert Haas wrote: > ISTR that you were going to try to look at this patch set. It seems > from the discussion that it's not really ready for serious > consideration for commit yet, but also that some high-level design > comments from you

Re: [HACKERS] GIN logging GIN_SEGMENT_UNMODIFIED actions?

2016-08-30 Thread Fujii Masao
On Tue, Aug 30, 2016 at 3:39 AM, Tom Lane wrote: > Fujii Masao writes: >> ISTM that the cause of this issue is that gin_desc() uses XLogRecGetData() to >> extract ginxlogVacuumDataLeafPage data from XLOG_GIN_VACUUM_DATA_LEAF_PAGE >> record. Since it's

Re: [HACKERS] standalone backend PANICs during recovery

2016-08-30 Thread Michael Paquier
On Wed, Aug 24, 2016 at 5:07 PM, Bernd Helmle wrote: > That said, i'm okay if --single is not intended to bring up a hot standby. > There are many other ways to debug such problems. This patch is still on the CF app: https://commitfest.postgresql.org/10/610/ Even after

[HACKERS] autonomous transactions

2016-08-30 Thread Peter Eisentraut
I would like to propose the attached patch implementing autonomous transactions for discussion and review. This work was mostly inspired by the discussion about pg_background a while back [0]. It seemed that most people liked the idea of having something like that, but couldn't perhaps agree on

Re: [HACKERS] ICU integration

2016-08-30 Thread Peter Eisentraut
On 8/30/16 11:27 PM, Craig Ringer wrote: > Speaking of which, have you had a chance to try it on Windows yet? nope > How stable are the UCU locales? Most importantly, does ICU offer any > way to "pin" a locale version, so we can say "we want de_DE as it was > in ICU 4.6" and get consistent

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Tom Lane
Michael Paquier writes: > [ malloc-nulls-v5.patch ] I've committed some form of all of these changes except the one in adt/pg_locale.c. I'm not entirely sure whether we need to do anything about that at all, but if we do, this doesn't cut it: thousands_sep =

Re: [HACKERS] ICU integration

2016-08-30 Thread Craig Ringer
On 31 August 2016 at 10:46, Peter Eisentraut wrote: > Here is a patch I've been working on to allow the use of ICU for sorting > and other locale things. Great to see you working on this. We've got some icky head-in-the-sand issues in this area when it comes to

Re: [HACKERS] standalone backend PANICs during recovery

2016-08-30 Thread Michael Paquier
On Tue, Aug 30, 2016 at 11:00 PM, Tom Lane wrote: > Michael Paquier writes: >> Does the attached suit better then? > > Thinking about it some more ... what we actually need to prevent, AFAICS, > is standby_mode becoming true in a standalone backend.

Re: [HACKERS] pageinspect: Hash index support

2016-08-30 Thread Michael Paquier
On Wed, Aug 31, 2016 at 2:06 AM, Alvaro Herrera wrote: > Jesper Pedersen wrote: >> Attached is a patch that adds support for hash indexes in pageinspect. >> >> The functionality (hash_metap, hash_page_stats and hash_page_items) follows >> the B-tree functions. > > I

[HACKERS] ICU integration

2016-08-30 Thread Peter Eisentraut
Here is a patch I've been working on to allow the use of ICU for sorting and other locale things. This is mostly complementary to the existing FreeBSD ICU patch, most recently discussed in [0]. While that patch removes the POSIX locale use and replaces it with ICU, my interest was on allowing

Re: [HACKERS] autonomous transactions

2016-08-30 Thread Joel Jacobson
I would love to see autonomous transactions in core. I just have one major concern, but thankfully it's easily addressed. There should be a way to within the session and/or txn permanently block autonomous transactions. This is important if you as a caller function want to be sure none of the

Re: [HACKERS] autonomous transactions

2016-08-30 Thread Jaime Casanova
On 30 August 2016 at 20:50, Peter Eisentraut wrote: > > - Patches to PL/pgSQL to implement Oracle-style autonomous transaction > blocks: > > AS $$ > DECLARE > PRAGMA AUTONOMOUS_TRANSACTION; > BEGIN > FOR i IN 0..9 LOOP > START TRANSACTION; > INSERT

[HACKERS] sequence data type

2016-08-30 Thread Peter Eisentraut
Here is a patch that adds the notion of a data type to a sequence. So it might be CREATE SEQUENCE foo AS integer. The types are restricted to int{2,4,8} as now. The main point of this is to make monitoring sequences less complicated. Right now, a serial column creates an int4 column but

Re: [HACKERS] ICU integration

2016-08-30 Thread Michael Paquier
On Wed, Aug 31, 2016 at 1:12 PM, Peter Eisentraut wrote: > On 8/30/16 11:27 PM, Craig Ringer wrote: >> Speaking of which, have you had a chance to try it on Windows yet? > > nope +SELECT a, b FROM collate_test2 ORDER BY b; + a | b +---+- + 1 | abc + 4 | ABC

Re: [HACKERS] autonomous transactions

2016-08-30 Thread Jaime Casanova
On 30 August 2016 at 23:10, Joel Jacobson wrote: > > There should be a way to within the session and/or txn permanently > block autonomous transactions. > This will defeat one of the use cases of autonomous transactions: auditing > > Coding conventions, rules and discipline

[HACKERS] identity columns

2016-08-30 Thread Peter Eisentraut
Here is another attempt to implement identity columns. This is a standard-conforming variant of PostgreSQL's serial columns. It also fixes a few usability issues that serial columns have: - need to set permissions on sequence in addition to table (*) - CREATE TABLE / LIKE copies default but

Re: [HACKERS] Logical decoding restart problems

2016-08-30 Thread Craig Ringer
On 25 Aug. 2016 20:03, "Stas Kelvich" wrote: > > > On 20 Aug 2016, at 15:59, Craig Ringer wrote: > > > > I'll wait for a test case or some more detail. > > Thanks for clarification about how restart_lsn is working. > > Digging slightly deeper into

[HACKERS] sequences and pg_upgrade

2016-08-30 Thread Peter Eisentraut
I was toying with a couple of ideas that would involve changing the storage of sequences. (Say, for the sake of discussion, removing the problematic/useless sequence_name field.) This would cause problems for pg_upgrade, because pg_upgrade copies the "heap" storage of sequences like it does for

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Tom Lane
Aleksander Alekseev writes: > I suggest to keep ShmemAlloc as is for backward compatibility and > introduce a new procedure ShmemAllocSafe. I think that's about the worst of all possible worlds, as it guarantees having to touch most call sites. If there were more than

Re: [HACKERS] RLS related docs

2016-08-30 Thread Dean Rasheed
On 28 August 2016 at 21:23, Joe Conway wrote: > Apologies for the delay, but new patch attached. Assuming no more > comments, will commit this, backpatched to 9.5, in a day or two. > Looking at this again, I think there is something fishy about these dump/restore flags. If

Re: [HACKERS] [WIP] Patches to enable extraction state of query execution from external session

2016-08-30 Thread Maksim Milyutin
On Mon, Aug 29, 2016 at 5:22 PM, maksim > wrote: Hi, hackers! Now I complete extension that provides facility to see the current state of query execution working on external session in form of EXPLAIN ANALYZE output.

Re: [HACKERS] standalone backend PANICs during recovery

2016-08-30 Thread Tom Lane
Michael Paquier writes: > On Wed, Aug 24, 2016 at 5:07 PM, Bernd Helmle wrote: >> That said, i'm okay if --single is not intended to bring up a hot standby. >> There are many other ways to debug such problems. > This patch is still on the CF app:

Re: [HACKERS] standalone backend PANICs during recovery

2016-08-30 Thread Michael Paquier
On Tue, Aug 30, 2016 at 10:24 PM, Tom Lane wrote: > Michael Paquier writes: >> On Tue, Aug 30, 2016 at 9:48 PM, Tom Lane wrote: >>> Hm, StartupXLOG seems like a pretty random place to check that, especially >>> since doing it

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-08-30 Thread Tom Lane
Heikki Linnakangas writes: > While profiling some queries and looking at executor overhead, I > realized that we're not making much use of TupleTableSlot's ability to > hold a buffer pin. In a SeqScan, the buffer is held pinned by the > underlying heap-scan anyway. Same with

Re: [HACKERS] old_snapshot_threshold documentation

2016-08-30 Thread Kevin Grittner
On Fri, Aug 26, 2016 at 1:34 PM, Peter Eisentraut wrote: > I doubt the documentation for old_snapshot_threshold is going to be > understood by many ordinary users. What is a "snapshot", first of all? > Why would a snapshot be old? Why is that a problem? What

Re: [HACKERS] Aggregate Push Down - Performing aggregation on foreign server

2016-08-30 Thread Pavel Stehule
Hi 2016-08-30 15:02 GMT+02:00 Jeevan Chalke : > Hi all, > > Attached is the patch which adds support to push down aggregation and > grouping > to the foreign server for postgres_fdw. Performing aggregation on foreign > server results into fetching fewer rows from

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Aleksander Alekseev
> > I think what we ought to do is make ShmemAlloc act like palloc > > (ie throw error not return NULL), and remove the duplicated error > > checks. For the one caller that that would be bad for, we could > > invent something like ShmemAllocNoError, or ShmemAllocExtended with > > a no_error flag,

Re: [HACKERS] Postgres abort found in 9.3.11

2016-08-30 Thread Tom Lane
"K S, Sandhya (Nokia - IN/Bangalore)" writes: > During the server restart, we are getting postgres crash with sigabrt. No > other operation being performed. > Attached the backtrace. What shows up in the postmaster log? > The occurrence is occasional. The issue is seen

Re: [HACKERS] 9.5.4: Segfault (signal 11) while running ALTER TABLE

2016-08-30 Thread Tom Lane
Devrim =?ISO-8859-1?Q?G=FCnd=FCz?= writes: > They wanted to change id column from uuid to int, so created this func first: > CREATE FUNCTION foofunc_id_uuidtoint(chartoconvert uuid) RETURNS integer > LANGUAGE sql IMMUTABLE STRICT > AS $_$ > SELECT newid FROM foo1 WHERE

Re: [HACKERS] sequences and pg_upgrade

2016-08-30 Thread Tom Lane
Peter Eisentraut writes: > I was toying with a couple of ideas that would involve changing the > storage of sequences. (Say, for the sake of discussion, removing the > problematic/useless sequence_name field.) This would cause problems for > pg_upgrade, because

Re: [HACKERS] standalone backend PANICs during recovery

2016-08-30 Thread Michael Paquier
On Tue, Aug 30, 2016 at 9:48 PM, Tom Lane wrote: > Michael Paquier writes: >> On Wed, Aug 24, 2016 at 5:07 PM, Bernd Helmle wrote: >>> That said, i'm okay if --single is not intended to bring up a hot standby. >>> There are

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Tom Lane
Michael Paquier writes: >> I've just realized that there is also malloc-compatible ShmemAlloc(). >> Apparently it's return value sometimes is not properly checked too. See >> attachment. > The funny part here is that ProcGlobal->allProcs is actually handled, > but not

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Tom Lane
Michael Paquier writes: > On Tue, Aug 30, 2016 at 10:18 PM, Tom Lane wrote: >> I think what we ought to do is make ShmemAlloc act like palloc >> (ie throw error not return NULL), and remove the duplicated error >> checks. > The only reason why I

Re: [HACKERS] Comment on GatherPath.single_copy

2016-08-30 Thread Tom Lane
Kyotaro HORIGUCHI writes: > - boolsingle_copy;/* path must not be executed >1x */ > + boolsingle_copy;/* path must not span on multiple > processes */ I agree that the existing comment sucks, but this isn't a lot better

Re: [HACKERS] standalone backend PANICs during recovery

2016-08-30 Thread Tom Lane
Michael Paquier writes: > On Tue, Aug 30, 2016 at 9:48 PM, Tom Lane wrote: >> Hm, StartupXLOG seems like a pretty random place to check that, especially >> since doing it there requires an extra stat() call. Why didn't you just >> make

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Michael Paquier
On Tue, Aug 30, 2016 at 10:18 PM, Tom Lane wrote: > I think what we ought to do is make ShmemAlloc act like palloc > (ie throw error not return NULL), and remove the duplicated error > checks. For the one caller that that would be bad for, we could > invent something like

Re: [HACKERS] [WIP] Patches to enable extraction state of query execution from external session

2016-08-30 Thread Maksim Milyutin
Hi, On 2016-08-29 18:22:56 +0300, maksim wrote: Now I complete extension that provides facility to see the current state of query execution working on external session in form of EXPLAIN ANALYZE output. This extension works on 9.5 version, for 9.6 and later it doesn't support detailed

Re: [HACKERS] pg_dump with tables created in schemas created by extensions

2016-08-30 Thread Martín Marqués
2016-08-30 2:02 GMT-03:00 Michael Paquier : > On Tue, Aug 30, 2016 at 5:43 AM, Martín Marqués > wrote: >> This is v4 of the patch, which is actually a cleaner version from the >> v2 one Michael sent. >> >> I stripped off the external index

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Michael Paquier
On Tue, Aug 30, 2016 at 5:08 PM, Aleksander Alekseev wrote: >> > The funny part here is that ProcGlobal->allProcs is actually handled, >> > but not the two others. Well yes, you are right, we really need to >> > fail on FATAL for all of them if ShmemAlloc returns NULL

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Aleksander Alekseev
> > The funny part here is that ProcGlobal->allProcs is actually handled, > > but not the two others. Well yes, you are right, we really need to > > fail on FATAL for all of them if ShmemAlloc returns NULL as they > > involve the shmem initialization at postmaster startup. > > And with an actual

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-30 Thread Aleksander Alekseev
> >> And with an actual patch things are better. > > > > Currently I can't think of any further improvements. I even would dare > > to say that patch is Ready for Committer. > > Thanks for the fruitful input by the way! You spotted many things. Thank _you_ for paying attention for such issues in

[HACKERS] 9.5.4: Segfault (signal 11) while running ALTER TABLE

2016-08-30 Thread Devrim Gündüz
Hi, I received an email offlist about a crash that a non-customer experienced recently. I asked them to send a few details about the crash, so here it is. Please note that I don't have access to their systems, but they will be fast enough to provide data if needed. (They asked me to mask some of

Re: [HACKERS] Patch: Implement failover on libpq connect level.

2016-08-30 Thread Mithun Cy
On Fri, Aug 26, 2016 at 10:10 AM, Mithun Cy wrote: > > >rpath,'/home/mithun/edbsrc/patch6bin/lib',--enable-new-dtags -lecpg > -lpgtypes -lpq -lpgcommon -lpgport -lz -lrt -lcrypt -ldl -lm -o test1 > >../../../../../src/interfaces/libpq/libpq.so: undefined reference

[HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-08-30 Thread Heikki Linnakangas
While profiling some queries and looking at executor overhead, I realized that we're not making much use of TupleTableSlot's ability to hold a buffer pin. In a SeqScan, the buffer is held pinned by the underlying heap-scan anyway. Same with an IndexScan, and the SampleScan. The only thing that

[HACKERS] Postgres abort found in 9.3.11

2016-08-30 Thread K S, Sandhya (Nokia - IN/Bangalore)
Hello, During the server restart, we are getting postgres crash with sigabrt. No other operation being performed. Attached the backtrace. The occurrence is occasional. The issue is seen once in 30~50 times. Recently we had performed postgres upgrade from 9.3.9 to 9.3.11. The issue is not seen

Re: [HACKERS] GIN logging GIN_SEGMENT_UNMODIFIED actions?

2016-08-30 Thread Fujii Masao
On Tue, Aug 30, 2016 at 3:13 PM, Fujii Masao wrote: > On Tue, Aug 30, 2016 at 3:39 AM, Tom Lane wrote: >> Fujii Masao writes: >>> ISTM that the cause of this issue is that gin_desc() uses XLogRecGetData() >>> to >>> extract