Re: Possible bug in logical replication.

2018-05-24 Thread Arseny Sher
Michael Paquier writes: > Maybe I am being too naive, but wouldn't it be just enough to update the > confirmed flush LSN to ctx->reader->ReadRecPtr? This way, the slot > advances up to the beginning of the last record where user wants to > advance, and not the beginning of the next record: Sam

Re: Keeping temporary tables in shared buffers

2018-05-24 Thread Andres Freund
On 2018-05-25 09:40:10 +0300, Heikki Linnakangas wrote: > On 25/05/18 09:25, Asim Praveen wrote: > > My parochial vision of the overhead is restricted to 4 * NBuffers of > > additional shared memory, as 4 bytes are being added to BufferTag. May I > > please get some enlightenment? > > Any extra f

Re: Possible bug in logical replication.

2018-05-24 Thread Arseny Sher
Hello, Kyotaro HORIGUCHI writes: > restart_lsn stays at the beginning of a transaction until the > transaction ends so just using restart_lsn allows repeated > decoding of a transaction, in short, rewinding occurs. The > function works only for inactive slot so the current code works > fine on t

Re: Redesigning the executor (async, JIT, memory efficiency)

2018-05-24 Thread Andres Freund
Hi, On 2018-05-25 09:26:47 +0300, Heikki Linnakangas wrote: > Cool stuff! Thanks! > On 25/05/18 06:35, Andres Freund wrote: > > For example, this converts this converts TPCH's Q01: > > > > tpch_1[26142][1]=# SET client_min_messages ='log'; > > tpch_1[26142][1]=# SELECT > > l_returnflag, > >

Re: Keeping temporary tables in shared buffers

2018-05-24 Thread Heikki Linnakangas
On 25/05/18 09:25, Asim Praveen wrote: On Thu, May 24, 2018 at 8:19 PM, Tom Lane wrote: So then you have to think about how to transition smoothly between "rel is in local buffers" and "rel is in shared buffers", bearing in mind that ever having the same page in two different buffers would be

Re: Redesigning the executor (async, JIT, memory efficiency)

2018-05-24 Thread Heikki Linnakangas
Cool stuff! On 25/05/18 06:35, Andres Freund wrote: For example, this converts this converts TPCH's Q01: tpch_1[26142][1]=# SET client_min_messages ='log'; tpch_1[26142][1]=# SELECT l_returnflag, l_linestatus, sum(l_quantity) AS sum_qty, sum(l_extendedprice) AS s

Re: Keeping temporary tables in shared buffers

2018-05-24 Thread Asim Praveen
On Thu, May 24, 2018 at 8:19 PM, Tom Lane wrote: > > So then you have to think about how to transition smoothly between "rel > is in local buffers" and "rel is in shared buffers", bearing in mind that > ever having the same page in two different buffers would be disastrous. Local buffers would no

Re: Possible bug in logical replication.

2018-05-24 Thread Michael Paquier
On Thu, May 24, 2018 at 10:14:01AM +0900, Kyotaro HORIGUCHI wrote: > At Wed, 23 May 2018 15:56:22 +0900, Masahiko Sawada > wrote in >> Another possible way might be to make XLogFindNextRecord valid in >> backend code and move startlsn to the first valid record with an lsn >> >= startlsn by using

Re: [GSoC] github repo and initial work

2018-05-24 Thread Aleksandr Parfenov
On Thu, 24 May 2018 18:25:28 -0700 Charles Cui wrote: > The second is to provide thrift type just like json or jsonb. When you > create a table, postgres knows ::thrift keywords. > I think method one should be easier to implement because it only > limits to this plugin. Method two needs modify pos

Re: XLogWrite uses palloc within a critical section

2018-05-24 Thread Heikki Linnakangas
On 25/05/18 07:45, Kyotaro HORIGUCHI wrote: Hello. I happened to see the following in XLogWrite. ereport(PANIC, (errcode_for_file_access(), errmsg("could not seek in log file %s to offset %u: %m", XLogFileNameP(ThisTimeLineID, openLogSegNo), startoffset)));

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2018-05-24 Thread Thomas Munro
On Thu, Apr 26, 2018 at 8:33 AM, Andres Freund wrote: > On 2018-04-25 14:41:44 -0400, Robert Haas wrote: >> On Mon, Apr 16, 2018 at 2:13 AM, Andrew Gierth >> wrote: >> > The code that detects sequential behavior can not distinguish between >> > pread() and lseek+read, it looks only at the actual

In what range of the code can we read debug_query_string?

2018-05-24 Thread Kyotaro HORIGUCHI
Hello. While looking into *some* tool that heavily reliant on debug_query_string, I found that the tool can miss it. However I believed that it is always set when post_parse_analyze_hook is called, it is not while processing DESCRIBE message of extended protocol. I believe that the assumption is

pg_replication_slot_advance to return NULL instead of 0/0 if slot not advanced

2018-05-24 Thread Michael Paquier
Hi all, When attempting to use multiple times pg_replication_slot_advance on a slot, then the caller gets back directly InvalidXLogRecPtr as result, for example: =# select * from pg_replication_slot_advance('popo', 'FF/0'); slot_name | end_lsn ---+--- popo | 0/60021E0 (1 ro

XLogWrite uses palloc within a critical section

2018-05-24 Thread Kyotaro HORIGUCHI
Hello. I happened to see the following in XLogWrite. > ereport(PANIC, > (errcode_for_file_access(), > errmsg("could not seek in log file %s to offset %u: %m", > XLogFileNameP(ThisTimeLineID, openLogSegNo), > startoffset))); where XLogFileNameP calls palloc within, a

Re: Fix some error handling for read() and errno

2018-05-24 Thread Kyotaro HORIGUCHI
At Wed, 23 May 2018 09:00:40 +0900, Michael Paquier wrote in <2018052340.ga3...@paquier.xyz> > On Tue, May 22, 2018 at 04:51:00PM +0900, Kyotaro HORIGUCHI wrote: > > I see the same issue in snapbuild.c(4 places). > > > > | readBytes = read(fd, &ondisk, SnapBuildOnDiskConstantSize); > > | pgs

Redesigning the executor (async, JIT, memory efficiency)

2018-05-24 Thread Andres Freund
Hi, The current executor structure limits us in a number of ways that are becoming problematic. I'll outline in this email how I think we should rework it. Including a prototype for the first, most drastic, steps. The primary problems with the current design that I want to address are: 1) Exec

Re: Keeping temporary tables in shared buffers

2018-05-24 Thread Tom Lane
Asim Praveen writes: > We are evaluating the use of shared buffers for temporary tables. The > advantage being queries involving temporary tables can make use of parallel > workers. Hm... > Challenges: > 1. We lose the performance benefit of local buffers. Yeah. This would be an absolute dea

RE: [Bug Fix]ECPG: cancellation of significant digits on ECPG

2018-05-24 Thread Higuchi, Daisuke
Hi, I have a question which is related to numeric data type bugs on ECPG sqlda. Currently, I think following functions have a problem when using numeric data type on sqlda. Please see the details of problem from the mail I sent before. [src/ecpg/ecpglib/sqlda.c] - ecpg_set_native_sqlda() - sq

Re: Portability concerns over pq_sendbyte?

2018-05-24 Thread Michael Paquier
On Thu, May 24, 2018 at 06:13:23PM +0100, Andrew Gierth wrote: > In PG11, pq_sendbyte got changed from taking an int parameter to taking > an int8. From 1de09ad8. > While that seems to work in general, it does mean that there are now > several places in the code that do the equivalent of: > >

Re: Add PostgreSQL 11 to feature matrix page?

2018-05-24 Thread Jonathan S. Katz
> On May 24, 2018, at 9:38 PM, Tatsuo Ishii wrote: > > Now that PostgreSQL 11 beta is out, I wonder if it's worth to add > PostgreSQL 11 to the feature matrix page. > > https://www.postgresql.org/about/featurematrix/ > > Adding PostgreSQL 11 to the page will give users at-a-glance-dfference >

Add PostgreSQL 11 to feature matrix page?

2018-05-24 Thread Tatsuo Ishii
Now that PostgreSQL 11 beta is out, I wonder if it's worth to add PostgreSQL 11 to the feature matrix page. https://www.postgresql.org/about/featurematrix/ Adding PostgreSQL 11 to the page will give users at-a-glance-dfference between 11 and previous releases, which could bring clearner image of

Re: [GSoC] github repo and initial work

2018-05-24 Thread Charles Cui
Hi guys, As mentioned in the last email that the current interface is based on bytea. To make users easily insert any thrift data structure and show them, we may want to have user friendly (user readable) format for this plugin. I can think about 2 ways to do this. One is to provide to_thrift(j

Re: A Japanese-unfriendy error message contruction

2018-05-24 Thread Kyotaro HORIGUCHI
At Thu, 24 May 2018 14:20:21 -0400, Tom Lane wrote in <24988.1527186...@sss.pgh.pa.us> > Kyotaro HORIGUCHI writes: > > Hello. Here is the patch set. > > Thanks! I pushed all this plus fixes for the OCLASS_PUBLICATION_REL > code. Thanks. > The only non-cosmetic change I made was that I didn't

Keeping temporary tables in shared buffers

2018-05-24 Thread Asim Praveen
Hello We are evaluating the use of shared buffers for temporary tables. The advantage being queries involving temporary tables can make use of parallel workers. Challenges: 1. We lose the performance benefit of local buffers. 2. Additional complexity in shared buffer manager - BufferTag needs to

add default parallel query to v10 release notes? (Re: [PERFORM] performance drop after upgrade (9.6 > 10))

2018-05-24 Thread Justin Pryzby
Moving to -hackers; On Sun, Jan 28, 2018 at 06:53:10PM -0500, Bruce Momjian wrote: > On Thu, Oct 26, 2017 at 02:45:15PM -0500, Justin Pryzby wrote: > > Is it because max_parallel_workers_per_gather now defaults to 2 ? > > > > BTW, I would tentatively expect a change in default to be documented in

Re: ParseDateTime in src/backend/utils/adt/datetime.c

2018-05-24 Thread Bruce Momjian
On Thu, May 3, 2018 at 12:21:45PM +0100, Vladimir Svedov wrote: > Hi guys, > https://stackoverflow.com/questions/50153122/ > how-come-this-postgres-query-is-working > Above guys are surprised by "ignore other punctuation but use as delimiter" > part. > Do you think this should be mentioned in docs

Re: contrib/sepgsql fails on Fedora 28

2018-05-24 Thread Mike Palmiotto
On 05/24/2018 03:30 PM, Tom Lane wrote: > I tried to run the regression test for sepgsql on F28 (so I could > fix the now-obsolete expected-file therein). It fails at this > preparatory step: > > $ sudo semodule -u sepgsql-regtest.pp > The --upgrade option is deprecated. Use --install instead. >

Re: Compiler warnings with --enable-dtrace

2018-05-24 Thread Peter Geoghegan
On Wed, May 23, 2018 at 5:17 AM, Christoph Berg wrote: > I'm not aware of any guidelines - I guess this stuff is still too new > to make general statements about it. Which other software offers > custom probes? As the LWN article mentions, user space probes are available for a number of other wid

Re: [PATCH v15] GSSAPI encryption support

2018-05-24 Thread Robbie Harwood
Thomas Munro writes: > On Thu, May 24, 2018 at 8:00 AM, Robbie Harwood wrote: > >> Zombie patch is back from the dead. > > Hi Robbie, > > Robots[1] vs zombies: > > + $postgres->RemoveFile('src/backennd/libpq/be-gssapi-common.c'); > > Typo, breaks on Windows. > > runtime.sgml:2032: parser error :

Re: PostgreSQL 11 beta1 : regressions failed on OpenBSD with JIT

2018-05-24 Thread Andres Freund
Hi, On 2018-05-24 21:47:27 +0200, Pierre-Emmanuel André wrote: > Hello, > > I'm the maintainer of PostgreSQL on OpenBSD. > Today I tried the beta1 of PostgreSQL 11 on OpenBSD -current @amd64. > > Without JIT, everything works fine. > > When i enable JIT, regress tests fail with this error : Th

PostgreSQL 11 beta1 : regressions failed on OpenBSD with JIT

2018-05-24 Thread Pierre-Emmanuel André
Hello, I'm the maintainer of PostgreSQL on OpenBSD. Today I tried the beta1 of PostgreSQL 11 on OpenBSD -current @amd64. Without JIT, everything works fine. When i enable JIT, regress tests fail with this error : indexing ... FAILED (test process exited with exit code 2) l

Re: Unexpected casts while using date_trunc()

2018-05-24 Thread Tom Lane
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> Yeah. There are two relevant variants of date_trunc(): > [...] > Tom> So we probably ought to change the docs here. > There's also the option of adding an explicit function > date_trunc(text,date) returns date, which is a workaround th

contrib/sepgsql fails on Fedora 28

2018-05-24 Thread Tom Lane
I tried to run the regression test for sepgsql on F28 (so I could fix the now-obsolete expected-file therein). It fails at this preparatory step: $ sudo semodule -u sepgsql-regtest.pp The --upgrade option is deprecated. Use --install instead. neverallow check failed at /var/lib/selinux/targeted/

Re: Unexpected casts while using date_trunc()

2018-05-24 Thread Andrew Gierth
> "Tom" == Tom Lane writes: Tom> Yeah. There are two relevant variants of date_trunc(): [...] Tom> So we probably ought to change the docs here. There's also the option of adding an explicit function date_trunc(text,date) returns date, which is a workaround that I (and probably quite a f

Re: Error on vacuum: xmin before relfrozenxid

2018-05-24 Thread Paolo Crosato
2018-05-24 8:30 GMT+02:00 Andrey Borodin : > Hi! > > > 24 мая 2018 г., в 0:55, Paolo Crosato > написал(а): > > > > 1) VACUUM FULL was issued after the first time the error occurred, and a > couple of times later. CLUSTER was never run. > > 2) Several failovers tests were perfomed before the clust

Re: Unexpected casts while using date_trunc()

2018-05-24 Thread Tom Lane
Chris Bandy writes: > The documentation explains that DATE is first cast to TIMESTAMP. (As I > understand it, this is an immutable cast; sounds find and appropriate.) > https://www.postgresql.org/docs/current/static/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC > But in my testing, the date va

Re: Should we add GUCs to allow partition pruning to be disabled?

2018-05-24 Thread Bruce Momjian
On Thu, May 24, 2018 at 02:23:17PM -0400, Alvaro Herrera wrote: > On 2018-May-24, Bruce Momjian wrote: > > > On Wed, May 2, 2018 at 07:49:42PM -0700, David G. Johnston wrote: > > > > I toss my +1 to removing it altogether. > > > > +1 We are terrible at removing old GUCs and having it around me

Re: Should we add GUCs to allow partition pruning to be disabled?

2018-05-24 Thread Alvaro Herrera
On 2018-May-24, Bruce Momjian wrote: > On Wed, May 2, 2018 at 07:49:42PM -0700, David G. Johnston wrote: > > I toss my +1 to removing it altogether. > > +1 We are terrible at removing old GUCs and having it around means > everyone has to decide if they need to change it, so having it is not a

Re: A Japanese-unfriendy error message contruction

2018-05-24 Thread Tom Lane
Kyotaro HORIGUCHI writes: > Hello. Here is the patch set. Thanks! I pushed all this plus fixes for the OCLASS_PUBLICATION_REL code. The only non-cosmetic change I made was that I didn't much like what you'd done with the OCLASS_DEFACL code: it seemed quite confusing to be renaming buffers like

Portability concerns over pq_sendbyte?

2018-05-24 Thread Andrew Gierth
In PG11, pq_sendbyte got changed from taking an int parameter to taking an int8. While that seems to work in general, it does mean that there are now several places in the code that do the equivalent of: unsigned char x = 128; pq_sendbyte(&buf, x); which I believe is not well-defined sin

Re: Should we add GUCs to allow partition pruning to be disabled?

2018-05-24 Thread Bruce Momjian
On Wed, May 2, 2018 at 07:49:42PM -0700, David G. Johnston wrote: > On Wednesday, May 2, 2018, Alvaro Herrera wrote: > > Robert Haas wrote: > > On Wed, May 2, 2018 at 9:28 AM, Alvaro Herrera > > wrote: > > > I admit I am more concerned about the possibility of bugs than I am >

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-24 Thread Bruce Momjian
On Wed, May 2, 2018 at 12:27:19PM -0400, Robert Haas wrote: > I've seen customer have very good luck going higher if it lets all the > data fit in shared_buffers, or at least all the data that is accessed > with any frequency. I think it's useful to imagine a series of > concentric working sets -

Re: computing z-scores

2018-05-24 Thread David G. Johnston
On Thursday, May 24, 2018, David G. Johnston wrote: > On Thu, May 24, 2018 at 8:15 AM, Martin Mueller < > martinmuel...@northwestern.edu> wrote: > >> You construct a z-score for a set of values by subtracting the average >> from the value and dividing the result by the standard deviation. I know

Re: computing z-scores

2018-05-24 Thread David G. Johnston
On Thu, May 24, 2018 at 8:15 AM, Martin Mueller < martinmuel...@northwestern.edu> wrote: > You construct a z-score for a set of values by subtracting the average > from the value and dividing the result by the standard deviation. I know > how to do this in a two-step procedure. First, I compute th

computing z-scores

2018-05-24 Thread Martin Mueller
You construct a z-score for a set of values by subtracting the average from the value and dividing the result by the standard deviation. I know how to do this in a two-step procedure. First, I compute the average and standard deviation. In a second run I use the formula and apply it to each valu

Unexpected casts while using date_trunc()

2018-05-24 Thread Chris Bandy
I have an application generating the following query on a DATE column in PostgreSQL 10.1: > SELECT TO_CHAR(DATE_TRUNC('month', jobs.active_until), '-MM') > FROM jobs > GROUP BY DATE_TRUNC('month', jobs.active_until) > LIMIT 500 I wanted to support it with an expression index, but was surpri

Re: PG11 jit failing on ppc64el

2018-05-24 Thread Tom Lane
Thomas Munro writes: > On Thu, May 24, 2018 at 3:48 PM, Tom Lane wrote: >> For entertainment's sake, I tried building --with-llvm on FreeBSD 12 >> arm64 (hey, gotta do something with this raspberry pi toy I got). > Neat. Quite tempted to get one! Fair warning: the newest "3B+" model contains n

Locks on unlogged tables are locked?!

2018-05-24 Thread Laurenz Albe
While looking at this: https://stackoverflow.com/q/50413623/6464308 I realized that "LOCK TABLE " puts a Standby/LOCK into the WAL stream, which causes a log flush at COMMIT time. That hurts performance, and I doubt that it is necessary. At any rate, DROP TABLE on an unlogged table logs nothing.

Re: Function Overloading

2018-05-24 Thread Tom Lane
ramsiddu007 writes: > I hope you are all doing good. Actually I got one problem. I > have 2 functions like below. Those are having 11 and 14 parameters with > commonly last parameter is variadic array type. > func_get_item_list_phy_param(smallint, numeric, numeric, character varying,

Re: A Japanese-unfriendy error message contruction

2018-05-24 Thread Kyotaro HORIGUCHI
Hello. Here is the patch set. At Wed, 23 May 2018 11:20:24 -0400, Tom Lane wrote in <4979.1527088...@sss.pgh.pa.us> > After thinking about this some more, I'd like to propose that we change > the English output to be "column COLNAME of ", using code > similar to what you suggested for O_POLICY e

Function Overloading

2018-05-24 Thread ramsiddu007
Hi Professionals, I hope you are all doing good. Actually I got one problem. I have 2 functions like below. Those are having 11 and 14 parameters with commonly last parameter is variadic array type. func_get_item_list_phy_param(smallint, numeric, numeric, character varying, character v

Re: Shared PostgreSQL libraries and ABI versioning

2018-05-24 Thread Pavel Raiskup
Hi Christoph, On Thursday, May 24, 2018 12:08:16 PM CEST Christoph Berg wrote: > Re: Pavel Raiskup 2018-05-24 <101829257.nn0xsvv...@nb.usersys.redhat.com> > > > Interesting, thanks. How this is implemented? What you mean by "newer > > > library" -- new soname, or just a newer package build (with

Proposal: Partitioning Advisor for PostgreSQL

2018-05-24 Thread Yuzuko Hosoya
Hello, I'm Yuzuko Hosoya. This is my first PostgreSQL project participation. I have been developing partitioning advisor prototype with Julien Rouhaud. It will be a new feature of HypoPG[1], which is a PostgreSQL extension, and will help partitioning design tuning. Currently, HypoPG only support

Re: Shared PostgreSQL libraries and symbol versioning

2018-05-24 Thread Christoph Berg
Re: Pavel Raiskup 2018-05-24 <101829257.nn0xsvv...@nb.usersys.redhat.com> > > Interesting, thanks. How this is implemented? What you mean by "newer > > library" -- new soname, or just a newer package build (without any other > > change) e.g.? > > I've already done some observation how libraries

Re: Shared PostgreSQL libraries and symbol versioning

2018-05-24 Thread Pavel Raiskup
On Wednesday, May 23, 2018 2:23:24 PM CEST Pavel Raiskup wrote: > Thanks Stephen, > > On Wednesday, May 23, 2018 1:13:08 PM CEST Stephen Frost wrote: > > Greetings, > > > > * Pavel Raiskup (prais...@redhat.com) wrote: > > > On Thursday, May 10, 2018 12:41:40 PM CEST Pavel Raiskup wrote: > > > > O

Re: PANIC during crash recovery of a recently promoted standby

2018-05-24 Thread Michael Paquier
On Mon, May 14, 2018 at 01:14:22PM +0530, Pavan Deolasee wrote: > Looks like I didn't understand Alvaro's comment when he mentioned it to me > off-list. But I now see what Michael and Alvaro mean and that indeed seems > like a problem. I was thinking that the test for (ControlFile->state == > DB_IN

Re: PG11 jit failing on ppc64el

2018-05-24 Thread Thomas Munro
On Thu, May 24, 2018 at 3:48 PM, Tom Lane wrote: > Thomas Munro writes: >> BTW It is working on arm64 too, starting with LLVM 6. 5 crashed the >> same way as it does on ppc. See build farm member eelpout which is >> running Debian. > > For entertainment's sake, I tried building --with-llvm on F