Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-06-04 Thread Andy Fan
On Fri, Jun 5, 2020 at 10:57 AM David Rowley wrote: > On Fri, 5 Jun 2020 at 14:36, Andy Fan wrote: > > On Mon, May 25, 2020 at 2:34 AM David Rowley > wrote: > >> > >> On Sun, 24 May 2020 at 04:14, Dmitry Dolgov <9erthali...@gmail.com> > wrote: > >> > > >> > > On Fri, May 22, 2020 at 08:40:17AM

Re: A wrong index choose issue because of inaccurate statistics

2020-06-04 Thread Andy Fan
> > > > Why will the (a, c) be choose? If planner think a = x has only 1 row .. > I just did more research and found above statement is not accurate, the root cause of this situation is because IndexSelectivity = 0. Even through I don't think we can fix anything here since IndexSelectivity is

Re: Atomic operations within spinlocks

2020-06-04 Thread Andres Freund
Hi, On 2020-06-04 15:13:29 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-06-04 14:50:40 -0400, Tom Lane wrote: > >> 2. The computed completePasses value would go backwards. I bet > >> that wouldn't matter too much either, or at least we could teach > >> BgBufferSync to cope. (I

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-06-04 Thread David Rowley
On Fri, 5 Jun 2020 at 14:36, Andy Fan wrote: > On Mon, May 25, 2020 at 2:34 AM David Rowley wrote: >> >> On Sun, 24 May 2020 at 04:14, Dmitry Dolgov <9erthali...@gmail.com> wrote: >> > >> > > On Fri, May 22, 2020 at 08:40:17AM +1200, David Rowley wrote: >> > > I imagine we'll set some required

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-06-04 Thread Andy Fan
On Mon, May 25, 2020 at 2:34 AM David Rowley wrote: > On Sun, 24 May 2020 at 04:14, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > > > On Fri, May 22, 2020 at 08:40:17AM +1200, David Rowley wrote: > > > I imagine we'll set some required UniqueKeys during > > > standard_qp_callback() > > > >

Re: Atomic operations within spinlocks

2020-06-04 Thread Andres Freund
Hi, On 2020-06-04 15:07:34 -0400, Tom Lane wrote: > Andres Freund writes: > > I'd still like to know which problem we're actually trying to solve > > here. I don't understand the "error" issues you mentioned upthread. > > If you error out of getting the inner spinlock, the outer spinlock > is

Re: REINDEX CONCURRENTLY and indisreplident

2020-06-04 Thread Michael Paquier
On Thu, Jun 04, 2020 at 11:23:36AM +0900, Michael Paquier wrote: > On Wed, Jun 03, 2020 at 12:40:38PM -0300, Euler Taveira wrote: > > On Wed, 3 Jun 2020 at 03:54, Michael Paquier wrote: > >> I have bumped into $subject, causing a replica identity index to > >> be considered as dropped if running

Re: significant slowdown of HashAggregate between 9.6 and 10

2020-06-04 Thread Andres Freund
Hi, On 2020-06-04 18:22:03 -0700, Jeff Davis wrote: > On Thu, 2020-06-04 at 11:41 -0700, Andres Freund wrote: > > +/* minimum number of initial hash table buckets */ > > +#define HASHAGG_MIN_BUCKETS 256 > > > > > > I don't really see much explanation for that part in the commit, > > perhaps > >

Re: v13: Performance regression related to FORTIFY_SOURCE

2020-06-04 Thread Tom Lane
Jeff Davis writes: > On Thu, 2020-06-04 at 16:35 -0400, Alvaro Herrera wrote: >> If it is something worth worrying about, let's discuss what's a good >> fix for it. > I did post a fix for it, but it's not a very clean fix. I'm slightly > inclined to proceed with that fix, but I was hoping

Re: significant slowdown of HashAggregate between 9.6 and 10

2020-06-04 Thread Jeff Davis
On Thu, 2020-06-04 at 11:41 -0700, Andres Freund wrote: > +/* minimum number of initial hash table buckets */ > +#define HASHAGG_MIN_BUCKETS 256 > > > I don't really see much explanation for that part in the commit, > perhaps > Jeff can chime in? I did this in response to a review comment

Re: v13: Performance regression related to FORTIFY_SOURCE

2020-06-04 Thread Jeff Davis
On Thu, 2020-06-04 at 16:35 -0400, Alvaro Herrera wrote: > If it is something worth worrying about, let's discuss what's a good > fix for it. I did post a fix for it, but it's not a very clean fix. I'm slightly inclined to proceed with that fix, but I was hoping someone else would have a better

Re: v13: Performance regression related to FORTIFY_SOURCE

2020-06-04 Thread Jeff Davis
On Sun, 2020-04-19 at 16:19 -0700, Peter Geoghegan wrote: > Is it possible that the issue has something to do with what the > compiler knows about the alignment of the tapes back when they were a > flexible array vs. now, where it's a separate allocation? Perhaps I'm > over reaching, but it occurs

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-06-04 Thread David Rowley
On Mon, 25 May 2020 at 19:14, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Mon, May 25, 2020 at 06:34:30AM +1200, David Rowley wrote: > > The difference will be that you'd be setting some distinct_uniquekeys > > in standard_qp_callback() to explicitly request that some skip scan > > paths

Re: SIGSEGV from START_REPLICATION 0/XXXXXXX in XLogSendPhysical () at walsender.c:2762

2020-06-04 Thread Alvaro Herrera
On 2020-Jun-04, Andres Freund wrote: > postgres[52656][1]=# SELECT 1; > ┌──┐ > │ ?column? │ > ├──┤ > │1 │ > └──┘ > (1 row) > > > I am very much not in love with the way that was implemented, but it's > there, and it's used as far as I know (cf tablesync.c). Ouch

Re: SIGSEGV from START_REPLICATION 0/XXXXXXX in XLogSendPhysical () at walsender.c:2762

2020-06-04 Thread Andres Freund
Hi, On 2020-06-04 16:44:53 -0400, Alvaro Herrera wrote: > A logical replication connection cannot run SQL anyway, can it? You can: andres@awork3:~/src/postgresql$ psql 'replication=database' postgres[52656][1]=# IDENTIFY_SYSTEM; ┌─┬──┬┬──┐ │

Re: what can go in root.crt ?

2020-06-04 Thread Tom Lane
Chapman Flack writes: > Sure. It seems sensible to me to start by documenting /what/ it is doing > now, and to what extent that should be called "its standard behavior" > versus "the way libpq is calling it", because even if nothing is to be > changed, there will be people who need to be able to

Re: what can go in root.crt ?

2020-06-04 Thread Chapman Flack
On 06/04/20 18:03, Tom Lane wrote: > It's possible that we could force openssl to validate cases it doesn't > accept now. Whether we *should* deviate from its standard behavior is > a fairly debatable question though. I would not be inclined to do so > unless we find that many other consumers of

Re: what can go in root.crt ?

2020-06-04 Thread Tom Lane
Chapman Flack writes: > On 06/04/20 17:31, Andrew Dunstan wrote: >> Do we actually do any of this sort of thing? I confess my impression was >> this is all handled by the openssl libraries, we just hand over the >> certs and let openssl do its thing. Am I misinformed about that? > By analogy to

Re: what can go in root.crt ?

2020-06-04 Thread Chapman Flack
On 06/04/20 17:31, Andrew Dunstan wrote: > Do we actually do any of this sort of thing? I confess my impression was > this is all handled by the openssl libraries, we just hand over the > certs and let openssl do its thing. Am I misinformed about that? I haven't delved very far into the code yet

Re: what can go in root.crt ?

2020-06-04 Thread Andrew Dunstan
On 6/3/20 7:57 PM, Chapman Flack wrote: > > In an ideal world, I think libpq would be using this algorithm: > > I'm looking at the server's certificate, s. > Is s unexpired and in the trust file? If so, SUCCEED. > > otherwise, loop: > get issuer certificate i from s (if s is

Re: repeat() function, CHECK_FOR_INTERRUPTS(), and unlikely()

2020-06-04 Thread Alvaro Herrera
On 2020-May-28, Joe Conway wrote: > I backpatched and pushed the changes to the repeat() function. Any other > opinions regarding backpatch of the unlikely() addition to > CHECK_FOR_INTERRUPTS()? We don't use unlikely() in 9.6 at all, so I would stop that backpatching at 10 anyhow. (We did

Re: repeat() function, CHECK_FOR_INTERRUPTS(), and unlikely()

2020-06-04 Thread Joe Conway
On 5/28/20 1:23 PM, Joe Conway wrote: > On 5/27/20 3:29 AM, Michael Paquier wrote: >>> I think that each of those tests should have a separate unlikely() marker, >>> since the whole point here is that we don't expect either of those tests >>> to yield true in the huge majority of

Re: SIGSEGV from START_REPLICATION 0/XXXXXXX in XLogSendPhysical () at walsender.c:2762

2020-06-04 Thread Alvaro Herrera
On 2020-Jun-04, Michael Paquier wrote: > On Wed, Jun 03, 2020 at 06:33:11PM -0700, Andres Freund wrote: > >> I don't think having a physical replication connection access catalog > >> data directly is a great idea. We already have gadgets like > >> IDENTIFY_SYSTEM for physical replication that

Re: v13: Performance regression related to FORTIFY_SOURCE

2020-06-04 Thread Alvaro Herrera
Speaking with my RMT hat on, I'm concerned that this item is not moving forward at all. ISTM we first and foremost need to decide whether this is a problem worth worrying about, or not. If it is something worth worrying about, let's discuss what's a good fix for it. -- Álvaro Herrera

Re: Atomic operations within spinlocks

2020-06-04 Thread Tom Lane
Andres Freund writes: > On 2020-06-04 14:50:40 -0400, Tom Lane wrote: >> 2. The computed completePasses value would go backwards. I bet >> that wouldn't matter too much either, or at least we could teach >> BgBufferSync to cope. (I notice the comments therein suggest that >> it is already

Re: Atomic operations within spinlocks

2020-06-04 Thread Tom Lane
Andres Freund writes: > I'd still like to know which problem we're actually trying to solve > here. I don't understand the "error" issues you mentioned upthread. If you error out of getting the inner spinlock, the outer spinlock is stuck, permanently, because there is no mechanism for spinlock

Re: Atomic operations within spinlocks

2020-06-04 Thread Andres Freund
Hi, On 2020-06-04 14:50:40 -0400, Tom Lane wrote: > Actually ... we could probably use this design with a uint32 counter > as well, on machines where the 64-bit operations would be slow. On skylake-x even a 32bit [i]div is still 26 cycles. That's more than an atomic operation 18 cycles. > 2.

Re: Atomic operations within spinlocks

2020-06-04 Thread Andres Freund
Hi, On 2020-06-04 13:57:19 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-06-03 14:19:45 -0400, Tom Lane wrote: > >> This seems to me to be very bad code. > > > I think straight out prohibiting use of atomics inside a spinlock will > > lead to more complicated and slower code, rather

Re: Atomic operations within spinlocks

2020-06-04 Thread Tom Lane
I wrote: > I think a good case could be made for ripping out what's there now > and just redefining nextVictimBuffer as a pg_atomic_uint64 that we > never reset (ie, make its comment actually true). Then ClockSweepTick > reduces to > pg_atomic_fetch_add_u64(>nextVictimBuffer, 1) % NBuffers > and

REL_13_STABLE Branch

2020-06-04 Thread Jonathan S. Katz
Hi, After conferring, the PostgreSQL 13 RMT[1] has decided that it is time to create the REL_13_STABLE branch. Tom has volunteered to create the branch this Sunday (2020-06-07). Please let us know if you have any questions. Thanks, Alvaro, Peter, Jonathan [1]

Re: significant slowdown of HashAggregate between 9.6 and 10

2020-06-04 Thread Andres Freund
Hi, On 2020-06-03 13:26:43 -0700, Andres Freund wrote: > On 2020-06-03 21:31:01 +0200, Tomas Vondra wrote: > > So there seems to be +40% between 9.6 and 10, and further +25% between > > 10 and master. However, plain hashagg, measured e.g. like this: As far as I can tell the 10->master difference

Re: Regarding TZ conversion

2020-06-04 Thread Rajin Raj
Thanks for the clarification. Is it advisable to modify the Default? Will it override when we apply a patch or upgrade the DB? What about creating a new file like below and update the postgres.conf with the new name. # New tz offset @INCLUDE Default @OVERRDIE IST 19800

Re: Removal of currtid()/currtid2() and some table AM cleanup

2020-06-04 Thread Andres Freund
Hi, On 2020-06-03 11:14:48 +0900, Michael Paquier wrote: > I would like to remove those two functions and the surrounding code > for v14, leading to some cleanup: > 6 files changed, 326 deletions(-) +1 > While on it, I have noticed that heap_get_latest_tid() is still > located within

Re: should INSERT SELECT use a BulkInsertState?

2020-06-04 Thread Andres Freund
Hi, On 2020-05-08 02:25:45 -0500, Justin Pryzby wrote: > Seems to me it should, at least conditionally. At least if there's a function > scan or a relation or .. Well, the problem is that this can cause very very significant regressions. As in 10x slower or more. The ringbuffer can cause

Re: Expand the use of check_canonical_path() for more GUCs

2020-06-04 Thread Tom Lane
Peter Eisentraut writes: > This (and some other messages in this thread) appears to assume that > canonicalize_path() turns relative paths into absolute paths, but > AFAICT, it does not do that. Ah, fair point --- I'd been assuming that we were applying canonicalize_path as cleanup for an

Re: Expand the use of check_canonical_path() for more GUCs

2020-06-04 Thread Peter Eisentraut
On 2020-06-03 20:45, Tom Lane wrote: Robert Haas writes: On Tue, Jun 2, 2020 at 5:04 AM Peter Eisentraut wrote: The archeology reveals that these calls where originally added to canonicalize the data_directory and config_file settings (7b0f060d54), but that was then moved out of guc.c to be

Re: libpq copy error handling busted

2020-06-04 Thread Tom Lane
I wrote: > * As for control-C not getting out of it: there is > if (CancelRequested) > break; > in pgbench's loop, but this does nothing in this scenario because > fe-utils/cancel.c only sets that flag when it successfully sends a > Cancel ... which it certainly

Re: Just for fun: Postgres 20?

2020-06-04 Thread Avinash Kumar
On Tue, Jun 2, 2020 at 2:45 PM Robert Haas wrote: > On Mon, Jun 1, 2020 at 3:20 PM Tom Lane wrote: > > Robert Haas writes: > > > As has already been pointed out, it could definitely happen, but we > > > could solve that by just using a longer version number, say, including > > > the month and,

Re: what can go in root.crt ?

2020-06-04 Thread Chapman Flack
On 06/04/20 11:04, Laurenz Albe wrote: > I was referring to the wish to *not* use a self-signed CA certificate, > but an intermediate certificate as the ultimate authority, based on > a distrust of the certification authority that your organization says > you should trust. Are you aware of any

Re: Possible bug on Postgres 12 (CASE THEN evaluated prematurely) - Change of behaviour compared to 11, 10, 9

2020-06-04 Thread Juan Fuentes
Thanks Tom! I was just hopping somebody could point out if this kind of issue has been reported before spending 2 days fabricating a simpler self contained example. Best, Juan > On 4 Jun 2020, at 16:26, Tom Lane wrote: > > Juan Fuentes writes: >> As you could see the query includes

Re: what can go in root.crt ?

2020-06-04 Thread Laurenz Albe
On Thu, 2020-06-04 at 08:25 -0400, Chapman Flack wrote: > > I feel bad about bending the basic idea of certificates and trust to suit > > some misbegotten bureaucratic constraints on good security. > > Can you elaborate on what, in the email message you replied to here, > represented a bending of

Re: Wrong width of UNION statement

2020-06-04 Thread Kenichiro Tanaka
Hello, Thank you for your quick response and sorry for my late reply. > (I suppose you're using UTF8 encoding...) It is right. As you said, my encoding of database is UTF8. >There's room for improvement there, but this is all bound up in the legacy >mess that we have in prepunion.c. At first,I

Re: Possible bug on Postgres 12 (CASE THEN evaluated prematurely) - Change of behaviour compared to 11, 10, 9

2020-06-04 Thread Tom Lane
Juan Fuentes writes: > As you could see the query includes castings, we noticed testing with > Postgres 12 that the castings of the CASE THEN statement (commented out > below) where failing in some cases, of course if you do the INNER JOIN and > CASE WHEN first our expectation is that the

Re: Regarding TZ conversion

2020-06-04 Thread Tom Lane
Rajin Raj writes: > Option 1: AT TIME ZONE 'IST' > Option 2: AT TIME ZONE 'Asia/Kolkata' > In the first option, I get +2:00:00 offset (when *timezone_abbrevations = > 'Default'*) and for option 2 , +5:30 offset. > I can see multiple entries for IST in pg_timezone_names with > different

Re: question regarding copyData containers

2020-06-04 Thread Jerome Wagner
Hello, thank you for your feedback. I agree that modifying the COPY subprotocols is hard to do because it would have an impact on the client ecosystem. My understanding (which seems to be confirmed by what Tom Lane said) is that the server discards the framing and manages to make sense of the

[PATCH] pg_dump: Add example and link for --encoding option

2020-06-04 Thread 이동욱
To let users know what kind of character set can be used add examples and a link to --encoding option. Thanks, Dong wook 0001-pg_dump-Add-example-and-link-for-encoding-option.patch Description: Binary data

Re: what can go in root.crt ?

2020-06-04 Thread Chapman Flack
On 06/04/20 02:07, Laurenz Albe wrote: > I feel bad about bending the basic idea of certificates and trust to suit > some misbegotten bureaucratic constraints on good security. Can you elaborate on what, in the email message you replied to here, represented a bending of the basic idea of

Re: Read access for pg_monitor to pg_replication_origin_status view

2020-06-04 Thread Martín Marqués
Hi Kyotaro-san, > Sorry for not mentioning it at that time, but about the following diff: > > +GRANT SELECT ON pg_replication_origin_status TO pg_read_all_stats; > > system_views.sql already has a REVOKE command on the view. We should > put the above just below the REVOKE command. > > I'm not

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-04 Thread Mahendra Singh Thalor
On Fri, 29 May 2020 at 15:52, Amit Kapila wrote: > > On Wed, May 27, 2020 at 5:19 PM Mahendra Singh Thalor wrote: >> >> On Tue, 26 May 2020 at 16:46, Amit Kapila wrote: >> >> Hi all, >> On the top of v16 patch set [1], I did some testing for DDL's and DML's to test wal size and performance.

Re: proposal - function string_to_table

2020-06-04 Thread movead...@highgo.ca
+{ oid => '2228', descr => 'split delimited text', + proname => 'string_to_table', prorows => '1000', proretset => 't', + prorettype => 'text', proargtypes => 'text text', + prosrc => 'text_to_table' }, +{ oid => '2282', descr => 'split delimited text with null string', + proname =>

Regarding TZ conversion

2020-06-04 Thread Rajin Raj
Hi , What is the right approach for using AT TIME ZONE function? Option 1: AT TIME ZONE 'IST' Option 2: AT TIME ZONE 'Asia/Kolkata' In the first option, I get +2:00:00 offset (when *timezone_abbrevations = 'Default'*) and for option 2 , +5:30 offset. I can see multiple entries for IST in

Re: libpq copy error handling busted

2020-06-04 Thread Thomas Munro
On Thu, Jun 4, 2020 at 6:22 PM Oleksandr Shulgin wrote: > On Thu, Jun 4, 2020 at 5:37 AM Thomas Munro wrote: >> Here's what I tested. First, I put this into pgdata/postgresql.conf: > Would it be feasible to capture this in a sort of a regression (TAP?) test? If I'm remembering correctly, it

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-04 Thread Dilip Kumar
On Wed, Jun 3, 2020 at 2:43 PM Amit Kapila wrote: > > On Tue, Jun 2, 2020 at 7:53 PM Dilip Kumar wrote: > > > > On Tue, Jun 2, 2020 at 4:56 PM Amit Kapila wrote: > > > > > > On Tue, Jun 2, 2020 at 3:59 PM Dilip Kumar wrote: > > > > > > > > I thin for our use case BufFileCreateShared is more

Re: Is it useful to record whether plans are generic or custom?

2020-06-04 Thread Atsushi Torikoshi
On Mon, May 25, 2020 at 10:54 AM Atsushi Torikoshi wrote: > On Thu, May 21, 2020 at 5:10 PM Kyotaro Horiguchi > wrote: > >> Cost numbers would look better if it is cooked a bit. Is it worth >> being in core? > > > I didn't come up with ideas about how to use them. > IMHO they might not so

Possible bug on Postgres 12 (CASE THEN evaluated prematurely) - Change of behaviour compared to 11, 10, 9

2020-06-04 Thread Juan Fuentes
Greetings, Our system uses an EAV like database and generates queries like the example below. As you could see the query includes castings, we noticed testing with Postgres 12 that the castings of the CASE THEN statement (commented out below) where failing in some cases, of course if you do

Re: Read access for pg_monitor to pg_replication_origin_status view

2020-06-04 Thread Kyotaro Horiguchi
Hi, Martin. At Wed, 3 Jun 2020 13:32:28 -0300, Martín Marqués wrote in > Hi Kyotaro-san, > > Thank you for taking the time to review my patches. Would you like to > set yourself as a reviewer in the commit entry here? > https://commitfest.postgresql.org/28/2577/ Done. > > 0002: > > > > It

Re: Atomic operations within spinlocks

2020-06-04 Thread Michael Paquier
On Thu, Jun 04, 2020 at 09:40:31AM +1200, Thomas Munro wrote: > Yeah. It'd be fine to move that after the spinlock release. Although > it's really just for informational purposes only, not for any data > integrity purpose, reading it before the spinlock acquisition would > theoretically allow it

Re: libpq copy error handling busted

2020-06-04 Thread Oleksandr Shulgin
On Thu, Jun 4, 2020 at 5:37 AM Thomas Munro wrote: > On Thu, Jun 4, 2020 at 1:53 PM Thomas Munro > wrote: > > On Thu, Jun 4, 2020 at 1:35 PM Tom Lane wrote: > > > Ah, it's better if I put the pqReadData call into *both* the paths > > > where 1f39a1c06 made pqSendSome give up. The attached

Re: what can go in root.crt ?

2020-06-04 Thread Laurenz Albe
On Wed, 2020-06-03 at 19:57 -0400, Chapman Flack wrote: > Ok, so a person in the situation described here, who is not in a position > to demand changes in an organizational policy (whether or not it seems > ill-conceived to you or even to him/her), is facing this question: > > What are the