Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Thomas Munro
On Fri, May 29, 2015 at 11:24 AM, Robert Haas robertmh...@gmail.com wrote: A. Most obviously, we should fix pg_upgrade so that it installs chkpnt_oldstMulti instead of chkpnt_nxtmulti into datfrozenxid, so that we stop creating new instances of this problem. That won't get us out of the hole

Re: [GENERAL] Fwd: Raster performance

2015-05-29 Thread PT
On Thu, 28 May 2015 10:06:24 -0500 David Haynes II dahay...@umn.edu wrote: The query run times are significantly slower on outdb as that using indb here are the run times on 2 queries. ST_Count/ST_Clip(select single band here)/Inner Join/ST_Transform (US Counties) OutDB: 873.564s (14

[GENERAL] Planner cost adjustments

2015-05-29 Thread Daniel Begin
Hi all, Running some queries, I found that the planner often selects sequential scan instead of an index scan, even if the latter is way faster (one order of magnitude faster if I consider some tests I made by setting enable_seqscan = ON/OFF). How can I figure out what parameter I should adjust

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Robert Haas
On Fri, May 29, 2015 at 10:17 AM, Tom Lane t...@sss.pgh.pa.us wrote: Thomas Munro thomas.mu...@enterprisedb.com writes: On Fri, May 29, 2015 at 11:24 AM, Robert Haas robertmh...@gmail.com wrote: B. We need to change find_multixact_start() to fail softly. Here is an experimental WIP patch that

Re: [GENERAL] Planner cost adjustments

2015-05-29 Thread Melvin Davidson
I have found that setting enable_seqscan = off will remedy that situation. Basically, it forces the planner to choose the index. However, if no correct index is available, it will use sequential scan anyway. The only time it will have a negative effect is if the seqscan is actually faster, which

Re: [GENERAL] Planner cost adjustments

2015-05-29 Thread Tomas Vondra
Hi, On 05/29/15 17:22, Melvin Davidson wrote: I have found that setting enable_seqscan = off will remedy that situation. Basically, it forces the planner to choose the index. However, if no correct index is available, it will use sequential scan anyway. The only time it will have a negative

Re: [GENERAL] Between with a list of ranges possible ?

2015-05-29 Thread Andreas Kretschmer
Arup Rakshit arupraks...@rocketmail.com wrote: Hi, Can I do the below 3 queries in a single query ? OR ? Andreas -- Really, I'm not out to destroy Microsoft. That will just be a completely unintentional side effect. (Linus Torvalds) If I was god, I would

Re: [GENERAL] Between with a list of ranges possible ?

2015-05-29 Thread John R Pierce
On 5/29/2015 9:32 AM, Arup Rakshit wrote: Can I do the below 3 queries in a single query ? select * from table where number * 3 between start_value1 and end_value2; select * from table where number * 3 between start_value2 and end_value2; select * from table where number * 3 between

Re: [GENERAL] Between with a list of ranges possible ?

2015-05-29 Thread Raymond O'Donnell
On 29/05/2015 17:32, Arup Rakshit wrote: Hi, Can I do the below 3 queries in a single query ? select * from table where number * 3 between start_value1 and end_value2; select * from table where number * 3 between start_value2 and end_value2; select * from table where number * 3

Re: [GENERAL] Between with a list of ranges possible ?

2015-05-29 Thread John R Pierce
On 5/29/2015 10:41 AM, John R Pierce wrote: On 5/29/2015 9:32 AM, Arup Rakshit wrote: Can I do the below 3 queries in a single query ? select * from table where number * 3 between start_value1 and end_value2; select * from table where number * 3 between start_value2 and end_value2; select

[GENERAL] Between with a list of ranges possible ?

2015-05-29 Thread Arup Rakshit
Hi, Can I do the below 3 queries in a single query ? select * from table where number * 3 between start_value1 and end_value2; select * from table where number * 3 between start_value2 and end_value2; select * from table where number * 3 between start_value3 and end_value3; --

Re: [GENERAL] Between with a list of ranges possible ?

2015-05-29 Thread Stephen Cook
Use UNION ALL: select * from table where number * 3 between start_value1 and end_value2 UNION ALL select * from table where number * 3 between start_value2 and end_value2 UNION ALL select * from table where number * 3 between start_value3 and end_value3; -- Stephen On 5/29/2015 12:32 PM,

Re: [GENERAL] Planner cost adjustments

2015-05-29 Thread Tomas Vondra
Hi, On 05/29/15 22:56, Daniel Begin wrote: Omg! I was not expecting such a step-by-step procedure, thanks! I'll follow the guide :-) Since I was about to provide a bit of context as asked by Tomas, here it is for those who are interested... Best regards, Daniel A bit of the required

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Andres Freund
On 2015-05-29 15:49:53 -0400, Bruce Momjian wrote: I think we need to step back and look at the brain power required to unravel the mess we have made regarding multi-xact and fixes. (I bet few people can even remember which multi-xact fixes went into which releases --- I can't.) Instead of

Re: [GENERAL] Planner cost adjustments

2015-05-29 Thread Daniel Begin
Thank Tomas, Here are few answers to your questions/comments - Write cache is the on-drive write cache and I agree on the risks... I plugged them on a UPS. - Spread over is done with tablespaces - 30% 40% ratios are that 3 indexes/10 and 4 tables/10 are larger than 10GB - Confused comments

[GENERAL] replacing jsonb field value

2015-05-29 Thread john.tiger
using 9.4.2 suppose we have create table test (id serial primary key, data jsonb); insert into test (data) values ({a:1, b:2}) want to replace b with 3 okay, we are retrieving entire record res = select * from test where data - b = 2 newrec = res newrec[b = 3 delete from test where data - b=

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Robert Haas
On Fri, May 29, 2015 at 3:08 PM, Robert Haas robertmh...@gmail.com wrote: It won't fix the fact that pg_upgrade is putting a wrong value into everybody's datminmxid field, which should really be addressed too, but I've been working on this for about three days virtually non-stop and I don't

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Alvaro Herrera
Bruce Momjian wrote: I think we need to step back and look at the brain power required to unravel the mess we have made regarding multi-xact and fixes. (I bet few people can even remember which multi-xact fixes went into which releases --- I can't.) Instead of working on actual features, we

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Alvaro Herrera
Andres Freund wrote: I considered for a second whether the solution for that could be to not truncate while inconsistent - but I think that doesn't solve anything as then we can end up with directories where every single offsets/member file exists. Hang on a minute. We don't need to scan

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Robert Haas
On Fri, May 29, 2015 at 9:46 PM, Andres Freund and...@anarazel.de wrote: On 2015-05-29 15:08:11 -0400, Robert Haas wrote: It seems pretty clear that we can't effectively determine anything about member wraparound until the cluster is consistent. I wonder if this doesn't actually hints at a

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Andres Freund
On 2015-05-29 15:08:11 -0400, Robert Haas wrote: It seems pretty clear that we can't effectively determine anything about member wraparound until the cluster is consistent. I wonder if this doesn't actually hints at a bigger problem. Currently, to determine where we need to truncate

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Thomas Munro
On Sat, May 30, 2015 at 1:46 PM, Andres Freund and...@anarazel.de wrote: On 2015-05-29 15:08:11 -0400, Robert Haas wrote: It seems pretty clear that we can't effectively determine anything about member wraparound until the cluster is consistent. I wonder if this doesn't actually hints at a

Re: [GENERAL] Fwd: Raster performance

2015-05-29 Thread David Haynes II
Hello, Let me explain this a bit more clearly. The dataset (table) that we are using in this analysis is produced from a satellite image (tif) called MODIS. This image can be placed into PostgreSQL using the raster2pgsql command (http://postgis.net/docs/using_raster_dataman.html). The

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Robert Haas
On Fri, May 29, 2015 at 12:43 PM, Robert Haas robertmh...@gmail.com wrote: Working on that now. OK, here's a patch. Actually two patches, differing only in whitespace, for 9.3 and for master (ha!). I now think that the root of the problem here is that DetermineSafeOldestOffset() and

Re: [GENERAL] Fwd: Raster performance

2015-05-29 Thread PT
On Fri, 29 May 2015 13:37:24 -0500 David Haynes II dahay...@umn.edu wrote: Let me explain this a bit more clearly. The dataset (table) that we are using in this analysis is produced from a satellite image (tif) called MODIS. This image can be placed into PostgreSQL using the raster2pgsql

Re: [GENERAL] Planner cost adjustments

2015-05-29 Thread Daniel Begin
Omg! I was not expecting such a step-by-step procedure, thanks! I'll follow the guide :-) Since I was about to provide a bit of context as asked by Tomas, here it is for those who are interested... Best regards, Daniel A bit of the required context... I am running all this on my personal PC:

Re: [GENERAL] Planner cost adjustments

2015-05-29 Thread PT
On Fri, 29 May 2015 09:39:00 -0400 Daniel Begin jfd...@hotmail.com wrote: Hi all, Running some queries, I found that the planner often selects sequential scan instead of an index scan, even if the latter is way faster (one order of magnitude faster if I consider some tests I made by setting

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Bruce Momjian
On Thu, May 28, 2015 at 07:24:26PM -0400, Robert Haas wrote: On Thu, May 28, 2015 at 4:06 PM, Joshua D. Drake j...@commandprompt.com wrote: FTR: Robert, you have been a Samurai on this issue. Our many thanks. Thanks! I really appreciate the kind words. So, in thinking through this

Re: [GENERAL] Between with a list of ranges possible ?

2015-05-29 Thread John McKown
On Fri, May 29, 2015 at 12:48 PM, Stephen Cook scli...@gmail.com wrote: Use UNION ALL: select * from table where number * 3 between start_value1 and end_value2 UNION ALL select * from table where number * 3 between start_value2 and end_value2 UNION ALL select * from table where number *

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Tom Lane
Thomas Munro thomas.mu...@enterprisedb.com writes: On Fri, May 29, 2015 at 11:24 AM, Robert Haas robertmh...@gmail.com wrote: B. We need to change find_multixact_start() to fail softly. Here is an experimental WIP patch that changes StartupMultiXact and SetMultiXactIdLimit to find the oldest

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-05-29 Thread Steve Kehlet
On Fri, May 29, 2015 at 12:08 PM Robert Haas robertmh...@gmail.com wrote: OK, here's a patch. I grabbed branch REL9_4_STABLE from git, and Robert got me a 9.4-specific patch. I rebuilt, installed, and postgres started up successfully! I did a bunch of checks, had our app run several thousand