Re: [PERFORM] High rate of transaction failure with the Serializable Isolation Level

2014-07-27 Thread Craig Ringer
On 07/26/2014 02:58 AM, Reza Taheri wrote: > Hi Craig, > >> According to the attached SQL, each frame is a separate phase in the >> operation and performs many different operations. >> There's a *lot* going on here, so identifying possible interdependencies >> isn't something I can do in a ten m

Re: [PERFORM] Very slow planning performance on partition table

2014-07-27 Thread Rural Hunter
Yes I checked. The connection I inspected is the longest running one. There was no other connections blocking it. And I also see all locks are granted for it. Does the planning phase require some internal locks? ?? 2014/7/28 0:28, Tom Lane : Yeah, that's what the stack trace suggests. Hav

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Vitalii Tymchyshyn
Well, that's why I said to apply regular algorithm to deduplicate after this step. Basically, what I expect is to have first pass with group by that do not require any joins and produces "dirty" set of identifiers. It should do next things: 1) Provide working set of dirty identifiers that has a hu

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Marc Mamin
[Craig] >>If you haven't looked at clustering algorithms yet, you might want to do so. >>Your problem is a special case of clustering, where you have a large number >>of small clusters. A good place to start is the overview on Wikipedia: >>http://en.wikipedia.org/wiki/Cluster_analysis According t

Re: [PERFORM] Very slow planning performance on partition table

2014-07-27 Thread Tom Lane
Rural Hunter writes: >> Does that indicate something? seems it's waiting for some lock. Yeah, that's what the stack trace suggests. Have you looked into pg_locks and pg_stat_activity to see which lock it wants and what's holding said lock? regards, tom lane -- Sent vi

Re: [PERFORM] Very slow planning performance on partition table

2014-07-27 Thread Rural Hunter
Anyone? I can see many pg processes are in BIND status with htop. Some of them could be hanging like 30 mins. I tried gdb on the same process many times and the trace shows same as my previous post. This happened after I partitioned my main tables to 60 children tables. And also, I'm experiecin

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Craig James
Jiri, If you haven't looked at clustering algorithms yet, you might want to do so. Your problem is a special case of clustering, where you have a large number of small clusters. A good place to start is the overview on Wikipedia: http://en.wikipedia.org/wiki/Cluster_analysis A lot of people have

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Jiří Nádvorník
Hi Vitalii, thank you for your reply. The problem you suggested can in the most pathological way be, that these observations are on one line. As you suggested it, the B would be in the middle. So A and C are not in 1 arcsec range of each other, but they must be within 1 arcsec of their commo