Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Vitalii Tymchyshyn
I am not sure I understand the problem fully, e.g. what to do if there are observations A,B and C with A to B and B to C less then treshold and A to C over treshold, but anyway. Could you first apply a kind of grid to your observations? What I mean is to round your coords to, say, 1/2 arcsec on

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Jiří Nádvorník
Hi Vitalii, thank you for your reply. The problem you suggested can in the most pathological way be, that these observations are on one line. As you suggested it, the B would be in the middle. So A and C are not in 1 arcsec range of each other, but they must be within 1 arcsec of their

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Craig James
Jiri, If you haven't looked at clustering algorithms yet, you might want to do so. Your problem is a special case of clustering, where you have a large number of small clusters. A good place to start is the overview on Wikipedia: http://en.wikipedia.org/wiki/Cluster_analysis A lot of people

Re: [PERFORM] Very slow planning performance on partition table

2014-07-27 Thread Rural Hunter
Anyone? I can see many pg processes are in BIND status with htop. Some of them could be hanging like 30 mins. I tried gdb on the same process many times and the trace shows same as my previous post. This happened after I partitioned my main tables to 60 children tables. And also, I'm

Re: [PERFORM] Very slow planning performance on partition table

2014-07-27 Thread Tom Lane
Rural Hunter ruralhun...@gmail.com writes: Does that indicate something? seems it's waiting for some lock. Yeah, that's what the stack trace suggests. Have you looked into pg_locks and pg_stat_activity to see which lock it wants and what's holding said lock? regards,

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Marc Mamin
[Craig] If you haven't looked at clustering algorithms yet, you might want to do so. Your problem is a special case of clustering, where you have a large number of small clusters. A good place to start is the overview on Wikipedia: http://en.wikipedia.org/wiki/Cluster_analysis According to this

Re: [PERFORM] Cursor + upsert (astronomical data)

2014-07-27 Thread Vitalii Tymchyshyn
Well, that's why I said to apply regular algorithm to deduplicate after this step. Basically, what I expect is to have first pass with group by that do not require any joins and produces dirty set of identifiers. It should do next things: 1) Provide working set of dirty identifiers that has a

Re: [PERFORM] Very slow planning performance on partition table

2014-07-27 Thread Rural Hunter
Yes I checked. The connection I inspected is the longest running one. There was no other connections blocking it. And I also see all locks are granted for it. Does the planning phase require some internal locks? ?? 2014/7/28 0:28, Tom Lane : Yeah, that's what the stack trace suggests.

Re: [PERFORM] High rate of transaction failure with the Serializable Isolation Level

2014-07-27 Thread Craig Ringer
On 07/26/2014 02:58 AM, Reza Taheri wrote: Hi Craig, According to the attached SQL, each frame is a separate phase in the operation and performs many different operations. There's a *lot* going on here, so identifying possible interdependencies isn't something I can do in a ten minute