Re: [OSM-dev] Potlatch causing DB inconsistency?
Tnx for the confirmation. I guess 0.6 API aims to fix that with atomic uploads, but Potlatch will remain using its own API, right? Is it now (version 0.10) preventing this somehow? so the events went something like: - user adjusts the way somehow [common operation] - potlatch decides to delete old way and create a new one for some reason [stupid (breaking history), but legal] - potlatch deletes old way and all the nodes it can (those that aren't used by other ways where other rivers join the riverbank) [stupid, but ok] - potlatch creates a new way, consisting of old (now mostly deleted) nodes [illegal] Not really sure this has anything to do with missing transactions, more likely some bad logic. Foreign keys in the db on current_ways - current_nodes could prevent that, but AFAIK they aren't used for performance reasons. But why is potlatch today STILL showing the riverbank as it would be there, but in fact it isn't? If this accident would be visible to the user then he could do something about it (re-draw it in worst case) immediately. Stefan -- Forwarded message -- From: Richard Fairhurst [EMAIL PROTECTED] Date: Tue, Jul 22, 2008 at 9:52 AM Subject: Re: [OSM-dev] Potlatch causing DB inconsistency? To: dev@openstreetmap.org Stefan Baebler wrote: Any ideas what's going on? I think it's reasonably well attested that, without transactions, this sort of thing is going to happen once in a while. As posted the other day, Potlatch's database access code has been rewritten almost beyond recognition for 0.10 (see http://tinyurl.com/5996mn) so there's not a whole lot of point filing bugs against 0.9c; let me know if you see anything like this happen with 0.10, of course. cheers Richard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Potlatch causing DB inconsistency?
Stefan Baebler wrote: - potlatch decides to delete old way and create a new one for some reason [stupid (breaking history), but legal] Er, no. Potlatch never decides of its own accord to delete an old way and create a new one. You look at the source and tell me where it does that. - potlatch deletes old way and all the nodes it can (those that aren't used by other ways where other rivers join the riverbank) [stupid, but ok] Er, no. That's not stupid, that's what deleting a way does. - potlatch creates a new way, consisting of old (now mostly deleted) nodes [illegal] Er, no. When Potlatch _created_ way 25397686 the nodes were live. Nothing illegal there. Not really sure this has anything to do with missing transactions, more likely some bad logic. Right, so try actually thinking about it rather than simply sounding off with the first assumptions that spring to mind, and throwing pejorative words around as if you know instantly where the blame lies and whatever muppet wrote Potlatch is clearly a moron not to have thought of it. http://www.openstreetmap.org/api/0.5/way/25397686/history shows that the first version of the way had 2 nodes, the second had 155 nodes. The extra 153 nodes were old nodes. So it looks like the user was merging ways. Because we don't have a server-side merge operation, a merge takes place by appending the nodes from way B to way A, saving way A, then deleting way B. Two separate operations. (There is no way to do this without breaking history.) Saving way A will set all its constituent nodes to visible=true (controllers/amf_controller.rb#L334). Deleting way B will not set nodes used in other ways (i.e. including A) to visible=false; only those that were used in this way only (models/way.rb#L237). If you can prove _how_ this is bad logic or point to an error in the code then that would actually be helpful. But if these operations execute at the same time, without transactions, you are going to get inconsistencies exactly like this. Potlatch tries its best to make sure that they don't: if you look at http://trac.openstreetmap.org/browser/applications/editors/potlatch/way.as#L264 , you'll see that the call to deleteMergedWays only happens after the new way has been successfully saved. Now my guess is that this is where something has gone askew. Experience also suggests that this is more likely to happen at a time of high server load. So if I were to make a random stab in the dark, I would guess that the user was merging ways hither and thither; that the first save operation (putway) took a long time to execute due to high load; and that somehow, perhaps due to Potlatch incorrectly chaining the list of ways to delete when ways are merged, a subsequent delete operation (deleteway) was sent while putway was still executing over some of the same nodes. Like I said, amf_controller has been so thoroughly rewritten for 0.10 that there is little point raising bugs against 0.9c now. If you can find steps to reproduce against 0.10, that starts to be helpful. But why is potlatch today STILL showing the riverbank as it would be there, but in fact it isn't? http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb#L160 There ain't nothing inherently flawed about that code (and I didn't even write it so can say that with impunity). I guess way.nodes.collect isn't fussy about node.visible. To be frank I'm not convinced that it's a bad thing: assuming that we can fix whatever's causing the inconsistency in the first place, then exposing such nodes means they'll be invisibly fixed over time as people edit and reupload the ways. Incidentally, Lester Caine HAS the trademark round here on RANDOM use of capitalised words in the MIDDLE of sentences. cheers Richard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] Potlatch causing DB inconsistency?
On Tue, Jul 22, 2008 at 10:18 AM, Stefan Baebler [EMAIL PROTECTED] wrote: Tnx for the confirmation. I guess 0.6 API aims to fix that with atomic uploads, but Potlatch will remain using its own API, right? Is it now (version 0.10) preventing this somehow? Well, it's API will get upgraded to use transactions and optimistic locking along with everything else. The 0.10 changes just regularise the code to use more rails and less SQL (or at least make that an option). so the events went something like: - user adjusts the way somehow [common operation] - potlatch decides to delete old way and create a new one for some reason [stupid (breaking history), but legal] - potlatch deletes old way and all the nodes it can (those that aren't used by other ways where other rivers join the riverbank) [stupid, but ok] - potlatch creates a new way, consisting of old (now mostly deleted) nodes [illegal] Not really sure this has anything to do with missing transactions, more likely some bad logic. Foreign keys in the db on current_ways - current_nodes could prevent that, but AFAIK they aren't used for performance reasons. It's almost certainly to do with transactions... or at least lack of any locking. There's very little you can do at the moment if requests don't get executed in the order which they are sent. And that can happen as there are multiple rails boxes/threads/processes dealing with requests. API 0.6 introduces optimistic locking which should cause these situations to generate errors rather than corrupt things. But why is potlatch today STILL showing the riverbank as it would be there, but in fact it isn't? For the same reason the history call to the API still shows it. The potlatch API works slightly differently to the normal API, and in this case it doesn't check for the visibility of the nodes before returning them. It assumes that if it's in a visible way then the node is visible, which unfortunately isn't now true. If you edit the way in potlatch it should actually resurrect the nodes and make everything consistent again -- I did this just now, and it now seems fine (hopefully :-)). Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
