On Tue, May 17, 2011 at 9:05 PM, Jaroslaw Wozny <[email protected]> wrote: > Hi, > >> interesting. i've just done a test with the latest git code (e120e59) >> and got no errors. unfortunately, my setup is different from yours: > > I have also the latest (head version e120e59). > >>> 187 tests, 2413 assertions, 3 failures, 0 errors > >> what's your log/test.log? are you using the postgres extensions in >> db/functions/? > > Ok. I have installed extra functions for PG and no fails on tests. But I > have still problem with seg fault. Problem is repeated by two independent > people on Ubuntu 11.04 (two different machines). Import is performed by > bulk_upload.py. :( > > Maybe something is specific in bulk_upload.py that shows problem during > import? I don't know.
yes, it appears to be. i was able to reproduce both the segfault you reported and the long-looked-for infinite loop bug, both with bulk_upload.py! unfortunately, the segfault seems to be happening because the libxml data structure is being reclaimed before the ruby object wrapping it is dead, and it's very difficult to debug stuff across that ruby/libxml interface. however, there is good news: i've updated rails_port to use libxml-ruby 2.0.5 and it seems much more robust - i haven't been able to get it to segfault. it'll be in git master soon, but if you'd like to try it out then a patch is attached. i'd like to know how you get on with it and whether it has any problems. cheers, matt
From 5472095287a651d9915805ab7bd3deadfe8d6c3d Mon Sep 17 00:00:00 2001 From: Matt Amos <[email protected]> Date: Sat, 21 May 2011 15:15:49 +0100 Subject: [PATCH] Updated to libxml-ruby 2.0.5 and fixed code accordingly. I've tested this through the unit tests and by hitting it with bulk_upload.py and neither fail or cause the server to crash or go into an infinite loop. Both of these things happened randomly with 1.1.3/4 due to an apparent early deregistration of the expanded nodes. --- config/environment.rb | 2 +- lib/diff_reader.rb | 19 ++++++++++++++++++- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/config/environment.rb b/config/environment.rb index 58d9432..d5cee81 100644 --- a/config/environment.rb +++ b/config/environment.rb @@ -18,7 +18,7 @@ Rails::Initializer.run do |config| unless STATUS == :database_offline config.gem 'composite_primary_keys', :version => '2.2.2' end - config.gem 'libxml-ruby', :version => '~> 1.1.1', :lib => 'libxml' + config.gem 'libxml-ruby', :version => '>= 2.0.5', :lib => 'libxml' config.gem 'rmagick', :lib => 'RMagick' config.gem 'oauth', :version => '>= 0.4.3' config.gem 'oauth-plugin', :version => '0.3.14' diff --git a/lib/diff_reader.rb b/lib/diff_reader.rb index 0f0492f..de2da3c 100644 --- a/lib/diff_reader.rb +++ b/lib/diff_reader.rb @@ -20,6 +20,10 @@ class DiffReader def initialize(data, changeset) @reader = XML::Reader.string(data) @changeset = changeset + # document that's (re-)used to handle elements expanded out of the + # diff processing stream. + @doc = XML::Document.new + @doc.root = XML::Node.new("osm") end ## @@ -85,8 +89,21 @@ class DiffReader model = MODELS[model_name] raise OSM::APIBadUserInput.new("Unexpected element type #{model_name}, " + "expected node, way or relation.") if model.nil? - yield model, @reader.expand + # new in libxml-ruby >= 2, expand returns an element not associated + # with a document. this means that there's no encoding parameter, + # which means basically nothing works. + expanded = @reader.expand + + # create a new, empty document to hold this expanded node + new_node = @doc.import(expanded) + @doc.root << new_node + + yield model, new_node @reader.next + + # remove element from doc - it will be garbage collected and the + # rest of the document is re-used in the next iteration. + @doc.root.child.remove! end end -- 1.7.4.1
_______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
