Re: [OSM-dev] osmium trouble

2011-11-21 Thread Hermann Peifer 

On 21/11/2011 08:13, Jochen Topf wrote:

On Sun, Nov 20, 2011 at 03:34:27PM -0700, Martijn van Exel wrote:

I also ran into trouble with files of around 40MB or larger. I still
get the same errors though, either using array or sparsetable for node
storage.
I'm on Ubuntu 10.04 64 bit with 16GB RAM.


Can you post the script you are using? Maybe we can find out if its doing
something that triggers a bug.



Just to confirm that osmjs works again for me after applying your fix 
from yesterday:


commit 10ec36fbf104c5e1ae8ae1e0c5199cf072acb143
Author: Jochen Topf joc...@topf.org
Date:   Sun Nov 20 15:06:09 2011 +0100
fix memory leak in Javascript glue code


I just can't confirm your comment from an earlier commit (22 Oct): This 
change does not lead to much extra overhead... [1]. Some more figures 
to illustrate the performance difference:


austria.osm.pbf (164M)

Before: After:
real0m58.012s   real1m19.850s
user0m52.899s   user1m16.013s
sys 0m2.980ssys 0m1.784s

france.osm.pbf (1.7G)

Before: After:
real5m43.047s   real11m4.719s
user5m25.260s   user10m49.733s
sys 0m13.881s   sys 0m14.349s

I was using again the same simple Javascript code [2].

Hermann

[1] 
https://github.com/joto/osmium/commit/17604decdd2f26d9fc831cc3c2412a8ca97dc2b0


[2]
Osmium.Callbacks.way = function() {
  if ( this.tags[highway] ) {
print(this.tags[highway] + '\t' + this.geom.toWKT());
  }
}

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osmium trouble

2011-11-20 Thread Hermann Peifer 

On 19/11/2011 21:58, Hermann Peifer wrote:

On 19/11/2011 20:06, ThomasB wrote:

Hi there,

I get the following error when using osmium
/thomas@suncobalt:~/osmium/osmjs/js$ ~/osmium/osmjs/osmjs -j count.js
~/osmlength/pbffiles/germany.osm.pbf
Start!

#
# Fatal error in CALL_AND_RETRY_2
# Allocation failed - process out of memory
#
/
The process used up to 2.5GB RAM and although there is still 9.5GB
physical
RAM available it stops.
osmjs and countjs are unmodified as they came from git.

Smaller files/extracts work without any issues



Same here on a Debian-Linux system with 8 GB RAM. Earlier today, things
were working fine when I was using an older version of osmjs (from
around mid-September). In the afternoon, I updated all dependencies,
refreshed from github and re-compiled osmjs. Now I can't process
germany.osm.pbf anymore: after about 1:30 min., the process aborts with
the above error.

The problem doesn't seem to be related to the type of node location
store, it even occurs with: -l none



As a workaround, I rolled my local git repo back to commit [1] from 15 
October, where osmjs still works fine. The changes in [2] are causing 
the memory allocation trouble with larger .pbf files.


Hermann

[1] 
https://github.com/joto/osmium/commit/61de862b54ffd879034c6f371df99182480fd613


[2] 
https://github.com/joto/osmium/commit/17604decdd2f26d9fc831cc3c2412a8ca97dc2b0


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osmium trouble

2011-11-20 Thread Hermann Peifer 

On 20/11/2011 15:09, Jochen Topf wrote:


I just pushed a fix for the memory leak. Please check whether it fixed your
problems.



Thanks for taking care. Your fix does indeed help and processing larger 
files like germany.osm.pbf is possible again. FYI: the current osmjs 
takes noticeably more time for processing the same file, compared to 
osmjs from 15 October, see below.


Thanks again for your time, Hermann


# Using osmjs from today..
$ time ./osmjs -l array -j ~/osm/tmp/ways.js 
~/osm/pbffiles/germany.osm.pbf  /dev/null


real8m17.119s
user8m11.415s
sys 0m7.156s


# Using osmjs from 15 October..
$ time osmjs -l array -j ~/osm/tmp/ways.js 
~/osm/pbffiles/germany.osm.pbf  /dev/null


real5m47.612s
user5m41.429s
sys 0m7.264s


$ cat ~/osm/tmp/ways.js
Osmium.Callbacks.way = function() {
  if ( this.tags[highway] ) {
print(this.tags[highway] + '\t' + this.geom.toWKT());
  }
}

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osmium trouble

2011-11-19 Thread Hermann Peifer 

On 19/11/2011 20:06, ThomasB wrote:

Hi there,

I get the following error when using osmium
/thomas@suncobalt:~/osmium/osmjs/js$ ~/osmium/osmjs/osmjs -j count.js
~/osmlength/pbffiles/germany.osm.pbf
Start!

#
# Fatal error in CALL_AND_RETRY_2
# Allocation failed - process out of memory
#
/
The process used up to 2.5GB RAM and although there is still 9.5GB physical
RAM available it stops.
osmjs and countjs are unmodified as they came from git.

Smaller files/extracts work without any issues



Same here on a Debian-Linux system with 8 GB RAM. Earlier today, things 
were working fine when I was using an older version of osmjs (from 
around mid-September). In the afternoon, I updated all dependencies, 
refreshed from github and re-compiled osmjs. Now I can't process 
germany.osm.pbf anymore: after about 1:30 min., the process aborts with 
the above error.


The problem doesn't seem to be related to the type of node location 
store, it even occurs with: -l none


:-(  Hermann


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev