Hi
no it was a mistake.
Basicly atm there is one huge nested map, that holds all required
information:
nodemap[Objekt-ID][Timestamp] = (lat, lon)
lat/lon is atm stored as double. The suggested memory layout is as follows:
1. An Array with a 32bit Pointer for each Node-ID
2. A Series of Timestamp-Lat-Lon Triplets, each stored as Integers, all
together on one row for all versions, ending with a 0-byte
Memory-Usage would be
8*(max. node-id) for 1.
4*(number of used node-ids) + 12*(number of used node-version) for 2.
Maximum memory usage for a complete planet would be ~40GB.
With a normal Array for 1. the minimum memory usage would be ~10GB.
This could be reduced by using a sparsetable
http://google-sparsehash.googlecode.com/svn/trunk/doc/sparsetable.html
Peter
Am 16.10.2012 10:50, schrieb Ab_fab:
Thanks for your answer Peter.
You did not answer to the list, was it intentional ?
I forwarded your reply to the French list thread [1], where the initial
discussion with Aurelien and others took place, back in july. I totally
miss the skills required to implement anything ...
Anyway, it could be of interest for some people, in particular for stats
purpose on the evolution of the database contents (not only the number
of buildings from cadastre ;-) ).
French community has also (I think) enough material ressources to setup
such low activity database, once this RAM issue for the import is solved.
Regards
Fabien
[1] http://lists.openstreetmap.org/pipermail/talk-fr/2012-July/045337.html
http://lists.openstreetmap.org/pipermail/talk-fr/2012-October/049849.html
2012/10/16 Peter Körner <pe...@mazdermind.de <mailto:pe...@mazdermind.de>>
Hi
Well, osm-history-renderer is - after all - a proof of concept. It's
impossible atm. to import a full country without having tons of RAM.
If you are interested in optimizing the memory usage I could provide
you with plans about a memory layout frederik ramm and I once talked
about; I never came around to implement it, while the necessary
interfaces are already in place.
Regards,
Peter
Am 15.10.2012 15:08, schrieb Ab_fab:
I think the memory problems occured during the
osm-history-importer [1]
process
Some people were successfull in building the database for
limited areas,
but France is extract is huge in comparison.
Is there a way to limit RAM usage for such large files during
import ?
[1] https://github.com/MaZderMind/__osm-history-renderer
<https://github.com/MaZderMind/osm-history-renderer>
2012/7/16 Jochen Topf <joc...@remote.org
<mailto:joc...@remote.org> <mailto:joc...@remote.org
<mailto:joc...@remote.org>>>
On Mon, Jul 16, 2012 at 01:28:50PM +0200, Aurélien FILEZ wrote:
> I'm searching history statistics of the french territory
contributions in
> order to make some graphics about :
> - Number of "named roads"
> - Numbers of "house numbers"
> - Number of kilometers of different types of roads
(highway, primary,
> secondary, tertiary, residential).
>
> I tried OSMIUM with the last avalable france.osh.pbf,
but data
are too many
> data and cause bad allocs errors on a 8 Gb memory server.
What exactly did you do?
If you use the RangeFromHistory handler to filter out one
specific
date in the
history and then the Statistics handler, you should be able to
generate some
statistics, though not exactly the kind of statistics you
are asking
for. But
you can use the Statistics handler as a model on how to
write your own.
You'd have to tun the thing several times, each time
filtering out a
different
date with RangeFromHistory.
Or you write your own handler that directyl works on
history data,
but thats
a bit more tricky.
In any case it should not need 8GB for all of this. You
might have
used the
MmapAnon node storage instead of SparseTable handler. This
explains the
different options:
http://wiki.openstreetmap.org/__wiki/Osmium/Storing_Node___Positions
<http://wiki.openstreetmap.org/wiki/Osmium/Storing_Node_Positions>
Jochen
--
Jochen Topf joc...@remote.org <mailto:joc...@remote.org>
<mailto:joc...@remote.org <mailto:joc...@remote.org>>
http://www.remote.org/jochen/ +49-721-388298
<tel:%2B49-721-388298> <tel:%2B49-721-388298>
_________________________________________________
dev mailing list
dev@openstreetmap.org <mailto:dev@openstreetmap.org>
<mailto:dev@openstreetmap.org <mailto:dev@openstreetmap.org>>
http://lists.openstreetmap.__org/listinfo/dev
<http://lists.openstreetmap.org/listinfo/dev>
--
ab_fab <http://wiki.openstreetmap.__org/wiki/User:Ab_fab
<http://wiki.openstreetmap.org/wiki/User:Ab_fab>>
"Il n'y a pas de pas perdus"
--
ab_fab <http://wiki.openstreetmap.org/wiki/User:Ab_fab>
"Il n'y a pas de pas perdus"
_______________________________________________
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
