Hi all,
OK, I think I can add code to detect the img files which were compiled in the
same call of mkgmap. So, with option --gmapi-minimal
- Files in this list always need to be written. It should be possible to
evaluate a parameter list that specifies additional file patterns.
- Files which don't yet exist in the gmapi folder are also written.
- other files in the Product subfolder would not be touched
- the Info.xml will contain all files listed as input
So, either just --gmapi-minimal or --gmapi-minimal[=<patterns for files which
should overwritten>]
Would that work well for all?
Gerd
________________________________________
Von: mkgmap-dev <[email protected]> im Auftrag von Felix
Hartmann <[email protected]>
Gesendet: Montag, 13. September 2021 20:44
An: Thomas Morgenstern; Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk
if used with --gmapi option
.img files are always reused and never rewritten - it's only about the gmapi
files (a gmapsupp.img has to be generated in one go anyhow - so it is not in
question here). So I do not see the sense of making it more complicated? It
would be fine with me too that way - but I think it's simpler to just assume
all .img files already have the converted files - because you could do it when
generating those .img files. The problem with your approach is - that when you
have a long list of tiles that were crashing due to too little maxnodes - then
regenerating it - will make it very complicated. It's much easier to assume
that all .img files are already converted. All other input not. If you need to
convert .img files too - then you just use the normal --gmapi option instead.
On Mon, 13 Sept 2021 at 18:58, Thomas Morgenstern
<[email protected]<mailto:[email protected]>> wrote:
I suggest the new option should be named reuse=<comma separated list of files, wildcards
enabled>. exampel reuse=40000001.img, 40005*.img. If this option is used, then mkgmap
should not overwrite the listed folders in the <name>gmap. Naturally mkgmap should
write a new tdb, preview.img, mdr and idx. In most cases the reuse folders contains
Contourlines or other static content.
Thomas
Am 13.09.2021 um 15:38 schrieb Felix Hartmann:
Hi Gerd, yes exactly. I have a large collection of .img files of contourlines.
When I realized that I trash my NVME disk very fast if I continue the way I did
I got into thinking and analysing what happens. It was clear that it all comes
down to the --gmapi option. I had believed that if I run several passes with
.o5m and .img files as input - only the .o5m input is converted into new .gmapi
folders - while the existing ones are not overwritten. Then I checked the
created date/last modified date (because windows has in my eyes a bug that if
you replace a file with a near identical file - it just shows a new modified
date - but keeps the original created date - even though it was a full
overwrite).
That got me thinking that in order to save writes - I have to find a way to not
only not recalculate the .img files - but also create a static set of .gmapi
folders. Those I just mklink into the directory name that will be used on the
next run of mkgmap.jar - I managed to do this by uncommenting this one line.
Because I noticed that the symlinked (by mklink) files are not rewritten I
changed my scripts to move them away and symlink them back. Then at the end
delete all symlinks - and move the files back (or to the location that I will
use for compressing). This step is a bit stupid if mkgmap could just have a
--gmapi-minimal mode in which only those files are converted - that are also
written out as .img files (if given --tdb-file option).
I know that many people keep a static set of contourlines .img files (also
containing DEM). You just add the show-profiles=1 option in case you include
contourlines - and leave it out if not. But actually it does not matter if the
contourlines files contain DEM or not.
I think the easiest way is the principle - --gmapi-minimal only converts those
files, it would write out as .img files if --tdb-files option were given (or is
given). --gmapi on the other hand should convert the all input files to .gmapi
format. Then mgkmap does not even need to test if those files are there or not.
This not only saves a lot of writes - but also a lot of compile time.
Because essentially if you only provide .img input files (including of course
the ovm-img for the overview map if you want) you only create a new set of
mapset files. The exception to this is the toolchain in which when a tile
failed to compile - you resplit the input files - and parse them again with the
same arguments so you have input of new o5m files and old .img files. But the
principle stays the same. If on the rerun of the failed tiles now newy split
with higher map-id you give --gmapi-minimal and tdb-file - only those new tiles
are written - while the old .img and old ovm.img are supplied to create the
correct mapset files. With this procedure you don't even need to put a symlink
for the contourlines into the gmap folder. As the input data to create the
contourlines rarely change - you can offer the contourlines as a separate
download.
Makes it much easier and faster.
On the map compilation server you then do not even need to have a copy of the
.gmapi contourlines files. You just need the new input data and the static
contourlines .img files. Thomas Morgenstern for example had also not realized
that he is writing the contourlines .gmapi files each time without any need. I
think many/most providers of garmin maps who offer many regions/worldwide
coverage put the contourlines separate. It's just a huge amount of compile time
if you merge the contourlines in o5m format with the map data in o5m data each
time before running mkgmap. The only actual advantage of this being that the
contourlines do not overlap roads (as they are supplied as transparent .img
files in another layer) - AND that when people select maps with a single click
instead of drag in MapInstall/Mapsource they get all maps they think they get
(though there is a different shading - each layer increases the darkness of the
shading for selected). And of course that the very old Mapsource still has
problems correctly showing the contourlines if they are in a separate layer.
However nearly everyone moved on to Basecamp which is fully compatible with
layered maps. The advantage of contourlines as separate layer is for the user
he can switch them on / off independant of the maps and does not need to
download and transfer them each time. So I think for most the advantages
heavily outweigh the disadvantages - hence contourlines into a separate
transparent layer. Same can be done of course for other things like buildings
or vegetation - though the advantage here is much less on the compile side
(while worldwide 10m coontourlines are 200GB of data - the same extracts are
only 15GB in data of buildings - if buildings are shown only at resolution 24)
On Mon, 13 Sept 2021 at 14:24, Gerd Petermann
<[email protected]<mailto:[email protected]>> wrote:
Hi Felix,
I have no clue how exactly your scripts work, how you manage to reuse *.img
files and so on. Also, I want to find a solution that works for all users, not
just you.
So, I expect that you have one step that compiles frequently changing OSM data
and other steps which are used to compile static data like contourlines or DEM.
I don't know if you do the latter for each country / continent or once for
planet and I don't care as long as it works for you.
My understanding is that you have a large collection of *.img files at some
stage and that you run mkgmap multiple times with different combinations of
those *.img files as input to produce different zip-files with gmapi format or
gmapsupp format.
I think that's the normal way to do it, so I try to support that way.
Gerd
________________________________________
Von: mkgmap-dev
<[email protected]<mailto:[email protected]>> im
Auftrag von Felix Hartmann <[email protected]<mailto:[email protected]>>
Gesendet: Montag, 13. September 2021 12:28
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk
if used with --gmapi option
I move away the info.xml - and use a different name for the mapset files and
then just have mkgmap any file that is input in osm.pbf / osm / o5m format.
Gmapi-minimal should not convert any file that is supplied as .img - as it can
be assumed that those exist already (if they do not - then create them with
--gmapi). That is in my opinion the best approach. So mkgmap does not even try
to convert them.
Afterwards I distribute a gmapi folder that includes all the data - and by
replacing the info.xml you can enable or disable contourlines. For big
countries the contourlines would be a separate download anyhow - so the user
only needs to download the maps (including mapset files) but not redownload the
contourlines. Same principle as in offering the contourlines as a separate
gmapsupp.img file. so you have maps.img contourlines10m.img contourlines20m.img
buildings.img ....
On Mon, 13 Sept 2021 at 13:17, Gerd Petermann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>
wrote:
Hi Felix,
sorry, the Data Deduplication as implemented in Windows Server would not help
here. It works after data was written.
And yes, files which are not just copies of the *.img are written as before. My
understanding is that you have different product directories in the gmapi
folder and that you write protect those files which should be kept.
If you have a script to zip the gmapi directory you have to exclude the
unwanted folders.
Didn't try it. Does it make sense?
Gerd
________________________________________
Von: mkgmap-dev
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>
im Auftrag von Felix Hartmann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>
Gesendet: Montag, 13. September 2021 12:09
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk
if used with --gmapi option
Oh - but data was certainly written - A rename will not show as data written in
both Task manager on Windows, as well as in the smart data (I'm using Windows
10 pro not Windows server however - maybe that functionality is limited to
windows server?)
On Mon, 13 Sept 2021 at 13:06, Felix Hartmann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>
wrote:
Not sure if it makes it possible to use read only attribute instead of moving and
mklink. Maybe yes - because that was not possible before. So it then would be set
read only - instead of of move & mklink - and at the end remove read only
instead of moving back.
On Mon, 13 Sept 2021 at 12:59, Felix Hartmann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>
wrote:
Thanks Gerd,
but that is just removing the warning if it cannot overwrite, correct?
If it can overwrite the gmap file it will still overwrite it as I see.. So in
essence just a bit more intelligent then my disabling that line.
I think it should not overwrite at all if present and --x-gmapi-minimal (then
you don't have to move away the files and link them back to the original
folder).
On Mon, 13 Sept 2021 at 10:43, Gerd Petermann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>
wrote:
Hi all,
attached is a quick patch that implements experimaltal option --x-gmapi-minimal.
If used instead of --gmapi mkgmap will not fail if a write-protected output
file exists in the gmapi output folder and mkgmap copies data from *.img. It
should still crash when other files like Info.xml are written.
BTW: no conversion is done when those files are written. I think mkgmap simply
copies data from sub files in *.img into single files. So, the same sequence of
Bytes exists two or more times on the Computer. Windows Server seems to support
automatic data deduplication (1). I am sure Linux offers similar support. No
idea how effective or reliable it is, but it might be worth trying.
Gerd
(1)
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable
________________________________________
Von: mkgmap-dev
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>
im Auftrag von Carlos Dávila
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>
Gesendet: Sonntag, 12. September 2021 22:45
An:
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk
if used with --gmapi option
I think it's a good idea if mkgmap checks the required files are present.
El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot
overwrite (write-protected) files in the output directory, right? No need to
verify the content of the exiting file(s)?
Gerd
________________________________________
Von: mkgmap-dev
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>
im Auftrag von Felix Hartmann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>
Gesendet: Sonntag, 12. September 2021 19:10
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk
if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling
multiple times. 130GB minimum per worldwide map update for all geofabrik
extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m
and 20m I had about 2TB of useless writes per weekly map update. I've got rid
of all of them with my uncommenting the line, plus saved about 500GB of writes
by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of
4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting
of tiles added another 5TB of writes - but that could have been fixed easily by
changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>
wrote:
because you need to do it if you want to show contourlines. Now compiling the
worldwide contourlines each time again - would burn SSD as well - besides
taking longer than just compiling the maps. So everyone who offers maps for
many countries as download puts the contourlines separate. If you want to offer
the choice of showing contourlines or not - mkmgap will write the gmap
contourlines once not needed, and once the maps not needed. If you don't want
to offer that option - it's only the contourlines being written without need.
Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m
would be 260GB.
If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need
128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk.
Same for North America extract (but few people offer that and just have Canada
and the US divided into the 4-5 zones). For other maps you will get by with a
15-20GB Ramdisk (which I have resorted to now for all but the windows
installers because I don't want to let the server wait for ages for the single
thread NSIS wrapping up the data and instead start the next country in
parallel). And yes going RAMDISK already using my patch and symlinking those
files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't
try in first place I think). If you want to write out the contourlines as well
besides the additional time - for Asia continent you may then go for a 90GB
Ramdisk minimum if offering windows and gmap installers. In this case
gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts.
For sm
aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot
of work and is prone to break - while I guess it's only 10 lines or so to add
to mkgmap code to have a mode that does not write them out if they are present
- or if you tell on commandline they are present already. For .img files they
aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>
wrote:
Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither
burning SSDs nor doing any excessive writing unless you call it multiple times
for the same input files. So the question is: Why do you do that?
Gerd
________________________________________
Von: mkgmap-dev
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>
im Auftrag von Felix Hartmann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>
Gesendet: Freitag, 10. September 2021 00:02
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk
if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are
not causing any actual excessive write.
As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the
thing that takes so much time - is the address search - and that is always the
same. The differences are tiny (just because MapInstall is crashing when files
are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go.
So family-name / family-name1 / family-name2
description / description1 / description2
input input1 input2
family-name..
show-profiles
overview-mapname
product-id
(and maybe I missed some options are those that would need to be given for each
set of input tiles. And then just an option where you tell mkgmap files
starting with which first 4 numbers are relevant for address search. No need to
analyze if those other supplied .img (e.g. buildings or contourlines) need to
be added to address search.). I know coded around the problem of the gmap files
causing excessive writes. But yeah that is actually really complicated be it on
windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>>
wrote:
Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less
than 12 processes 64GB ram available). But that is not enough for Asia
continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit
is the gmap files not needeed to be written.... Also would save quite some CPU
time.
On Wed, 8 Sep 2021, 14:31 ael
<[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>>
wrote:
On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick.
BUT it is not needed if you have the files and even more - it burns your
NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd
write cycles, even without contours and such. It seems to be profligate
in its use of disk cycles. I did try using RAM disk, but even with 16GB
on a laptop, that was soon exhausted.
ael
_______________________________________________
mkgmap-dev mailing list
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
--
Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________
mkgmap-dev mailing list
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
--
Felix Hartman - Openmtbmap.org & VeloMap.org
--
Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________
mkgmap-dev mailing list
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
--
Felix Hartman - Openmtbmap.org & VeloMap.org
--
Felix Hartman - Openmtbmap.org & VeloMap.org
--
Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________
mkgmap-dev mailing list
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
--
Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________
mkgmap-dev mailing list
[email protected]<mailto:[email protected]>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
--
Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________
mkgmap-dev mailing list
[email protected]<mailto:[email protected]>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
--
Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________
mkgmap-dev mailing list
[email protected]
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
