Re: [mkgmap-dev] mkgmap doing excessive writing

[Felix Hartmann]

I just answered in the old topic - but it would make it very complicated if
you had some tiles that you need to split again. I feel --gmapi converts
everything - while --gmapi-minimal only converts input given in
.o5m/osm.pbf/osm is the much better approach. The first time you use
--gmapi - all subsequent times you use --gmapi-minimal. Otherwise it would
not only need to be fine with asterisk, but also like reuse=<4134.img
Sorry but I really don't see why any .img files should be converted again.
Do it the first time you generate them - or just use --gmapi. I do not see
the workflow where a reuse= whatever makes sense. While any o5m / osm /
osm.pbf input would be strange to already exist as gmapi format but not as
img format. Even more as mkgmap so far can use .img as input, but cannot
use gmapi folders as input. If gmapi was possible as input as well - then
your reuse list would make sense. But I do not have a use case for gmapi
input.
And if gmapi input was possible  then the correct approach would be to
write the input like this:
123400??.img gmapi/123400?? 1234002?.o5m
In that case as we passed the map in img and gmapi format up to 19 as input
- those shall not be written either as .img nor as gmapi.
However then
gmapi/123400?? 1234002?.05m would need to write out all .img files
(overwrite those already present) and write the o5m files in both gmapi and
img format. Mkgmap would then need to be able to convert into both
directions and people who only need gmapi files would not need the .img
files. So the call here should be - are there any people using mkgmap
consistently and only need gmapi but not .img output? But I think that
would be way too much work. Because so far mkgmap is simply converting to
gmapi - not writing gmapi in first place from input data without writing
.img files.

On Mon, 13 Sept 2021 at 19:46, Gerd Petermann <
gpetermann_muenc...@hotmail.com> wrote:

> Hi Thomas,
>
> yes, I also thought about something like that. I still have to learn some
> details of the gmapi format and the generated files and dependencies
> between them.
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von
> Thomas Morgenstern 
> Gesendet: Montag, 13. September 2021 18:01
> An: Development list for mkgmap
> Betreff: [mkgmap-dev] mkgmap doing excessive writing
>
> I suggest the new option should be named reuse= files, wildcards enabled>. exampel  reuse=4001.img, 40005*.img. If this
> option is used, then mkgmap should not overwrite the listed folders in the
> gmap. Naturally mkgmap should write a new tdb, preview.img, mdr and
> idx. In most cases the reuse folders contains Contourlines or other static
> content.
>
> Thomas
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>


-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

[Felix Hartmann]

.img files are always reused and never rewritten - it's only about the
gmapi files (a gmapsupp.img has to be generated in one go anyhow - so it is
not in question here). So I do not see the sense of making it more
complicated? It would be fine with me too that way - but I think it's
simpler to just assume all .img files already have the converted files -
because you could do it when generating those .img files. The problem with
your approach is - that when you have a long list of tiles that were
crashing due to too little maxnodes - then regenerating it - will make it
very complicated. It's much easier to assume that all .img files are
already converted. All other input not. If you need to convert .img files
too - then you just use the normal --gmapi option instead.

On Mon, 13 Sept 2021 at 18:58, Thomas Morgenstern 
wrote:

> I suggest the new option should be named *reuse= files, wildcards enabled>. *exampel  *reuse=4001.img, 40005*.img. *If
> this option is used, then mkgmap should not overwrite the listed folders
> in the gmap. Naturally mkgmap should write a new tdb, preview.img,
> mdr and idx. In most cases the *reuse folders *contains Contourlines or
> other static content.
>
> Thomas
>
> Am 13.09.2021 um 15:38 schrieb Felix Hartmann:
>
> Hi Gerd, yes exactly. I have a large collection of .img files of
> contourlines. When I realized that I trash my NVME disk very fast if I
> continue the way I did I got into thinking and analysing what happens. It
> was clear that it all comes down to the --gmapi option. I had believed that
> if I run several passes with .o5m and .img files as input - only the .o5m
> input is converted into new .gmapi folders - while the existing ones are
> not overwritten. Then I checked the created date/last modified date
> (because windows has in my eyes a bug that if you replace a file with a
> near identical file - it just shows a new modified date - but keeps the
> original created date - even though it was a full overwrite).
>
> That got me thinking that in order to save writes - I have to find a way
> to not only not recalculate the .img files - but also create a static set
> of .gmapi folders. Those I just mklink into the directory name that will be
> used on the next run of mkgmap.jar - I managed to do this by uncommenting
> this one line. Because I noticed that the symlinked (by mklink) files are
> not rewritten I changed my scripts to move them away and symlink them back.
> Then at the end delete all symlinks - and move the files back (or to the
> location that I will use for compressing). This step is a bit stupid if
> mkgmap could just have a --gmapi-minimal mode in which only those files are
> converted - that are also written out as .img files (if given --tdb-file
> option).
>
> I know that many people keep a static set of contourlines .img files (also
> containing DEM). You just add the show-profiles=1 option in case you
> include contourlines - and leave it out if not. But actually it does not
> matter if the contourlines files contain DEM or not.
> *I think the easiest way is the principle - --gmapi-minimal only converts
> those files, it would write out as .img files if --tdb-files option were
> given (or is given). --gmapi on the other hand should convert the all input
> files to .gmapi format.* Then mgkmap does not even need to test if those
> files are there or not. This not only saves a lot of writes - but also a
> lot of compile time.
>
> Because essentially if you only provide .img input files (including of
> course the ovm-img for the overview map if you want) you only create a new
> set of mapset files. The exception to this is the toolchain in which when a
> tile failed to compile - you resplit the input files - and parse them again
> with the same arguments so you have input of new o5m files and old .img
> files. But the principle stays the same. If on the rerun of the failed
> tiles now newy split with higher map-id you give --gmapi-minimal and
> tdb-file - only those new tiles are written - while the old .img and old
> ovm.img are supplied to create the correct mapset files. With this
> procedure you don't even need to put a symlink for the contourlines into
> the gmap folder. As the input data to create the contourlines rarely change
> - you can offer the contourlines as a separate download.
> Makes it much easier and faster.
>
> On the map compilation server you then do not even need to have a copy of
> the .gmapi contourlines files. You just need the new input data and the
> static contourlines .img files. Thomas Morgenstern for example had also not
> realized that he is writing the contourlines .gmapi files each time without
> any need. I think many/most providers of garmin maps who offer many
> regions/worldwide coverage put the contourlines separate. It's just a huge
> amount of compile time if you merge the contourlines in o5m format with the
> map data in o5m data each time before running mkgmap. The only actual
> advantage of this being 

Re: [mkgmap-dev] mkgmap doing excessive writing

[Gerd Petermann]

Hi Thomas,

yes, I also thought about something like that. I still have to learn some 
details of the gmapi format and the generated files and dependencies between 
them.

Gerd


Von: mkgmap-dev  im Auftrag von Thomas 
Morgenstern 
Gesendet: Montag, 13. September 2021 18:01
An: Development list for mkgmap
Betreff: [mkgmap-dev] mkgmap doing excessive writing

I suggest the new option should be named reuse=. exampel  reuse=4001.img, 40005*.img. If this option is 
used, then mkgmap should not overwrite the listed folders in the gmap. 
Naturally mkgmap should write a new tdb, preview.img, mdr and idx. In most 
cases the reuse folders contains Contourlines or other static content.

Thomas
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

[mkgmap-dev] mkgmap doing excessive writing

[Thomas Morgenstern]

I suggest the new option should be named /reuse=files, wildcards enabled>. /exampel /reuse=4001.img, 40005*.img. /If 
this option is used, then mkgmap should not overwrite the listed folders 
in the gmap. Naturally mkgmap should writea new tdb, preview.img, 
mdr and idx. In most cases the /reuse folders /contains Contourlines or 
other static content.


Thomas
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

[Felix Hartmann]

Hi Gerd, yes exactly. I have a large collection of .img files of
contourlines. When I realized that I trash my NVME disk very fast if I
continue the way I did I got into thinking and analysing what happens. It
was clear that it all comes down to the --gmapi option. I had believed that
if I run several passes with .o5m and .img files as input - only the .o5m
input is converted into new .gmapi folders - while the existing ones are
not overwritten. Then I checked the created date/last modified date
(because windows has in my eyes a bug that if you replace a file with a
near identical file - it just shows a new modified date - but keeps the
original created date - even though it was a full overwrite).

That got me thinking that in order to save writes - I have to find a way to
not only not recalculate the .img files - but also create a static set of
.gmapi folders. Those I just mklink into the directory name that will be
used on the next run of mkgmap.jar - I managed to do this by uncommenting
this one line. Because I noticed that the symlinked (by mklink) files are
not rewritten I changed my scripts to move them away and symlink them back.
Then at the end delete all symlinks - and move the files back (or to the
location that I will use for compressing). This step is a bit stupid if
mkgmap could just have a --gmapi-minimal mode in which only those files are
converted - that are also written out as .img files (if given --tdb-file
option).

I know that many people keep a static set of contourlines .img files (also
containing DEM). You just add the show-profiles=1 option in case you
include contourlines - and leave it out if not. But actually it does not
matter if the contourlines files contain DEM or not.
*I think the easiest way is the principle - --gmapi-minimal only converts
those files, it would write out as .img files if --tdb-files option were
given (or is given). --gmapi on the other hand should convert the all input
files to .gmapi format.* Then mgkmap does not even need to test if those
files are there or not. This not only saves a lot of writes - but also a
lot of compile time.

Because essentially if you only provide .img input files (including of
course the ovm-img for the overview map if you want) you only create a new
set of mapset files. The exception to this is the toolchain in which when a
tile failed to compile - you resplit the input files - and parse them again
with the same arguments so you have input of new o5m files and old .img
files. But the principle stays the same. If on the rerun of the failed
tiles now newy split with higher map-id you give --gmapi-minimal and
tdb-file - only those new tiles are written - while the old .img and old
ovm.img are supplied to create the correct mapset files. With this
procedure you don't even need to put a symlink for the contourlines into
the gmap folder. As the input data to create the contourlines rarely change
- you can offer the contourlines as a separate download.
Makes it much easier and faster.

On the map compilation server you then do not even need to have a copy of
the .gmapi contourlines files. You just need the new input data and the
static contourlines .img files. Thomas Morgenstern for example had also not
realized that he is writing the contourlines .gmapi files each time without
any need. I think many/most providers of garmin maps who offer many
regions/worldwide coverage put the contourlines separate. It's just a huge
amount of compile time if you merge the contourlines in o5m format with the
map data in o5m data each time before running mkgmap. The only actual
advantage of this being that the contourlines do not overlap roads (as they
are supplied as transparent .img files in another layer) - AND that when
people select maps with a single click instead of drag in
MapInstall/Mapsource they get all maps they think they get (though there is
a different shading - each layer increases the darkness of the shading for
selected). And of course that the very old Mapsource still has problems
correctly showing the contourlines if they are in a separate layer. However
nearly everyone moved on to Basecamp which is fully compatible with layered
maps. The advantage of contourlines as separate layer is for the user he
can switch them on / off independant of the maps and does not need to
download and transfer them each time. So I think for most the advantages
heavily outweigh the disadvantages - hence contourlines into a separate
transparent layer. Same can be done of course for other things like
buildings or vegetation - though the advantage here is much less on the
compile side (while worldwide 10m coontourlines are 200GB of data - the
same extracts are only 15GB in data of buildings - if buildings are shown
only at resolution 24)

On Mon, 13 Sept 2021 at 14:24, Gerd Petermann <
gpetermann_muenc...@hotmail.com> wrote:

> Hi Felix,
>
> I have no clue how exactly your scripts work, how you manage to reuse
> *.img files and so on. Also, I want to find a

Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

[Gerd Petermann]

Hi Felix,

I have no clue how exactly your scripts work, how you manage to reuse *.img 
files and so on. Also, I want to find a solution that works for all users, not 
just you.

So, I expect that you have one step that compiles frequently changing OSM data 
and other steps which are used to compile static data like contourlines or DEM. 
I don't know if you do the latter for each country / continent or once for 
planet and I don't care as long as it works for you.
My understanding is that you have a large collection of *.img files at some 
stage and that you run mkgmap multiple times with different combinations of 
those *.img files as input to produce different zip-files with gmapi format or 
gmapsupp format.
I think that's the normal way to do it, so I try to support that way.

Gerd


Von: mkgmap-dev  im Auftrag von Felix 
Hartmann 
Gesendet: Montag, 13. September 2021 12:28
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk 
if used with --gmapi option

I move away the info.xml - and use a different name for the mapset files and 
then just have mkgmap any file that is input in osm.pbf / osm / o5m format. 
Gmapi-minimal should not convert any file that is supplied as .img - as it can 
be assumed that those exist already (if they do not - then create them with 
--gmapi). That is in my opinion the best approach. So mkgmap does not even try 
to convert them.

Afterwards I distribute a gmapi folder that includes all the data - and by 
replacing the info.xml you can enable or disable contourlines. For big 
countries the contourlines would be a separate download anyhow - so the user 
only needs to download the maps (including mapset files) but not redownload the 
contourlines.  Same principle as in offering the contourlines as a separate 
gmapsupp.img file. so you have maps.img contourlines10m.img contourlines20m.img 
buildings.img 


On Mon, 13 Sept 2021 at 13:17, Gerd Petermann 
mailto:gpetermann_muenc...@hotmail.com>> wrote:
Hi Felix,

sorry, the Data Deduplication as implemented in Windows Server would not help 
here. It works after data was written.

And yes, files which are not just copies of the *.img are written as before. My 
understanding is that you have different product directories in the gmapi 
folder and that you write protect those files which should be kept.
If you have a script to zip the gmapi directory you have to exclude the 
unwanted folders.
Didn't try it. Does it make sense?

Gerd


Von: mkgmap-dev 
mailto:mkgmap-dev-boun...@lists.mkgmap.org.uk>>
 im Auftrag von Felix Hartmann 
mailto:extremecar...@gmail.com>>
Gesendet: Montag, 13. September 2021 12:09
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk 
if used with --gmapi option

Oh - but data was certainly written - A rename will not show as data written in 
both Task manager on Windows, as well as in the smart data (I'm using Windows 
10 pro not Windows server however - maybe that functionality is limited to 
windows server?)

On Mon, 13 Sept 2021 at 13:06, Felix Hartmann 
mailto:extremecar...@gmail.com>>>
 wrote:
Not sure if it makes it possible to use read only attribute instead of moving 
and mklink. Maybe yes - because that was not possible before. So it then would 
be set read only - instead of of move & mklink - and at the end remove read 
only instead of moving back.

On Mon, 13 Sept 2021 at 12:59, Felix Hartmann 
mailto:extremecar...@gmail.com>>>
 wrote:
Thanks Gerd,

but that is just removing the warning if it cannot overwrite, correct?
If it can overwrite the gmap file it will still overwrite it as I see.. So in 
essence just a bit more intelligent then my disabling that line.

I think it should not overwrite at all if present and --x-gmapi-minimal (then 
you don't have to move away the files and link them back to the original 
folder).

On Mon, 13 Sept 2021 at 10:43, Gerd Petermann 
mailto:gpetermann_muenc...@hotmail.com>>>
 wrote:
Hi all,

attached is a quick patch that implements experimaltal option --x-gmapi-minimal.

If used instead of --gmapi mkgmap will not fail if a write-protected output 
file exists in the gmapi output folder and mkgmap copies data from *.img. It 
should still crash when other files like Info.xml are written.

BTW: no conversion is done when those files are written. I think mkgmap simply 
copies data from sub files in *.img into single files. So, the same sequence of 
Bytes exists two or more times on the Computer. Windows Server seems to support 
automatic data deduplication (1). I am sure Linux offers similar support. No 
idea how effective or reliable it is, but it might be worth trying.

G

Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

[Felix Hartmann]

I move away the info.xml - and use a different name for the mapset files
and then just have mkgmap any file that is input in osm.pbf / osm / o5m
format. Gmapi-minimal should not convert any file that is supplied as .img
- as it can be assumed that those exist already (if they do not - then
create them with --gmapi). That is in my opinion the best approach. So
mkgmap does not even try to convert them.

Afterwards I distribute a gmapi folder that includes all the data - and by
replacing the info.xml you can enable or disable contourlines. For big
countries the contourlines would be a separate download anyhow - so the
user only needs to download the maps (including mapset files) but not
redownload the contourlines.  Same principle as in offering the
contourlines as a separate gmapsupp.img file. so you have maps.img
contourlines10m.img contourlines20m.img buildings.img 


On Mon, 13 Sept 2021 at 13:17, Gerd Petermann <
gpetermann_muenc...@hotmail.com> wrote:

> Hi Felix,
>
> sorry, the Data Deduplication as implemented in Windows Server would not
> help here. It works after data was written.
>
> And yes, files which are not just copies of the *.img are written as
> before. My understanding is that you have different product directories in
> the gmapi folder and that you write protect those files which should be
> kept.
> If you have a script to zip the gmapi directory you have to exclude the
> unwanted folders.
> Didn't try it. Does it make sense?
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von
> Felix Hartmann 
> Gesendet: Montag, 13. September 2021 12:09
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to
> disk if used with --gmapi option
>
> Oh - but data was certainly written - A rename will not show as data
> written in both Task manager on Windows, as well as in the smart data (I'm
> using Windows 10 pro not Windows server however - maybe that functionality
> is limited to windows server?)
>
> On Mon, 13 Sept 2021 at 13:06, Felix Hartmann  > wrote:
> Not sure if it makes it possible to use read only attribute instead of
> moving and mklink. Maybe yes - because that was not possible before. So it
> then would be set read only - instead of of move & mklink - and at the end
> remove read only instead of moving back.
>
> On Mon, 13 Sept 2021 at 12:59, Felix Hartmann  > wrote:
> Thanks Gerd,
>
> but that is just removing the warning if it cannot overwrite, correct?
> If it can overwrite the gmap file it will still overwrite it as I see.. So
> in essence just a bit more intelligent then my disabling that line.
>
> I think it should not overwrite at all if present and --x-gmapi-minimal
> (then you don't have to move away the files and link them back to the
> original folder).
>
> On Mon, 13 Sept 2021 at 10:43, Gerd Petermann <
> gpetermann_muenc...@hotmail.com>
> wrote:
> Hi all,
>
> attached is a quick patch that implements experimaltal option
> --x-gmapi-minimal.
>
> If used instead of --gmapi mkgmap will not fail if a write-protected
> output file exists in the gmapi output folder and mkgmap copies data from
> *.img. It should still crash when other files like Info.xml are written.
>
> BTW: no conversion is done when those files are written. I think mkgmap
> simply copies data from sub files in *.img into single files. So, the same
> sequence of Bytes exists two or more times on the Computer. Windows Server
> seems to support automatic data deduplication (1). I am sure Linux offers
> similar support. No idea how effective or reliable it is, but it might be
> worth trying.
>
> Gerd
> (1)
>
> https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable
>
> 
> Von: mkgmap-dev  mkgmap-dev-boun...@lists.mkgmap.org.uk>> im Auftrag von Carlos Dávila <
> car...@alternativaslibres.org>
> Gesendet: Sonntag, 12. September 2021 22:45
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to
> disk if used with --gmapi option
>
> I think it's a good idea if mkgmap checks the required files are present.
>
> El 12/9/21 a las 21:16, Gerd Petermann escribió:
> > Hi Felix,
> >
> > so you just want a new option so that mkgmap doesn't fail if it cannot
> overwrite (write-protected) files in the output directory, right? No need
> to verify the content of the exiting file(s)?
> >
> > Gerd
> >
> > 
> > Von: mkgmap-dev  mkgmap-dev-boun...@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <
> extremecar...@gmail.com>
> > Gesendet: Sonntag, 12. September 2021 19:10
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writin

Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

[Gerd Petermann]

Hi Felix,

sorry, the Data Deduplication as implemented in Windows Server would not help 
here. It works after data was written.

And yes, files which are not just copies of the *.img are written as before. My 
understanding is that you have different product directories in the gmapi 
folder and that you write protect those files which should be kept.
If you have a script to zip the gmapi directory you have to exclude the 
unwanted folders.
Didn't try it. Does it make sense?

Gerd


Von: mkgmap-dev  im Auftrag von Felix 
Hartmann 
Gesendet: Montag, 13. September 2021 12:09
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk 
if used with --gmapi option

Oh - but data was certainly written - A rename will not show as data written in 
both Task manager on Windows, as well as in the smart data (I'm using Windows 
10 pro not Windows server however - maybe that functionality is limited to 
windows server?)

On Mon, 13 Sept 2021 at 13:06, Felix Hartmann 
mailto:extremecar...@gmail.com>> wrote:
Not sure if it makes it possible to use read only attribute instead of moving 
and mklink. Maybe yes - because that was not possible before. So it then would 
be set read only - instead of of move & mklink - and at the end remove read 
only instead of moving back.

On Mon, 13 Sept 2021 at 12:59, Felix Hartmann 
mailto:extremecar...@gmail.com>> wrote:
Thanks Gerd,

but that is just removing the warning if it cannot overwrite, correct?
If it can overwrite the gmap file it will still overwrite it as I see.. So in 
essence just a bit more intelligent then my disabling that line.

I think it should not overwrite at all if present and --x-gmapi-minimal (then 
you don't have to move away the files and link them back to the original 
folder).

On Mon, 13 Sept 2021 at 10:43, Gerd Petermann 
mailto:gpetermann_muenc...@hotmail.com>> wrote:
Hi all,

attached is a quick patch that implements experimaltal option --x-gmapi-minimal.

If used instead of --gmapi mkgmap will not fail if a write-protected output 
file exists in the gmapi output folder and mkgmap copies data from *.img. It 
should still crash when other files like Info.xml are written.

BTW: no conversion is done when those files are written. I think mkgmap simply 
copies data from sub files in *.img into single files. So, the same sequence of 
Bytes exists two or more times on the Computer. Windows Server seems to support 
automatic data deduplication (1). I am sure Linux offers similar support. No 
idea how effective or reliable it is, but it might be worth trying.

Gerd
(1)
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable


Von: mkgmap-dev 
mailto:mkgmap-dev-boun...@lists.mkgmap.org.uk>>
 im Auftrag von Carlos Dávila 
mailto:car...@alternativaslibres.org>>
Gesendet: Sonntag, 12. September 2021 22:45
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk 
if used with --gmapi option

I think it's a good idea if mkgmap checks the required files are present.

El 12/9/21 a las 21:16, Gerd Petermann escribió:
> Hi Felix,
>
> so you just want a new option so that mkgmap doesn't fail if it cannot 
> overwrite (write-protected) files in the output directory, right? No need to 
> verify the content of the exiting file(s)?
>
> Gerd
>
> 
> Von: mkgmap-dev 
> mailto:mkgmap-dev-boun...@lists.mkgmap.org.uk>>
>  im Auftrag von Felix Hartmann 
> mailto:extremecar...@gmail.com>>
> Gesendet: Sonntag, 12. September 2021 19:10
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to 
> disk if used with --gmapi option
>
> And well - it is burning SSD for the contourlines currently even if not 
> calling multiple times. 130GB minimum per worldwide map update for all 
> geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple 
> times and 10m and 20m I had about 2TB of useless writes per weekly map 
> update. I've got rid of all of them with my uncommenting the line, plus saved 
> about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in 
> Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. 
> (and yeah the resplitting of tiles added another 5TB of writes - but that 
> could have been fixed easily by changing order of commands too).
>
> On Sun, 12 Sept 2021 at 20:04, Felix Hartmann 
> mailto:extremecar...@gmail.com>>>
>  wrote:
> because you need to do it if you want to show contourlines. Now compiling the 
> worldwide contourlines each time again - would burn SSD as well - besides 
> taking longer than just compiling the maps. So everyone who offers maps for 
> many countries as download puts the contourl

Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

[Felix Hartmann]

Oh - but data was certainly written - A rename will not show as data
written in both Task manager on Windows, as well as in the smart data (I'm
using Windows 10 pro not Windows server however - maybe that functionality
is limited to windows server?)

On Mon, 13 Sept 2021 at 13:06, Felix Hartmann 
wrote:

> Not sure if it makes it possible to use read only attribute instead of
> moving and mklink. Maybe yes - because that was not possible before. So it
> then would be set read only - instead of of move & mklink - and at the end
> remove read only instead of moving back.
>
> On Mon, 13 Sept 2021 at 12:59, Felix Hartmann 
> wrote:
>
>> Thanks Gerd,
>>
>> but that is just removing the warning if it cannot overwrite, correct?
>> If it can overwrite the gmap file it will still overwrite it as I see..
>> So in essence just a bit more intelligent then my disabling that line.
>>
>> I think it should not overwrite at all if present and --x-gmapi-minimal
>> (then you don't have to move away the files and link them back to the
>> original folder).
>>
>> On Mon, 13 Sept 2021 at 10:43, Gerd Petermann <
>> gpetermann_muenc...@hotmail.com> wrote:
>>
>>> Hi all,
>>>
>>> attached is a quick patch that implements experimaltal option
>>> --x-gmapi-minimal.
>>>
>>> If used instead of --gmapi mkgmap will not fail if a write-protected
>>> output file exists in the gmapi output folder and mkgmap copies data from
>>> *.img. It should still crash when other files like Info.xml are written.
>>>
>>> BTW: no conversion is done when those files are written. I think mkgmap
>>> simply copies data from sub files in *.img into single files. So, the same
>>> sequence of Bytes exists two or more times on the Computer. Windows Server
>>> seems to support automatic data deduplication (1). I am sure Linux offers
>>> similar support. No idea how effective or reliable it is, but it might be
>>> worth trying.
>>>
>>> Gerd
>>> (1)
>>>
>>> https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable
>>>
>>> 
>>> Von: mkgmap-dev  im Auftrag von
>>> Carlos Dávila 
>>> Gesendet: Sonntag, 12. September 2021 22:45
>>> An: mkgmap-dev@lists.mkgmap.org.uk
>>> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
>>> to disk if used with --gmapi option
>>>
>>> I think it's a good idea if mkgmap checks the required files are present.
>>>
>>> El 12/9/21 a las 21:16, Gerd Petermann escribió:
>>> > Hi Felix,
>>> >
>>> > so you just want a new option so that mkgmap doesn't fail if it cannot
>>> overwrite (write-protected) files in the output directory, right? No need
>>> to verify the content of the exiting file(s)?
>>> >
>>> > Gerd
>>> >
>>> > 
>>> > Von: mkgmap-dev  im Auftrag
>>> von Felix Hartmann 
>>> > Gesendet: Sonntag, 12. September 2021 19:10
>>> > An: Development list for mkgmap
>>> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and
>>> converting to disk if used with --gmapi option
>>> >
>>> > And well - it is burning SSD for the contourlines currently even if
>>> not calling multiple times. 130GB minimum per worldwide map update for all
>>> geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple
>>> times and 10m and 20m I had about 2TB of useless writes per weekly map
>>> update. I've got rid of all of them with my uncommenting the line, plus
>>> saved about 500GB of writes by now doing all the gmapsupp.img and gmap
>>> stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm
>>> down to 1TB.. (and yeah the resplitting of tiles added another 5TB of
>>> writes - but that could have been fixed easily by changing order of
>>> commands too).
>>> >
>>> > On Sun, 12 Sept 2021 at 20:04, Felix Hartmann >> > wrote:
>>> > because you need to do it if you want to show contourlines. Now
>>> compiling the worldwide contourlines each time again - would burn SSD as
>>> well - besides taking longer than just compiling the maps. So everyone who
>>> offers maps for many countries as download puts the contourlines separate.
>>> If you want to offer the choice of showing contourlines or not - mkmgap
>>> will write the gmap contourlines once not needed, and once the maps not
>>> needed. If you don't want to offer that option - it's only the contourlines
>>> being written without need. Contourlines for all geofabrik extracts 20m
>>> equidistance are about 130GB, 10m would be 260GB.
>>> > If you offer 10m and 20m it will be even more not needed writes.
>>> >
>>> > The only solution is to go for a Ramdisk instead - However you likely
>>> will need 128GB of RAM to do that - because for Asia or Europe you need a
>>> 64GB Ramdisk. Same for North America extract (but few people offer that and
>>> just have Canada and the US divided into the 4-5 zones). For other maps you
>>> will get by with a 15-20GB Ramdisk (which I have resorted to now for all
>>> but the windows installe

Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

[Felix Hartmann]

Not sure if it makes it possible to use read only attribute instead of
moving and mklink. Maybe yes - because that was not possible before. So it
then would be set read only - instead of of move & mklink - and at the end
remove read only instead of moving back.

On Mon, 13 Sept 2021 at 12:59, Felix Hartmann 
wrote:

> Thanks Gerd,
>
> but that is just removing the warning if it cannot overwrite, correct?
> If it can overwrite the gmap file it will still overwrite it as I see.. So
> in essence just a bit more intelligent then my disabling that line.
>
> I think it should not overwrite at all if present and --x-gmapi-minimal
> (then you don't have to move away the files and link them back to the
> original folder).
>
> On Mon, 13 Sept 2021 at 10:43, Gerd Petermann <
> gpetermann_muenc...@hotmail.com> wrote:
>
>> Hi all,
>>
>> attached is a quick patch that implements experimaltal option
>> --x-gmapi-minimal.
>>
>> If used instead of --gmapi mkgmap will not fail if a write-protected
>> output file exists in the gmapi output folder and mkgmap copies data from
>> *.img. It should still crash when other files like Info.xml are written.
>>
>> BTW: no conversion is done when those files are written. I think mkgmap
>> simply copies data from sub files in *.img into single files. So, the same
>> sequence of Bytes exists two or more times on the Computer. Windows Server
>> seems to support automatic data deduplication (1). I am sure Linux offers
>> similar support. No idea how effective or reliable it is, but it might be
>> worth trying.
>>
>> Gerd
>> (1)
>>
>> https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable
>>
>> 
>> Von: mkgmap-dev  im Auftrag von
>> Carlos Dávila 
>> Gesendet: Sonntag, 12. September 2021 22:45
>> An: mkgmap-dev@lists.mkgmap.org.uk
>> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
>> to disk if used with --gmapi option
>>
>> I think it's a good idea if mkgmap checks the required files are present.
>>
>> El 12/9/21 a las 21:16, Gerd Petermann escribió:
>> > Hi Felix,
>> >
>> > so you just want a new option so that mkgmap doesn't fail if it cannot
>> overwrite (write-protected) files in the output directory, right? No need
>> to verify the content of the exiting file(s)?
>> >
>> > Gerd
>> >
>> > 
>> > Von: mkgmap-dev  im Auftrag
>> von Felix Hartmann 
>> > Gesendet: Sonntag, 12. September 2021 19:10
>> > An: Development list for mkgmap
>> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
>> to disk if used with --gmapi option
>> >
>> > And well - it is burning SSD for the contourlines currently even if not
>> calling multiple times. 130GB minimum per worldwide map update for all
>> geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple
>> times and 10m and 20m I had about 2TB of useless writes per weekly map
>> update. I've got rid of all of them with my uncommenting the line, plus
>> saved about 500GB of writes by now doing all the gmapsupp.img and gmap
>> stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm
>> down to 1TB.. (and yeah the resplitting of tiles added another 5TB of
>> writes - but that could have been fixed easily by changing order of
>> commands too).
>> >
>> > On Sun, 12 Sept 2021 at 20:04, Felix Hartmann > > wrote:
>> > because you need to do it if you want to show contourlines. Now
>> compiling the worldwide contourlines each time again - would burn SSD as
>> well - besides taking longer than just compiling the maps. So everyone who
>> offers maps for many countries as download puts the contourlines separate.
>> If you want to offer the choice of showing contourlines or not - mkmgap
>> will write the gmap contourlines once not needed, and once the maps not
>> needed. If you don't want to offer that option - it's only the contourlines
>> being written without need. Contourlines for all geofabrik extracts 20m
>> equidistance are about 130GB, 10m would be 260GB.
>> > If you offer 10m and 20m it will be even more not needed writes.
>> >
>> > The only solution is to go for a Ramdisk instead - However you likely
>> will need 128GB of RAM to do that - because for Asia or Europe you need a
>> 64GB Ramdisk. Same for North America extract (but few people offer that and
>> just have Canada and the US divided into the 4-5 zones). For other maps you
>> will get by with a 15-20GB Ramdisk (which I have resorted to now for all
>> but the windows installers because I don't want to let the server wait for
>> ages for the single thread NSIS wrapping up the data and instead start the
>> next country in parallel). And yes going RAMDISK already using my patch and
>> symlinking those files so mkgmap cannot overwrite them (would still be
>> faster if mkgmap didn't try in first place I think). If you want to write
>> out the contourlines as well besides the addition

Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

[Felix Hartmann]

Thanks Gerd,

but that is just removing the warning if it cannot overwrite, correct?
If it can overwrite the gmap file it will still overwrite it as I see.. So
in essence just a bit more intelligent then my disabling that line.

I think it should not overwrite at all if present and --x-gmapi-minimal
(then you don't have to move away the files and link them back to the
original folder).

On Mon, 13 Sept 2021 at 10:43, Gerd Petermann <
gpetermann_muenc...@hotmail.com> wrote:

> Hi all,
>
> attached is a quick patch that implements experimaltal option
> --x-gmapi-minimal.
>
> If used instead of --gmapi mkgmap will not fail if a write-protected
> output file exists in the gmapi output folder and mkgmap copies data from
> *.img. It should still crash when other files like Info.xml are written.
>
> BTW: no conversion is done when those files are written. I think mkgmap
> simply copies data from sub files in *.img into single files. So, the same
> sequence of Bytes exists two or more times on the Computer. Windows Server
> seems to support automatic data deduplication (1). I am sure Linux offers
> similar support. No idea how effective or reliable it is, but it might be
> worth trying.
>
> Gerd
> (1)
>
> https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable
>
> 
> Von: mkgmap-dev  im Auftrag von
> Carlos Dávila 
> Gesendet: Sonntag, 12. September 2021 22:45
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to
> disk if used with --gmapi option
>
> I think it's a good idea if mkgmap checks the required files are present.
>
> El 12/9/21 a las 21:16, Gerd Petermann escribió:
> > Hi Felix,
> >
> > so you just want a new option so that mkgmap doesn't fail if it cannot
> overwrite (write-protected) files in the output directory, right? No need
> to verify the content of the exiting file(s)?
> >
> > Gerd
> >
> > 
> > Von: mkgmap-dev  im Auftrag von
> Felix Hartmann 
> > Gesendet: Sonntag, 12. September 2021 19:10
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting
> to disk if used with --gmapi option
> >
> > And well - it is burning SSD for the contourlines currently even if not
> calling multiple times. 130GB minimum per worldwide map update for all
> geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple
> times and 10m and 20m I had about 2TB of useless writes per weekly map
> update. I've got rid of all of them with my uncommenting the line, plus
> saved about 500GB of writes by now doing all the gmapsupp.img and gmap
> stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm
> down to 1TB.. (and yeah the resplitting of tiles added another 5TB of
> writes - but that could have been fixed easily by changing order of
> commands too).
> >
> > On Sun, 12 Sept 2021 at 20:04, Felix Hartmann  > wrote:
> > because you need to do it if you want to show contourlines. Now
> compiling the worldwide contourlines each time again - would burn SSD as
> well - besides taking longer than just compiling the maps. So everyone who
> offers maps for many countries as download puts the contourlines separate.
> If you want to offer the choice of showing contourlines or not - mkmgap
> will write the gmap contourlines once not needed, and once the maps not
> needed. If you don't want to offer that option - it's only the contourlines
> being written without need. Contourlines for all geofabrik extracts 20m
> equidistance are about 130GB, 10m would be 260GB.
> > If you offer 10m and 20m it will be even more not needed writes.
> >
> > The only solution is to go for a Ramdisk instead - However you likely
> will need 128GB of RAM to do that - because for Asia or Europe you need a
> 64GB Ramdisk. Same for North America extract (but few people offer that and
> just have Canada and the US divided into the 4-5 zones). For other maps you
> will get by with a 15-20GB Ramdisk (which I have resorted to now for all
> but the windows installers because I don't want to let the server wait for
> ages for the single thread NSIS wrapping up the data and instead start the
> next country in parallel). And yes going RAMDISK already using my patch and
> symlinking those files so mkgmap cannot overwrite them (would still be
> faster if mkgmap didn't try in first place I think). If you want to write
> out the contourlines as well besides the additional time - for Asia
> continent you may then go for a 90GB Ramdisk minimum if offering windows
> and gmap installers. In this case gmapsupp.img donwnloads aren't possible
> anyhow due to the huge data amounts. For sm
> >   aller countries I have them too
> >
> > I coded around this now by using symlinks - but yeah that will be quite
> a lot of work and is prone to break - while I guess it's only 10 lines or
> so to a

Re: [mkgmap-dev] How to create an empty overview map (for transparent maps)

[Felix Hartmann]

I think just remove the warning. I could not notice any problem due to it (
and if there would be problems, by now at least one user of my maps should
have complained)

On Mon, 13 Sept 2021 at 12:09, Gerd Petermann <
gpetermann_muenc...@hotmail.com> wrote:

> Hi,
>
> yes, I think that message makes no sense in this situation. I am not sure
> why I decided to log this as an error.
> See https://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=3679
>
> I've attached a patch to suppress the message if the input file is a
> ovm_*.img.
> I could also simply reduce severity to warning or remove it for good.
>
> Gerd
>
>
>
>
> 
> Von: mkgmap-dev  im Auftrag von
> Felix Hartmann 
> Gesendet: Mittwoch, 1. September 2021 12:18
> An: Development list for mkgmap
> Betreff: [mkgmap-dev] How to create an empty overview map (for
> transparent  maps)
>
> If you create a separate map for buildings or contourlines only - usually
> there is no need for an overview map - or better no need for any content in
> the overview map - or am I missing something?
>
> Because those maps sometimes may have very large tiles, but very few
> levels (maybe only 24) - mkgmap creates an overview map at the next
> resolution - and will then often throw warnings such as:
>
> SEVERE (OverviewBuilder): Tile selection (0x4a) polygon for tile 8490
> cannot be written in level 0 resolution 21, using 20 instead
>
>
> There is still a need for an empty overview map - as without it you cannot
> show the map in Basecamp/Mapsource - or transfer it with MapInstall.
> I know the overview map is only empty 0x4a polygons - just wondered if
> there is a better solution...
>
> --overview-levels=
> is not given.
>
> --
> Felix Hartman - Openmtbmap.org & VeloMap.org
>
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev



-- 
Felix Hartman - Openmtbmap.org & VeloMap.org
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] How to create an empty overview map (for transparent maps)

[Gerd Petermann]

Hi,

yes, I think that message makes no sense in this situation. I am not sure why I 
decided to log this as an error.
See https://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=3679

I've attached a patch to suppress the message if the input file is a ovm_*.img.
I could also simply reduce severity to warning or remove it for good.

Gerd





Von: mkgmap-dev  im Auftrag von Felix 
Hartmann 
Gesendet: Mittwoch, 1. September 2021 12:18
An: Development list for mkgmap
Betreff: [mkgmap-dev] How to create an empty overview map (for transparent  
maps)

If you create a separate map for buildings or contourlines only - usually there 
is no need for an overview map - or better no need for any content in the 
overview map - or am I missing something?

Because those maps sometimes may have very large tiles, but very few levels 
(maybe only 24) - mkgmap creates an overview map at the next resolution - and 
will then often throw warnings such as:

SEVERE (OverviewBuilder): Tile selection (0x4a) polygon for tile 8490 
cannot be written in level 0 resolution 21, using 20 instead


There is still a need for an empty overview map - as without it you cannot show 
the map in Basecamp/Mapsource - or transfer it with MapInstall.
I know the overview map is only empty 0x4a polygons - just wondered if there is 
a better solution...

--overview-levels=
is not given.

--
Felix Hartman - Openmtbmap.org & VeloMap.org



no-error-for-0x4a.patch
Description: no-error-for-0x4a.patch
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option

[Gerd Petermann]

Hi all,

attached is a quick patch that implements experimaltal option --x-gmapi-minimal.

If used instead of --gmapi mkgmap will not fail if a write-protected output 
file exists in the gmapi output folder and mkgmap copies data from *.img. It 
should still crash when other files like Info.xml are written.

BTW: no conversion is done when those files are written. I think mkgmap simply 
copies data from sub files in *.img into single files. So, the same sequence of 
Bytes exists two or more times on the Computer. Windows Server seems to support 
automatic data deduplication (1). I am sure Linux offers similar support. No 
idea how effective or reliable it is, but it might be worth trying.

Gerd
(1)
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable


Von: mkgmap-dev  im Auftrag von Carlos 
Dávila 
Gesendet: Sonntag, 12. September 2021 22:45
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk 
if used with --gmapi option

I think it's a good idea if mkgmap checks the required files are present.

El 12/9/21 a las 21:16, Gerd Petermann escribió:
> Hi Felix,
>
> so you just want a new option so that mkgmap doesn't fail if it cannot 
> overwrite (write-protected) files in the output directory, right? No need to 
> verify the content of the exiting file(s)?
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von Felix 
> Hartmann 
> Gesendet: Sonntag, 12. September 2021 19:10
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to 
> disk if used with --gmapi option
>
> And well - it is burning SSD for the contourlines currently even if not 
> calling multiple times. 130GB minimum per worldwide map update for all 
> geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple 
> times and 10m and 20m I had about 2TB of useless writes per weekly map 
> update. I've got rid of all of them with my uncommenting the line, plus saved 
> about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in 
> Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. 
> (and yeah the resplitting of tiles added another 5TB of writes - but that 
> could have been fixed easily by changing order of commands too).
>
> On Sun, 12 Sept 2021 at 20:04, Felix Hartmann 
> mailto:extremecar...@gmail.com>> wrote:
> because you need to do it if you want to show contourlines. Now compiling the 
> worldwide contourlines each time again - would burn SSD as well - besides 
> taking longer than just compiling the maps. So everyone who offers maps for 
> many countries as download puts the contourlines separate. If you want to 
> offer the choice of showing contourlines or not - mkmgap will write the gmap 
> contourlines once not needed, and once the maps not needed. If you don't want 
> to offer that option - it's only the contourlines being written without need. 
> Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m 
> would be 260GB.
> If you offer 10m and 20m it will be even more not needed writes.
>
> The only solution is to go for a Ramdisk instead - However you likely will 
> need 128GB of RAM to do that - because for Asia or Europe you need a 64GB 
> Ramdisk. Same for North America extract (but few people offer that and just 
> have Canada and the US divided into the 4-5 zones). For other maps you will 
> get by with a 15-20GB Ramdisk (which I have resorted to now for all but the 
> windows installers because I don't want to let the server wait for ages for 
> the single thread NSIS wrapping up the data and instead start the next 
> country in parallel). And yes going RAMDISK already using my patch and 
> symlinking those files so mkgmap cannot overwrite them (would still be faster 
> if mkgmap didn't try in first place I think). If you want to write out the 
> contourlines as well besides the additional time - for Asia continent you may 
> then go for a 90GB Ramdisk minimum if offering windows and gmap installers. 
> In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge 
> data amounts. For sm
>   aller countries I have them too
>
> I coded around this now by using symlinks - but yeah that will be quite a lot 
> of work and is prone to break - while I guess it's only 10 lines or so to add 
> to mkgmap code to have a mode that does not write them out if they are 
> present - or if you tell on commandline they are present already. For .img 
> files they aren't overwritten either...
>
> On Sun, 12 Sept 2021 at 18:40, Gerd Petermann 
> mailto:gpetermann_muenc...@hotmail.com>> 
> wrote:
> Hi Felix,
>
> did not read all the posts in detail. I understand that mkgmap is neither 
> burning SSDs nor doing any excessive writing unless you call it multiple 
> times for the same input files. So the question is: Why do you do that