Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Mike,

I've compiled r4107 with your patch:
http://files.mkgmap.org.uk/download/418/mkgmap.jar

I did not yet try it because I am still fighting with the crash in
MapSource.

I am not sure if the usage of classes like
com.sun.management.OperatingSystemMXBean
are supported with openjdk? Will this also work on a Mac?

Gerd




--
Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Henning Scholland]

Hi Gerd,

these are my runtimes of several compilations of my Turkey map, splitted
with 1.500.000 nodes, compiled witout index, gmapi and gmappsupp, but
including DEM and routing etc.

RAM in GB    Threads     Time1 in ms    Time2 in ms
12                    8                328106            323605
8                      8                323135            341242
4                      8                378460            408714
4                      7                364123            361501
4                     6                 349048            343866
4                      4                371786            366581
4                       1                845778            797376
2                   4                527267            509698
2                        1           804117            853805

It seems a lack of RAM slows more down if there are two many jobs. For
4GB Ram, 6 threads are faster than 8 threads, even using only 4 threads
is faster than using 8.

Henning


On 10.02.2018 16:41, Gerd Petermann wrote:
On 10.02.2018 16:41, Gerd Petermann wrote:
> Hi Henning,
>
> thanks for the numbers. I assume that your JRE sets xmx=4G because you 
> probably have 16G.
> My machine has 4 cores and 8G and the JRE sets xmx=2G when I don't specify a 
> value.
> When you compile a large number of tiles peak memory is reached when building 
> the index.
>
> Maybe you can do some more tests with a smaller set of tiles?
> I'd like to know if the run time changes much when you compile e.g. 32 tiles 
> with
> a) -Xmx8GB and --max-jobs=8
> b) -Xmx4GB and --max-jobs=8
> c) -Xmx4GB and --max-jobs=7
> d) -Xmx4GB and --max-jobs=6
>
> I suggest to do each test at least 2 times and start with one unmeasured test 
> to reduce
> IO caching effects. You should also not use --index, --gmapi or --gmapsupp 
> for these tests.
>
> Gerd
>
>
> 
> Von: Henning Scholland 
> Gesendet: Samstag, 10. Februar 2018 07:23
> An: Gerd Petermann; Development list for mkgmap
> Betreff: Re: AW: [mkgmap-dev] max-jobs patch
>
> Hi Gerd,
> Hi Mike,
>
> I gave it a try on all my maps (splitted with max-nodes=1.600.000).
> Removed xmx and max jobs. mkgmap used all 8 logical cores and about 4GB
> of memory. So it seems, Gerd was correct with the 500mb per job. So I
> would suggest to limit max-jobs with 32bit java to 3 jobs and to
> RAM/500mb on 64bit java. The used jobs should be minimum of both CPU and
> RAM.
>
> Regarding to higher amount of RAM I was mentioning before, I was calling
> mkgmap with xmx=12G, maybe java is not freeing the heap in that case. It
> was during dem-branch. Maybe during that time mkgmap wasn't also that
> efficient.
>
> Henning
>

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Mike Baggaley]

Hi Gerd, I have put together the attached patch that should result in a
reasonable value for max-jobs if it is defaulted. This version attempts to
calculate a suitable value by processing one task, seeing how much memory it
used, then dividing the available memory by the memory used to determine the
value before running the remaining tasks. I have tried a variety of values
of -Xmx on my PC, and it seems to work for both a county and the whole of
the UK. I haven't tried it on a continent, as I don't have suitable data. I
can't guarantee that every combination of memory and CPU will work, but if
it does fail, you will still get a sensible error message telling you what
to change. At the end of the run, if insufficient memory is allocated to the
Java heap to allow all the CPU cores to be used, it will suggest a value for
-Xmx. For my maps, it seems to suggest about 400Mb per thread. My build of
the UK without specifying --max-jobs has reduced from about 3 hours to 45
minutes.

I have also made a few further minor improvements to the documentation to
make the form more consistent, and changed the logged execution time from
milliseconds to a more useful hours, minutes and seconds (or milliseconds
when less than 1 second).

Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com] 
Sent: 07 February 2018 19:06
To: Development list for mkgmap 
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

I did not yet try it. My understanding is that the Garbage Collection will
try hard
to avoid an out of heap exception. In other words: A user might see better
throughput
with fewer parallel jobs. What are your experiences? Did you try with
large maps, say > 40 large tiles ?

Gerd


Von: mkgmap-dev  im Auftrag von Mike
Baggaley 
Gesendet: Mittwoch, 7. Februar 2018 01:58
An: 'Development list for mkgmap'
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd, please find attached v2 of the max-jobs patch.

This version additionally catches out of heap memory exceptions and displays
a message suggesting either the use of the Java -Xmx option to increase the
available heap memory or the mkgmap --max-jobs option with a smaller value
to reduce the memory requirement (the latter is suggested only if using more
than one thread).

How does that seem?
Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
Sent: 05 February 2018 07:28
To: Mike Baggaley ; 'Development list for mkgmap'

Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

thought about this again. Maybe this change is too simple. With multiple
jobs one also
needs more heap (-Xmx JRE option). If you create rather large tiles with
splitter (max-nodes=160)
you need 0.5 - 1 GB for each job. Not sure what happens when a user creates
a map with
10 tiles on an 8-core machine without any -Xmx option.

I fear this will result in OutOfMemory exception, so better check the
available heap as well.

Gerd


Von: mkgmap-dev  im Auftrag von Mike
Baggaley 
Gesendet: Sonntag, 4. Februar 2018 15:14
An: 'Development list for mkgmap'
Betreff: [mkgmap-dev] max-jobs patch

Hi Gerd,

Please find attached a patch that amends the default behaviour if the
--max-jobs option is not specified, to using a value equal to the number of
CPU cores, as suggested in a previous post. The documentation is also
amended to reflect the change. This halves the execution time of mkgmap for
building a map of Staffordshire on my 8-core PC when --max-jobs is not
specified (I didn't know about this option previously and was unaware the
performance could have been improved).

Cheers,
Mike




maxjobs-v3.patch
Description: Binary data
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Bernd,

not sure what you want to express here.
When you use --max-jobs or max-jobs=8 with the unpatched version you should see
exactly the same numbers on an 8 core machine, the patch only changes the 
default
for max-jobs.

Gerd


Von: mkgmap-dev  im Auftrag von Bernd 
Weigelt 
Gesendet: Samstag, 10. Februar 2018 10:19:09
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd

the unpatched version uses up to 8.7 GB with the options for mkgmap

I'm using maxnodes=512000 to create an area.list and use this file for every
build.

At the moment i see no problem with the patched version


Bernd



Am Samstag, 10. Februar 2018, 09:56:41 CET schrieb Gerd Petermann:
> Hi Bernd,
>
> okay, I think that confirms my estimate of ~500M per thread.
> See also my previous answer to Henning.
>
> Gerd
>
> 
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

[mkgmap-dev] WG: AW: max-jobs patch

[Gerd Petermann]

Sorry, answered to Henning instead of the list, see below...

Gerd

Von: Gerd Petermann 
Gesendet: Samstag, 10. Februar 2018 09:52
An: Henning Scholland
Betreff: AW: AW: [mkgmap-dev] max-jobs patch

Hi Henning,

splitter uses multiple threads by default, but up to now only during the last 
phase that writes the tiles it can use
more than 2. Not a big problem, most of the work is I/O bound.

The workload in splitter is completely different to mkgmap. Peak memory depends 
on the --keep-complete option and
the max-areas option, not so much on the number of parallel threads. In some 
cases I see slightly better throughput when
I use --max-threads=6 on my 4 core machine, but only a few percent.

The last time when I measured it I was quite happy with the defaults.

Gerd


Von: Henning Scholland 
Gesendet: Samstag, 10. Februar 2018 08:46
An: Gerd Petermann; Development list for mkgmap
Betreff: Re: AW: [mkgmap-dev] max-jobs patch

Hi Gerd,

maybe splitter can be modified  similar.

Henning
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Bernd,

okay, I think that confirms my estimate of ~500M per thread.
See also my previous answer to Henning.

Gerd


Von: mkgmap-dev  im Auftrag von Bernd 
Weigelt 
Gesendet: Samstag, 10. Februar 2018 09:43
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi

tested on my linux box, looks good

Heap is 4.4 GB with 8 threads.

With the unpatched version i'm using -Xmx8G for the heap


Bernd


Am Donnerstag, 8. Februar 2018, 15:34:23 CET schrieb Gerd Petermann:
> binary is here:
> http://files.mkgmap.org.uk/download/415/mkgmap.jar
>
> For those users who already used --max-jobs with or without a paramter
> before there should be no difference.

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Bernd Weigelt]

Hi

tested on my linux box, looks good

Heap is 4.4 GB with 8 threads.

With the unpatched version i'm using -Xmx8G for the heap


Bernd


Am Donnerstag, 8. Februar 2018, 15:34:23 CET schrieb Gerd Petermann:
> binary is here:
> http://files.mkgmap.org.uk/download/415/mkgmap.jar
> 
> For those users who already used --max-jobs with or without a paramter
> before there should be no difference.



signature.asc
Description: This is a digitally signed message part.
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev