Re: [mkgmap-dev] max-jobs patch

[Mike Baggaley]

Hi Gerd, 

I would prefer removing the static modifier as you suggest, but accessing
the variable via mm.maxJobs rather than mm.getMaxJobs(), and having no
getter implementation. However, I am happy with using the getter if you
prefer it.

I think getters and setters in mkgmap have been overused. I would only use
getters and/or setters for a class that wants to allow another class to
access some internal variable, but also wants to prevent it from changing it
directly. If a class implements get and set as direct copies from/to the
internal variable, all it is doing is adding a small overhead each time the
variable is accessed with no obvious benefit. I would primarily use them
where either the get or set does more than just copy (e.g. it needs to
recalculate something when the value is changed) or there is a likelihood
that another class based on it would need to override the interface.
However, my background is C++ and C#, not Java, so I may be missing some
subtlety.

Cheers,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com] 
Sent: 19 February 2018 09:12
To: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

sorry for the late response.
I've just recognized the static modifier in maxjobs. I think it is better to
use a getter instead, see
attached modifed version of the patch. If you agree, I'll commit this
version.

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <m...@tvage.co.uk>
Gesendet: Freitag, 16. Februar 2018 17:18:15
An: 'Development list for mkgmap'
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd, I misread the --max-jobs with no value functionality. Please find
attached a revised patch that does function that way when the max-jobs
option is specified with no value.

Cheers,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
Sent: 16 February 2018 10:42
To: Mike Baggaley <m...@tvage.co.uk>
Subject: AW: [mkgmap-dev] max-jobs patch

Hi Mike,

without the patch the option --max-jos means: use as many threads as cores.
Since mkgmap is normally very silent I would not print an information that
just confirms that anything is normal.

My personal preference would be to print much more information about
performance, but my understanding is that
the mkgmap users are happy with the silent behaviour.

BTW: I'd also print a message that tells the user how to activate logging if
it is not already activated.

Gerd


Von: Mike Baggaley <m...@tvage.co.uk>
Gesendet: Freitag, 16. Februar 2018 11:20:03
An: 'Gerd Petermann'
Betreff: RE: [mkgmap-dev] max-jobs patch

Hi Gerd, as the functionality of --max-jobs without a value is the same as
not specifying the parameter (i.e. it is the default), it seems to me to be
better for them to function the same. If you have specified --max-jobs
without a value, why would you not want to know how many threads had been
assigned?

Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
Sent: 15 February 2018 08:54
To: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

looks good to me. I am just not sure about the new stdout message
"Setting max-jobs to 4"
It appears when --max-jobs is not used (ok), but also when it is used
without a parameter.
In  the latter case I would only print this message if the number of CPU
cores is higher.

Gerd



Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Gerd
Petermann <gpetermann_muenc...@hotmail.com>
Gesendet: Mittwoch, 14. Februar 2018 19:38:06
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

okay, I'll try it tomorrow.

Gerd



Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <m...@tvage.co.uk>
Gesendet: Mittwoch, 14. Februar 2018 18:30:28
An: 'Steve Ratcliffe'; mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] max-jobs patch

I tried renaming the java library to force a failure and got a java error
"Error occurred during initialization of VM" before any mkgmap code was
executed, so to be on the safe side, I have removed that code and amended
the message in the attached patch.

Regards,
Mike

-Original Message-
From: Steve Ratcliffe [mailto:st...@parabola.me.uk]
Sent: 13 February 2018 23:01
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike

> I could take out that bit of code, which is just determining how
> much physical memory is installed, so that mkgmap won't suggest that the
> user increases the available heap too much. It isn't used in determin

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Mike,

sorry for the late response.
I've just recognized the static modifier in maxjobs. I think it is better to 
use a getter instead, see
attached modifed version of the patch. If you agree, I'll commit this version.

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike 
Baggaley <m...@tvage.co.uk>
Gesendet: Freitag, 16. Februar 2018 17:18:15
An: 'Development list for mkgmap'
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd, I misread the --max-jobs with no value functionality. Please find
attached a revised patch that does function that way when the max-jobs
option is specified with no value.

Cheers,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
Sent: 16 February 2018 10:42
To: Mike Baggaley <m...@tvage.co.uk>
Subject: AW: [mkgmap-dev] max-jobs patch

Hi Mike,

without the patch the option --max-jos means: use as many threads as cores.
Since mkgmap is normally very silent I would not print an information that
just confirms that anything is normal.

My personal preference would be to print much more information about
performance, but my understanding is that
the mkgmap users are happy with the silent behaviour.

BTW: I'd also print a message that tells the user how to activate logging if
it is not already activated.

Gerd


Von: Mike Baggaley <m...@tvage.co.uk>
Gesendet: Freitag, 16. Februar 2018 11:20:03
An: 'Gerd Petermann'
Betreff: RE: [mkgmap-dev] max-jobs patch

Hi Gerd, as the functionality of --max-jobs without a value is the same as
not specifying the parameter (i.e. it is the default), it seems to me to be
better for them to function the same. If you have specified --max-jobs
without a value, why would you not want to know how many threads had been
assigned?

Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
Sent: 15 February 2018 08:54
To: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

looks good to me. I am just not sure about the new stdout message
"Setting max-jobs to 4"
It appears when --max-jobs is not used (ok), but also when it is used
without a parameter.
In  the latter case I would only print this message if the number of CPU
cores is higher.

Gerd



Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Gerd
Petermann <gpetermann_muenc...@hotmail.com>
Gesendet: Mittwoch, 14. Februar 2018 19:38:06
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

okay, I'll try it tomorrow.

Gerd



Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <m...@tvage.co.uk>
Gesendet: Mittwoch, 14. Februar 2018 18:30:28
An: 'Steve Ratcliffe'; mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] max-jobs patch

I tried renaming the java library to force a failure and got a java error
"Error occurred during initialization of VM" before any mkgmap code was
executed, so to be on the safe side, I have removed that code and amended
the message in the attached patch.

Regards,
Mike

-Original Message-
From: Steve Ratcliffe [mailto:st...@parabola.me.uk]
Sent: 13 February 2018 23:01
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike

> I could take out that bit of code, which is just determining how
> much physical memory is installed, so that mkgmap won't suggest that the
> user increases the available heap too much. It isn't used in determining a
> value for maxjobs.

Since it is just used for an informational message, you could just
surround it with a try/catch block.  Then if it doesn't
exist on a particular it isn't a problem when using the
downloaded pre-compiled distribution.

Of course it would fail to compile on such a platform, but then
hopefully someone will let us know and we can decide what to do then.

Steve

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev




maxjobs-v6.patch
Description: maxjobs-v6.patch
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Mike Baggaley]

Hi Gerd, I misread the --max-jobs with no value functionality. Please find
attached a revised patch that does function that way when the max-jobs
option is specified with no value.

Cheers,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com] 
Sent: 16 February 2018 10:42
To: Mike Baggaley <m...@tvage.co.uk>
Subject: AW: [mkgmap-dev] max-jobs patch

Hi Mike,

without the patch the option --max-jos means: use as many threads as cores.
Since mkgmap is normally very silent I would not print an information that
just confirms that anything is normal.

My personal preference would be to print much more information about
performance, but my understanding is that
the mkgmap users are happy with the silent behaviour.

BTW: I'd also print a message that tells the user how to activate logging if
it is not already activated.

Gerd


Von: Mike Baggaley <m...@tvage.co.uk>
Gesendet: Freitag, 16. Februar 2018 11:20:03
An: 'Gerd Petermann'
Betreff: RE: [mkgmap-dev] max-jobs patch

Hi Gerd, as the functionality of --max-jobs without a value is the same as
not specifying the parameter (i.e. it is the default), it seems to me to be
better for them to function the same. If you have specified --max-jobs
without a value, why would you not want to know how many threads had been
assigned?

Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
Sent: 15 February 2018 08:54
To: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

looks good to me. I am just not sure about the new stdout message
"Setting max-jobs to 4"
It appears when --max-jobs is not used (ok), but also when it is used
without a parameter.
In  the latter case I would only print this message if the number of CPU
cores is higher.

Gerd



Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Gerd
Petermann <gpetermann_muenc...@hotmail.com>
Gesendet: Mittwoch, 14. Februar 2018 19:38:06
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

okay, I'll try it tomorrow.

Gerd



Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <m...@tvage.co.uk>
Gesendet: Mittwoch, 14. Februar 2018 18:30:28
An: 'Steve Ratcliffe'; mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] max-jobs patch

I tried renaming the java library to force a failure and got a java error
"Error occurred during initialization of VM" before any mkgmap code was
executed, so to be on the safe side, I have removed that code and amended
the message in the attached patch.

Regards,
Mike

-Original Message-
From: Steve Ratcliffe [mailto:st...@parabola.me.uk]
Sent: 13 February 2018 23:01
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike

> I could take out that bit of code, which is just determining how
> much physical memory is installed, so that mkgmap won't suggest that the
> user increases the available heap too much. It isn't used in determining a
> value for maxjobs.

Since it is just used for an informational message, you could just
surround it with a try/catch block.  Then if it doesn't
exist on a particular it isn't a problem when using the
downloaded pre-compiled distribution.

Of course it would fail to compile on such a platform, but then
hopefully someone will let us know and we can decide what to do then.

Steve

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev




maxjobs-v5.patch
Description: Binary data
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Mike,

okay, I'll try it tomorrow.

Gerd



Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike 
Baggaley <m...@tvage.co.uk>
Gesendet: Mittwoch, 14. Februar 2018 18:30:28
An: 'Steve Ratcliffe'; mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] max-jobs patch

I tried renaming the java library to force a failure and got a java error 
"Error occurred during initialization of VM" before any mkgmap code was 
executed, so to be on the safe side, I have removed that code and amended the 
message in the attached patch.

Regards,
Mike

-Original Message-
From: Steve Ratcliffe [mailto:st...@parabola.me.uk]
Sent: 13 February 2018 23:01
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike

> I could take out that bit of code, which is just determining how
> much physical memory is installed, so that mkgmap won't suggest that the
> user increases the available heap too much. It isn't used in determining a
> value for maxjobs.

Since it is just used for an informational message, you could just
surround it with a try/catch block.  Then if it doesn't
exist on a particular it isn't a problem when using the
downloaded pre-compiled distribution.

Of course it would fail to compile on such a platform, but then
hopefully someone will let us know and we can decide what to do then.

Steve

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Mike Baggaley]

I tried renaming the java library to force a failure and got a java error 
"Error occurred during initialization of VM" before any mkgmap code was 
executed, so to be on the safe side, I have removed that code and amended the 
message in the attached patch.

Regards,
Mike

-Original Message-
From: Steve Ratcliffe [mailto:st...@parabola.me.uk] 
Sent: 13 February 2018 23:01
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike

> I could take out that bit of code, which is just determining how
> much physical memory is installed, so that mkgmap won't suggest that the
> user increases the available heap too much. It isn't used in determining a
> value for maxjobs.

Since it is just used for an informational message, you could just
surround it with a try/catch block.  Then if it doesn't
exist on a particular it isn't a problem when using the
downloaded pre-compiled distribution.

Of course it would fail to compile on such a platform, but then
hopefully someone will let us know and we can decide what to do then.

Steve



maxjobs-v4.patch
Description: Binary data
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Steve Ratcliffe]

Hi Mike


I could take out that bit of code, which is just determining how
much physical memory is installed, so that mkgmap won't suggest that the
user increases the available heap too much. It isn't used in determining a
value for maxjobs.


Since it is just used for an informational message, you could just
surround it with a try/catch block.  Then if it doesn't
exist on a particular it isn't a problem when using the
downloaded pre-compiled distribution.

Of course it would fail to compile on such a platform, but then
hopefully someone will let us know and we can decide what to do then.

Steve
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Mike Baggaley]

Hi Gerd,

I would expect the code to work on a MAC, because com.sun.management is part
of the JDK, and defines the interface by which specific platforms can
provide subinterfaces with further information (AFAIK there is only one for
UNIX). However, I don't have a MAC, so can't test it on there. If it is a
problem, I could take out that bit of code, which is just determining how
much physical memory is installed, so that mkgmap won't suggest that the
user increases the available heap too much. It isn't used in determining a
value for maxjobs.

Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com] 
Sent: 10 February 2018 15:45
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

I've compiled r4107 with your patch:
http://files.mkgmap.org.uk/download/418/mkgmap.jar

I did not yet try it because I am still fighting with the crash in
MapSource.

I am not sure if the usage of classes like
com.sun.management.OperatingSystemMXBean
are supported with openjdk? Will this also work on a Mac?

Gerd




--
Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Mike,

I've compiled r4107 with your patch:
http://files.mkgmap.org.uk/download/418/mkgmap.jar

I did not yet try it because I am still fighting with the crash in
MapSource.

I am not sure if the usage of classes like
com.sun.management.OperatingSystemMXBean
are supported with openjdk? Will this also work on a Mac?

Gerd




--
Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Henning Scholland]

Hi Gerd,

these are my runtimes of several compilations of my Turkey map, splitted
with 1.500.000 nodes, compiled witout index, gmapi and gmappsupp, but
including DEM and routing etc.

RAM in GB    Threads     Time1 in ms    Time2 in ms
12                    8                328106            323605
8                      8                323135            341242
4                      8                378460            408714
4                      7                364123            361501
4                     6                 349048            343866
4                      4                371786            366581
4                       1                845778            797376
2                   4                527267            509698
2                        1           804117            853805

It seems a lack of RAM slows more down if there are two many jobs. For
4GB Ram, 6 threads are faster than 8 threads, even using only 4 threads
is faster than using 8.

Henning


On 10.02.2018 16:41, Gerd Petermann wrote:
On 10.02.2018 16:41, Gerd Petermann wrote:
> Hi Henning,
>
> thanks for the numbers. I assume that your JRE sets xmx=4G because you 
> probably have 16G.
> My machine has 4 cores and 8G and the JRE sets xmx=2G when I don't specify a 
> value.
> When you compile a large number of tiles peak memory is reached when building 
> the index.
>
> Maybe you can do some more tests with a smaller set of tiles?
> I'd like to know if the run time changes much when you compile e.g. 32 tiles 
> with
> a) -Xmx8GB and --max-jobs=8
> b) -Xmx4GB and --max-jobs=8
> c) -Xmx4GB and --max-jobs=7
> d) -Xmx4GB and --max-jobs=6
>
> I suggest to do each test at least 2 times and start with one unmeasured test 
> to reduce
> IO caching effects. You should also not use --index, --gmapi or --gmapsupp 
> for these tests.
>
> Gerd
>
>
> 
> Von: Henning Scholland 
> Gesendet: Samstag, 10. Februar 2018 07:23
> An: Gerd Petermann; Development list for mkgmap
> Betreff: Re: AW: [mkgmap-dev] max-jobs patch
>
> Hi Gerd,
> Hi Mike,
>
> I gave it a try on all my maps (splitted with max-nodes=1.600.000).
> Removed xmx and max jobs. mkgmap used all 8 logical cores and about 4GB
> of memory. So it seems, Gerd was correct with the 500mb per job. So I
> would suggest to limit max-jobs with 32bit java to 3 jobs and to
> RAM/500mb on 64bit java. The used jobs should be minimum of both CPU and
> RAM.
>
> Regarding to higher amount of RAM I was mentioning before, I was calling
> mkgmap with xmx=12G, maybe java is not freeing the heap in that case. It
> was during dem-branch. Maybe during that time mkgmap wasn't also that
> efficient.
>
> Henning
>

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Mike Baggaley]

Hi Gerd, I have put together the attached patch that should result in a
reasonable value for max-jobs if it is defaulted. This version attempts to
calculate a suitable value by processing one task, seeing how much memory it
used, then dividing the available memory by the memory used to determine the
value before running the remaining tasks. I have tried a variety of values
of -Xmx on my PC, and it seems to work for both a county and the whole of
the UK. I haven't tried it on a continent, as I don't have suitable data. I
can't guarantee that every combination of memory and CPU will work, but if
it does fail, you will still get a sensible error message telling you what
to change. At the end of the run, if insufficient memory is allocated to the
Java heap to allow all the CPU cores to be used, it will suggest a value for
-Xmx. For my maps, it seems to suggest about 400Mb per thread. My build of
the UK without specifying --max-jobs has reduced from about 3 hours to 45
minutes.

I have also made a few further minor improvements to the documentation to
make the form more consistent, and changed the logged execution time from
milliseconds to a more useful hours, minutes and seconds (or milliseconds
when less than 1 second).

Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com] 
Sent: 07 February 2018 19:06
To: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

I did not yet try it. My understanding is that the Garbage Collection will
try hard
to avoid an out of heap exception. In other words: A user might see better
throughput
with fewer parallel jobs. What are your experiences? Did you try with
large maps, say > 40 large tiles ?

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <m...@tvage.co.uk>
Gesendet: Mittwoch, 7. Februar 2018 01:58
An: 'Development list for mkgmap'
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd, please find attached v2 of the max-jobs patch.

This version additionally catches out of heap memory exceptions and displays
a message suggesting either the use of the Java -Xmx option to increase the
available heap memory or the mkgmap --max-jobs option with a smaller value
to reduce the memory requirement (the latter is suggested only if using more
than one thread).

How does that seem?
Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
Sent: 05 February 2018 07:28
To: Mike Baggaley <m...@tvage.co.uk>; 'Development list for mkgmap'
<mkgmap-dev@lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

thought about this again. Maybe this change is too simple. With multiple
jobs one also
needs more heap (-Xmx JRE option). If you create rather large tiles with
splitter (max-nodes=160)
you need 0.5 - 1 GB for each job. Not sure what happens when a user creates
a map with
10 tiles on an 8-core machine without any -Xmx option.

I fear this will result in OutOfMemory exception, so better check the
available heap as well.

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <m...@tvage.co.uk>
Gesendet: Sonntag, 4. Februar 2018 15:14
An: 'Development list for mkgmap'
Betreff: [mkgmap-dev] max-jobs patch

Hi Gerd,

Please find attached a patch that amends the default behaviour if the
--max-jobs option is not specified, to using a value equal to the number of
CPU cores, as suggested in a previous post. The documentation is also
amended to reflect the change. This halves the execution time of mkgmap for
building a map of Staffordshire on my 8-core PC when --max-jobs is not
specified (I didn't know about this option previously and was unaware the
performance could have been improved).

Cheers,
Mike




maxjobs-v3.patch
Description: Binary data
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Bernd,

not sure what you want to express here.
When you use --max-jobs or max-jobs=8 with the unpatched version you should see
exactly the same numbers on an 8 core machine, the patch only changes the 
default
for max-jobs.

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Bernd 
Weigelt <weigelt.be...@web.de>
Gesendet: Samstag, 10. Februar 2018 10:19:09
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd

the unpatched version uses up to 8.7 GB with the options for mkgmap

I'm using maxnodes=512000 to create an area.list and use this file for every
build.

At the moment i see no problem with the patched version


Bernd



Am Samstag, 10. Februar 2018, 09:56:41 CET schrieb Gerd Petermann:
> Hi Bernd,
>
> okay, I think that confirms my estimate of ~500M per thread.
> See also my previous answer to Henning.
>
> Gerd
>
> 
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Bernd,

okay, I think that confirms my estimate of ~500M per thread.
See also my previous answer to Henning.

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Bernd 
Weigelt <weigelt.be...@web.de>
Gesendet: Samstag, 10. Februar 2018 09:43
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi

tested on my linux box, looks good

Heap is 4.4 GB with 8 threads.

With the unpatched version i'm using -Xmx8G for the heap


Bernd


Am Donnerstag, 8. Februar 2018, 15:34:23 CET schrieb Gerd Petermann:
> binary is here:
> http://files.mkgmap.org.uk/download/415/mkgmap.jar
>
> For those users who already used --max-jobs with or without a paramter
> before there should be no difference.

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Bernd Weigelt]

Hi

tested on my linux box, looks good

Heap is 4.4 GB with 8 threads.

With the unpatched version i'm using -Xmx8G for the heap


Bernd


Am Donnerstag, 8. Februar 2018, 15:34:23 CET schrieb Gerd Petermann:
> binary is here:
> http://files.mkgmap.org.uk/download/415/mkgmap.jar
> 
> For those users who already used --max-jobs with or without a paramter
> before there should be no difference.



signature.asc
Description: This is a digitally signed message part.
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Henning Scholland]

Hi Gerd,

maybe splitter can be modified  similar.

Henning
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Henning Scholland]

Hi Gerd,
Hi Mike,

I gave it a try on all my maps (splitted with max-nodes=1.600.000).
Removed xmx and max jobs. mkgmap used all 8 logical cores and about 4GB
of memory. So it seems, Gerd was correct with the 500mb per job. So I
would suggest to limit max-jobs with 32bit java to 3 jobs and to
RAM/500mb on 64bit java. The used jobs should be minimum of both CPU and
RAM.

Regarding to higher amount of RAM I was mentioning before, I was calling
mkgmap with xmx=12G, maybe java is not freeing the heap in that case. It
was during dem-branch. Maybe during that time mkgmap wasn't also that
efficient.

Henning
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Henning,

maybe you see those high heap values with DEM? How do measure it?
I think the monitor function of VisualVM shows quite well if Garbage Collection 
is under stress or not.

binary is here:
http://files.mkgmap.org.uk/download/415/mkgmap.jar

For those users who already used --max-jobs with or without a paramter before 
there should be no difference.

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Henning 
Scholland <o...@hscholland.de>
Gesendet: Donnerstag, 8. Februar 2018 09:56
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd,
Maybe my map is somehow special, but mkgmap definitely need more than 1gb heap 
per job. I will check on the weekend. Can you please post a patched mkgmap.jar?

How about doing it the other way round and suggest the user to use max-job 
option if mkgmap think it is useful?

Henning
On 8 Feb 2018, at 16:38, Gerd Petermann 
<gpetermann_muenc...@hotmail.com<mailto:gpetermann_muenc...@hotmail.com>> wrote:

Hi Manfred,

thanks for testing. Just to make this clear: I always use --max-jobs, because I 
also
always use -Xmx whenever I compile many tiles.

In this case a large tile is one that results in a large *.img
(one with many MB). Typically this depends on the --max-nodes option for 
splitter
and the style and the mkgmap options (routable or not, without or without DEM 
etc).

The max-jobs option allows to specify the number of parallel jobs,
e.g. --max-jobs=5 would start 5 parallel jobs (threads). Whenever one input 
file is done,
the next one is started until all are finished. The patch only changes the 
default from 1
to "number of cores". You can also use max-jobs=10 on a 2-core cpu, but this 
will slow down
the run time.

What I tried to point out is that a user who doesn't know about the --max--jobs 
option
might as well not know about the meaning of the -Xmx JRE option.

What happens if you start mkgmap on an 8-core machine with max-jobs=8 and
without -Xmx option? I think it will crash, and with the 2nd version of the 
patch it will tell
the user to look at the -Xmx option. So far so good.
What happens on a 4 core machine with max-jobs=4 and no -Xmx option?
What if the OS is 32 bit only? This would not allow much more than -Xmx1700m.
What if the option is used but set to a rather low value, e.g. -Xmx=1000m?

My current thinking is this:
If --max-jobs is not specified mkgmap should check the number of cores and the 
available heap
and assume that one thread needs ~500M.

So, I would use something like num = Math.min(Runtime.getRuntime().maxMemory() 
/ (512 * 1024 * 1024),
 Runtime.getRuntime().availableProcessors()) as default, maybe also for the 
case that one
specifies --max-jobs .

How does that sound?

Gerd








Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Manfred 
Haiduk <mhai...@t-online.de>
Gesendet: Mittwoch, 7. Februar 2018 22:28
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] max-jobs patch

I don't know, if Nordrhein-Westfalen is that kind of large tiles you
meant. Splitter ended up with 43 files and i tried to make mapsource
maps with and without --max-jobs option and mkgmap r4000. My computer
does the job with 43 files about 12minutes faster than without this
option. My laptop is a core i5 with 8Gb of memory. The command i use is
like this line :

java -Xmx7500m -XX:-UseGCOverheadLimit -ea -jar
-Dlog.config=logging.properties<http://logging.properties>
C:\PROGRA~2\KartenTools\mkgmap\mkgmap.jar
--location-autofill=is_in,nearest --mapname=70060001 --family-id=7006
.\styles\mystyle.txt --series-name="STYLE TEST"
--family-name="OpenStreetmap" --country-name=STYLETEST
--country-abbr=STY --area-name=STY --overview-mapname="overview"
--latin1 --style-file=.\styles --style=myland --max-jobs --keep-going
--check-roundabouts --drive-on=detect,right --output-dir=.\out\MYSTYLE
--index --bounds=mybounds --route
--name-tag-list=name,place_name,loc_name --housenumbers
--x-split-name-index --add-pois-to-areas --precomp-sea=.\input\sea.zip
--tdbfile --draw-priority=10 --gmapsupp -c .\data\MYSTYLE\template.args
--description=MYSTYLE --remove-ovm-work-files=true

Hope this gave you some useful hints


Am 07.02.2018 um 20:06 schrieb Gerd Petermann:
 Hi Mike,

 I did not yet try it. My understanding is that the Garbage Collection will try 
hard
 to avoid an out of heap exception. In other words: A user might see better 
throughput
 with fewer parallel jobs. What are your experiences? Did you try with
 large maps, say > 40 large tiles ?

 Gerd



 Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike 
Baggaley <m...@tvage.co.uk>
 Gesendet: Mittwoch, 7. Februar 2018 01:58
 An: 'Development list for mkgmap'
 Betreff: Re: [mkgmap-dev] max-jobs patch

Re: [mkgmap-dev] max-jobs patch

[Henning Scholland]

Hi Gerd,
Maybe my map is somehow special, but mkgmap definitely need more than 1gb heap 
per job. I will check on the weekend. Can you please post a patched mkgmap.jar?

How about doing it the other way round and suggest the user to use max-job 
option if mkgmap think it is useful?

Henning

On 8 Feb 2018, 16:38, at 16:38, Gerd Petermann 
<gpetermann_muenc...@hotmail.com> wrote:
>Hi Manfred,
>
>thanks for testing. Just to make this clear: I always use --max-jobs,
>because I also
>always use -Xmx whenever I compile many tiles.
>
>In this case a large tile is one that results in a large *.img
>(one with many MB). Typically this depends on the --max-nodes option
>for splitter
>and the style and the mkgmap options (routable or not, without or
>without DEM etc).
>
>The max-jobs option allows to specify the number of parallel jobs,
>e.g. --max-jobs=5 would start 5 parallel jobs (threads). Whenever one
>input file is done,
>the next one is started until all are finished. The patch only changes
>the default from 1
>to "number of cores". You can also use max-jobs=10 on a 2-core cpu, but
>this will slow down
>the run time.
>
>What I tried to point out is that a user who doesn't know about the
>--max--jobs option
>might as well not know about the meaning of the -Xmx JRE option.
>
>What happens if you start mkgmap on an 8-core machine with max-jobs=8
>and
>without -Xmx option? I think it will crash, and with the 2nd version of
>the patch it will tell
>the user to look at the -Xmx option. So far so good.
>What happens on a 4 core machine with max-jobs=4 and no -Xmx option?
>What if the OS is 32 bit only? This would not allow much more than
>-Xmx1700m.
>What if the option is used but set to a rather low value, e.g.
>-Xmx=1000m?
>
>My current thinking is this:
>If --max-jobs is not specified mkgmap should check the number of cores
>and the available heap
>and assume that one thread needs ~500M.
>
>So, I would use something like num =
>Math.min(Runtime.getRuntime().maxMemory() / (512 * 1024 * 1024),
>Runtime.getRuntime().availableProcessors()) as default, maybe also for
>the case that one
>specifies --max-jobs .
>
>How does that sound?
>
>Gerd
>
>
>
>
>
>
>
>Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von
>Manfred Haiduk <mhai...@t-online.de>
>Gesendet: Mittwoch, 7. Februar 2018 22:28
>An: mkgmap-dev@lists.mkgmap.org.uk
>Betreff: Re: [mkgmap-dev] max-jobs patch
>
>I don't know, if Nordrhein-Westfalen is that kind of large tiles you
>meant. Splitter ended up with 43 files and i tried to make mapsource
>maps with and without --max-jobs option and mkgmap r4000. My computer
>does the job with 43 files about 12minutes faster than without this
>option. My laptop is a core i5 with 8Gb of memory. The command i use is
>like this line :
>
>java -Xmx7500m -XX:-UseGCOverheadLimit -ea -jar
>-Dlog.config=logging.properties
>C:\PROGRA~2\KartenTools\mkgmap\mkgmap.jar
>--location-autofill=is_in,nearest --mapname=70060001 --family-id=7006
>.\styles\mystyle.txt --series-name="STYLE TEST"
>--family-name="OpenStreetmap" --country-name=STYLETEST
>--country-abbr=STY --area-name=STY --overview-mapname="overview"
>--latin1 --style-file=.\styles --style=myland --max-jobs --keep-going
>--check-roundabouts --drive-on=detect,right --output-dir=.\out\MYSTYLE
>--index --bounds=mybounds --route
>--name-tag-list=name,place_name,loc_name --housenumbers
>--x-split-name-index --add-pois-to-areas --precomp-sea=.\input\sea.zip
>--tdbfile --draw-priority=10 --gmapsupp -c .\data\MYSTYLE\template.args
>--description=MYSTYLE --remove-ovm-work-files=true
>
>Hope this gave you some useful hints
>
>
>Am 07.02.2018 um 20:06 schrieb Gerd Petermann:
>> Hi Mike,
>>
>> I did not yet try it. My understanding is that the Garbage Collection
>will try hard
>> to avoid an out of heap exception. In other words: A user might see
>better throughput
>> with fewer parallel jobs. What are your experiences? Did you try with
>> large maps, say > 40 large tiles ?
>>
>> Gerd
>>
>> 
>> Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag
>von Mike Baggaley <m...@tvage.co.uk>
>> Gesendet: Mittwoch, 7. Februar 2018 01:58
>> An: 'Development list for mkgmap'
>> Betreff: Re: [mkgmap-dev] max-jobs patch
>>
>> Hi Gerd, please find attached v2 of the max-jobs patch.
>>
>> This version additionally catches out of heap memory exceptions and
>displays
>> a message suggesting either the use of the Java -Xmx 

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Manfred,

thanks for testing. Just to make this clear: I always use --max-jobs, because I 
also
always use -Xmx whenever I compile many tiles.

In this case a large tile is one that results in a large *.img
(one with many MB). Typically this depends on the --max-nodes option for 
splitter
and the style and the mkgmap options (routable or not, without or without DEM 
etc).

The max-jobs option allows to specify the number of parallel jobs,
e.g. --max-jobs=5 would start 5 parallel jobs (threads). Whenever one input 
file is done,
the next one is started until all are finished. The patch only changes the 
default from 1
to "number of cores". You can also use max-jobs=10 on a 2-core cpu, but this 
will slow down
the run time.

What I tried to point out is that a user who doesn't know about the --max--jobs 
option
might as well not know about the meaning of the -Xmx JRE option.

What happens if you start mkgmap on an 8-core machine with max-jobs=8 and
without -Xmx option? I think it will crash, and with the 2nd version of the 
patch it will tell
the user to look at the -Xmx option. So far so good.
What happens on a 4 core machine with max-jobs=4 and no -Xmx option?
What if the OS is 32 bit only? This would not allow much more than -Xmx1700m.
What if the option is used but set to a rather low value, e.g. -Xmx=1000m?

My current thinking is this:
If --max-jobs is not specified mkgmap should check the number of cores and the 
available heap
and assume that one thread needs ~500M. 

So, I would use something like num = Math.min(Runtime.getRuntime().maxMemory() 
/ (512 * 1024 * 1024),
 Runtime.getRuntime().availableProcessors()) as default, maybe also for the 
case that one
specifies --max-jobs .

How does that sound?

Gerd







Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Manfred 
Haiduk <mhai...@t-online.de>
Gesendet: Mittwoch, 7. Februar 2018 22:28
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] max-jobs patch

I don't know, if Nordrhein-Westfalen is that kind of large tiles you
meant. Splitter ended up with 43 files and i tried to make mapsource
maps with and without --max-jobs option and mkgmap r4000. My computer
does the job with 43 files about 12minutes faster than without this
option. My laptop is a core i5 with 8Gb of memory. The command i use is
like this line :

java -Xmx7500m -XX:-UseGCOverheadLimit -ea -jar
-Dlog.config=logging.properties
C:\PROGRA~2\KartenTools\mkgmap\mkgmap.jar
--location-autofill=is_in,nearest --mapname=70060001 --family-id=7006
.\styles\mystyle.txt --series-name="STYLE TEST"
--family-name="OpenStreetmap" --country-name=STYLETEST
--country-abbr=STY --area-name=STY --overview-mapname="overview"
--latin1 --style-file=.\styles --style=myland --max-jobs --keep-going
--check-roundabouts --drive-on=detect,right --output-dir=.\out\MYSTYLE
--index --bounds=mybounds --route
--name-tag-list=name,place_name,loc_name --housenumbers
--x-split-name-index --add-pois-to-areas --precomp-sea=.\input\sea.zip
--tdbfile --draw-priority=10 --gmapsupp -c .\data\MYSTYLE\template.args
--description=MYSTYLE --remove-ovm-work-files=true

Hope this gave you some useful hints


Am 07.02.2018 um 20:06 schrieb Gerd Petermann:
> Hi Mike,
>
> I did not yet try it. My understanding is that the Garbage Collection will 
> try hard
> to avoid an out of heap exception. In other words: A user might see better 
> throughput
> with fewer parallel jobs. What are your experiences? Did you try with
> large maps, say > 40 large tiles ?
>
> Gerd
>
> 
> Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike 
> Baggaley <m...@tvage.co.uk>
> Gesendet: Mittwoch, 7. Februar 2018 01:58
> An: 'Development list for mkgmap'
> Betreff: Re: [mkgmap-dev] max-jobs patch
>
> Hi Gerd, please find attached v2 of the max-jobs patch.
>
> This version additionally catches out of heap memory exceptions and displays
> a message suggesting either the use of the Java -Xmx option to increase the
> available heap memory or the mkgmap --max-jobs option with a smaller value
> to reduce the memory requirement (the latter is suggested only if using more
> than one thread).
>
> How does that seem?
> Regards,
> Mike
>
> -Original Message-
> From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
> Sent: 05 February 2018 07:28
> To: Mike Baggaley <m...@tvage.co.uk>; 'Development list for mkgmap'
> <mkgmap-dev@lists.mkgmap.org.uk>
> Subject: Re: [mkgmap-dev] max-jobs patch
>
> Hi Mike,
>
> thought about this again. Maybe this change is too simple. With multiple
> jobs one also
> needs more heap (-Xmx JRE option). If you create rather large tiles with
> splitter (max-nodes=160)
> you 

Re: [mkgmap-dev] max-jobs patch

[Manfred Haiduk]

I don't know, if Nordrhein-Westfalen is that kind of large tiles you 
meant. Splitter ended up with 43 files and i tried to make mapsource 
maps with and without --max-jobs option and mkgmap r4000. My computer 
does the job with 43 files about 12minutes faster than without this 
option. My laptop is a core i5 with 8Gb of memory. The command i use is 
like this line :


java -Xmx7500m -XX:-UseGCOverheadLimit -ea -jar 
-Dlog.config=logging.properties 
C:\PROGRA~2\KartenTools\mkgmap\mkgmap.jar 
--location-autofill=is_in,nearest --mapname=70060001 --family-id=7006 
.\styles\mystyle.txt --series-name="STYLE TEST" 
--family-name="OpenStreetmap" --country-name=STYLETEST 
--country-abbr=STY --area-name=STY --overview-mapname="overview" 
--latin1 --style-file=.\styles --style=myland --max-jobs --keep-going 
--check-roundabouts --drive-on=detect,right --output-dir=.\out\MYSTYLE 
--index --bounds=mybounds --route 
--name-tag-list=name,place_name,loc_name --housenumbers 
--x-split-name-index --add-pois-to-areas --precomp-sea=.\input\sea.zip  
--tdbfile --draw-priority=10 --gmapsupp -c .\data\MYSTYLE\template.args 
--description=MYSTYLE --remove-ovm-work-files=true


Hope this gave you some useful hints


Am 07.02.2018 um 20:06 schrieb Gerd Petermann:

Hi Mike,

I did not yet try it. My understanding is that the Garbage Collection will try 
hard
to avoid an out of heap exception. In other words: A user might see better 
throughput
with fewer parallel jobs. What are your experiences? Did you try with
large maps, say > 40 large tiles ?

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley 
<m...@tvage.co.uk>
Gesendet: Mittwoch, 7. Februar 2018 01:58
An: 'Development list for mkgmap'
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd, please find attached v2 of the max-jobs patch.

This version additionally catches out of heap memory exceptions and displays
a message suggesting either the use of the Java -Xmx option to increase the
available heap memory or the mkgmap --max-jobs option with a smaller value
to reduce the memory requirement (the latter is suggested only if using more
than one thread).

How does that seem?
Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
Sent: 05 February 2018 07:28
To: Mike Baggaley <m...@tvage.co.uk>; 'Development list for mkgmap'
<mkgmap-dev@lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

thought about this again. Maybe this change is too simple. With multiple
jobs one also
needs more heap (-Xmx JRE option). If you create rather large tiles with
splitter (max-nodes=160)
you need 0.5 - 1 GB for each job. Not sure what happens when a user creates
a map with
10 tiles on an 8-core machine without any -Xmx option.

I fear this will result in OutOfMemory exception, so better check the
available heap as well.

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <m...@tvage.co.uk>
Gesendet: Sonntag, 4. Februar 2018 15:14
An: 'Development list for mkgmap'
Betreff: [mkgmap-dev] max-jobs patch

Hi Gerd,

Please find attached a patch that amends the default behaviour if the
--max-jobs option is not specified, to using a value equal to the number of
CPU cores, as suggested in a previous post. The documentation is also
amended to reflect the change. This halves the execution time of mkgmap for
building a map of Staffordshire on my 8-core PC when --max-jobs is not
specified (I didn't know about this option previously and was unaware the
performance could have been improved).

Cheers,
Mike

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


--

 Mit freundlichen Grüßen

#
Manfred Haiduk,
Zum Fischbach 9,
52393 Hürtgenwald
e-mail mhai...@t-online.de
#

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Mike,

I did not yet try it. My understanding is that the Garbage Collection will try 
hard
to avoid an out of heap exception. In other words: A user might see better 
throughput
with fewer parallel jobs. What are your experiences? Did you try with
large maps, say > 40 large tiles ?

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike 
Baggaley <m...@tvage.co.uk>
Gesendet: Mittwoch, 7. Februar 2018 01:58
An: 'Development list for mkgmap'
Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd, please find attached v2 of the max-jobs patch.

This version additionally catches out of heap memory exceptions and displays
a message suggesting either the use of the Java -Xmx option to increase the
available heap memory or the mkgmap --max-jobs option with a smaller value
to reduce the memory requirement (the latter is suggested only if using more
than one thread).

How does that seem?
Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com]
Sent: 05 February 2018 07:28
To: Mike Baggaley <m...@tvage.co.uk>; 'Development list for mkgmap'
<mkgmap-dev@lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

thought about this again. Maybe this change is too simple. With multiple
jobs one also
needs more heap (-Xmx JRE option). If you create rather large tiles with
splitter (max-nodes=160)
you need 0.5 - 1 GB for each job. Not sure what happens when a user creates
a map with
10 tiles on an 8-core machine without any -Xmx option.

I fear this will result in OutOfMemory exception, so better check the
available heap as well.

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <m...@tvage.co.uk>
Gesendet: Sonntag, 4. Februar 2018 15:14
An: 'Development list for mkgmap'
Betreff: [mkgmap-dev] max-jobs patch

Hi Gerd,

Please find attached a patch that amends the default behaviour if the
--max-jobs option is not specified, to using a value equal to the number of
CPU cores, as suggested in a previous post. The documentation is also
amended to reflect the change. This halves the execution time of mkgmap for
building a map of Staffordshire on my 8-core PC when --max-jobs is not
specified (I didn't know about this option previously and was unaware the
performance could have been improved).

Cheers,
Mike

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Mike Baggaley]

Hi Gerd, please find attached v2 of the max-jobs patch.

This version additionally catches out of heap memory exceptions and displays
a message suggesting either the use of the Java -Xmx option to increase the
available heap memory or the mkgmap --max-jobs option with a smaller value
to reduce the memory requirement (the latter is suggested only if using more
than one thread).

How does that seem?
Regards,
Mike

-Original Message-
From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com] 
Sent: 05 February 2018 07:28
To: Mike Baggaley <m...@tvage.co.uk>; 'Development list for mkgmap'
<mkgmap-dev@lists.mkgmap.org.uk>
Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

thought about this again. Maybe this change is too simple. With multiple
jobs one also
needs more heap (-Xmx JRE option). If you create rather large tiles with
splitter (max-nodes=160)
you need 0.5 - 1 GB for each job. Not sure what happens when a user creates
a map with
10 tiles on an 8-core machine without any -Xmx option.

I fear this will result in OutOfMemory exception, so better check the
available heap as well.

Gerd


Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von Mike
Baggaley <m...@tvage.co.uk>
Gesendet: Sonntag, 4. Februar 2018 15:14
An: 'Development list for mkgmap'
Betreff: [mkgmap-dev] max-jobs patch

Hi Gerd,

Please find attached a patch that amends the default behaviour if the
--max-jobs option is not specified, to using a value equal to the number of
CPU cores, as suggested in a previous post. The documentation is also
amended to reflect the change. This halves the execution time of mkgmap for
building a map of Staffordshire on my 8-core PC when --max-jobs is not
specified (I didn't know about this option previously and was unaware the
performance could have been improved).

Cheers,
Mike



maxjobs-v2.patch
Description: Binary data
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Henning Scholland]

Hi Gerd,
I think it's even more in combination with DEM. I usually use 6 of my 8
cores and ending up with 10 to 12 GB of heap. So I definitely agree with
you available heap somehow needs to be considered as well. Not only CPU
cores. Btw. for CFD/FEM analysis the simulation is usually is faster if
you only use physical cores, not the 'virtual' ones. Don't know how
mkgmap will behave.

Henning

On 05.02.2018 15:28, Gerd Petermann wrote:
> Hi Mike,
>
> thought about this again. Maybe this change is too simple. With multiple jobs 
> one also
> needs more heap (-Xmx JRE option). If you create rather large tiles with 
> splitter (max-nodes=160)
> you need 0.5 - 1 GB for each job. Not sure what happens when a user creates a 
> map with
> 10 tiles on an 8-core machine without any -Xmx option.
>
> I fear this will result in OutOfMemory exception, so better check the 
> available heap as well.
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von Mike 
> Baggaley 
> Gesendet: Sonntag, 4. Februar 2018 15:14
> An: 'Development list for mkgmap'
> Betreff: [mkgmap-dev] max-jobs patch
>
> Hi Gerd,
>
> Please find attached a patch that amends the default behaviour if the
> --max-jobs option is not specified, to using a value equal to the number of
> CPU cores, as suggested in a previous post. The documentation is also
> amended to reflect the change. This halves the execution time of mkgmap for
> building a map of Staffordshire on my 8-core PC when --max-jobs is not
> specified (I didn't know about this option previously and was unaware the
> performance could have been improved).
>
> Cheers,
> Mike
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] max-jobs patch

[Gerd Petermann]

Hi Mike,

thought about this again. Maybe this change is too simple. With multiple jobs 
one also
needs more heap (-Xmx JRE option). If you create rather large tiles with 
splitter (max-nodes=160)
you need 0.5 - 1 GB for each job. Not sure what happens when a user creates a 
map with
10 tiles on an 8-core machine without any -Xmx option.

I fear this will result in OutOfMemory exception, so better check the available 
heap as well.

Gerd


Von: mkgmap-dev  im Auftrag von Mike 
Baggaley 
Gesendet: Sonntag, 4. Februar 2018 15:14
An: 'Development list for mkgmap'
Betreff: [mkgmap-dev] max-jobs patch

Hi Gerd,

Please find attached a patch that amends the default behaviour if the
--max-jobs option is not specified, to using a value equal to the number of
CPU cores, as suggested in a previous post. The documentation is also
amended to reflect the change. This halves the execution time of mkgmap for
building a map of Staffordshire on my 8-core PC when --max-jobs is not
specified (I didn't know about this option previously and was unaware the
performance could have been improved).

Cheers,
Mike
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev