The discussions around "Commit: r1566: Drop all tags from the osm file
that are not used" inspired me to address the memory footprints of the tags.
The patch reduces the memory footprint by using the String.intern()
method. This ensures that the String "highway" exists only once when
stored as tag. Additionally the values of the most common tags with a
limited number of different values are also interned. (It makes sense to
intern the value of a highway tag but it does not make sense to intern
the value of a name tag because the share rate of the name tag values is
too low).
This all might sound embarrasing but the String objects that are
returned from the XML reader are not interned. You can test this easily
by exchanging the put method in class tag:
public String put(String key, String value) {
if ("highway".equals(key) && "highway" != key) {
log.error("Tag is not interned");
}
ensureSpace();
Integer ind = keyPos(key);
if (ind == null)
assert false : "keyPos(" + key + ") returns null - size = " + size +
", capacity = " + capacity;
keys[ind] = key;
String old = values[ind];
if (old == null)
size++;
values[ind] = value;
return old;
}
You will see lots of "Tag is not interned" errors.
I have seen memory reductions of > 10%.
Please test this patch against your well known tiles. It would be
perfect if someone has a tile and knows its minimum memory requirement
for mkgmap. This patch should lower it.
WanMil
Index: src/uk/me/parabola/mkgmap/reader/osm/Tags.java
===================================================================
--- src/uk/me/parabola/mkgmap/reader/osm/Tags.java (revision 1566)
+++ src/uk/me/parabola/mkgmap/reader/osm/Tags.java (working copy)
@@ -19,6 +19,7 @@
import java.util.AbstractMap;
import java.util.Arrays;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
@@ -45,6 +46,18 @@
private String[] keys;
private String[] values;
+
+ /**
+ * Stores all tags which values should be stored as String intern. The
values of
+ * these tags should have a limited number of different values to get a
+ * reasonable memory footprint effect.
+ */
+ private final static HashSet<String> interableValueTags = new
HashSet<String>(
+ Arrays.asList("highway", "building",
"addr:housenumber", "access",
+ "natural", "waterway", "amenity", "oneway",
"surface",
+ "landuse", "lanes", "place", "layer",
"tracktype", "maxspeed",
+ "foot", "bridge", "height", "area", "railway",
"admin_level",
+ "power", "type", "leisure", "barrier"));
public Tags() {
keys = new String[INIT_SIZE];
@@ -65,11 +78,19 @@
Integer ind = keyPos(key);
if (ind == null)
assert false : "keyPos(" + key + ") returns null - size
= " + size + ", capacity = " + capacity;
- keys[ind] = key;
+ // use .intern() to reduce memory footprint
+ keys[ind] = key.intern();
String old = values[ind];
if (old == null)
size++;
+
+ if (interableValueTags.contains(key)) {
+ // use .intern() to reduce memory footprint for the most
+ // common tags with a limited range of values
+ value = value.intern();
+ }
+
values[ind] = value;
return old;
_______________________________________________
mkgmap-dev mailing list
[email protected]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
