(cc'ing to osmosis-dev) Clearly it's the ListIdTracker in the Used Node Filter task that is causing the problem. The ListIdTracker is good for smallish data sets, but for processing planet-wide data sets a BitSetIdTracker is more memory-efficient. So, try adding idTrackerType=BitSet to the --used-node task (i.e., java -Xmx1024m -jar e:\osmosis-0.30\osmosis.jar --read-xml file="planet-090218.osm" enableDateParsing="no" --log-progress interval="10" --way-key-value keyValueList="..." --used-node idTrackerType=BitSet --migrate --write-xml-0.6 file="E:\world\dump\roads\roads.osm"
Karl On Thu, Feb 26, 2009 at 5:48 AM, Emilie Laffray <[email protected]>wrote: > > > > Are you sure to be using DMA for HDD-access? > > It should be IO-bound for such simple tasks. > > > > What do you mean with "maxed out"? "99% user" or > > just just "0% idle"? > > IO-processes (Windows: [x]show kernel times) use > > cpu-time too. > > > > (It's sad that Java-NIO does not get async file-io until > > Java7.) > > > > Well, whether it is IO bound or not in this case doesn't really > matter. Osmosis just crashed again on me, while I increased the VM to > 1024MB. > The exact message that i am getting is this: > > INFO: Processing Way 15717214. > 26-Feb-2009 12:45:54 > com.bretth.osmosis.core.progress.v0_5.EntityProgressLogger > process > INFO: Processing Way 15815970. > 26-Feb-2009 12:46:00 > com.bretth.osmosis.core.pipeline.common.ActiveTaskManager w > aitForCompletion > SEVERE: Thread for task 1-read-xml failed > java.lang.OutOfMemoryError: Java heap space > at > com.bretth.osmosis.core.filter.common.ListIdTracker.extendIdList(List > IdTracker.java:66) > at > com.bretth.osmosis.core.filter.common.ListIdTracker.set(ListIdTracker > .java:117) > at > com.bretth.osmosis.core.filter.v0_5.UsedNodeFilter.process(UsedNodeFi > lter.java:89) > at > com.bretth.osmosis.core.container.v0_5.WayContainer.process(WayContai > ner.java:60) > at > com.bretth.osmosis.core.filter.v0_5.UsedNodeFilter.process(UsedNodeFi > lter.java:58) > at > com.bretth.osmosis.core.filter.v0_5.WayKeyValueFilter.process(WayKeyV > alueFilter.java:90) > at > com.bretth.osmosis.core.container.v0_5.WayContainer.process(WayContai > ner.java:60) > at > com.bretth.osmosis.core.filter.v0_5.WayKeyValueFilter.process(WayKeyV > alueFilter.java:53) > at > com.bretth.osmosis.core.progress.v0_5.EntityProgressLogger.process(En > tityProgressLogger.java:84) > at > com.bretth.osmosis.core.xml.v0_5.impl.WayElementProcessor.end(WayElem > entProcessor.java:103) > at > com.bretth.osmosis.core.xml.v0_5.impl.OsmHandler.endElement(OsmHandle > r.java:109) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endEleme > nt(Unknown Source) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp > l.scanEndElement(Unknown Source) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp > l$FragmentContentDriver.next(Unknown Source) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(U > nknown Source) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp > l.scanDocument(Unknown Source) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U > nknown Source) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U > nknown Source) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown So > urce) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Un > known Source) > at > com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.p > arse(Unknown Source) > at javax.xml.parsers.SAXParser.parse(Unknown Source) > at javax.xml.parsers.SAXParser.parse(Unknown Source) > at > com.bretth.osmosis.core.xml.v0_5.XmlReader.run(XmlReader.java:109) > at java.lang.Thread.run(Unknown Source) > 26-Feb-2009 12:46:00 com.bretth.osmosis.core.Osmosis main > SEVERE: Execution aborted. > com.bretth.osmosis.core.OsmosisRuntimeException: One or more tasks failed. > at > com.bretth.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pi > peline.java:146) > at com.bretth.osmosis.core.Osmosis.run(Osmosis.java:85) > at com.bretth.osmosis.core.Osmosis.main(Osmosis.java:30) > > The command line was: > java -Xmx1024m -jar e:\osmosis-0.30\osmosis.jar --read-xml > file="planet-090218.osm" enableDateParsing="no" --log-progress > interval="10" --way-key-value > > keyValueList="highway.motorway,highway.motorway_link,highway.trunk,highway.trunk_link,highway.primary,highway.primary_link,highway.secondary,highway.tertiary,highway.unclassified,highway.road,highway.residential,highway.living_street,highway.service,highway.track,highway.pedestrian,highway.bus_guideway,highway.path,highway.cycleway,highway.footway,highway.bridleway,highway.byway,highway.steps,highway.toll_booth,highway.ford,cycleway.lane,cycleway.track,cycleway.opposite_lane,opposite_track,cycleway.opposite,tracktype.grade1,tracktype.grade2,tracktype.grade3,tracktype.grade4,tracktype.grade5,waterway.stream,waterway.river,waterway.riverbank,waterway.canal,waterway.dock,waterway.weir,waterway.dam" > --used-node --migrate --write-xml-0.6 > file="E:\world\dump\roads\roads.osm" > > I am a bit at a loss on why it is failing here. > > Emilie Laffray > > _______________________________________________ > dev mailing list > [email protected] > http://lists.openstreetmap.org/listinfo/dev >
_______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
