Re: [OSM-dev] issue with geofabrik europe update
Hi Flo, Am 17.05.2018 um 16:49 schrieb Florian Lohoff: > On Wed, May 16, 2018 at 05:50:19PM +0200, Frederik Ramm wrote: >> Hi, >> >> On 05/16/18 14:43, Julien Fastré wrote: >>> We had a strange issue with a europe diff update from geofabrik: the >>> diff file is not a valid xml. >> >> And that's entirely my fault for using "sed" to modify a couple of .osc >> files around the beginning of May. Sorry for that! It was after we got >> rid of uid/user fields in .osc files, and it turned out that some people >> has issues with the reduced files, so we decided to put dummy uid/user >> fields back in, and for the old files I quickly did that with a too >> broad search-and-replace command ;) > > So you actually say that geofabrik pbf/osc files dont contain valid > usernames anymore? > > I was wondering why i saw numerical usernames but if thats the case > i can drop usernames from my QA tools. OSM files which are available on our public server without log-in don't contain user names, user IDs and changeset IDs. To be precise, these fields are set to zeros/empty strings because Libosmium < 2.14 (and Osm2pgsql < 0.96) and Osmosis (all versions) expect these fields in the DenseInfo block of PBF files. You can get files with full metadata at https://osm-internal.download.geofabrik.de Some information on the access using scripts can be found at https://blog.geofabrik.de/?p=484 https://github.com/geofabrik/sendfile_osm_oauth_protector#documentation I am working on cookie support on Osmosis. A preview can be found at [1]. Someone else is working on the same for pyosmium-up-to-date. Best regards Michael [1] https://github.com/openstreetmap/osmosis/pull/40 -- Michael Reichert www.geofabrik.de Geofabrik GmbHHandelsregister: HRB Mannheim 703657 Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm 76133 Karlsruhe Tel: 0721-1803560-3 reich...@geofabrik.de Fax: 0721-1803560-9 signature.asc Description: OpenPGP digital signature ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] issue with geofabrik europe update
On Wed, May 16, 2018 at 05:50:19PM +0200, Frederik Ramm wrote: > Hi, > > On 05/16/18 14:43, Julien Fastré wrote: > > We had a strange issue with a europe diff update from geofabrik: the > > diff file is not a valid xml. > > And that's entirely my fault for using "sed" to modify a couple of .osc > files around the beginning of May. Sorry for that! It was after we got > rid of uid/user fields in .osc files, and it turned out that some people > has issues with the reduced files, so we decided to put dummy uid/user > fields back in, and for the old files I quickly did that with a too > broad search-and-replace command ;) So you actually say that geofabrik pbf/osc files dont contain valid usernames anymore? I was wondering why i saw numerical usernames but if thats the case i can drop usernames from my QA tools. Flo -- Florian Lohoff f...@zz.de UTF-8 Test: The 🐈 ran after a 🐁, but the 🐁 ran away signature.asc Description: PGP signature ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] issue with geofabrik europe update
Le 16/05/18 à 17:50, Frederik Ramm a écrit : > > Sorry for that - I'm sure it took you some time to figure out what was > wrong! > Thanks for the explanation ! No problem for me: this is part of the job. I am happy to hear that there is an explanation for this issue ! Julien Fastré ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] issue with geofabrik europe update
Hi, On 05/16/18 14:43, Julien Fastré wrote: > We had a strange issue with a europe diff update from geofabrik: the > diff file is not a valid xml. And that's entirely my fault for using "sed" to modify a couple of .osc files around the beginning of May. Sorry for that! It was after we got rid of uid/user fields in .osc files, and it turned out that some people has issues with the reduced files, so we decided to put dummy uid/user fields back in, and for the old files I quickly did that with a too broad search-and-replace command ;) Fixed the file now (and another one in the "georgia-updates" dir). > I wonder if we were the only one affected and, if not, how did you cope > to pass this diff without error ? Since the bug was introduced a day or two after the diff was published, it is possible that other consumers of the diff who loaded it immediately didn't run into the issue. Sorry for that - I'm sure it took you some time to figure out what was wrong! Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] issue with geofabrik europe update
Hi, We had a strange issue with a europe diff update from geofabrik: the diff file is not a valid xml. The affected file: http://download.geofabrik.de/europe-updates/000/001/872.osc.gz We encounter an error when parsing this file with osmosis: ``` org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file 872.osc. publicId=(null), systemId=(null), lineNumber=972870, columnNumber=158. at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:114) at java.lang.Thread.run(Thread.java:748) Caused by: org.xml.sax.SAXParseException: Element type "tag" must be followed by either attribute specifications, ">" or "/>". at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(SAXParser.java:189) at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:109) ... 1 more ``` This come from the line 972870, which is not a valid xml tag indeed: > v="http://ka.wikipedia.org/w/index.php?title=%E1%83%A4%E1%83%90%E1%83%98%E1%83%9A%E1%83%98:Tbilisi_Admin_Map.jpg&fileuid="0"; > user="" changeset="0" timestamp=20080104164655"/> Removing this line (`sed '972870d' 872.osc > 872.fixed.osc`) make osmosis able to parse the file. I wonder if we were the only one affected and, if not, how did you cope to pass this diff without error ? Thanks, Julien ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev