Re: [OSM-dev] issue with geofabrik europe update

2018-05-17 Thread Michael Reichert
Hi Flo,

Am 17.05.2018 um 16:49 schrieb Florian Lohoff:
> On Wed, May 16, 2018 at 05:50:19PM +0200, Frederik Ramm wrote:
>> Hi,
>>
>> On 05/16/18 14:43, Julien Fastré wrote:
>>> We had a strange issue with a europe diff update from geofabrik: the
>>> diff file is not a valid xml.
>>
>> And that's entirely my fault for using "sed" to modify a couple of .osc
>> files around the beginning of May. Sorry for that! It was after we got
>> rid of uid/user fields in .osc files, and it turned out that some people
>> has issues with the reduced files, so we decided to put dummy uid/user
>> fields back in, and for the old files I quickly did that with a too
>> broad search-and-replace command ;)
> 
> So you actually say that geofabrik pbf/osc files dont contain valid
> usernames anymore?
> 
> I was wondering why i saw numerical usernames but if thats the case
> i can drop usernames from my QA tools.

OSM files which are available on our public server without log-in don't
contain user names, user IDs and changeset IDs. To be precise, these
fields are set to zeros/empty strings because Libosmium < 2.14 (and
Osm2pgsql < 0.96) and Osmosis (all versions) expect these fields in the
DenseInfo block of PBF files.

You can get files with full metadata at
https://osm-internal.download.geofabrik.de

Some information on the access using scripts can be found at
https://blog.geofabrik.de/?p=484
https://github.com/geofabrik/sendfile_osm_oauth_protector#documentation

I am working on cookie support on Osmosis. A preview can be found at
[1]. Someone else is working on the same for pyosmium-up-to-date.

Best regards

Michael


[1] https://github.com/openstreetmap/osmosis/pull/40

-- 
Michael Reichert  www.geofabrik.de
Geofabrik GmbHHandelsregister: HRB Mannheim 703657
Amalienstr. 44Geschaeftsfuehrung: C. Karch, F. Ramm
76133 Karlsruhe   Tel: 0721-1803560-3
reich...@geofabrik.de Fax: 0721-1803560-9



signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] issue with geofabrik europe update

2018-05-17 Thread Florian Lohoff
On Wed, May 16, 2018 at 05:50:19PM +0200, Frederik Ramm wrote:
> Hi,
> 
> On 05/16/18 14:43, Julien Fastré wrote:
> > We had a strange issue with a europe diff update from geofabrik: the
> > diff file is not a valid xml.
> 
> And that's entirely my fault for using "sed" to modify a couple of .osc
> files around the beginning of May. Sorry for that! It was after we got
> rid of uid/user fields in .osc files, and it turned out that some people
> has issues with the reduced files, so we decided to put dummy uid/user
> fields back in, and for the old files I quickly did that with a too
> broad search-and-replace command ;)

So you actually say that geofabrik pbf/osc files dont contain valid
usernames anymore?

I was wondering why i saw numerical usernames but if thats the case
i can drop usernames from my QA tools.

Flo
-- 
Florian Lohoff f...@zz.de
 UTF-8 Test: The  ran after a , but the  ran away


signature.asc
Description: PGP signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] issue with geofabrik europe update

2018-05-17 Thread Julien Fastré


Le 16/05/18 à 17:50, Frederik Ramm a écrit :

> 
> Sorry for that - I'm sure it took you some time to figure out what was
> wrong!
> 

Thanks for the explanation !

No problem for me: this is part of the job. I am happy to hear that
there is an explanation for this issue !

Julien Fastré

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] issue with geofabrik europe update

2018-05-16 Thread Frederik Ramm
Hi,

On 05/16/18 14:43, Julien Fastré wrote:
> We had a strange issue with a europe diff update from geofabrik: the
> diff file is not a valid xml.

And that's entirely my fault for using "sed" to modify a couple of .osc
files around the beginning of May. Sorry for that! It was after we got
rid of uid/user fields in .osc files, and it turned out that some people
has issues with the reduced files, so we decided to put dummy uid/user
fields back in, and for the old files I quickly did that with a too
broad search-and-replace command ;)

Fixed the file now (and another one in the "georgia-updates" dir).

> I wonder if we were the only one affected and, if not, how did you cope
> to pass this diff without error ?

Since the bug was introduced a day or two after the diff was published,
it is possible that other consumers of the diff who loaded it
immediately didn't run into the issue.

Sorry for that - I'm sure it took you some time to figure out what was
wrong!

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] issue with geofabrik europe update

2018-05-16 Thread Julien Fastré
Hi,

We had a strange issue with a europe diff update from geofabrik: the
diff file is not a valid xml.

The affected file:

http://download.geofabrik.de/europe-updates/000/001/872.osc.gz

We encounter an error when parsing this file with osmosis:


```
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse
xml file 872.osc.  publicId=(null), systemId=(null), lineNumber=972870,
columnNumber=158.
at
org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:114)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.xml.sax.SAXParseException: Element type "tag" must be
followed by either attribute specifications, ">" or "/>".
at
org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown
Source)
at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown 
Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:189)
at
org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:109)
... 1 more
```

This come from the line 972870, which is not a valid xml tag indeed:

>  v="http://ka.wikipedia.org/w/index.php?title=%E1%83%A4%E1%83%90%E1%83%98%E1%83%9A%E1%83%98:Tbilisi_Admin_Map.jpgfileuid="0;
>  user="" changeset="0" timestamp=20080104164655"/>

Removing this line (`sed '972870d' 872.osc > 872.fixed.osc`) make
osmosis able to parse the file.

I wonder if we were the only one affected and, if not, how did you cope
to pass this diff without error ?

Thanks,
Julien

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev