Creating a true history is impossible. The information about the
concurrent updates across different non-atomic changesets is
presumably lost.
You can easily recreate that by sorting everything by timestamp. Which
I think is the easiest/most obvious ordering there is.
Cheers,
Lars
On Mon, Aug 23, 2010 at 5:40 PM, Scott Crosby scro...@cs.rice.edu wrote:
On Mon, Aug 23, 2010 at 9:48 AM, Lars Francke lars.fran...@gmail.com wrote:
Creating a true history is impossible. The information about the
concurrent updates across different non-atomic changesets is
presumably lost
Anyone have or know of any example code for doing oauth with OSM in PHP or
perl?
http://oauth.net/code/
Just remember that OSM uses OAuth 1.0 and not 1.0a.
osm-f...@googlegroups.com? Got anything done yet?
Cheers,
Lars
___
dev mailing list
So no?
Well...OSM does nothing special in regards to OAuth so any tutorial will do.
This gets the access token no problem. But then what do you do with
the access token?
The access token is called token credentials now.
And this explains how an authenticated request is made:
Hi,
2: I am beginning a project to parallelize OSM data processing
with Hadoop, and the postgreSQL copy-format output is perfect
for loading into HDFS. (If this goes well, I'd want to discuss
ideas for adapting Osmosis to talk to Hadoop, eventually.)
that is very interesting. I'm doing the
Will a period in the key name for a tag cause a problem?
Generally: No.
You were asking about database problems: No problems at all in regards
to PostgreSQL. MongoDB is the only database I know that has problems
with dots in keys but you'd have to escape everything anyway, so no
problem there
I'd use name:full, name:prefix etc. for it. As far as I can tell it's
pretty much established practice and you'd make _at least_ my life
easier :)
How?
I've just rewritten parts of OSMdoc.com to support : in keys[1]
because they were the only delimiter in widespread use. So I naturally
hope
Hi guys,
Matt Amos was so nice to run the history export again.
The result is available here:
http://planet.openstreetmap.org/full-experimental/full-planet-100801.osm.bz2
and it's grown from 13 GB in February to 17 GB. The regular planet has
grown from 8 to 10 GB in the same time.
Have fun and
At SOTM I talked to Lars Francke about his history dump script that was run
once or twice a few months ago. He said it would definitely be feasible to
implement an extent parameter but that would require some significant
rewriting effort. I am willing to look into this but it's in Java
On Wed, Jul 21, 2010 at 15:42, Brett Henderson br...@bretth.com wrote:
I haven't looked at the full history dump to be honest so I'm not in a great
position to comment, but I'll comment anyway ;-)
I'm curious what the format of the full history dump is. I'd like to
understand how nodes, ways
I'm wondering if anyone out there could talk about how resource-intensive it
would be to set up a system to show the usage of any arbitrary tag/value
pair (or maybe just the top 5000?) over the course of time. Presumably it
would be something like what tagstat does, but the results would be
The code is here: http://github.com/iandees/mongosm
I just had a quick look at it because I've never used MongoDB before
and was interested.
This line[1] makes me want to cry (not your fault). There's always
this one catch with a solution that looks perfect otherwise. Now
you've got to escape
It looks like our schemas are mostly identical, but in my experience,
MongoDB used more and more time importing the index as the import continued.
An import of the dataset for TX took several hours, but import speeds
dropped off markably as the import continued and, presumably, as the
On Fri, Jul 2, 2010 at 20:52, Serge Wroclawski emac...@gmail.com wrote:
Similarly, Ian Dees and I have written a server using MongoDB, which
also provides functionality such as auto-sharding and built in
map/reduce.
Ah I remember that we've talked about this. Great!
Will any of you be at the
were there any successful attempts to read OSM data into CouchDB and
Geocouch? Does somebody know of a backend?
I have done something like that and can provide some code at the end
of July (I won't be back home before then). It really is just a
different kind of schema. But the exact schema
Mostly likely we will wind up adding a moderation queue and moderating
new users first few diary entries.
I don't have any developer experience with it but Akismet[1] seems to
work quite well on Wordpress and we seem to get the same kind of spam
I regularly see in my Wordpress spam queue. They
I currently use AMQP (RabbitMQ) for message processing and it works
very well. It is very flexible and it'd be easy to extend it with a
PubSubHubBub or XMPP output.
Mitja (of OpenStreetBugs) proposed just yesterday a filter that
filters changes by the tags/changes involved so it would be
I started working on a streaming XML output plugin for Osmosis. I was
intending to take advantage of PuSH/PubSubHub messaging and maybe even XMPP
(so that you get a 1-min delayed IM when someone changes something in your
bbox).
Anyway, TRAPI could use this same plugin to apply updates to
On Fri, Mar 5, 2010 at 20:59, Jeffrey Ollie j...@ocjtech.us wrote:
Has anyone looked at importing the planet into a Hadoop/HBase cluster?
I'm just learning about the whole clustering/cloud computing stuff so
it might not be a good fit but I might give it a try, if only as a
learning project.
On Sun, Feb 21, 2010 at 23:30, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote:
We were discussing maybe moving over to Git in #osm and I thought I'd
do a quick conversion just to show how easy it is:
Was there any discussion about alternatives to git?
I favour Mercurial myself. I know that
Disclaimer: This is untested as I didn't feel like setting up tagwatch
to be able to test it myself. I hope there is no escaping necessary.
Escaping is necessary[1]! Not escaping can cause problems for example
with the | character as you might receive a lot more data than you
expected[2].
Hi,
as you might know there is an experimental database dump of the full
history of the OSM data[1]. Good thing that Matt labeled it as
experimental as several small bugs were discovered (most by Anthony,
thanks).
These are the changes:
* Timestamps were off during GMT summer time. This should
Also it isn't the phonetic spelling that is really the problem - the
problem is that a lot of the time mistakes are typos not
miss-spellings i.e. Lonon which is phonetically completely different
to London despite being obvious that someone just missed the 'd' when
they were typing.
In that
I'd love to do something like this but have so far been unable to find
a suggest / phonetic algorithm that scales to 5.4 million
words/phrases across more than a hundered languages. The geocoder you
mention doesn't seem to do street level - which simplifies the problem
by at least a couple
A quick status update.
I've done some more work and pushed a first incomplete version to my
bitbucket repository[1]. I've never tested this so there are bound to
be bugs. Tom pointed out a mistake I made with the private traces. So
there'll be one public.gpx with all the trackpoints from all
On Sat, Jan 2, 2010 at 19:17, Anthony o...@inbox.org wrote:
On Tue, Dec 22, 2009 at 11:28 PM, Lars Francke lars.fran...@gmail.com
wrote:
Are there any other bugs/strange things you've encountered? I haven't
had the time to look at the final file myself.
The num_changes data is weird.
http
On Wed, Dec 23, 2009 at 04:28, Anthony o...@inbox.org wrote:
Hmm, the full-planet dump seems to be doing something weird around daylight
saving time. See, for example changeset id=2975
created_at=2006-03-26T00:51:16Z closed_at=2006-03-26T02:51:23Z
open=false num_changes=7 min_lat=52.5450951
Ah yes. Hmm. That said, most of the characters actually in the database
are carriage returns, which along with tabs and line feeds (also in the db)
are valid in XML. Other characters are present - for instance ASCII 3 in
http://www.openstreetmap.org/browse/changeset/1325382 - those will be
Have you been considering how to handle the history of old anonymous edits?
This new history data should not reveal those user names but keep them
anonymous.
User IDs and Usernames for those anonymous edits are left out of the
dump. So the elements may not have an uid or user attribute.
Currently I need about four days for importing the whole planet file. Is
that a normal computing time? The bottleneck seems to be the database
(PostgreSQL). Are there any hints for the database configuration? For
conversion I use a quadcore machine with 8GB RAM. Data files are
located on an
I assume it doesn't include the history that was deleted when segments were
converted to ways?
No, it doesn't.
I recall that it was archived somewhere. Would it be possible to make that
available in some form?
Personal data etc. needs to be stripped from that data first.
Tom tried to load
Hi Peter,
as far as I can see all that you've described will be available in the
new OSMdoc version (including 'minor' versions when a node for a way
is updated).
This is if everything goes according to plan and I don't have to
delete a whole bunch of data because of the license change.
But the
I'd take a look at it if no one else is working on it but if someone
else is interested I'd gladly step back :)
As no one has stepped forward with any ideas or offers to implement
this I'll take a shot at it. This also means that I get to define the
format of the metadata. Once I've worked that
On Sat, Dec 5, 2009 at 10:55, Tom Hughes t...@compton.nu wrote:
On 05/12/09 01:27, Lars Francke wrote:
The easiest for now would be to just throw all the GPX files that are
marked as Identifiable or Public in a compressed file and add a file
with (XML?) metadata about those traces.
Actually
We've (Matt Amos, Tom Hughes, Ævar Arnfjörð Bjarmason and myself) just
had a discussion on IRC about an export of the GPX/GPS-data (Matt
called it planet.gpx) and I'd like to post our findings here and
hope to get some more ideas/feedback in.
A few notes:
* All the uploaded GPX files are kept in
The only thing I see potentially getting in your way is that the OSM
database doesn't store enough significant digits of lat/lon
coordinates to make indoor mapping viable, but perhaps it does. I
couldn't find documentation on how many digits it stores and how that
translates approximately
Both apps tag the change set [2] rather than the
node or way that is created. I'm wondering if this is why there are
no stats on the editor's usage in Tagwatch? If this is the case, how
can I help update Tagwatch to look at change sets?
That is almost certainly the case. Tagwatch uses a
I am partly done with my Java version. There are a few
questions/problems/remarks:
Is Java really up to this job from a performance point of view?
I haven't done any performance comparisons between planet.c and my
program but I believe that there won't be much of a difference. I
could rip out
A quick status update and a link to the code.
- I decided to dump num_changes too
- One thing that startled me: planet.c converts _all_ relation member
roles to lower case before dumping them. I'd consider this a bug but
I'm sure there is a reason for this. Considering that neither the API
nor
There is a dump, but it's a mysql dump so not easily readable. There may
be
a planet around somewhere as well but I don't think it will be
synchronised
to the actual shutdown time or have any history.
If you'd be willing to share the mysql dump (I of course wouldn't need
user- or any other
I understand that a lot of this data is
available throughout the web using old snapshots and diffs but this
comes in outdated formats and is by no way complete or easy to use.
Keep in mind that while a full database dump will give you some things that
are not in the old planet files, but the
Andy,
Ultimately the formation of a mini project is probably needed. Input from
those like yourself willing to work on it and the will and time from others
who would need to support the work, including sysadmins.
I had hoped that this could be (re-)solved rather unbureaucratic and
without the
- Am I correct in assuming that there are no general objections from
the OSM server folks against such a dump? (Which would render the rest
of this E-Mail useless ;-)
the response has always been if someone writes it, and it's good,
we'll run it :-)
That's all I wanted to hear :)
(3) is
I had not thought of that. When I first used OSM, segments were long
gone so I tend to forget those. There are a few planet dumps from
those time but none before 060403 and no diffs (which would be
required to fully reconstruct the history).
No, diffs only show the changes between two points
Hi!
I and many (okay at least a few) others have shown interest in the
complete history data of OSM. I understand that a lot of this data is
available throughout the web using old snapshots and diffs but this
comes in outdated formats and is by no way complete or easy to use. I
also had a look at
I'd like to include full changeset information in diffs but it's not
trivial. I'm not sure if I'll ever get to this personally. I'd love to
see somebody take it on though.
I'll have a look at it but I don't want to get your hopes up :)
I had a look and my initial enthusiasm has been
2)
It seems as if none of the diffs contain the changesets. I may have
missed something here but the only way to get these seems to be the
weekly dump of all changesets or by using the API? As I use the
changeset tags for OSMdoc I'd be very interested why this is the case
and if there are
Hi!
I'd like to set up a new consistent database of the OSM data for
OSMdoc. I've been out of the loop for a while so I've got a few
questions.
1)
If I understood it correctly there is a new type of diffs - the
minute-replicate - which are guaranteed to contain every change but
they are not
No, I meant the tokens which would be used if authentication is done with
https and
data transmission without. Somehow these two parts have to be connected, a
token transfered
with the authentication would be one solution.
This token could be intercepted and used to send unauthenticated map
If HTTPS is ever offered we have two options (as we do now):
A third option with a non-standard auth token being generated was discussed
in this thread, and that's probably what Stefan was referring to.
I must have overlooked that. In that case I'm sorry Stefan, seems as
if _I_ was the one
- signing and checking a signature is essentally encrypting and
decrypting (CPU?), but with less data
Indeed. But as far as I know HMAC-SHA1 is reasonably quick and
shouldn't be a performance hit. Even more so because it needs to be
done only once for each 'consumer' (JOSM in this case).
-
On a related note: There was a change between PostgreSQL 8.3 and 8.4
which increased the default_statistics_target from 10 to 100 and its
maximum from 1.000 to 10.000 which makes ANALYZE runs take longer but
at least for me it helps a lot.
Have you, too, noticed the 10% performance
1. Doing a vacuum immediately after the import isn't useful, there is
nothing to vacuum by definition.
While this is true an ANALYZE should still be done to initialize the statistics.
2. Doing a vacuum after a diff is also useless, since the vacuum will
probably takes much much longer than
it would be nice to have the API operate in several different formats,
for reading and writing.
Is that not just another form of API bloat? Could people not offer any
format they want by writing either a proxy or, if they don't need write
access, a mirror server?
I don't know about Ruby on
+1
This has needed a tidyup for a long time.
I'm done with this change.
I have renamed every relevant page to API v0.x and API changes
between v0.x and v0.y and I tried to correct every link to point to
the new name. I might have missed some references but...it should
work, because the old
Is the old page redirecting to the new page?
All old pages should redirect to their new versions, yes:
http://wiki.openstreetmap.org/wiki/OSM_Protocol_Version_0.3
http://wiki.openstreetmap.org/wiki/OSM_Protocol_Version_0.4
http://wiki.openstreetmap.org/wiki/OSM_Protocol_Version_0.5
Hi,
I already asked this on the Talk page for the API
(http://wiki.openstreetmap.org/wiki/Talk:API) but I don't know how
many people will be reading that.
* I'd like to rename (move) Protocol to API on the Wiki:
OSM_Protocol_Version_0.6 - OSM_API_Version_0.6. API is the commonly
used word and it
In case anyone else missed this (at least I couldn't find any
announcement): OAuth seems to be live on openstreetmap.org.
Cheers,
Lars
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
On Tue, Aug 18, 2009 at 09:54, Jochen Topfjoc...@remote.org wrote:
I'll need output in the following form:
tag-key, number of changesets, nodes, relations and ways this key is
used on, number of distinct values
tag-value, the tag-key this value belongs to, number of changesets,
nodes,
Hi,
I'm in the process of updating osmdoc.com after several people have
reminded me that the data is a bitold :)
The following explanations are rather technical and Java centric but
the underlying problem should be language independent.
Previously I used several Map/Reduce jobs running on
http://repo.or.cz/w/handlerosm.git?a=blob;f=osmsucker-ywk.c;h=1cf6e1dee9f22d7b4e161984ce3b404923c7cf89;hb=HEAD
This is rather /fast/ ;)
But does it do what I need? It seems to process a .osm file and save
it in its entirety in a database (or write files suitable for
import/COPY), just like
I need to aggregate data about the number of tags, tag combinations,
keys and values. See http://osmdoc.com for the kind of data I need.
Do something smart on line 346 for on the fly counting. Though I think
that I can generate the page that you provide maybe faster using just
group by on an
Hi,
You might want to have a look at the source code for tagstat at
http://trac.openstreetmap.org/browser/applications/utils/tagstat
that looks very interesting. I didn't even know that there was another
Tagwatch like program. Thanks!
I had a look at your PlanetReader.java and it very much
So please elaborate what combinations you need ;) Since the output is
csv /anything/ you imagine is there ;) and if you don't want to code,
just add what you want and run uniq -c on it.
I'll need output in the following form:
tag-key, number of changesets, nodes, relations and ways this key is
Can we start a thread discussing what is required/wanted in a changeset
tool? What should such a tool do besides revert changes from changeset x?
I think it would be best to first concentrate on some kind of OpenAuth
support (or something homebuilt) that would allow a third-party
application
Can we start a thread discussing what is required/wanted in a changeset
tool? What should such a tool do besides revert changes from changeset x?
After re-reading your mail I see that I too missed your point, sorry.
I'd like to have a tool which can revert a single changeset by id and
it
But honestly, what would you say from a user interface perspective;
should we try and keep changesets open? What JOSM currently does is open
changeset - upload - close, for each upoad action. Some people have a
habit of uploading once every five minutes and they are perhaps annoyed
to have to
Hi,
I just found the following line in the ruby code for the tags:
validates_length_of :k, :v, :maximum = 255, :allow_blank = true
Does this mean that Key and Value may only be 255 characters long? And
that both may be empty? I don't know Rails very well but it sure seems
so
Lars
I just answered my own question. Matt already updated the API page.
255 unicode characters it is.
Thanks,
Lars
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
The 255 limit is from MySQL. For the timebeing we are are trying to be
compatible with MySQL for people who are still using MySQL. Many developers
will not have had time to switch their development/production databases from
MySQL to Postgres yet.
But MySQL must have a way to save text with
I finished most of the methods now. Some may be incomplete
but...well..work in progress ;-)
A few comments though:
- The changeset methods allow multiple changeset XML elements and
the resulting changeset will be a combination of them all. The
node/way/element methods allow multiple elements,
- In routes.rb there is no api/0.6/relation/:id/:version to retrieve
an old version of a relation but all the neccessary methods are there
(old_relation_controller#version) but there is a type (see attached
diff)
type - typo
- For ways for node and relations for element there is no error
- In routes.rb there is no api/0.6/relation/:id/:version to retrieve
an old version of a relation but all the neccessary methods are there
(old_relation_controller#version) but there is a type (see attached
diff)
I see that someone by the name of smsm1 has fixed a typo.
Unfortunately it is
Hi,
I have updated quite a lot on these two pages:
- http://wiki.openstreetmap.org/wiki/OSM_Protocol_Version_0.6
- http://wiki.openstreetmap.org/wiki/OSM_Protocol_Changes_V0.5_to_V0.6
(changed from a redirect to its own page, not complete)
My main changes are on the API 0.6 page. I tried to
Hi,
Osmosis used to create a bbox column for the ways table. As I didn't
need this I replaced all its queries with queries to create and update
a LINESTRING-column instead. This works flawlessly and as I understand
this feature has since been added to Osmosis (I'm using 0.29.2). As
this update
Hi,
I doubt if I'll
be able to look at this myself soon so feel free to experiment with
improvements.
and thank you for your comments and clarifications. I will have a look
at this but it will take me some time. I'll get back to you when I
have something to report.
Lars
Hi!
Thank you for this tool and for the recent update.
I had a look in my area and saw two errors of the railway crossings
without tag kind.
But these nodes were indeed tagged with railway=crossing which is
documented in the wiki as A point where pedestrians may cross.
Could you adopt this check
Hi,
I'm using this command line to force JOSM to use my c\OSM\JOSM\data
directory instead of the one in Documents and Settings:
java -Djosm.home=c:\OSM\JOSM\data -jar josm-latest.jar
And so far it seems to work perfectly.
Lars
___
josm-dev mailing
With Potlatch it shouldn't be hard to add something, although this does
require registering and maybe a bit of reading.
I had wondered before why there wasn't a link from the main map, so I'm
all for someone adding it.
There's no link because I'm waiting for somebody to do a properly
79 matches
Mail list logo
