[OSM-dev] Shutting down opestreetmap-mirror, can hand it off if anyone cares
Some of the repositories at https://github.com/openstreetmap, I think https://github.com/openstreetmap/josm is the most popular, are mirrored from Git or SVN via scripts hosted here: https://github.com/openstreetmap/openstreetmap-mirror I set this up in 2010 and have been running it ever since. I'm not active in OpenStreetMap these days, and the server it's been running on will be decommissioned in the next few days. It's done via the https://github.com/openstreetmap-mirror user. If anyone's interested in continuing this service I'd be happy to hand off that GitHub user to someone (not that a new one couldn't be created). The scripts / crontabs are already in the GitHub repo above. They currently take around 2GB of storage. The problem is not that I don't have a server to run this on (I have one). I'd need to set this up again, and don't want to spend time on doing that or maintaining a service that I myself no longer use. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] An update on github.com/openstreetmap
On Tue, Jan 3, 2012 at 07:49, Philipp Borgers borg...@mi.fu-berlin.de wrote: Not sure, if this is the right place... It seems someone's running a buggy cron job, that creates new branches on mapnik-stylesheets. In my github news feed I'm getting items like: openstreetmap-mirror created branch mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/master at openstreetmap/mapnik-stylesheets Ævar Arnfjörð Bjarmason told me yesterday a bug fix is on the way. Yeah I screwed up the pushing of multiple git-svn branches. I'll try to get around to fixing it soon-ish. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Can I get distance across dry land with OpenStreetMap
I have the following problems I want to solve with OpenStreetMap data: * For points A and B, detect if either one is on an island, and that you therefore can't traverse between them without a swim. * For points A and B, get the distance between them by land. E.g. if A and B are on opposite sites of a fjord. The routing libraries I've found seem to all assume that I want to route across roads, whereas for this task all I need is just a coastline shapfile of the planet. I'd then find out if A and B are on different coastlines, or the shortest path between them on land. Is there anything that does this already? I'm fairly sure I could do some cleverness with PostGIS to find out whether the line between points A and B crosses the ocean, but what I actually want is the distance between them sans ocean. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] https://github.com/openstreetmap is now not only a mirror
Paul Hartmann requested permission to use github.com/openstreetmap for a non-mirrored project, beboj. I thought it was a great idea and the openstreetmap github account was subsequently turned into an organization by Paul. It's still mainly a mirroring account, but if anyone else would like to develop openstreetmap-related code there adding you as a member of the organization would be most welcome. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Release candidate for OSM binary format is in osmosis trunk.
On Wed, Sep 22, 2010 at 23:50, Frederik Ramm frede...@remote.org wrote: I always viewed the Geofabrik extracts as a service to the community who build interesting things, or try out stuff - and who can also be counted on for a little bit of willingness to adapt to change, or add a little extra program to their toolchain if required. You're making it sound as if the Geofabrik extracts are meanwhile a staple ingredient of consumer-facing services (just download gpsmid here and get your data from Geofabrik and you're good to go) - consumers who need to have things served on a silver platter and who cannot be bothered to install software or convert files or something. I'm not entirely comfortable with that. I think if you really are in a situation where you need a guaranteed 6 month notice period before anything changes, you should calculate some buffer on *your* side - e.g. you should definitely not assume that certain files will be at a certain place on a certain server all the time, but instead have some sort of directory that is under your control. You cannot burden Geofabrik - or any OSMer running something in his spare time - with keeping your application happy. Especially not if you only tell them about your application depending on them when they want to make a change! Having said that, I won't switch off the bz2 stuff tomorrow but I certainly think that I'd prefer to use Geofabrik resources to produce more extracts for more people than to keep things running smoothly for end users of an application somewhere. The Geofabrik download stuff is primarily intended to be low-level - a building block for members of our community, not a shrink-wrapped service for the world. I use the geofabrik services and I must say I'm slightly irked at the tone this discussion has taken. That you're providing these services for free *at all* is not a given, and it's great that you continue to do so. If you dropped osm.bz2 tomorrow with zero warning I'd at most have to download some file and add another osmosis command to some shellscripts. I can't see how that could possibly be a big deal for anyone, and hopefully the resources being freed up on your end by not grinding through gigabytes of XML every day could be spent on something more worthwhile as a result. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Marble-devel] NOTICE: gazetteer.osm.org being retired
On Wed, Sep 1, 2010 at 13:26, Chris Browet c...@semperpax.com wrote: If you want to use an OSM service in one of your applications, then I would suggest firstly, and most importantly, that the application doesn't have a *.openstreetmap.org URL embedded anywhere in the code. Come on. Does that mean that we cannot use www.openstreetmap.org/api to update the database? Well, in the context of stable OS releases that distribute programs like Merkaartor the API is pretty much *guaranteed* to break. An Ubuntu LTS release is 5 years, Red Hat and Debian have something similar. That's a lot less time than the entire 0.5 - 0.6 transition took. Didn't we have existing packaged programs that used 0.5 break for this exact reason when we switched to 0.6? If not we're certain to have it if we ever switch to 0.7, depending on the nature of the API changes. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Marble-devel] NOTICE: gazetteer.osm.org being retired
On Thu, Sep 2, 2010 at 15:00, Chris Browet c...@semperpax.com wrote: On Thu, Sep 2, 2010 at 16:39, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: On Wed, Sep 1, 2010 at 13:26, Chris Browet c...@semperpax.com wrote: If you want to use an OSM service in one of your applications, then I would suggest firstly, and most importantly, that the application doesn't have a *.openstreetmap.org URL embedded anywhere in the code. Come on. Does that mean that we cannot use www.openstreetmap.org/api to update the database? Well, in the context of stable OS releases that distribute programs like Merkaartor the API is pretty much *guaranteed* to break. An Ubuntu LTS release is 5 years, Red Hat and Debian have something similar. That's a lot less time than the entire 0.5 - 0.6 transition took. Sure, but again, LTS is for business, and I don't see a business using Merkaartor ;-) And the 0.5 - 0.6 was announced, what, 6 months in advance? With discussions and plannification over 1 year... It really doesn't matter in this context how long it was planned for. With API 0.6 we didn't have full editor support until relatively shortly before the deployment (actually only proper support long after that). Any distro that took a snapshot of an OSM editor a few months before the switchover for its stable release would have ended up with an editor that didn't work anymore. As long as our API transitions are shorter than major distro releases (i.e. shorter than 6months - 2 years or so) this sort of thing is bound to happen. But there's ways to mitigate that, from an earlier E-Mail you wrote: Now, name search on 0.16.x which is going to be included in the newest Fedora and Debian won't work :-( Have you tried to contact upstream about this? This sounds like something that they'd be willing to patch even in a stable release. It's a very important feature (in a relatively unimportant[1]. program) that won't work *at all*. Aren't they willing to patch that? You'd want me to spend $12 a day to provide geolocalisation for an OSM editor (if you didn't read the thread, I remind you I'm speaking of Merkaartor)!!?? What sort of crazy hosting costs $360 per month (asking Nic Roets)? I bet you could host basic redirection services on a 512 Linode for $20/month. Possibly even a whole nominatim instance (I don't know its resource demands). Anyway, just pointing the service to your own domain costs next to nothing. And that seems like a very basic precaution that programs like Merkaartor should take so they can deal with cases like these. Have fun. 1. Unimportant in the context of the OS, i.e. it's not like Merkaartor is glibc here ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] no latest built for 5 days
On Fri, Aug 13, 2010 at 18:27, Dirk Stöcker openstreet...@dstoecker.de wrote:
On Fri, 13 Aug 2010, Ævar Arnfjörð Bjarmason wrote:
What sort of automated job is it? If it's in cron you can easily make
cron send you an E-Mail on error.
It is a cronjob, but normally it is too noisy, so all outputs go to
/dev/null. It is not so easy to get cronjobs to only produce ouput when
something fails.
It is actually, just install this: http://search.cpan.org/dist/App-Cronjob/
Then put something like this in your crontab:
@daily cronjob -E -j josm-build -E /path/to/build.sh
cronjob(1) will consume all the output, and either print it all or
nothing, depending on the exit code of the program it's running.
If you're running a POSIX shell script you can add this to the top of
the script:
# Exit on errors
trap 'fail' ERR
fail () {
code=$?
echo Failed with exit code $code
exit 1
}
Then you don't have to check the exit code of everything individually.
I use this for all my cronjobs, see
e.g. http://github.com/avar/openstreetmap-mirror for an example, it's
what I use to get E-Mails when the JOSM GitHub mirror fails, but
*only* when it fails.
___
josm-dev mailing list
josm-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] OSM Wiki now uses the Vector skin
On Sun, Aug 8, 2010 at 10:31, Eugene Alvin Villar sea...@gmail.com wrote: Nice! When did this happen? I only realized that the OSM Wiki was using the new skin just now. :-) It was recently upgraded to MediaWiki 1.16, then the skin settings for logged-in users were changed. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] licensing question
On Wed, Aug 4, 2010 at 21:55, Jim Burt jimb...@gmail.com wrote: Greetings, I'm interested in using OpenStreetMap for an internally facing website for my company (no public access). This is going to be a small app with very limited users and page views. I've reviewed the licensing, and this appears to be acceptable use. Can anyone confirm this? Yes, I'm also pretty sure that you can just completely ignore the license since you're only using the data within your own organization. The licence only kicks in when you're doing re-distribution to third parties. (See e.g. Google's use of Linux, they don't re-distribute their changes, and don't have to) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] licensing question
On Thu, Aug 5, 2010 at 12:31, Pieren pier...@gmail.com wrote: On Thu, Aug 5, 2010 at 1:32 PM, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: Yes, I'm also pretty sure that you can just completely ignore the license since you're only using the data within your own organization. The licence only kicks in when you're doing re-distribution to third parties. Could you point out in the license text where own organization and third party are defined and where it says that CC or BY do not apply for the first ? Copyright law covers distribution. As long as you're not distributing a work to third parties you don't have to worry about it. If you haven't signed a contract nobody can force you to adhare to a license for a copyrighted work until you start distributing it. I can scribble on my Sunday newspaper without caring about copyright, but if I distribute it to other people I have to start worrying. The same thing applies to corporations. The Sunday Newspaper Inc. can distribute a proprietary third-party image around their office while they're deciding what to do with it. They don't have to worry about contacting someone else for royalties until they put it in print and thus start distributing the work to third parties. So I don't see why someone couldn't completely ignore the CC-BY-SA when using OSM for personal or intra-office use. They're not re-distributing anything, so the licenes doesn't apply. The above is just my understanding of copyright law. I'm not a lawyer and it may all be BS. But as far as I can see it's backed up by several examples in the wild. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
On Sun, Aug 1, 2010 at 17:33, Andreas Kalsch andreaskal...@gmx.de wrote: What about some metrics (performance, size)? Data is the same, whether binary or not. So binary really has to pay off significantly. What performance metrics would you like that haven't already be covered earlier in this thread, and in the initial announcement? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
On Sun, Aug 1, 2010 at 21:24, Frederik Ramm frede...@remote.org wrote: I prefer SVN over git for the simple reason that I only have to svn up and everything is there but I'm sure it is going to be a matter of minutes before someone from Iceland points out that the same convenience can be had with git if one knows what they're doing ;) Why would you think that, I live in Germany now :) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Maposmatic-dev] Re: Growth of on-disk size of OSM database
On Wed, Jul 14, 2010 at 10:56, Thomas Petazzoni thomas.petazz...@enix.org wrote: Unfortunately, as far as I understand, doing a vacuum full is going to block accesses to the database for a fairly long amount of time (days ?). So the best solution is probably to regularly (every few months) do a new full import in a separate database, and switch to this new database when the full import is completed. This is why PostgreSQL administrators usually put VACUUM FULL; in their cron.daily, but that doesn't help you at this point. You could also, as you say, import a new DB and swap the old one out. That's what I do with my own (not full Planet) import. Or you could use MySQL instead of PostgreSQL (if that even works anymore). It doesn't use the same model for deletion as PostgreSQL, so you'll reclaim your space without expensive VACUUM operations. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Today's hacking project: GitHub OpenStreetMap mirror
On Tue, Jun 29, 2010 at 18:48, Ian Dees ian.d...@gmail.com wrote: Looks like github just came out with organizations. Would it make sense to convert the openstreetmap account to an organization? Link to convert is here: https://github.com/account/organizations Info about organizations feature: http://github.com/blog/674-introducing-organizations Thanks for the heads-up. It seems like the main benefit is unified account management, which we don't have to deal with since the openstreetmap account is (still) only for mirroring. If some people are interested in commiting there it'd be useful to change it over. But as it is I'd rather have Public Activity on the right pane of http://github.com/openstreetmap, rather than Organization members. All other things being equal. Maybe I've missed something and there are other benefits. Thanks for the heads-up, in any case. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Field lengths in Rails port
On Wed, Jun 30, 2010 at 08:11, John Smith deltafoxtrot...@gmail.com wrote: On 30 June 2010 17:46, Andy Allan gravityst...@gmail.com wrote: Postgres ignores them, IIRC, since it doesn't have such a concept. I'm sure one day some numbers will get too big for mysql, but hey. Doesn't MySQL support BigInt (64bit) and UUID (128bit) numbers? Yes, but it generally handles larger types much worse than real databases. E.g. you can't have an index for a TEXT, but you can index VARCHAR(255) on InnoDB. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Field lengths in Rails port
On Thu, Jul 1, 2010 at 01:50, John Smith deltafoxtrot...@gmail.com wrote: On 1 July 2010 00:32, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: Yes, but it generally handles larger types much worse than real databases. E.g. you can't have an index for a TEXT, but you can index VARCHAR(255) on InnoDB. I thought we were only discussing numeric types, but I think you can index varchar regardless of the DB type, and you avoid TEXT fields like the plague since it will use the hdd to build the query... I was interjecting a factoid about MySQL's type handling in general as a point of reference. And as I said the difference is that text fields have to be VARCHAR in MySQL for it to index them, unless you use MyISAM FULLTEXT indexes. You don't have to avoid TEXT at all in databases that don't have such arbitrary restrictions. PostgreSQL handles it just fine, so does SQLite for that matter. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Feature Request: Timed Save
On Wed, Jun 23, 2010 at 20:51, MP singular...@gmail.com wrote: On my old computer JOSM was quite a favorite target for OOM killer (1 gb java heap) or prone to OutOfMemory exceptions (500-600 mb java heap). I think neither of these can be fixed in JOSM. Well, JOSM could use on-disk storage (like SQLite or PostgreSQL), but that would be a pretty major change. Although perhaps it's easier now with more encapsulated access to the storage backends. But a feature like this sounds useful, although personally I just get into the habit of making very quick and granular uploads. ___ josm-dev mailing list josm-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Feature Request: Timed Save
On Wed, Jun 23, 2010 at 10:37, Nick Elliott nickelli...@ntlworld.com wrote: Is there any chance the option of timed saves could be added to JOSM? I find the program does occasionally exit without warning and I just lost about 25mins of editing.. Doh. That may be a good idea, but if JOSM is randomly crashing on you we should try to fix that first. Do you get anything in the console log when it does this? What are you doing at the time etc. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Rails port - current source, and tracking changes
On Thu, Jun 10, 2010 at 07:09, Frederik Ramm frede...@remote.org wrote: now that the rails port has moved to git, is there a way to find out which source the servers are currently running? No, the only way for that to happen would be if the rails port itself was changed to output `git describe` on some version page, or if the live version was otherwise made public. git clone git://git.openstreetmap.org/rails.git magically yield the live version? And is there a way to track changes to the live version (as opposed to track stuff that someone has uploaded somewhere which may or may not ever go live)? While git://git.openstreetmap.org/rails.git may contain changes that haven't gone live yet it's a pretty sure bet that changes there /will/ go live. So you should track it if you want the real version. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] building mapnik tirex - missing dependency information
On Thu, Jun 3, 2010 at 18:07, Peter Körner osm-li...@mazdermind.de wrote: Am 02.06.2010 23:32, schrieb Frederik Ramm: Better still, build a proper Mapnik Debian package which is what I usually do. Yan you explain in short how to do this? I'm new to the process of building packages myself and would like to use this way in my HowTo. I's nowhere explaines on the Mapnik-Wiki and it seems not be part of the scons.py -- but I don't know scons either. Frederik probably has a better way, but one way of doing it to just add unstable as a apt archive and install libmapnik-dev from there instead of stable. That'll get you the latest 0.7.1 mapnik, instead of 0.5.1 in stable. You can also apt-get source libmapnik-dev from unstable to get the source, including the debian/ directory used to build it. That'll allow you to build your own from source using Debian's scripts. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Complex JOSM filtering rules
On Wed, May 26, 2010 at 13:27, Gregory nomoregra...@googlemail.com wrote: If you do Edit-Search in JOSM it gives you a load of examples of what you can do. I presume (and Ævar seems to be saying) this is the same searching engine/code that the filters use. Yes, the filtering engine is powered by the search engine. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Complex JOSM filtering rules
On Wed, May 26, 2010 at 13:50, John Smith deltafoxtrot...@gmail.com wrote: On 26 May 2010 23:22, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: Filters are just search queries, so you can do e.g.: I'm aware of that... but... boundary=administrative -(highway=* | waterway=* | natural=*) Doesn't do what I'm trying to do... If you want to hide boundary relations you can put that query in a () and OR it together with one that searches for ways that are part of a boundary relation, use the child/parent operators for that. That's what I'm asking, how do I hide all admin boundaries that don't have highway, waterway, natural tags because I haven't figured out a combination of filters that will do what I'm trying to do. Your original question had a double negative. If you want to inverse the query I gave you just remove the - from it. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Complex JOSM filtering rules
On Wed, May 26, 2010 at 14:10, John Smith deltafoxtrot...@gmail.com wrote: On 27 May 2010 00:07, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: Your original question had a double negative. If you want to inverse the query I gave you just remove the - from it. Perhaps I'm not explaining myself clearly, ideally I want to completely hide admin boundaries all the time, except some admin boundaries are also tagged as waterways and highways and so on, and I'd like to be only able to see those admin boundaries, long term I'm hoping to de-merge these ways completely, but for now finding them and dealing with them would be nice. I think you've explained yourself clearly enough, and I've given you a search query that finds boundary=administrative ways, either with the waterway/highway tags, or not. You just have to feed that into the search/filter engine. Why doesn't that work for you? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Complex JOSM filtering rules
On Wed, May 26, 2010 at 14:32, Alan Mintz alan_mintz+...@earthlink.net wrote: At 2010-05-26 06:27, Gregory wrote: On 26 May 2010 14:22, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: On Wed, May 26, 2010 at 13:08, John Smith deltafoxtrot...@gmail.com wrote: I want to be able to hide all admin boundaries in JOSM using the filter method, except admin boundaries that have highway=* or waterway=* or natural=* tags also, I haven't been able to figure out the right combination of filters to achieve this. Filters are just search queries, so you can do e.g.: boundary=administrative -(highway=* | waterway=* | natural=*) Maybe it was just me, but it wasn't obvious that you logically AND search terms by just space-separating them. I realize, upon close reading of the examples, that the first one implies this*, but maybe it should be re-emphasized down by the Use | or OR to combine with logical OR. *When I originally read this example, I was thinking it only works with simple text searches, just like putting two space-separated words in a search engine. It wasn't obvious that the terms could be more complex expressions. Maybe JOSM's documentation sholud be better, but that's how pretty much all search engines work. When you search for foo bar you're searching for foo AND bar. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] start of gwtosm the google webtoolkit port of josm
On Wed, May 26, 2010 at 19:17, Frederik Ramm frede...@remote.org wrote: jamesmikedup...@googlemail.com wrote: I have refactored some core classes in josm to run in gwt. this includes most of data classes. I am porting the graphic output to use gwt graphics and svg output. Might be worth taking this to the josm-dev list. Also, to those unfamiliar with gwt, explain the advantages. These examples might be useful: http://code.google.com/webtoolkit/examples/ Basically it's a Java to JavaScript compiler and UI toolkit. You can write Java which is then magically compiled to a web application. This is a great development. It might also help to generalize the JOSM core more, right now it's very tightly bound to its current UI. It would be neat if there was a Java libosm that could be used for stuff like this, and things like Android apps. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Java Applet
It would be very interesting if were un-buggy enough to run via the Edit button on openstreetmap.org. I've wanted to make the editor the user runs via the Edit button optional for some time. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] How to test against the API ?
On Mon, May 10, 2010 at 13:19, sorel johann.so...@geomatys.fr wrote: I am wondering if I can test my code against this server : *http://api06.dev.openstreetmap.org/* ? Can I mess up on this server ? like transactions that modify datas ? Or must I use something else ? You can test against api06.dev.openstreetmap.org, that's what it's for. You can also run a version of the API locally which might be faster easier. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] drop in users
On Sat, May 8, 2010 at 23:42, Gregory nomoregra...@googlemail.com wrote: As such, I think it might be nice if a geodata blog post was made or a message sent to the talk list. Simply saying a system has been rolled in that detects and suspends accounts for checking should suffice, without giving away spam-secret formulas. There's no secret sauce; The source code is all in Git. Good spam systems don't suffer from their internals being known. Besides, I don't think that those spamming OpenStreetMap blogs are proficient enough to read the code and come up with a targeted attack even if that were the case. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] tile caching for webkit?
On Fri, May 7, 2010 at 13:01, Claudius claudiu...@gmx.de wrote: A user from a country with very limited internet access it asking if there are any possibilities to make webkit use a cache for the tiles it fetches from a WMS-server? Any ideas? I don't know, but doesn't webkit use the system proxy settings if they're available? If you you can set up a squid cache in front if it with an aggressive cache policy. E.g. cache all images for 1 month no matter what the cache headers say. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] sporadic wiki access problems?
On Sat, May 1, 2010 at 15:45, Grant Slater openstreet...@firefishy.com wrote: On 1 May 2010 16:32, Michael Kugelmann michaelk_...@gmx.de wrote: Also yesterday there was some kind of hang short after lunch. And I would exclude network problems at my side because the observations had been made on totally different networks/providers on my side. Is there something known at the server side? Can it be fixed? Yes, there was a problem but should hopefully now be fixed. Unfortunately yesterday I was slow to respond. (day job and all) I'm still up for helping to admin it. And I'm doing more MediaWiki hacking these days too:) 1) Mediawiki Page hit counter is enabled. Seems under high load the hitcounter table has been deadlocking, causing the wiki to slowly ground to a halt. I've increased the '$wgHitcounterUpdateFreq' and am considering disabling the page hit counter completely. Yeah, you should do that. It's disabled on Wikipedia for exactly this reason, and it's meaningless in any case if you wrap MediaWiki in Squid. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Today's hacking project: GitHub OpenStreetMap mirror
I'm not at the hack weekend but today I moved the josm/openstreetmap-website mirrors to a new account on GitHub: http://github.com/openstreetmap I also set up a mirror of Merkaartor (whose primary hosting is on Gitorious). This makes it easy for GitHub users to monitor progress on these projects, and makes it easy to develop/update patches against them. If anyone's interested in having something mirrored from OpenStreetMap's SVN (or even converted), tell me. I'll try to make it happen. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Git mirror of JOSM on GitHub
On Sat, Apr 24, 2010 at 02:05, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: I got tired of waiting for `svn up' when rebuilding JOSM and wanted to commit some local patches. So I made a Git mirror of JOSM that I and others can use. It's very handy for getting faster updates, a local history and the ability to patch JOSM without SVN write access: http://github.com/avar/josm I've now moved it to http://github.com/openstreetmap/josm Update your bookmarks in the unlikely case that you had any. avar/josm will be deleted soon-ish in favor of the new mirror. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] OAuth secure ?
On Thu, Apr 29, 2010 at 12:04, colliar colliar4e...@aol.com wrote: I thought at least with semi-automatic use OAuth was transfering with encryption ( and should also now with https) , but there is still a warning about no secure possibility on the wiki. Am I wrong or do we need to change this page. The wiki is wrong and needs to be brought up to date. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
[josm-dev] Open offer to convert JOSM to Git
In case it eluded anyone I'm pretty fond of Git [1][2]. The rails_port has now switched over (with some help of mine), as well as Merkaartor (with no help of mine). I've also set up Git mirror of JOSM on GitHub[3] which I plan to keep up to date. If the JOSM project is ever interested, I'd be willing to do all the work of doing a full and proper conversion of the master repository. This would mean: * Rewriting the history so that patches by other authors (through Trac) that have been applied by commiters get correct Author/Commiter metadata. * Adding tags to the repository to mark all previous r releases of JOSM to the correct Git commit. * Generally helping with any conversion issues. There was some discussion of switching over the official repo last year[2], but Dirk Stöcker didn't seem very impressed. That's fine, but if that ever changes conversion won't be an issue, because I'll do it. That's all. 1. http://lists.openstreetmap.org/pipermail/josm-dev/2010-April/004310.html 2. http://lists.openstreetmap.org/pipermail/josm-dev/2009-November/thread.html#3694 3. http://github.com/avar/josm ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Open offer to convert JOSM to Git
On Thu, Apr 29, 2010 at 19:27, Sebastian Klein basti...@googlemail.com wrote: Ævar Arnfjörð Bjarmason wrote: In case it eluded anyone I'm pretty fond of Git [1][2]. The rails_port has now switched over (with some help of mine), as well as Merkaartor (with no help of mine). I've also set up Git mirror of JOSM on GitHub[3] which I plan to keep up to date. If the JOSM project is ever interested, I'd be willing to do all the work of doing a full and proper conversion of the master repository. This would mean: * Rewriting the history so that patches by other authors (through Trac) that have been applied by commiters get correct Author/Commiter metadata. * Adding tags to the repository to mark all previous r releases of JOSM to the correct Git commit. * Generally helping with any conversion issues. There was some discussion of switching over the official repo last year[2], but Dirk Stöcker didn't seem very impressed. That's fine, but if that ever changes conversion won't be an issue, because I'll do it. That's all. 1. http://lists.openstreetmap.org/pipermail/josm-dev/2010-April/004310.html 2. http://lists.openstreetmap.org/pipermail/josm-dev/2009-November/thread.html#3694 3. http://github.com/avar/josm How would we deal with the plugins on openstreetmap.org? It should stay open for developers with access to the main subversion repository. The JOSM Plugin directory in OSM works now, and is not dependant (aside from myabe a script or two) on the VCS used by JOSM itself. Even if JOSM moves to Git the plugins don't have to. There's actually very little that ties down plugins to svn.openstreetmap.org. It's the default plugin provider, but there are actually already 4 plugins that are part of the plugin list that aren't hosted there. If you put your plugin in any Git repository capable of offering a .jar download (like GitHub, Gitorious and most do), you should be able to move plugins to Git one-by-one. Or you could not move, it would be up to the individual developer. You'd just have to start a Git repository *somewhere* and add links to the .jar on the Plugins page on the wiki, actually there seems to be no reason for why you couldn't do this today. And what about automatic update of the subversion externals? Git itself has svn externals-like support in the form of submodules. Unlike SVN you can only check out whole repositories, not paths of a repository (since Git is tree based, there's no such thing as a subset of the repository). There are currently 4 externals: * apache/bzip2 * apache/codec These aren't going to move since Apache maintains them. But they don't change a lot either. I could either set up our own Git mirror of these, or just copy/paste the relevant code regularly to the JOSM Git. * openstreetmap/gui Could either be moved to Git + used as a submodule (the main contributor, stotz, would have to agree). Or the approach with Apache could be used. * map-icons/classic.small Same as the above. In principle would be quite cool to switch to git. However subversion works reasonably well for us and there should be real advantages (other than being hip) to make up the work and potential problems of the change. Indeed. Ultimately it's up to you guys. I'm not going to parrot all the advantages of doing so, that's been covered before. I just wanted to indicate that I'd like to help if JOSM chooses to go this way. Thanks for setting up the github anyway, it made me start learning git and i feel a little enlightened now. :) It was very nice to see h4ck3rm1k3 use it right away, good that it's interesting for other people as well. You can use that checkout b.t.w. to do local development and commit back to SVN. I use this approach with some SVN projects I contribute to. What you'd do is: 1. check out the repository on the mirror branch (default) 2. Add git-svn metadata: http://trac.parrot.org/parrot/wiki/git-svn-tutorial 3. branch from the mirror branch: git checkout mirror git checkout -b some/silly-branch-name 3. *hack on the train* 4. Use git-rebase to apply only your commits (none of the mirror stuff) to master 5. git svn dcommit With Git that would just be: 1. git clone some/project 3. *hack* 4. git push So git-svn makes things a lot harder, since you're always speaking in a foreign language and translating between two fundamentally incompatible systems. But stuff like this comes in handy when I'm hacking on 8 hour network-less train rides across Germany. 1. http://josm.openstreetmap.de/wiki/Styles and http://josm.openstreetmap.de/styles 2. http://josm.openstreetmap.de/wiki/Plugins and http://josm.openstreetmap.de/plugin $ lwp-request http://josm.openstreetmap.de/plugin | ack '^\S.*http://([^/]+)' --output='$1' | sort | uniq -c | sort -nr 42 svn.openstreetmap.org 2 web.me.com 1 topo.geofabrik.de 1 mappin.hp2.jp 3. http://josm.openstreetmap.de/ticket/4967 4
[OSM-dev] Git mirror of JOSM on GitHub
I got tired of waiting for `svn up' when rebuilding JOSM and wanted to commit some local patches. So I made a Git mirror of JOSM that I and others can use. It's very handy for getting faster updates, a local history and the ability to patch JOSM without SVN write access: http://github.com/avar/josm It's made with `git-svn`. It has two branches, the master branch is unaltered from git-svn and can be used to bootstrap from git-svn[1]: http://github.com/avar/josm/tree/master The second one is the mirror branch. This one is checked out by default and includes all the svn externals plus this hack to get it to build under Git: http://github.com/avar/josm http://github.com/avar/josm/commit/f5f16565e It would be nice if the core build.xml supported getting the revision from the git history if applicable, but my ant fu didn't suffice to make this happen. Finally this is the script that does the mirroring: http://github.com/avar/josm-mirror It runs through cron every 30 minutes and hits the josm.openstreetmap.org server with `git svn fetch`. I can adjust the frequency if the server admins think this is too much. 1. Example of git-svn bootstrapping: http://trac.parrot.org/parrot/wiki/git-svn-tutorial ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[josm-dev] Git mirror of JOSM on GitHub
I got tired of waiting for `svn up' when rebuilding JOSM and wanted to commit some local patches. So I made a Git mirror of JOSM that I and others can use. It's very handy for getting faster updates, a local history and the ability to patch JOSM without SVN write access: http://github.com/avar/josm It's made with `git-svn`. It has two branches, the master branch is unaltered from git-svn and can be used to bootstrap from git-svn[1]: http://github.com/avar/josm/tree/master The second one is the mirror branch. This one is checked out by default and includes all the svn externals plus this hack to get it to build under Git: http://github.com/avar/josm http://github.com/avar/josm/commit/f5f16565e It would be nice if the core build.xml supported getting the revision from the git history if applicable, but my ant fu didn't suffice to make this happen. Finally this is the script that does the mirroring: http://github.com/avar/josm-mirror It runs through cron every 30 minutes and hits the josm.openstreetmap.org server with `git svn fetch`. I can adjust the frequency if the server admins think this is too much. 1. Example of git-svn bootstrapping: http://trac.parrot.org/parrot/wiki/git-svn-tutorial ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] git.openstreetmap.org
On Wed, Apr 21, 2010 at 15:42, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: On Wed, Apr 21, 2010 at 15:11, Tom Hughes t...@compton.nu wrote: On 21/04/10 15:06, Chris Browet wrote: The real question is whether it is wanted/desirable to have the myriad of OSM related projects hosted under a single repository (or at least under an openstreetmap.org http://openstreetmap.org umbrella). I don't actually have any opinion on this. Well one problem is that managing it will be a pain if we have lots of projects as an admin will have to create the repos and things. The git security model also makes it a pain because it's OS security based so would mean adding lots of system users and creating some complicated system of group ownership to control who owned each repo. If you want to give people ssh:// access it can be (although GitHub, Gitorious et al manage with git-shell). But are you aware of git-http-backend(1)? You could set it up to do .htaccess push authentication via a CGI script exactly like SVN is set up now. Here's more info on this from GitHub which just rolled it out: http://github.com/blog/642-smart-http-support ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] git.openstreetmap.org
On Wed, Apr 21, 2010 at 09:28, Tom Hughes t...@compton.nu wrote: I've seen that the rails_port has (quite silently) moved to a OSM owned git repository. I'm a bit surprised I didn't hear more of that (or maybe I wasn't listening at the right place). But whatever... It only happened about 24 hours ago! Give us a chance and we'll get all the details sorted out and make some announcements... The actual question is what is the status of this git repo. Is this more or less private or open to the other OSM projects? That repo is the master repo for the live site and is only likely to be open to a limited number of committers. The nature of a DVCS is that it doesn't need to be open to everybody as people can just fork and then ask for changes to be pulled in to the live site. Here's a mirror I set up for those interested in forking/watching it from GitHub: http://github.com/avar/morbo It'll probably be taken down in lieu of a more official mirror soon-ish. If you're asking whether other projects can have repos on that server then we may consider it for some other core tools but I'd like to let what we have settle down a bit first - if nothing else I need to get used to git. If you need some admin help with it you know who to call :) I don't see it becoming the kind of vast dumping ground for all and sundry that the svn repo is - once again that isn't really the spirit of a DVCS anyway. *nod*, but on the other hand we're offering free hosting services for SVN now, it would be neat to eventually have a hosting plan if those projects want to move to Git (or if we decide to mandate it because we don't want to host SVN anymore). Wherever the master repos are hosted it would be nice to set up a project-wide mirror for all the osm Git stuff on GitHub, simiral to e.g. http://github.com/apache and http://github.com/facebook. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] git.openstreetmap.org
On Wed, Apr 21, 2010 at 14:06, Chris Browet c...@semperpax.com wrote: But I'd really like to move Merkaartor to git ;-) Do you need any help with converting it? The OSM website was converted via some hacky work I did (http://gist.github.com/373809). I could do the same for Merkaartor. If a decision could me made quite rapidly on what could be hosted in git.openstreetmap.org and what couldn't, it would be great. You don't need to wait for osm.org to move Merkaartor to git. Isn't it mostly maintained by 5 authors or so? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Can someone add a natural=volcano rendering to mapnik?
There's a ticket/icon for it here: http://trac.openstreetmap.org/ticket/2687 I don't have the experience with the stylesheets to patch it. It should render like natural=peak (with ele=*) except the icon should be more red. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Where are the slippy map chooser tiles cached?
On Thu, Apr 8, 2010 at 22:55, Matthias Julius li...@julius-net.net wrote: Under Linux this is usually not a problem because /tmp typically gets cleaned out at boot time (at least on Debian it does). That depends on your usage patterns. I generally don't reboot my Linux machine more than monthly and servers even more infrequently. You can run something like tmpreaper though. I wonder if JOSM should delete the cached tiles on exit. There's a cache class that wmsplugin and others use that allows you to keep files around until they collectively reach a given size / age. Maybe it could use that. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] osm.org userpage broken with farsi localization?
On Wed, Apr 7, 2010 at 14:28, Claudius claudiu...@gmx.de wrote: Am 07.04.2010 15:21, Tom Hughes: On 07/04/10 13:56, Claudius wrote: That line should run: abbr_day_names: [ی, د, س, چ, پ, ج, ش] Can we somehow push that fix upstream or is this a local plugin? OK. I've put that in now if you want to give it a try and make sure it's right now - the signup date on your user page should be a Saturday... Looking good now. Thanks for the quick fix. Maybe Ævar can comment on the export... The reason they aren't being shown in the default Translatewiki interface is because they're on a list of ignored messages that aren't shown to translators by default. Here's the list of these messages: http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/Translate/groups/OpenStreetMap/OpenStreetMap.yml To change them you have to pick I want to: View optional messages from in the translation drop-down list, here's the full list of hidden messages for Farsi: http://translatewiki.net/w/i.php?title=Special:Translatetask=optionalgroup=out-osm-sitelanguage=falimit=2500uselang=en Hope that helps. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Talk-GB] OSM shortlinks problem
On Mon, Mar 29, 2010 at 23:28, Matt Amos zerebub...@gmail.com wrote: On Mon, Mar 29, 2010 at 11:34 PM, Gregory nomoregra...@googlemail.com wrote: On 29 March 2010 15:15, Matt Amos zerebub...@gmail.com wrote: http://trac.openstreetmap.org/changeset/16271 yeah, it was originally =, but changed to - to work with twitter. given that the characters A-Z, a-z, 0-9, _, @, - and = have already been used, what's the next best character? ~? +? Well I suggested that you remove problematic symbols, and just use A-Z, a-z, 0-9 if need be. we're already using those symbols in a way that makes it difficult to reuse them for trailing characters without breaking existing links. if anyone can find a character which isn't in the list above, and doesn't break at least one URL-detector out there, then that's the best way to fix this. the alternative is to use some other scheme, for example appending .1/.2 instead of -/-- which makes some zoom levels a char longer. or possibly pre-pending the dashes, since they're usually picked up in the middle of an URL - but this makes char deletion from the end work slightly differently. it's also worth noting that, in the current scheme, shortlinks can end in @ or _ - do these also break the URL detectors? Yes, and not just if they end in @, e.g. if I paste http://osm.org/go/e...@3k9-- into Twitter that yields a link to http://osm.org/go/e0z followed by a reply to the user 3K9 followed by two trailing -- I think you could get maximum comparability with the most amount of silly URL extractors this way: ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Talk-GB] OSM shortlinks problem
On Tue, Mar 30, 2010 at 08:32, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: I think you could get maximum comparability with the most amount of silly URL extractors this way: Eek, accidentally hit send, continued: * Replace @ with ~. I haven't seen anything that fails on ~ in the middle or end of a URL. * End the link with .$zoom as you suggest. This would also have the added advantage of it being easy to tweak the zoom level after you've generated the shortlink. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Gource animations of OpenStreetMap website and JOSM development histories
I was playing with Gource / git-svn and made animations of the entire history of the OpenStreetMap website (the rails_port) and JOSM from a SVN-Git conversion. Here they are: OpenStreetMap website: http://blip.tv/file/3344055 http://blip.tv/file/get/Avar-GourceAnimationOfTheOpenStreetMapWebsiteDevelopment159.mp4 JOSM: http://blip.tv/file/3344236 http://blip.tv/file/get/Avar-GourceAnimationOfJOSMDevelopment592.mp4 I recommend downloading the mp4 instead of watching it on the web, the flash video quality is horrible. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Student Project Ideas?
On Wed, Mar 10, 2010 at 08:52, Graham Jones grahamjones...@googlemail.com wrote: Please give this a bit of thought, and add any ideas to the Wiki page! If you don't have chance to do that, an email to me will do and I will add it. Here's my idea: Can we please not make things like Develop a Simple, Stand-Alone Editor for New Users part of the GSOC list. I've seen numerous failed and dead-on-arrival GSOC projects with various projects that usually turned out that way because inexperienced students were being handed projects that were too ambitious and even if they were finished saw decay because nobody else was interested in maintaining them. I think it would be most useful to have students contribute to existing projects that need help such as JOSM, Potlatch, the rails port or something similar. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Student Project Ideas?
On Wed, Mar 10, 2010 at 16:31, Ian Dees ian.d...@gmail.com wrote: On Wed, Mar 10, 2010 at 10:21 AM, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: On Wed, Mar 10, 2010 at 08:52, Graham Jones grahamjones...@googlemail.com wrote: Please give this a bit of thought, and add any ideas to the Wiki page! If you don't have chance to do that, an email to me will do and I will add it. Here's my idea: Can we please not make things like Develop a Simple, Stand-Alone Editor for New Users part of the GSOC list. I've seen numerous failed and dead-on-arrival GSOC projects with various projects that usually turned out that way because inexperienced students were being handed projects that were too ambitious and even if they were finished saw decay because nobody else was interested in maintaining them. I think a more useful criticism would include some specific ideas... You mean specific GSOC ideas? We'll probably have plenty of those. What I was pointing out that just because something would be neat to do that doesn't mean that it's appropriate for being handed to a student for 3 months. Once you have those ideas how are you gong to pick one? I for one think: * You should try to make students work on existing /active/ projects instead of sending them off on their own for 3 months * In particular, assume that they'll be working for 3 months and we'll never hear from them again. I think there are some numbers on the % of GSOC students that stay around after the 3 months and IIRC they're alarmingly low * Try to recruit people with programming experience who're already contributing to the project in interesting ways that happen to be students (and no, I'm not eligible). This will reduce load on mentors * Don't underestimate the load on mentors. I've heard from people that did mentoring (albeit for complex projects) that spent more time on mentoring than it would have taken them to implement the student work themselves, and the student disappeared after 3 months so there was no long-term gain from it. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] how does the transition to tested work ?
On Mon, Mar 8, 2010 at 15:04, Sebastian Klein basti...@googlemail.com wrote: #4366 (validator: repair: overlapping ways distroys data) Plugin issue, can be fixed after release. #4584 (cursor not show in textfields of relations-editor) Cannot reproduce. We don't have enough info to fix this. I think this is the same issue as I've just described in this bug report: https://josm.openstreetmap.de/ticket/4686 I've made the old one a duplicate of this new report. #4631 (Plugin download broken) [3090] should have addressed most points of this ticket. #4401 (JOSM does not remember what has been uploaded) It's more like an enhancement. Too complicated, will be fixed after release. I've used JOSM a lot recently to upload objects in the one object at a time mode, if it fails with a conflict it will start the whole thing over again (even if it doesn't need to). I'm not 100% sure but I think that #4401 has been causing a lot of duplicate data created by JOSM that I've had to clean up. Uploading duplicates is definitely a bug. #4371 (Changeset comment reverts to previous when switching options) The blocker flag wasn't set by a developer. Should be fixed, but isn't all that serious. FWIW I have at least 10 large changesets uploaded to the main API that have inaccurate changeset summaries because of this and related bugs. So, I'd say it was OK to release, but tested version should be updated before next development cycle. I think tested should be a version without any blocker/critical bug reported on and being tested as unstable/latest for a week. This is how we try to do it, but some critical bus are not easy to fix, so we postpone these issues and move on. Aren't all these bugs already in the latest tested? A new tested with the same bugs (+ unrelated fixes) wouldn't be any worse would it? ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] how does the transition to tested work ?
On Mon, Mar 8, 2010 at 16:15, Apollinaris Schoell ascho...@gmail.com wrote: On 8 Mar 2010, at 7:04 , Sebastian Klein wrote: #4401 (JOSM does not remember what has been uploaded) It's more like an enhancement. Too complicated, will be fixed after release. for me this is really a bug not an enhancement. Josm is used for semi automatic imports a lot and this creates big mess of duplicates on server/network interrupts. I know it wasn't designed for that but Josm is just such a great workhorse. on normal editing this can leave a user with conflicts after a partial upload, maybe we should add a comment that uploads should be done in 1 transaction only until this bug is fixed. Anyway thanks for the great work and it's better to have a stable again. Josm is improving big with every release:) The problem with recommending that is that the only reason for using one object at a time uploading is to narrow down conflicts you may be having, right now due to this bug at the cost of adding lots of duplicates to the database. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] how does the transition to tested work ?
On Mon, Mar 8, 2010 at 16:30, Dirk Stöcker openstreet...@dstoecker.de wrote: On Mon, 8 Mar 2010, Ævar Arnfjörð Bjarmason wrote: for me this is really a bug not an enhancement. Josm is used for semi automatic imports a lot and this creates big mess of duplicates on server/network interrupts. I know it wasn't designed for that but Josm is just such a great workhorse. on normal editing this can leave a user with conflicts after a partial upload, maybe we should add a comment that uploads should be done in 1 transaction only until this bug is fixed. Anyway thanks for the great work and it's better to have a stable again. Josm is improving big with every release:) The problem with recommending that is that the only reason for using one object at a time uploading is to narrow down conflicts you may be having, right now due to this bug at the cost of adding lots of duplicates to the database. To have better conflict solving you may use Update modified from first menu. This will produce all conflicts in one single run and allows them to fix in one go. Yes, but due to the bad interface for conflict resolution in JOSM this isn't an option sometimes. There isn't support for mass-resolution of similar conflicts (I've filed bugs for this) so you run into situations where you can either: * Solve 300 conflicts which are of the same nature manually. Which requires for each one: 1. Click on an item in the conflict list 2. Move to tags/whatever in the dialog 3. Click merge your/remote changes 4. Apply 5 Repeat After I'm done with that I'll have manually clicked UI elements at least 300*4 times or so. So instead of doing that when I run into it I just cut my losses and try to upload at least /some/ of my data. I do this by turning on single-object upload, trying to select subsets of the dataset and see if I can upload them without conflicts. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] how does the transition to tested work ?
On Mon, Mar 8, 2010 at 16:59, Dirk Stöcker openstreet...@dstoecker.de wrote: On Mon, 8 Mar 2010, Ævar Arnfjörð Bjarmason wrote: To have better conflict solving you may use Update modified from first menu. This will produce all conflicts in one single run and allows them to fix in one go. Yes, but due to the bad interface for conflict resolution in JOSM this isn't an option sometimes. There isn't support for mass-resolution of similar conflicts (I've filed bugs for this) so you run into situations where you can either: Well, I think about a year ago we had NO working conflict resolution at all and I also think JOSM is the only editor which has that feature working reliable. So be happy you can click 300 dialogs :-) Beside that a semi-automatic conflict resolution for very common cases should be implemented. Probably we should add a send workflow data to server option, so we can get data about the real user needs (for these cases where users don't know at all what the would like to have). We already have rudimentary support (plugin list and version string) for such analysis and it influenced development already. Sure. I know this is a hard problem to get right and we're all aware of these issues already. I just wanted to point out why Update modified isn't always what you want because you sometimes want to explicitly avoid the conflict resolution and upload as much of what you can from your local dataset. In my experience if you get more than 30 conflicts in JOSM it's going to be less work just to close JOSM and redo all your edits than dealing with the resolving them (although I do save frequently). ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] how does the transition to tested work ?
On Mon, Mar 8, 2010 at 17:01, Sebastian Klein basti...@googlemail.com wrote: Ævar Arnfjörð Bjarmason wrote: This is how I've been running into this: 1. Download data from the OSM server, say 20 ways 2. While I edit 10 of those 2 have been changed already, so 2/10 conflict 3. Press upload in single-object mode, will upload 1/10, 2/10... 4. It conflicts at say the 5th object out of 10 and stops the entire upload process 6. If I try again it'll still start from 1/10 instead of 1/5 As I said I haven't tracked this down in any detail but it seems that in some situations JOSM doesn't track the stuff it has uploaded already when it runs into conflicts during uploads. I forgot to ask: With recently you mean after 5th February or so? (The server behavior has changed at that time due to a reimplementation.) Recently meaning from January up to March 1st. I ran into his a lot when processing the ourFootPrints import as blogged about here: http://www.openstreetmap.org/user/%C3%86var%20Arnfj%C3%B6r%C3%B0%20Bjarmason/diary/9698 What edits did you perform? Did you add more nodes, deleted nodes or just updated tags? These are pretty much the edits we were doing: http://wiki.openstreetmap.org/wiki/WikiProject_Iceland/ourFootPrints#Dealing_with_the_ofp_data The conflicts occured when we stepped on each others toes. This happened a bit frequently since we were all editing with day-old Iceland.osm dumps so we'd occasionally step on each others toes. Don't fear to clutter the database for testing. If you can help us resolving this, it's for the greater good. :) I'll see if I can reproduce this on api06.dev. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] how does the transition to tested work ?
On Mon, Mar 8, 2010 at 19:21, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: Don't fear to clutter the database for testing. If you can help us resolving this, it's for the greater good. :) I'll see if I can reproduce this on api06.dev. I just tried and I can't reproduce this or the issue with the number of objects to be uploaded drifting from what really needs to be uploaded. Now if I upload a per-object upload and it fails partway the objects that were uploaded up until that point will be counted as uploaded and won't be re-uploaded. But like I said I don't know if this issue in particular was what was causing some of the duplicates I was seeing initially. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Using osmosis for a daily from-scratch import into PostgreSQL and SQLite support
On Wed, Mar 3, 2010 at 22:03, Brett Henderson br...@bretth.com wrote: On Wed, Mar 3, 2010 at 8:32 PM, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: On Wed, Mar 3, 2010 at 01:21, Brett Henderson br...@bretth.com wrote: Hi Ævar, Can you provide the full error output from Osmosis. I suspect there's a caused by nested exception that provides the root cause for the error. After some more debugging it turns out that the issue is simply that the script/*.sql files are very out of date and in no way reflect the current DB schema or the tables that osmosis tries to insert into. Um, I think you may be using the wrong tasks against the wrong schema. It sounds like you're trying to use apidb tasks against the Osmosis simple schema. The pgsql tasks are the ones required for the Osmosis simple schema that you've created. This schema doesn't have a concept of history tables so in fact every table is a current table. It is a PostGIS schema with optional linestring and bbox columns on the way table. http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage#PostGIS_Tasks The apidb tasks are the ones that write to the production rails API schema. There is a script called script/contrib/apidb_0.6.sql included with Osmosis to create it. It is somewhat out of date because I don't have a working rails port, but should still be compatible if you switch on the allowIncorrectSchemaVersion=yes on all database tasks. http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage#API_Database_Tasks In the attached patch I fixed some of these but it doesn't fix all the issues, the current_* tables are still missing. It would be nice if one could skip importing into current_* for one-off imports like these. I had a quick look and it appears that you're adding APIDB tables to the simple schema. As for current_* tables, the apidb tasks do have a populateCurrentTables=no option, but I don't think this is what you're looking for. http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage#--write-apidb_.28--wd.29 Unless you really need apidb tasks, I'd stick with pgsql tasks because the associated schema is much simpler. Yes. The problem was that I was mixing the apidb task with the postgis task as you point out. I tried using the api .sql in the past but it failed with similar errors, it would probably work with allowIncorrectSchemaVersion=yes though as you point out. Thanks for your help. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Using osmosis for a daily from-scratch import into PostgreSQL and SQLite support
On Wed, Mar 3, 2010 at 01:21, Brett Henderson br...@bretth.com wrote:
Hi Ævar,
Can you provide the full error output from Osmosis. I suspect there's a
caused by nested exception that provides the root cause for the error.
After some more debugging it turns out that the issue is simply that
the script/*.sql files are very out of date and in no way reflect the
current DB schema or the tables that osmosis tries to insert into.
In the attached patch I fixed some of these but it doesn't fix all the
issues, the current_* tables are still missing. It would be nice if
one could skip importing into current_* for one-off imports like
these.
As for SQLite support, it's a nice idea but a non-trivial amount of work.
I'm happy to give somebody guidance on adding it, but I don't have time
myself.
Ok :)
Index: script/pgsql_simple_schema_0.6.sql
===
--- script/pgsql_simple_schema_0.6.sql (revision 20253)
+++ script/pgsql_simple_schema_0.6.sql (working copy)
@@ -3,6 +3,8 @@
-- Drop all tables if they exist.
DROP TABLE IF EXISTS actions;
DROP TABLE IF EXISTS users;
+DROP TABLE IF EXISTS changesets;
+DROP TABLE IF EXISTS changeset_tags;
DROP TABLE IF EXISTS nodes;
DROP TABLE IF EXISTS node_tags;
DROP TABLE IF EXISTS ways;
@@ -16,6 +18,12 @@
-- Drop all stored procedures if they exist.
DROP FUNCTION IF EXISTS osmosisUpdate();
+-- Create enum
+CREATE TYPE nwr_enum AS ENUM (
+'Node',
+'Way',
+'Relation'
+);
-- Create a table which will contain a single row defining the current schema version.
CREATE TABLE schema_info (
@@ -25,18 +33,57 @@
-- Create a table for users.
CREATE TABLE users (
-id int NOT NULL,
-name text NOT NULL
+email character varying(255) NOT NULL,
+id bigint NOT NULL,
+active integer DEFAULT 0 NOT NULL,
+pass_crypt character varying(255) NOT NULL,
+creation_time timestamp without time zone NOT NULL,
+display_name character varying(255) DEFAULT ''::character varying NOT NULL,
+data_public boolean DEFAULT false NOT NULL,
+description text DEFAULT ''::text NOT NULL,
+home_lat double precision,
+home_lon double precision,
+home_zoom smallint DEFAULT 3,
+nearby integer DEFAULT 50,
+pass_salt character varying(255),
+image text,
+email_valid boolean DEFAULT false NOT NULL,
+new_email character varying(255),
+visible boolean DEFAULT true NOT NULL,
+creation_ip character varying(255),
+languages character varying(255)
);
+-- Create a table for changesets,
+CREATE TABLE changesets (
+id bigint NOT NULL,
+user_id bigint NOT NULL,
+created_at timestamp without time zone NOT NULL,
+min_lat integer,
+max_lat integer,
+min_lon integer,
+max_lon integer,
+closed_at timestamp without time zone NOT NULL,
+num_changes integer DEFAULT 0 NOT NULL
+);
+-- Create a table for changeset tags.
+CREATE TABLE changeset_tags (
+id bigint NOT NULL,
+k character varying(255) DEFAULT ''::character varying NOT NULL,
+v character varying(255) DEFAULT ''::character varying NOT NULL
+);
+
-- Create a table for nodes.
CREATE TABLE nodes (
id bigint NOT NULL,
-version int NOT NULL,
-user_id int NOT NULL,
-tstamp timestamp without time zone NOT NULL,
-changeset_id bigint NOT NULL
+latitude integer NOT NULL,
+longitude integer NOT NULL,
+changeset_id bigint NOT NULL,
+visible boolean NOT NULL,
+timestamp timestamp without time zone NOT NULL,
+tile bigint NOT NULL,
+version bigint NOT NULL
);
-- Add a postgis point column holding the location of the node.
SELECT AddGeometryColumn('nodes', 'geom', 4326, 'POINT', 2);
@@ -44,65 +91,67 @@
-- Create a table for node tags.
CREATE TABLE node_tags (
-node_id bigint NOT NULL,
-k text NOT NULL,
-v text NOT NULL
+id bigint NOT NULL,
+version bigint NOT NULL,
+k character varying(255) DEFAULT ''::character varying NOT NULL,
+v character varying(255) DEFAULT ''::character varying NOT NULL
);
-
-- Create a table for ways.
CREATE TABLE ways (
-id bigint NOT NULL,
-version int NOT NULL,
-user_id int NOT NULL,
-tstamp timestamp without time zone NOT NULL,
-changeset_id bigint NOT NULL
+id bigint DEFAULT 0 NOT NULL,
+changeset_id bigint NOT NULL,
+timestamp timestamp without time zone NOT NULL,
+version bigint NOT NULL,
+visible boolean DEFAULT true NOT NULL
);
-- Create a table for representing way to node relationships.
CREATE TABLE way_nodes (
-way_id bigint NOT NULL,
+id bigint NOT NULL,
node_id bigint NOT NULL,
-sequence_id int NOT NULL
+version bigint NOT NULL,
+sequence_id bigint NOT NULL
);
-- Create a table for way tags.
CREATE TABLE way_tags (
-way_id bigint NOT NULL,
-k text NOT NULL,
-v text
+id bigint DEFAULT 0 NOT NULL,
+k character varying(255) NOT NULL,
+v character
Re: [OSM-dev] Using osmosis for a daily from-scratch import into PostgreSQL and SQLite support
On Wed, Mar 3, 2010 at 01:57, Jochen Plumeyer joc...@plumeyer.org wrote:
Dear Ævar Arnfjörð,
On Mar 02 Mar 2010, Ævar Arnfjörð Bjarmason wrote:
It would be even better if it could write to an SQLite database, then
it would be trivial to turn a .osm file into a .sqlite file for easy
querying of the data. There's a lot of tools in the OSM SVN that could
use something like this but right now they just implement their own
ad-hoc XML parsing in a while () {}.
I wrote scripts which are using the expat library, no dirty parsing, to
SQLite:
http://plumeyer.org/osm/osm2sqlite.pl
The bottlenecks of this rather slow script is conversion from date strings to
UNIX epoch seconds, and the whole SQLite latency, although the data is
committed in larger chunks.
That's neat. I hacked it up to use prepared statements changed the
schema which brought the runtime from c.a. 4m52.121s to 2m44.241s
(real).
My changes are here: http://github.com/avar/osm2sqlite
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Using osmosis for a daily from-scratch import into PostgreSQL and SQLite support
I maintain some statistics on the country dump for Iceland so as part
of a daily cronjob I do:
* drop database osm_iceland;
* cd rails_port rake db:migrate
* osmosis import Iceland.osm
Is it possible to to skip the step where I have to maintain a database
copy of the rails port? I've tried script/contrib/apidb_0.6.sql in the
osmosis distribution as well as using
examples/pgsql_simple_schema_0.6.sql with the appropriate
createdb/createlang statements a postgis setup but I always get
Unable to check if user with id -1 exists in the database. from
org.openstreetmap.osmosis.core.apidb.v0_6.impl.UserManager.doesUserExistInDb(UserManager.java:96).
It would be really nice if there was an option in osmosis that I could
use to tell it to create a simple database from a given .osm file from
scratch, I.e. do the required CREATE TABLE operations to populate
something like an apidb with nodes/ways/relations and their tags.
It would be even better if it could write to an SQLite database, then
it would be trivial to turn a .osm file into a .sqlite file for easy
querying of the data. There's a lot of tools in the OSM SVN that could
use something like this but right now they just implement their own
ad-hoc XML parsing in a while () {}.
Consider this mainly a feature request. If someone's interested in
implementing SQLite support I can help with that, I'm just lost with
Java.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] I converted the rails port to Git
On Mon, Feb 22, 2010 at 08:14, Frederik Ramm frede...@remote.org wrote: Ævar Arnfjörð Bjarmason wrote: We discussed briefly on IRC where we would host our Git repositories if we hypothetically converted all of the OpenStreetMap SVN to it. Personally I think Github would be great for that, the alternative of just setting up our own git ssh:// and gitweb really misses the added benefit of the web-based forking, pull requests, multi-repostiory tracking and the other collaborative aspects of git which really shine through with Github. I have no experience with Github. Is it clean from a privacy / owning one's data viewpoint, i.e. doesn't try to sell stuff and annoy users (like Sourceforge does with endless buy services for ... ads), doesn't require committer's real names or e-mail addresses, and so on - or is it one of those sites with an evil business plan where sooner or later you'll be ashamed that you used them at all (as I have heard people say about Sourcefourge)? Because if there is any lingering doubt that Github might one day send targeted e-mails advertising the latest Ruby book to those who have checked in lots of Ruby code, or whatever, then I'd rather forgo these collaborative aspects. They have a non-evil business model: Selling access if you need large or private repositories. This is their privacy policy, it's written in Human: http://help.github.com/privacy/ (All this comes from someone with almost not Git exposure; if Git is built in a way that even if OSM uses Github people can easily and equally well contribute without registering at Github then the point is probably less important. But if everyone is told why don't you register at Github then you can participate better then Github really needs to be squeaky clean for that to happen.) You'll always be able to contribute without registering with Github, although of course if the master repository is hosted there and you want to push directly to it you'll need an account there. Github just makes things easier, especially if the people you're working with use it too. Here's an example: Earlier in this thread I mentioned that I back up my Github repositories, but just after writing that E-Mail I found a bug in the program I'm using to perform that backup. So I do: 1. Click Fork 2. git clone g...@github.com:avar/git-megapull.git 3. Patch it, the commits are here: http://github.com/avar/git-megapull/commits/master/ 4. Notify the author that I made the changes: http://i.imgur.com/K97zP.png If the author doesn't respond in a timely fashion by applying the patches other people that run into the same problem can see that the bug is fixed in my branch and trivially merge from it or just use it instead of his. This process is much easier than making a change to a program in a subversion repository you don't have access to. Then you have to fix it locally, do svn diff attach your patch (which won't have 4 commit messages like the above) to an E-Mail / trac ticket. Someone who wants to use your patch has to download it somewhere manually, may have difficulties merging it etc. Even if you do have commit access to the primary repository it's great to be able to easily fork programs in this manner. I've fixed a few issues with some programs in OpenStreetMap's repository but you always feel like you're playing with somebody else's toys. Maybe they don't like the changes you made and then you've just fouled up their trunk. It's much easier just to fix it locally, push it and tell them that they can merge from you if they like the changes. The biggest feature of Git or any distributed version control system is that it brings things that were previously outside the repository inside the repository, for instance getting multiple commit messages and easy merging for things that were previously huge patches attached in trac somewhere. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] I converted the rails port to Git
On Mon, Feb 22, 2010 at 11:56, Jonathan Bennett openstreet...@jonno.cix.co.uk wrote: On 21/02/2010 22:30, Ęvar Arnfjörš Bjarmason wrote: We were discussing maybe moving over to Git in #osm and I thought I'd do a quick conversion just to show how easy it is: Sorry I missed the original conversation. Can you, for the benefit of those of us not present, give a few bullet points of the benefits to OSM as a project of a move to Git (or any other SCM)? I'm interested in what problems this move would solve, not what features Git has that SVN doesn't. For what it's worth, I have nothing particularly against Git, or for SVN, but change has its cost. And I'd prefer Mercurial, purely because I use NetBeans, and it has better hg support. If OSM moves to Git we'll all be way awesomer and get more girls. We might even eclipse Google Maps sometime next month. More seriously though I didn't mean to sound as if everyone should switch to git /right now/ because it's so awesome. I did the experimental rails_port - Git conversion because there was some discussion in the #osm channel from some developers that work on it (zere shaun) about potentially using Git. I just thought I'd spend 10 minutes bringing that from the hypothetical to something that worked. Furthermore the OpenStreetMap SVN is a collection of a vast amount of unrelated projects who reraly talk together or share any code. My posting was mainly aimed at on of the authors (or group of authors) which already maintain a project there, want to switch to Git but don't know how. Honstly I'd rather not see some project-wide discussion about whether we should all use Git or Subversion. I've seen a few of those and it ultimately degenerates into the equivalent of a room full of people collectively deciding what their favorite color is; Or to pick an analogy closer to home: Whether we should all use Potlatch or JOSM. Rather I think it's useful to recognize that we like different things, that there are boundaries in the OSM SVN over which code never crosses, and if people want to split up a part of it into Git, Mecurial, CVS or whatever that's fine, and if they want to stay with SVN that's also just fine :) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] I converted the rails port to Git
We were discussing maybe moving over to Git in #osm and I thought I'd do a quick conversion just to show how easy it is: http://github.com/avar/openstreetmap-website Here are the commands you need to execute to convert it yourself: http://wiki.github.com/avar/openstreetmap-website/subversion-to-git-commands One thing I didn't do was rewrite commits so that they have proper author names / emails. The best I could find for this purpose was this: http://wiki.openstreetmap.org/wiki/Subversion#Subversion_Users Is anyone maintaining a file with the E-Mail addresses of commiters? Tom? As for converting other bits of our SVN here are the programs that use branches: $ find . -type d -name '*branch*' ./applications/routing/yours/branches ./applications/rendering/tilesAtHome-dev/branches ./applications/editors/merkaartor-branches ./applications/utils/srtm2postgis/branches ./applications/utils/osmosis/branches ./applications/utils/downloading/JTileDownloader/branches ./sites/other/route-altitude-profile/branches ./sites/rails_port_branches Those have to be converted with git-svn's --branch command, the rest can just be filtered out of a conversion of the whole repo with git-filter-branch's --subdirectory-filter. We could convert stuff incrementally from Subversion as developers want. If you have a project in openstreetmap's subversion and you've recently joined the Sainthood of the Church of Git and would like to convert your project: speak up. We can probably figure something out. We discussed briefly on IRC where we would host our Git repositories if we hypothetically converted all of the OpenStreetMap SVN to it. Personally I think Github would be great for that, the alternative of just setting up our own git ssh:// and gitweb really misses the added benefit of the web-based forking, pull requests, multi-repostiory tracking and the other collaborative aspects of git which really shine through with Github. Some people on IRC were worried about the case where Github might be down and they wanted to deploy the site. It is down every once in a while but in those cases it should be just a matter of doing: git push ssh://deploy.openstreetmap.org/repos/web.git master From your local machine which also has those changes, and of course it would also be trivial to automatically replicate Github's repositories somewhere else as commits come in due to the distributed nature of Git. In the very worst case of it getting nuked from orbit by invading aliens you'll just copy the ssh keys you've allowed to somewhere else and tell commiters to do: git config remote.origin.url git://aliens-ate-github.org/openstreetmap/web.git :) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] I converted the rails port to Git
On Mon, Feb 22, 2010 at 00:09, Tom Hughes t...@compton.nu wrote: On 21/02/10 22:30, Ęvar Arnfjörš Bjarmason wrote: We were discussing maybe moving over to Git in #osm and I thought I'd do a quick conversion just to show how easy it is: http://github.com/avar/openstreetmap-website Here are the commands you need to execute to convert it yourself: http://wiki.github.com/avar/openstreetmap-website/subversion-to-git-commands Whilst I'm prepared to consider moving to git we need to give serious consideration to whether we want to host on an external site, if only from the point of view of maintaining a solid set of backups. Yes, of course. Backups need to be done wherever this ends up. Is anyone maintaining a file with the E-Mail addresses of commiters? Tom? I don't have that as such, but if you can give me a list of usernames I should be able to come up with a list of addresses. Just for the rails port this is everyone who's ever commited to it as produced by: git log --pretty=format:%an | sort | uniq -c | sort -nr http://paste.scsys.co.uk/39613 We discussed briefly on IRC where we would host our Git repositories if we hypothetically converted all of the OpenStreetMap SVN to it. Personally I think Github would be great for that, the alternative of just setting up our own git ssh:// and gitweb really misses the added benefit of the web-based forking, pull requests, multi-repostiory tracking and the other collaborative aspects of git which really shine through with Github. Well apparently we can setup our own github if we want, but I will need to look at that. It's possible to set up your own ssh:// or git:// server and a Gitweb installation. Github however provides a lot more on top of that, mainly having to do with the buzzwordsocial/buzzword aspects of coding. I.e. it's really easy for anyone to fork your repository and easy for you to monitor what they're doing and merge stuff back if you want. Check out for example this network graph for the recently released hiphop-php: http://github.com/facebook/hiphop-php/network You can drag it see that people have been passing patches around and merging back and forth as they need some features that other people have added. A better example perhaps is rails-i18n which gets a lot of merge requests, and it's something we use: http://github.com/svenfuchs/rails-i18n/network All of this is possible without using something like Github of course but it's much harder (set up your own hosting instead of just clicking fork) and it's not as easy to see what's being added to your code elsewhere to merge those changes back. Some people on IRC were worried about the case where Github might be down and they wanted to deploy the site. It is down every once in a while but in those cases it should be just a matter of doing: git push ssh://deploy.openstreetmap.org/repos/web.git master From your local machine which also has those changes, and of course it would also be trivial to automatically replicate Github's repositories somewhere else as commits come in due to the distributed nature of Git. What is deploy.openstreetmap.org exactly? In order to make deployment work we need somewhere we can pull from to multiple machines. A hypothetical local mirror of the official repository used for deployment. Just like the webserver (presumably) has a Subversion checkout now. In the very worst case of it getting nuked from orbit by invading aliens you'll just copy the ssh keys you've allowed to somewhere else and tell commiters to do: git config remote.origin.url git://aliens-ate-github.org/openstreetmap/web.git That presumes that aliens-ate-github.org has somehow magically gained a copy of the code from github before the aliens arrived. No. In this case you'd do: # Doesn't work, aliens ate github git pull # Go to http://aliens-ate-github.org and create a fresh empty repository, then: git config remote.origin.url git://aliens-ate-github.org/openstreetmap/web.git # Populate it with your local copy git push Since every Git repository is a full cryptographically signed copy of the full repository history everyone working on the code has a full backup up to the time that he or she last did git pull. Of course you want a more reliable backup than just counting on commiters to have a recent copy. It's easy to do this by setting up a cronjob somewhere that does git pull from github to a local copy somewhere every hour or so, that's how I backup my repositories on Github. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] South Pole at 0,0 ?
On Tue, Jan 26, 2010 at 22:05, Matthias Julius li...@julius-net.net wrote: Mercator is defined right up to the poles - the scale just grows larger and larger and at the poles it is infinite. Our limit of about 85° comes from us wanting a square map. Would it be possible to switch to a different projection at some latitude? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Can't connect to local PostgreSQL database with osmosis
I can't connect to postgresql database that I can connect to with pgsql(1) with osmosis and I can't figure out why this is: osmosis --read-xml-0.6 /var/www/osm.nix.is/latest/Iceland.osm.bz2 --write-apidb-0.6 populateCurrentTables=yes host=localhost database=osmistmp user=osmistmp password=osmistmp validateSchemaVersion=no org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to establish a database connection. at org.openstreetmap.osmosis.core.apidb.common.DatabaseContext.getPostgresConnection(DatabaseContext.java:111) [...] Caused by: org.postgresql.util.PSQLException: The connection attempt failed. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:150) [...] Caused by: java.net.SocketException: Network is unreachable at java.net.PlainSocketImpl.socketConnect(Native Method) [...] Nothing shows up in postgresql's log which suggests that it really can't reach the server, but with the psql tool I can talk to the server over TCP just fine: $ psql -h localhost -W -U osmistmp osmistmp Password for user osmistmp: [...] osmistmp= And I'm not having some DNS issues where localhost isn't resolved (I've also tried to tell it to connect to127.0.0.1): $ dig +short localhost 127.0.0.1 This is what my pg_hba.conf is like: $ egrep -v -e ^# -e ^$ /etc/postgresql/8.4/main/pg_hba.conf local all postgres ident local all all ident hostall all 127.0.0.1/32 md5 hostall all ::1/128 md5 I added some debugging strings to DatabaseContext.java and found out that it's getting the connection as: newConnection = DriverManager.getConnection(jdbc:postgresql://localhost/osmistmp, osmistmp, osmistmp); Which looks fine. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Can't connect to local PostgreSQL database with osmosis
On Tue, Jan 19, 2010 at 14:10, Frederik Ramm frede...@remote.org wrote: Could it be some strange ipv6 issue where one of the applications tries to talk to localhost through ipv6? I am by no means an ipv6 expert but I have had strange things like this happen to me. That turned out to be the issue, running: ifconfig lo down sysctl -w net.ipv6.conf.all.disable_ipv6=1 ifconfig lo up Made everything work, thanks! ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Potlatch 1.3d: Impoved Haiti crisis support
I've commited Potlatch 1.3d, this release adds better support for mapping in Haiti: * Adding DigitalGlobe layer to presets * Now with 2 GeoEye layers: GravityStorm's and the NYPL extended coverage * Pressing B with GeoEye/DigitalGlobe (and NearMap) layers now adds the correct source=* tag to elements * Added GeoEye and digitalglobe to source=* tag autocompletion (See http://trac.openstreetmap.org/changeset/19544 for the commit) It's just in SVN now, I have no idea when it'll be on http://openstreetmap.org/edit ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql = default.style = add a new line to get a new column in the database postgis
Is your osm2pgsql definitely reading the right style file? You can specify a path to a style file with the -S switch (see --help). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Can I style by user in mappaint?
On Sat, Jan 16, 2010 at 04:25, Ulf Lamping ulf.lamp...@googlemail.com wrote: Am 15.01.2010 09:05, schrieb Jiri Klement: I think it would be nice to use search patterns in mappaint. The style information is cached in OsmPrimitive so it shouldn't have impact on overall performance. And there are probably other usecases when key=value matching is not sufficient. Actually I was planning to do some prototype to see if performance really isn't a problem Might be a problem for initial reading of large datasets, but you cannot say wether it's acceptable or not until you've really tried it. That's the conclusion I've got when doing the first round of mappaint performance improvements a lot of time ago :-) and then ask here if it is really a good idea to switch to search patterns. But when avar brought it up, we can start the discussion now. So are there any other usecases when search patterns would be usefull in mappaint or any objections agains search patterns? Avar was the first one I've seen to come up with show me stuff from user xy in a special way. I'm unsure if this is really a common use case. FWIW potlatch does this every day by rendering things differently if user == the guy who uploaded tiger tiger:reviewed == Personally, I'd like to keep things simple here - so even none developers have at least a chance to write their own rendering rules. Otherwise complexity grows and grows and ends up like the mapnik rendering rules with embedded SQL statements - the learning curve is now really high :-( Sure, but condition user=foo / isn't that much harder than condition k= v=/ :) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Can I style by user in mappaint?
On Sat, Jan 16, 2010 at 05:39, Ulf Lamping ulf.lamp...@googlemail.com wrote:
Am 16.01.2010 05:38, schrieb Ævar Arnfjörð Bjarmason:
Avar was the first one I've seen to come up with show me stuff from
user xy in a special way. I'm unsure if this is really a common use
case.
FWIW potlatch does this every day by rendering things differently if
user == the guy who uploaded tiger tiger:reviewed ==
tiger has special tags that are easy to display, JOSM renders:
condition k=tiger:reviewed v=no/
already for quite a while in a special way.
I can't see why a special user condition is needed here.
It's rendering TIGER data that hasn't been touched since import in a
special manner:
if (preferences.data.tiger this.uid==7168 this.version==1
this.clean this.attr[tiger:tlid]) {
Someone may edit the data and not change the tiger:reviewed tag, not
every user reads a manual on TIGER editing before clicking Edit.
Sure, butcondition user=foo / isn't that much harder than
condition k= v=/ :)
Exercise: Can you tell - not looking at the help - what's the difference
between the two search expressions:
type=*
foot:
Both of those are exact key matches for type foot with any value.
... I'm pretty sure very few people can.
The JOSM search box help lists 21 conditions, offers case (in)sensitiveness
and optional regular expressions, still misses a simple logical AND and
offers a myriad of possible combinations.
Yeah, it needs to be friendlier.
In contrast, we currently have four different conditions that are easy to
grasp:
condition k=key v=value/
condition k=key b=yes/
condition k=key b=no/
condition k=key/
I do see a clear difference between how hard it is to learn one of the two
approaches.
The search is exposed to users, there are currently 5 custom
stylesheets for JOSM. I can't the easy of use concern here,
stylesheets are either written by developers or automatically
generated by user-friendly programs.
___
josm-dev mailing list
josm-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Localisation policy for quotes etc.
On Wed, Jan 13, 2010 at 14:53, Matthias Julius li...@julius-net.net wrote: Marc Schütz schue...@gmx.net writes: Also, I want to change all those don'ts and won'ts to proper English. This doesn't change the meaning of the string, so all the translations (hopefully) just need to be unfuzzied. I thought, these don'ts and won'ts is proper english. Am I wrong? +1 Furthermore, don't and do not etc. are not freely interchangeable, as they have different connotations and for both there are situations where they are not appropriate. Well, this is true - especially for spoken language. This has probably much to do with the emphasis that is also put on it. There is certainly a difference between I won't do that. and I WILL NOT do that. But, in written language these shortcuts are consided casual and they are not used in formal texts, AFAIK. That's why I use this in emails, but not in the user interface of a program. Do we have any native English speakers here? It is somewhat silly to discuss subtle meanings of the English language between Germans. Perhaps you can make do with me? :) Well Marc Schütz is in theory completely wrong that don't and do not are not freely interchangeable. Don't is a contraction of do not and means exactly the same thing by definition. However as you point out the meanings are in practice subtly different. Contraction-free speech has an air of authority about it Do not touch that, it is hot as opposed to Don't touch that, it's hot. Contraction are accepted almost everywhere in English, although you might get into trouble if you use them in something like a scholarly paper, they're not looked down upon like reductions are, such as want a - wanna. All of these contractions and reductions will no doubt become perfectly accepted English within a few decades, just try reading newspaper articles en English from the early 20th century where authors avoid writing things that are universally acceptable today, such as writing automobile without a hyphen (just as we're slowly converting E-Mail into email nowadays). But to comment on the general issue here interface messages should in my opinion by clear, short and unambiguous, JOSM has sometimes lacked in this respect mostly due to being written by people for whom English is a second language. I don't think programs should take themselves overly serious though, Potlatch doesn't, nor does the OpenStreetMap website itself or programs like git which are all programs whose UI messages I like. I don't like using programs whose messages look like they're extracted from a speech to be given at a fancy dinner party. It's OK to say Oops in your error messages and not clutter up the UI by saying please in every second word. But that's just my opinion :) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] mod_tile/renderd logging
On Tue, Jan 12, 2010 at 11:08, Jaak Laineste jaak.laine...@gmail.com wrote: Hello, Is there a way to save renderd debug log to a file? I can see the debug log when I start renderd with -f option, but I'd like to run it as normal service, silently, but have debug log in e.g. /var/log/renderd.log You can run it under inet.d and have it capture its output iirc. I was getting occasional Segmentation Faults in renderd (when I use 2 threads), this may be useful for troubleshooting. It usually doesn't help at all to capture program output when something is segfaulting. Compile it with the -ggdb3 flag and run it under gdb instead. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] IRC JOSM
On Mon, Jan 11, 2010 at 13:15, Minh Quang chumkhungbo2...@yahoo.com wrote: Yooo, Do you have your own IRC for JOSM ?? I really need your help, please There's no IRC channel for JOSM, but you can ask in #osm on irc.oftc.net. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] Localisation policy for quotes etc.
On Sun, Jan 10, 2010 at 15:00, Andre Hinrichs andre.hinri...@gmx.de wrote:
Hi List!
Since I currently check all translatable texts I found that sometimes
quotes are single and sometimes double. E.g. isn't is sometimes simply
isn't and sometimes isn''t.
Which is the correct way?
It's always a single quite in English, but perhaps they're double
quotes in the JOSM source to work around something?:
aoeu josm (r2766) $ ack tr\(.*isn''t src
src/org/openstreetmap/josm/gui/download/BoundingBoxSelection.java
219:setErrorMessage(tfLatValue,tr(The string ''{0}''
isn''t a valid double value., tfLatValue.getText()));
251:setErrorMessage(tfLonValue,tr(The string ''{0}''
isn''t a valid double value., tfLonValue.getText()));
src/org/openstreetmap/josm/gui/widgets/BoundingBoxSelectionPanel.java
160:feedbackInvalid(tr(The string ''{0}'' isn''t a
valid double value., getComponent().getText()));
200:feedbackInvalid(tr(The string ''{0}'' isn''t a
valid double value., getComponent().getText()));
src/org/openstreetmap/josm/gui/dialogs/changeset/query/AdvancedChangesetQueryPanel.java
533:throw new IllegalStateException(tr(Current
value ''{0}'' for user ID isn''t valid, tfUid.getText()));
Forthermore I find sometimes american english (e.g. 'initialize').
Wasn't the overall policy to use british english?
I thought the source default was en_US since if it was en_GB the
existing en_GB translation would be pointless (and we'd need a en_US
one).
And finally sometimes I find XHTML style end tags in translated texts
(e.g. 'br /' instead of 'br'). By now I haven't found any in the
source. Is it ok to use them or should these be fixed?
The HTML is rendered by some internal Java stuff, both of those tags
work just as well when fed to it so I don't think it needs to be fixed
either way.
___
josm-dev mailing list
josm-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] New translations system
On Thu, Jan 7, 2010 at 13:05, Dirk Stöcker openstreet...@dstoecker.de wrote: I switched the translations from the gettext support to an own version using Java mechanisms. This now means adding new languages results in a small change in josm (adding the plural mode - one line). As a result the final josm jar is a bit smaller again (no longer multiple english strings). Please all of you test if josm is still working reliable in your native language! It works fine for Icelandic, neat that we can continue translating without JOSM blowing up. Out of curiosity though since you were changing the internal storage away from Gettext why did you go with something custom instead of using Java ResourceBundles or whatever the system Java projects usually use is? I'm not familiar with Java i18n issues so it would be interesting to know. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] OSM Book
On Tue, Dec 29, 2009 at 15:12, Frederik Ramm frede...@remote.org wrote: And another thing. If you had room to present four JOSM plugins to the newcomer - which four would you choose? The book currently covers WMS, Validator, RemoteControl and Surveyor/LiveGPS. But I'm prepared to change that if you think another selection would make more sense. A brief mention of walkingpapers would be good given the target audience, not at the expense of the others though. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] UpdateModifiedAction ?
On Sun, Dec 27, 2009 at 16:14, Karl Guggisberg karl.guggisb...@guggis.ch wrote: r2682 adds an UpdateModifiedAction. The changeset looks fine http://josm.openstreetmap.de/changeset/2682/ The file was indeed added and checked in. Stragenly, UpdateModifiedAction.java isn't checked out when I update to HEAD, in contrast to the updates in MainMenu.java referring to it. Any idea what is wrong here? It works for me, I get src/org/openstreetmap/josm/actions/UpdateModifiedAction.java when I check out HEAD. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Changeset files (was Removing Minutely and Hourly Changesets)
On Mon, Dec 21, 2009 at 11:44, Jon Burgess jburgess...@googlemail.com wrote:
On Mon, 2009-12-21 at 01:08 -0500, Anthony wrote:
Cool. If anyone familiar with the planet dumper tool is listening...
In
http://svn.openstreetmap.org/applications/utils/planet.osm/C/output_osm.c
} else if ((*in = 0) (*in 32)) {
escape_tmp[len] = '?';
len++;
should be something like
} else if ((*in 0) (*in 32)) {
len+=sprintf(escape_tmp[len], #%d;, *in);
Something like as in I haven't even checked if that compiles :).
Most of the control characters are not allowed in a valid XML file. It
makes no difference whether they are present as an ASCII character or as
the equivalent entity.
$ echo foo#01;/foo | xmllint -format -
-:1: parser error : xmlParseCharRef: invalid xmlChar value 1
foo#01;/foo
Indeed. These were all created by Potlatch while it still wasn't doing
proper input validation (http://trac.openstreetmap.org/ticket/2072).
Would anything break if these values were just cleaned up in the
database so we don't have to deal with them in other tools in the
future?
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] renderd in a multi-user multi-style environment
On Fri, Dec 18, 2009 at 10:01, Frederik Ramm frede...@remote.org wrote: for our German dev server, I want to give everyone who has an Unix account the opportunity to install as many Mapnik styles as they want, and make the rendered tiles available. I don't want everyone to run their own renderd though! I'm prepared to code the necessary changes to renderd and mod_tile but would like to solicit your opinion before I do. I think it could be feasible to have an /etc/renderd.d directory where you can put as many config snippets as you want, each representing a config section of the renderd.conf file, e.g. if you before had the section [default] max_zoom=18 URI=/osm/ XML=/etc/mapnik/osm.xml then you would now have a file named /etc/renderd.d/default containing max_zoom=18 URI=/osm/ XML=/etc/mapnik/osm.xml We would then also need some way to make renderd re-read the config dir or a Mapnik config file so that the whole thing does not have to be restarted every time someone makes a change. The same is probably true for mod_tile which also reads the renderd.conf file. Also, we currently have one global planet import timestamp file, but in a multi-style environment it is possible that some of them have their own little database and thus have their own timestamp. We should possibly specify the timestamp file in the renderd config so that this is catered for. Anything stupid or unworkable in this? Better ideas? Sounds good. We have the same setup on the Wikimedia Toolserver except there's a preprocessor that writes out one huge .ini so renderd/mod_tile isn't parsing different files. What you need to change is stuff like XMLCONFIGS_MAX. We had around ~270 configs so I changed it to ~300 but things like that shouldn't be allocated on the stack, there's also a few related areas where it'll malloc() according to some #define in one go instead of doing malloc()/realloc() as it goes. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tagwatch Editor Counts
On Wed, Dec 16, 2009 at 18:58, Matt Amos zerebub...@gmail.com wrote: Thanks for those language stats, they were very informative. not sure what the null language results are - presumably at some point the editors weren't putting a language in their changeset comments or something? No, I only added it recently to Potlatch and it looks like JOSM hasn't had it always either. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Delete a node
On Sun, Dec 13, 2009 at 19:23, Ian Dees ian.d...@gmail.com wrote: On Sun, Dec 13, 2009 at 1:16 PM, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: On Sun, Dec 13, 2009 at 17:54, hy-soft hy-s...@sha-mash.de wrote: the wiki says: http://wiki.openstreetmap.org/wiki/API_v0.6 Delete: DELETE /api/0.6/[node|way|relation]/#id Expects a valid XML representation of the element to be deleted. --- What exactly does the term 'valid XML representation of the element' mean? Is it the same what I would get when downloading the item? The API docs suck, I beg to differ. The API 0.6 docs are quite good. You may think so, but obviously they aren't good enough to prevent questions such as the one that started this thread. Hence my suggestion to do as more mature editors do when in doubt. Obviously they aren't hopeless in all cases, but I've frequently run into areas where it has been more useful to just read the source / network sniff than try to figure out what they're talking about. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Delete a node
On Sun, Dec 13, 2009 at 19:38, Ian Dees ian.d...@gmail.com wrote: Perhaps you (or others) that spend the time to do the network sniffing could update the wiki API docs so that in the future these questions are answered without a trip to the mailing list? I already know how it works so there's no personal gain for me in spending time in updating the documentation. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Delete a node
On Sun, Dec 13, 2009 at 20:14, Ian Dees ian.d...@gmail.com wrote: On Sun, Dec 13, 2009 at 2:06 PM, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: On Sun, Dec 13, 2009 at 19:38, Ian Dees ian.d...@gmail.com wrote: Perhaps you (or others) that spend the time to do the network sniffing could update the wiki API docs so that in the future these questions are answered without a trip to the mailing list? I already know how it works so there's no personal gain for me in spending time in updating the documentation. Is there personal gain for you in spending time on an Open Street Map? :-) Yes, it's fun and I get cool map data for areas I care about. It's not just about helping yourself, we are a community after all. There are an infinite useful things I could be doing but I have finite time, which these days I spend answering a few questions on this mailing list and working on other interesting projects (and work) as opposed to spending a few days getting API documentation I very rarely use up to shape. But I'll make you a deal, I'll fix the API docs up if you come over and map my backyard, we're a community after all. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Open changeset(s) with potlatch
On Sun, Dec 13, 2009 at 17:52, hy-soft hy-s...@sha-mash.de wrote: Hi, I just noticed that if you open Potlatch and skip to another tab above [view|edit|history|export] a new changeset is opened each time - even if you have saved your editings. Maybe it's not important (and I actually don't care) but people might find it annoying to have lots of changesets still open though the actual work is saved. In save mode potlatch opens a changeset every time you click save and keeps it open for subsequent saves within the same Potlatch session, but of course if you navigate away from Potlatch you stop the flash player it's using destroy the state. So if you go from Potlatch/edit - History - Potlatch/edit - View - Potlatch/edit - export you'll end up with 3 open changesets. I don't think Potlatch has support for using existing open changesets on the server like say JOSM, but you can work around this by opening view/edit/history/export in a new tab instead of navigating away from Potlatch. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Translating into Catalan
On Wed, Jun 3, 2009 at 22:17, Jordi Julià jjul...@wanadoo.es wrote: I saw on the osm blog an entry which said that if we wanted to help translating osm into any languages we had to subscrive to this mailing list. I would like to make a translation into Catalan. What can I do? I searched a little in the wiki but I'm not sure about what I have to do. Jordi: Since you initially asked about this we've changed the way we do translations. We're now using Translatewiki and not manually commiting YAML files to SVN. See this URL: http://wiki.openstreetmap.org/wiki/Translation#OpenStreetMap_website_interface I reverted your changes to ca.yml since you broke the rails port by commiting a file with invalid syntax: http://trac.openstreetmap.org/changeset/19076 And previously Edgemaster had done the same. Don't commit YAML files, edit Translatewiki instead. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 16:20, Ian Dees ian.d...@gmail.com wrote: On Thu, Dec 10, 2009 at 10:16 AM, Matthias Julius li...@julius-net.net wrote: There are requests in JOSM's trac to improve the handling of API errors. To do that JOSM needs to get a better understanding on what is wrong with the data. Currently, JOSM is parsing the error strings the API is returning. This is far from ideal because they are not structured, not documented and might change without warning. To improve things I would like to see the API extended to return meta data about errors (error type, id of offending element, ...) in a structured way. There are a couple of ways to that (that came to my mind): - change the Error header - add home-made HTTP headers (X-Error-Type ...) - add pseudo headers to the body - return a XML document containing all the info (see also http://trac.openstreetmap.org/ticket/2544) What do you think? How important is that to people on the receiving end (application developers)? On a related note, I would be very interested in seeing a progress of some kind being returned when doing a osmChange upload. I realize that it is difficult (because it could fail after spitting out lots of seemingly valid IDs), but if it was documented it would be doable. You mean showing upload progress in JOSM as opposed to the current cylon impression? That could be implemented by counting the number of bytes of the osmChange request that have been successfully sent over the wire. That's how upload progress bars are usually implemented. Obviously the upload could fail but that's another issue. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 17:01, Peter Körner osm-li...@mazdermind.de wrote:
You mean showing upload progress in JOSM as opposed to the current
cylon impression? That could be implemented by counting the number of
bytes of the osmChange request that have been successfully sent over
the wire. That's how upload progress bars are usually implemented.
Obviously the upload could fail but that's another issue.
That looks somehow more intelligent than streaming the progress from the
server back to the client.
It is:) When I upload a file with Google Chrome it always shows the
progress of the upload. This isn't done with some ad-hoc streaming XML
from the server, the client is just counting how many bytes it has
sent over the wire and how large a percentage that is of the total.
I looked at the relevant code in JOSM once but I couldn't find a way
to do it (but I'm not familiar with Java). In Perl with
HTTP::Request::Common you can do e.g.:
$HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1
my $req = HTTP::Request::Common-new( ... );
$req-content( sub {
my $chunk = $gen();
my $length = length $chunk;
warn I'm now uploading a chunk of length $length, of a total
of $total bytes;
return $chunk;
} );
my $result = LWP::UserAgent-request( $req );
(See this code for a practical example of this that I wrote:
http://cpansearch.perl.org/src/CPB/Flickr-Upload-1.32/Upload.pm)
I presume Java has some library to do this as well but I couldn't find it.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Thu, Dec 10, 2009 at 17:25, Karl Guggisberg karl.guggisb...@guggis.ch wrote: You mean showing upload progress in JOSM as opposed to the current cylon impression? That could be implemented by counting the number of bytes of the osmChange request that have been successfully sent over the wire. That's how upload progress bars are usually implemented. It's not the upload or download which takes most of the time but the processing on the server. Uploading an osmChange has three phases: Indeed, but I understood it from Ian's post when doing a[sic] osmChange upload that he was only interested in step #1. 1. upload the osmChange document - can take some time, we could give feedback information based on the bytes sent, but can be neglected compared to phase 2 Actually when I do uploads uploading the file itself and processing on the server seems to take around the same time, judging from network sniffing I've done. Results may vary though. 2. process on the server - this takes most of the time, no feedback here Yup, PostgreSQL can't tell you where it's at in the query: http://wiki.postgresql.org/wiki/Query_progress_indication 3. download the diffResult - can be neglected compared to phase 2. Again, we could give feedback based on the bytes read from the server, but it's not worth the effort Actually, I implemented something based on the approach Avar proposes but I wasn't satisfied and didn't check it in. I've filed a bug to track this, for what it's worth: http://josm.openstreetmap.de/ticket/4145 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] How to make the filter feature foolproof
On Sun, Dec 6, 2009 at 17:31, Sebastian Klein basti...@googlemail.com wrote: The Filter feature [1] is great, but still not ready for general use. I, for one, fool myself every now and then and think I have lost data, when it was actually just hidden by the filter. So how can we improve the situation? First of all I think that the vast majority of all filter texts will consist of _one_ pattern, only. (E.g. boundary, FIXME, user:me) So one could sacrifice the possible modes of combining different filters, for an easier interface: Just use normal Layers to represent filters. The advantage would be that one usually keeps an eye on the layer list and so you don't forget about it so easily. Layer visibility would correspond to Filter enabled/disabled. 'Hide' and 'Invert' would be icons on the layer entry that can be toggled. The other options (if any) would be accessible through the right click menu - 'edit'. You would insert a filter from the main menu: - 'add new Filter' - 'add preset Filter' ... - 'add saved Filter' ... Then the dialog could look something like this: Pattern: __ o show only the specified elements o hide the specified elements -- ☐ case sensitive ☐ regular expression ☐ apply to child elements When multiple filter layers are present, they would simply combine like 'AND' because this is what one would expect. To compensate this, one could introduce something similar to the filter in Thunderbird. (Multiple conditions can be combined.) I haven't done any coding, so feel free to object. :) This mostly sounds good, but more generally it would be nice if we had a preset system for search queries just like we have one for tags now. Then users could in the search menu select from some drop-down list of presets like edited by me (to search for user:$myusername), or boundaries and so on. Then you could just re-use these presets for the filter presets since the presets are just search-powered views on the OSM data. The interface in the filter dialog could be exactly as you suggest but the presets would be shared with the search engine. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Tagwatch Editor Counts
On Fri, Dec 4, 2009 at 20:09, Matt Amos zerebub...@gmail.com wrote: as a massive self-plug, here's a more in-depth look at some of the editor data. unfortunately, there isn't enough data to do this for any editor other than the big three, but hopefully in six months time... This looks nice. It's certainly a lot better than my ad-hoc statistics I posted in september: http://lists.openstreetmap.org/pipermail/talk/2009-September/042902.html One thing that I missed about your stats though is language statistics like the ones I posted (but done better with fancy graphs, obviousl :). Since I posted my stats I've added information about the user language to Potlatch (previously only JOSM had them), but unfortunately it looks like the Merkaartor people have ignored my request of adding it to their created_by string. It would be really cool to see statistics per-editor and per-language presented in such a way that one could gauge whether an editor being localized had an effect on its update. I recently found out for example that the Potlatch translation into Italian was really incomplete while JOSM was almost 100% translated into Italian (and JOSM has like 4000 strings while Potlatch has around 300). I contacted some Italian translators about this and Potlatch now has a much better Italian translation. Perhaps some Italians where shunning Potlatch because of this, it would be interesting to see stats to confirm or disprove that. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Distance Grid
On Tue, Dec 1, 2009 at 17:06, Matt Amos zerebub...@gmail.com wrote: On Tue, Dec 1, 2009 at 4:04 PM, Vitor George vitor.geo...@gmail.com wrote: I want to start a project in Brazil that is similar to the Tiger Fixup 250 Cities [1], and I looking forward to build a script that generates a distance grid, like this: http://matt.sandbox.cloudmade.com/usa-routes.html I've tried to contact the authors via wiki, but I was unsuccessful, so I want to develop it by myself. I'm not a professional programmer, but I have some experience in java. Is the LibOSM the easiest way tool to implement it? sorry you missed us on the wiki. i have to say, i don't really check the discussion pages like a good wikian should :-( the script is very simple, and uses cloudmade's ruby API [1] to get the routes. you'll need to sign up for a cloudmade API key, if you haven't already. the script i've attached will output CSV on stdout, but i usually put them in a database for easy access. run it like ruby routes.rb conf file. i've attached an example configuration file, which is just a yaml map of string to lat/lon array. i look forward to seeing brazil's 250 cities :-) I made a hacky script to generate .yml from XAPI output (attached): wget 'http://www.informationfreeway.org/api/0.6/node[place=city|town|village|hamlet][bbox=-25.74085,62.84553,-12.41708,67.50085]' -O iceland-places.osm perl -CI place2yaml.pl place.osm place.yml But I just get internal server errors from the CM API: $ ruby routes.rb place.yml [Tue Dec 01 18:59:57 + 2009] HTTP error: Couldn't read data. HTTP status: #Net::HTTPInternalServerError:0xb769348c, retrying... Anyway do you have a script to generate that cute HTML matrix? And how can I link to a route on CM's website. I have to supply lat/lon/zoom it seems and not just starting/ending lat/lon (at least the URLs I've seen are all like that). place2yaml.pl Description: Binary data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Distance Grid
On Tue, Dec 1, 2009 at 20:27, Matt Amos zerebub...@gmail.com wrote:
On Tue, Dec 1, 2009 at 7:03 PM, Ævar Arnfjörð Bjarmason
ava...@gmail.com wrote:
On Tue, Dec 1, 2009 at 17:06, Matt Amos zerebub...@gmail.com wrote:
On Tue, Dec 1, 2009 at 4:04 PM, Vitor George vitor.geo...@gmail.com wrote:
I want to start a project in Brazil that is similar to the Tiger Fixup 250
Cities [1], and I looking forward to build a script that generates a
distance grid, like this:
http://matt.sandbox.cloudmade.com/usa-routes.html
I've tried to contact the authors via wiki, but I was unsuccessful, so I
want to develop it by myself. I'm not a professional programmer, but I have
some experience in java. Is the LibOSM the easiest way tool to implement
it?
sorry you missed us on the wiki. i have to say, i don't really check
the discussion pages like a good wikian should :-(
the script is very simple, and uses cloudmade's ruby API [1] to get
the routes. you'll need to sign up for a cloudmade API key, if you
haven't already. the script i've attached will output CSV on stdout,
but i usually put them in a database for easy access.
run it like ruby routes.rb conf file. i've attached an example
configuration file, which is just a yaml map of string to lat/lon
array.
i look forward to seeing brazil's 250 cities :-)
I made a hacky script to generate .yml from XAPI output (attached):
wget
'http://www.informationfreeway.org/api/0.6/node[place=city|town|village|hamlet][bbox=-25.74085,62.84553,-12.41708,67.50085]'
-O iceland-places.osm
perl -CI place2yaml.pl place.osm place.yml
But I just get internal server errors from the CM API:
$ ruby routes.rb place.yml
[Tue Dec 01 18:59:57 + 2009] HTTP error: Couldn't read data. HTTP
status: #Net::HTTPInternalServerError:0xb769348c, retrying...
interesting. if you could send me a trace of what's going on that
would be very helpful.
Silly me, I was generating an invalid YAML file. still this could use
a better error message:
{{{
GET http://routes.cloudmade.com/MY_API_KEY/api/0.3/,,,/car.js?lang=enunits=km
HTTP/1.1
Accept: */*
Host: routes.cloudmade.com
075.101.134.058.00080-192.168.002.101.41914: HTTP/1.1 500 Internal Server Error
Server: Apache-Coyote/1.1
Content-Length: 40
Date: Tue, 01 Dec 2009 20:28:09 GMT
Connection: close
Sorry, some unrecoverable error happened
}}}
Anyway do you have a script to generate that cute HTML matrix? And how
can I link to a route on CM's website. I have to supply lat/lon/zoom
it seems and not just starting/ending lat/lon (at least the URLs I've
seen are all like that).
i've just thrown something together here - might be buggy, etc...
you'll need to play with the factor variable to see what looks right
for your area.
Sweet, I'll try it out.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Distance Grid
On Tue, Dec 1, 2009 at 20:27, Matt Amos zerebub...@gmail.com wrote: i've just thrown something together here - might be buggy, etc... you'll need to play with the factor variable to see what looks right for your area. I had to set it to 4. Anyway here's the scripts I used (including yours, hacked): http://git.nix.is/?p=avar/osm-place-matrix;a=summary And here's the output: http://osm.is/tmp/road-dist.html Thanks! ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Split osm line with perl
On Sun, Nov 29, 2009 at 12:10, Simone Cortesi sim...@cortesi.com wrote:
On Sun, Nov 29, 2009 at 12:16, Maarten Deen md...@xs4all.nl wrote:
I've tried a few things, but I'm not fluent in perl. My problem at the
moment is
that splitting a line on the space character seems logical, but you run into
problems if a value has a space in it.
wouldnt be wiser to use a DOM/XML parser. which is native able to interpret
XML?
Yes it would. Unfortunately some Perl programmers seem to be unaware
of the existence of CPAN and insist on solving non-trivial problems
like XML parsing over and over again with the wrong tools, namely
regular expressions;
If you want a Perl one-liner to get all tag values from a OSM file
here's one on the house that isn't insane:
perl -CI -MXML::Parser -E 'my $x = XML::Parser-new(Handlers = {
Start = sub { my ($p, $e, %kv) = @_; return unless $e eq tag; say
$kv{k} = $kv{v} } }); $x-parse(*STDIN)' File.osm
This could probably done in an easier way using something higher level
than XML::Parser (which is just a raw interface to expat) but I'm not
that familiar with Perl XML parsing. If I were to acquaint myself with
it I'd be sure not to start by writing the millionth buggy tagsoup
parser using regexes though.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Split osm line with perl
On Sun, Nov 29, 2009 at 18:43, Frederik Ramm frede...@remote.org wrote:
Ævar Arnfjörð Bjarmason wrote:
If I were to acquaint myself with
it I'd be sure not to start by writing the millionth buggy tagsoup
parser using regexes though.
As I said, a good craftsperson will know all available tools and choose the
one that best suits the job, and not disregard a whole family of tools just
because he believes them to be inferior (or uncool).
If you are dealing with the kind of XML emitted by Osmosis, you can make
assumptions about the structure. Assumptions that will break if you try to
deal with other files of course, but assumptions that make things faster as
long as you stay within the envelope.
Assume you want to count the different values for the highway tag.
The following millionth buggy tagsoup parser (which anyone familar with Perl
can write without looking up the details of an XML parser library) does
this for Germany in 108 seconds:
perl -e 'while() {$count{$1}++ if (/tag k=highway v=([^]*)/); };
foreach (sort { $count{$b}=$count{$a}} keys %count) { printf %6d
%s\n,$count{$_},$_; }'
Your XML parser based code into which I injected the same counting routine,
perl -CI -MXML::Parser -E 'my $x = XML::Parser-new(Handlers = {
Start = sub { my ($p, $e, %kv) = @_; return unless $e eq tag; return
unless $kv{k} eq highway; $count{$kv{v}}++; } }); $x-parse(*STDIN);
foreach (sort { $count{$b}=$count{$a}} keys %count) { printf %6d
%s\n,$count{$_},$_; }'
arrives at the same result in 915 seconds, that's a 850% performance
penalty.
Yes, the primitive version will choke if there's a line break or if someone
uses ' instead of ; it doesn't decode UTF-8 properly and it will not
resolve entities. Your version does all this, and precisely because it does,
takes four times longer.
A good programmer should be aware of this, and not pay for the XML parser
bells and whistles if he doesn't need them.
I may be a bit old-fashioned but I had to take exception to the arrogance
that spoke from your post. It is exactly that kind of attitude that I often
see in young programmers: I implemented this by the book and it doesn't go
any faster. - But how do you expect us to run this on a nightly basis when
your code takes 28 hours to run? - Use more machines, dude. Never heard of
map/reduce? - and all that because they are too snotnosed to parse XML with
a regex if required.
I'm not calling for premature optimisation, and nothing would be more stupid
than trying to parse a 100-line user-written config file with anything else
than a proper and tested XML parser. But discounting regex-based XML parsing
outright, without having some knowledge about the cost incurred, is
imprudent, and does not go well with the air of superiority that you gave
off.
I think Perl's regex engine is cool, in fact if you're using it you're
using my code.
However when a self-admitted Perl newbie starts a thread saying he's
already split up an XML file by lines and inquires about how he can
parse those lines it's worth stepping back and asking if that's really
the approach he wants to be taking. In most cases the answers given in
this thread are the right answers to the wrong question.
Admittedly my response was a bit snotty mostly because I've spent
untold hours maintaining large swaths of Perl code which for no good
reason reinvented something for which there was a perfectly good
library in a buggy manner with no documentation.
A lot of Perl programmers really do have no idea how to use CPAN
judging by the amount of code they churn out which duplicates
well-known and tested CPAN modules with their own badly reinvented
wheels.
Of course there are cases where the libraries aren't sufficient as you
rightly point out but nothing about Maarten's question indicated that
this was the case. Sometimes you have to dig yourself into the hole of
implementing maintaining your own tagsoup parser but I wouldn't help
a newbie dig that hole for himself unless I was certain that was what
he really needed.
And by the way your program would be slightly faster if you used
(.*?) instead of ([^]*). Minimally greedy matching is faster than
using negated character classes.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] indoor wifi geopositionning - openstreetmap precision, collaborators?
On Mon, Nov 30, 2009 at 02:35, Jonathan-David SCHRODER jonathan.schro...@gmail.com wrote: Hello, I am working on a student project (team of 6 people) whose goal is to build a solution allowing mobile devices to display indoor data along with wifi geopositioning. (I am willing citing this because contrary to http://www.micello.com or http://www.aws.cit.ie/mapume/ who we just found about, we want to have something fully open source/free software). As part of this project, we have decided to use the openstreetmap server software technology. Someone at University of Calford (UK) did the same as us and managed to draw into a self-hosted openstreetmap server, the inside of some campus's bookstore. See here : http://www.ja.net/development/network-access/location-awareness/investigations-la.html = B2. Interactive Maps = http://www.ja.net/documents/development/network-access/location-awareness/investigations/B2-interactive-maps-2.pdf @page 14 (or 270 in page footers) What worked well was the fact that whilst OpenStreetMap is intended for outdoor maps it can be made to work equally well for indoor maps. The interactive server can be configured to provide further zoom levels to display the resolution required for indoor maps. The provision of interactive elements can then be achieved by editing the PostGIS database to include additional objects, which can then be rendered by the mapnik renderer. Each additional interactive object will require its own unique style to be predefined to ensure that they appear correctly on the portable device. I have put Nigel Linge who I believe is the author of this PDF and paragraph as a recipient of this e-mail too. Could someone tell precisely what config changes need to be done for indoor precision objects, starting from a regular openstreetmap server setup such as that described on http://weait.com/content/build-your-own-openstreetmap-server ? I basically would like to be able to draw every possible object contained in a building/home/construction. I have started creating a tags draft for indoor which is very limited at : http://wiki.openstreetmap.org/wiki/Proposed_features#Proposed_Features_-_Indoor By the way, who would like to collaborate with my team on our indoor self- and objects geopositionning project relying on openstreetmap and java-based mobile applications ? That's a very interesting project, but I don't see why you need to set up your own OpenStreetMap-like infrastructure for it. Why not just save this data to the main OpenStreetMap API which will take care of storing it for you and then retrieve daily dumps for your area and render the map from those? Then you don't have worry about hosting your own API, others can easily access your data from OSM and you only have to worry about rendering. To render you need small PostGIS database (for mapnik) you refresh daily along with a custom stylesheet to actually render the data you're putting in, along with small configuration changes to mapnik/OpenLayers to render more zoom levels than normally. That can all be done on something as unpowerful as someones laptop which runs generate_tiles.py overnight and then uploads tiles / HTML to some web hosting space. The only thing I see potentially getting in your way is that the OSM database doesn't store enough significant digits of lat/lon coordinates to make indoor mapping viable, but perhaps it does. I couldn't find documentation on how many digits it stores and how that translates approximately into real-world meters/centimeters. Perhaps someone else can chime in with that information? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Switch to Java 6?
On Sat, Nov 28, 2009 at 19:24, Dirk Stöcker openstreet...@dstoecker.de wrote: Yes. #ifdef DEBUG and other conditionals are one of the features I most miss in Java. With a bit of build.xml hacking I think there's nothing stopping you from calling the C preprocessor before you compile the Java code :) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] potlatch breaking B38a
On Tue, Nov 24, 2009 at 05:22, Marcus Wolschon marcus.wolsc...@googlemail.com wrote: I have the source. I have inserted my call to setAdvice() to notify the user that he just merged 2 ways instead of selecting multiple ways. But where do I add the translated texts for a new constant advice_merged given to iText() ? I cannot find the text-ressources. You add it to sites/rails_port/config/potlatch/locales/en.yml, see e.g. this commit: http://trac.openstreetmap.org/changeset/18694 The reason for it being there is that the rails port tells Potlatch what i18n strings it should use. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] I reverted commit 18774 to the OpenStreetMap repository
Hi I reverted your change to it.yml here: http://trac.openstreetmap.org/changeset/18775 With this revert commit: http://trac.openstreetmap.org/changeset/18775 The reason is that we now use Translatewiki for editing translations for the rails port. If I hadn't reverted your commit it would have been automatically overwritten by the translatewiki import script (and potentially caused conflicts). Here's info on how to sign up to contribute to TranslatewikI: http://www.openstreetmap.org/user/Siebrand/diary/8342 Once you're signed up you can edit the Italian translation here: http://translatewiki.net/w/i.php?title=Special:Translatetask=untranslatedgroup=out-osm-sitelanguage=itlimit=500 (CC-ing to dev@ for future reference) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[josm-dev] Duplicate placeholder IDs in the latest JOSM tested (2510)
On Tue, Nov 17, 2009 at 11:50, Dirk Stöcker openstreet...@dstoecker.de wrote: Code consolidation until end of month, so we can have a new release by then. Is the current latest tested release based on 2510 the aforementioned end of the month release? It was released with this bug I filed 4 weeks ago: http://josm.openstreetmap.de/ticket/3812 One user already ran into it during normal editing (see comments on the bug). It's fairly easy to trigger and really hard to fix for normal users who aren't clueful enough to save the file and manually edit it with a text editor until it's uploadable. It would be good if someone could take a look at this bug fix it and release a new JOSM tested soon. Otherwise I think we're going to have a lot of support requests over the next weeks with people running into this bug, and lost data for those that don't have the patience to deal with it. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
