Steve,
SteveC wrote:
I don't mean just another big import. Sadly I can't be public
but I had a conversation with a large company over a year ago (no,
it's not MS or CM) who speculated about putting OSM on the front
page of their maps product, which would approximately turn all of our
yearly statistics to daily or weekly numbers. We went through a
decision tree about how that could happen. Every leaf node on that
tree came back as basically we couldn't do it.
This was not purely a techical issue. If we were set up, technically, to
handle something like what you're describing here, the "eternal
september" effect would kill off the community for good.
So in a way, we must be glad that we have this technical limitation,
because it protects us against someone dumping 1000s of new users every
week onto our mailing lists, IRC channels, forums, onto help.osm.org,
the Wiki, and int our regular pub meets.
If you think through it,
within any reasonable time frame (like 6-12 months) it's very hard to
make that happen, and so you may as well go build your own things.
Which I think sucks and is a loss for OSM.
I honestly think that the project would degrade tremendously were
something like this to happen. We can handle growth, and I also believe
we can handle super-linear growth, but something that "turns all our
yearly statistics into daily or weekly numbers" would be the end of
everything that is sane about OSM.
Now this conversation has come up a few more times recently with
other large mapping companies. And I feel like I'm rehashing those
conversations above. I'd love to be public about it, but those
companies aren't ready to talk yet.
Good for them because they would have to take the heat for killing OSM -
or nonchalantly thinking about doing that.
Even if people weren't privately proposing notching up our traffic a
few orders of magnitude
Please stop thinking of this as a technical issue. It isn't!
But the direction of supporting and encouraging basic
things like scaling is I think well within the bounds.
I haven't a clue what we should use to scale horizontally.
I think "encouraging scaling" is within the bounds of what OSMF could
think about. Whether or not such scaling needs to be horizontal,
vertical, deep-bore or whatever other method is currently en vogue,
that's perhaps something best left to the TWG.
If OSMF board wants to realistically construct a scenario that has the
number of new users, and the amount of edits and traffic, multiplied by
100 in the frame of a few months, then I'm sure that TWG can provide
ideas how to get there. But you would *have* to find someone who creates
that scenario for the social side and looks into the issues I've brought
up above or else we'll be fucked on a level that's much harder to fix
than a slow database.
I don't know how much your personal view might be tainted by the US
perspective. Compared to Europe, ans especially taking into account the
population base, community size in the US is still negligible. You might
be tempted to think that you can play va banque there as there isn't a
lot to lose. If that's the case then maybe we should work on providing
more separation - allowing you to supercahrge and horizontally scale
OSM-US with the help of one megacorp or the other, but keeping us here
in Europe out of harm's way.
(Indeed one way of scaling would be to "devolve" the central database -
have several of them for different regions. That would make it more
difficult for people to write bots that wreak havoc with the world-wide
dataset but I wouldn't consider that a loss ;)
So, how do we get from here to there? Speaking strictly personally, I
think one of the best uses of funds in or out of OSM has been bug
bounties.
Does anyone have an example where this has worked in OSM? I always
perceived these bounties as your personal hobby horse that never worked
for us but maybe my thinking has clouded my vision here.
Personally, I think that people tend to be encouraged by the hope that
their solution might become THE solution used in OSM. Tell a hacker that
he gets $1000 if he gets something to work, and he'll say "oh well,
let's leave that to those who need the cash". Tell a hacker that you'll
roll out his idea big time if he proves it can work, and he'll happily
do it...
So, what do you think? And if you agree it's worth doing, how do we
achieve it either as individuals or the board or companies supporting
it?
First of all, stop talking of the kind of explosive growth you're
talking about. Set yourselves a realistic growth scenario - I'd say
something like "number of contributors & data size double every year" or
so. Sit down with TWG and find out how long the traditional approach
will hold out, and how and when to scale, then make plans for that. I'm
sure TWG will be mature enough to say what they can do and what they
can't, and where they will need input & concepts from the project at
large. Find suitable ways to solicit such.
If you are looking for a project that can support the kind of explosive
growth you envisage - factor 100 in a few months -, then start a new
project, one that relies much less on individual human beings. A project
where you can simply scale by adding another server, without also having
to tear down the pub and erect a town hall in its place.
Bye
Frederik
--
Frederik Ramm ## eMail [email protected] ## N49°00'09" E008°23'33"
_______________________________________________
dev mailing list
[email protected]
http://lists.openstreetmap.org/listinfo/dev
