[Wikitech-l] More news on Wikidata Query Indexing Strategy

2015-02-04 Thread Nikolas Everett
tl/dr: The technology we started building against (Titan) is probably
dead.  We're reopening the investigation for a backing technology.

Yesterday DataStax http://www.datastax.com/ announced
http://www.datastax.com/2015/02/datastax-acquires-aurelius-the-experts-behind-titandb
that they'd acquired
http://www.datastax.com/2015/02/datastax-acquires-aurelius-the-experts-behind-titandb
ThinkAurelius http://thinkaurelius.com/, the company for whom almost all
the Titan developers work. The ZDNet article
http://www.zdnet.com/article/datastax-snaps-up-aurelius-and-its-titan-team-to-build-new-graph-database/
made it pretty clear that they are killing the project

 We're not going to do an integration. The play here is we'll take
 everything that's been done on Titan as inspiration, and maybe some of the
 Titan project will make it into DSE Graph, DataStax engineering VP Martin
 Van Ryswyk said.


While its certainly possible that someone from the community will come out
of the woodwork and continue Titan its now lost almost all of its top
developers.  It looks like there is some secret succession discussions
going on but I'm not holding out hope that anything will come of it.  This
pretty much blows this project's schedule of having a hardware request by
the end of the month and a publicly released beta at the end of March.

Anyway, we're reopening the investigation to pick a new backend.  We're
including more options than we had before as its become clear that open
source graph databases is a bit of a wild west space.  But there are people
waiting on this.  The developer summit made that clear.  So we're not going
to do the month long dive into each choice like we did last time.  I'm not
100% sure exactly what we'll do but I can assure you we'll be careful.

I know you might want to talk about other options - you may as well stuff
them on
https://www.mediawiki.org/wiki/Wikibase/Indexing#Other_possible_candidates
and we'll get to them.  As always, you can check out our workboard
https://phabricator.wikimedia.org/project/board/37/query/DwEBx9K4vaHo/ to
see what we're actually working on.

Titan is still in the running assuming it gets active maintainers.
OrientDB, which we evaluated last round, is still in there too.  So too are
GraphX and Neo4j.  And ArangoDB.  And Magnus' WDQ.  We'd get much more
involved in maintenance, I think.  And writing a TinkerPop implementation
Elasticsearch.  That's not a serious contender.  It'd get geo support for
free but its really just a low bar to compare all the other options to.

Thanks,

Nik https://phabricator.wikimedia.org/T88550
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] More news on Wikidata Query Indexing Strategy

2015-02-04 Thread Nikolas Everett
Top posting to add context: this is for the initiative to get a version of
Magnus' wonderful http://wdq.wmflabs.org/ running in production at WMF.

On Wed, Feb 4, 2015 at 4:50 PM, Nikolas Everett never...@wikimedia.org
wrote:

 tl/dr: The technology we started building against (Titan) is probably
 dead.  We're reopening the investigation for a backing technology.

 Yesterday DataStax http://www.datastax.com/ announced
 http://www.datastax.com/2015/02/datastax-acquires-aurelius-the-experts-behind-titandb
 that they'd acquired
 http://www.datastax.com/2015/02/datastax-acquires-aurelius-the-experts-behind-titandb
 ThinkAurelius http://thinkaurelius.com/, the company for whom almost
 all the Titan developers work. The ZDNet article
 http://www.zdnet.com/article/datastax-snaps-up-aurelius-and-its-titan-team-to-build-new-graph-database/
 made it pretty clear that they are killing the project

 We're not going to do an integration. The play here is we'll take
 everything that's been done on Titan as inspiration, and maybe some of the
 Titan project will make it into DSE Graph, DataStax engineering VP Martin
 Van Ryswyk said.


 While its certainly possible that someone from the community will come out
 of the woodwork and continue Titan its now lost almost all of its top
 developers.  It looks like there is some secret succession discussions
 going on but I'm not holding out hope that anything will come of it.  This
 pretty much blows this project's schedule of having a hardware request by
 the end of the month and a publicly released beta at the end of March.

 Anyway, we're reopening the investigation to pick a new backend.  We're
 including more options than we had before as its become clear that open
 source graph databases is a bit of a wild west space.  But there are people
 waiting on this.  The developer summit made that clear.  So we're not going
 to do the month long dive into each choice like we did last time.  I'm not
 100% sure exactly what we'll do but I can assure you we'll be careful.

 I know you might want to talk about other options - you may as well stuff
 them on
 https://www.mediawiki.org/wiki/Wikibase/Indexing#Other_possible_candidates
 and we'll get to them.  As always, you can check out our workboard
 https://phabricator.wikimedia.org/project/board/37/query/DwEBx9K4vaHo/
 to see what we're actually working on.

 Titan is still in the running assuming it gets active maintainers.
 OrientDB, which we evaluated last round, is still in there too.  So too are
 GraphX and Neo4j.  And ArangoDB.  And Magnus' WDQ.  We'd get much more
 involved in maintenance, I think.  And writing a TinkerPop implementation
 Elasticsearch.  That's not a serious contender.  It'd get geo support for
 free but its really just a low bar to compare all the other options to.

 Thanks,

 Nik https://phabricator.wikimedia.org/T88550


And, too add more context, we chose not to just immediately deploy Magnus'
WDQ because we didn't want to maintain a graph database ourselves.  You
should now be able to appreciate the irony of the situation more
thoroughly.  Its healthy to find humor where you can.

Nik
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l