[
https://issues.apache.org/jira/browse/CONNECTORS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181155#comment-13181155
]
Karl Wright edited comment on CONNECTORS-286 at 1/6/12 6:23 AM:
----------------------------------------------------------------
bq. Not sure what Wharthog is...
See
https://svn.apache.org/repos/asf/incubator/lcf/branches/CONNECTORS-286/warthog.
"Warthog" is a potential future spinoff technology of ManifoldCF - if all this
stuff actually works. ;-)
bq. But I wanted to suggest you consider using HBase for underlying storage.
Maybe. One of the apparent requirements of ManifoldCF is the ability run on
top of practically anything. I don't think we'd be able to throw away
PostgreSQL and MySQL and HSQLDB and Derby support for instance. So if Warthog
is the actual API layer ManifoldCF uses then we'd need implementations of
Warthog for as many backends as possible. Right now I've only got one going
for a testing key-value store, but almost certainly the next step would be a
SQL database. Then moving on from then HBase (or Hive/Pig) may also be
possibilities. Still proving the concept however...
was (Author: kwri...@metacarta.com):
bq. Not sure what Wharthog is...
See
https://svn.apache.org/repos/asf/incubator/lcf/branches/CONNECTORS-286/warthog.
"Warthog" is a potential future spinoff technology of ManifoldCF - if all this
stuff actually works. ;-)
bq. But I wanted to suggest you consider using HBase for underlying storage.
Maybe. One of the apparent requirements of ManifoldCF is the ability run on
top of practically anything. I don't think we'd be able to throw away
PostgreSQL and MySQL and HSQLDB and Derby support for instance. So if Warthog
is the actual API layer ManifoldCF uses then we'd need implementations of
Warthog for as many backends as possible. Right now I've only going one for a
testing key-value store, but almost certainly the next step would be a SQL
database. Then moving on from then HBase (or Hive/Pig) may also be
possibilities. Still proving the concept however...
> Get ManifoldCF to run on top of a key/value store like Voldemort, for
> potential massive scalability improvements and speed gains
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-286
> URL: https://issues.apache.org/jira/browse/CONNECTORS-286
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Framework core
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF next
>
>
> ManifoldCF's reliance on a relational database limits its throughput and
> scalability. I am now convinced it is possible to build all the structures
> we need within a distributed key-value store like Voldemort, which has the
> nice side effect of permitting massive scaling. I envision there will be
> several layers to this project, some of which may have broader utility in the
> open-source community at large:
> (1) An atomic serialization layer, which adds serialization capabilities to
> an non-transactional substrate;
> (2) A transaction layer, which uses atomic serialization to build a notion of
> light transactions;
> (3) A table and index layer, which defines SQL-like concepts of tables and
> btree indexes on top of the transaction layer, via a Java API;
> (4) A generic "database abstraction" layer, which is capable of representing
> both standard SQL databases as well as this NoSQL variant, so that ManifoldCF
> can support both models.
> This is obviously a major development task, and as such is not envisioned to
> be completed by the next standard release. Work will indeed need to be done
> in a branch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
