[
https://issues.apache.org/jira/browse/CONNECTORS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747450#comment-13747450
]
Karl Wright commented on CONNECTORS-286:
----------------------------------------
Experiments with warthog show the following:
(1) Performance on a representative single-threaded sequence of reads and
writes improves up to a max of some 30%, if you have 8 processors involved.
(2) Beyond 8 processors, performance degrades, as contention overwhelms the
gains due to lack of locks.
My conclusion is that for a ManifoldCF-style access pattern, some mechanism of
gating access (i.e. locking) becomes important at some point for maximum
throughput. It is not clear exactly how you would determine when switching to
a locking paradigm would be appropriate. This project has therefore entered
the "research project" stage, in my view.
> Get ManifoldCF to run on top of a key/value store like Voldemort, for
> potential massive scalability improvements and speed gains
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-286
> URL: https://issues.apache.org/jira/browse/CONNECTORS-286
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Framework core
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF next
>
>
> ManifoldCF's reliance on a relational database limits its throughput and
> scalability. I am now convinced it is possible to build all the structures
> we need within a distributed key-value store like Voldemort, which has the
> nice side effect of permitting massive scaling. I envision there will be
> several layers to this project, some of which may have broader utility in the
> open-source community at large:
> (1) An atomic serialization layer, which adds serialization capabilities to
> an non-transactional substrate;
> (2) A transaction layer, which uses atomic serialization to build a notion of
> light transactions;
> (3) A table and index layer, which defines SQL-like concepts of tables and
> btree indexes on top of the transaction layer, via a Java API;
> (4) A generic "database abstraction" layer, which is capable of representing
> both standard SQL databases as well as this NoSQL variant, so that ManifoldCF
> can support both models.
> This is obviously a major development task, and as such is not envisioned to
> be completed by the next standard release. Work will indeed need to be done
> in a branch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
