For term "heavy/complex stack", I have different opinion, Kylin, from the begging is designed to serve TB to PB level dataset and queries, and scale out for high concurrency, for that scenario, such complex is worth to do. Distribution + Big Data is not simple world we could live with;-)
But I agree with you about development cycle part, that's why we are introduce mini cluster, and also off-hadoop-cli installation guide for people to develop on own laptop but can share one or more dev cluster even just VM. And, please refer to Yang's recent thread for https://issues.apache.org/jira/browse/KYLIN-875 about more generic storage interface. Thanks. Best Regards! --------------------- Luke Han On Wed, Jul 15, 2015 at 4:47 PM, Li Yang <[email protected]> wrote: > For dev/test purpose, I personally uses HDP 2.2 on a single node sandbox > with 8GM memory. Not lightweight at all, but works quite well. > > Kylin has clearly defined interface between query module and storage module > -- IStorageEngine. Somehow so far the only implementation is HBase. > > > > On Mon, Jul 13, 2015 at 8:08 AM, Stephen Boesch <[email protected]> wrote: > > > HBase/Zookeeper is a heavy/complex stack when considering small-scale > > development and testing . The Mini HBase cluster is tricky to configure > > and consumes a fair amount of memory. Zookeeper suffers from timeout > > issues that complicate debugging cycles. Region server management also > > complicates testing It may be preferable to have an option to avoid > these > > considerations altogether when working on/developing portions of logic > that > > do not interface directly with the indexing and metadata logic. > > > > In the eBay blog there is a single sentence mentioning it "may" be > possible > > to use a different K-V backend than HBase: > > > > http://www.ebaytech > > blog.com/2014/10/20/announcing-kylin-extreme-olap-engine-for-big-data/ > > > > > > > > - *Storage Engine: *This engine manages the underlying storage – > > specifically the cuboids, which are stored as key-value pairs. The > > Storage > > Engine uses HBase ... *Kylin can also be extended to support other K-V > > systems, such as Redis <http://redis.io/>.* > > > > Is there any documentation on how that extension may be achieved? A > > pluggable interface? I would for example like to see Cassandra as a > > drop-in replacement for HBase. > > > > Thanks > > > > stephenb > > >
