Hi Sarnth,
Thanks to share this, we are using different methodologies, but the
basic idea
are same, to using KV storage to reduce query latency.
I have to say OLAP is a complex system which not only about storage,
there were some discussions about Redis/HBase/Cassandra for underline
K-V storage. HBase may not the perfect option yet but it works so far,
at least in our cases. I do not know too much about ES so that I can't give
you too much comments about that, but would like to suggest to do more
testing
with different data and scenarios, we actually made such progress with many
extreme cases to help verify and validate.
BTW, are you just using ES REST api for query? Or bring SQL on top of
your engine?
Thanks.
Luke
Best Regards!
---------------------
Luke Han
On Fri, Nov 20, 2015 at 12:01 AM, Sarnath <[email protected]> wrote:
> Hi Luke,
> I will definitely post back here when we make things public.
>
> My choice of ES is based on simple reasoning:
>
> When we build a Cuboid -- we basically build Key-Value pairs. i.e.
> 1) "Country=US", "State=Texas", "City=Houston", "Quantity=500"
> 2) "Country=India", "State=Karnataka", "City=Bangalore", "Quantity=1000"
> and so on...
>
> That looked to me like an ES document (or) a SOLR document
>
> And, When we search it, We search it like this:
>
> 1) Fetch me all documents that have "Country=US", "State=Texas"
> 2) Fetch me all documents that correspond to "Country=India".
> 3) Fetch me all documents that correspond to "Country=US"
> and so on..
>
> All these are essentially search problems solved by Elastic Search/SOLR and
> they are scalable as well..
>
> So, armed with this simple insight - we started the project. Do you find
> this simple? simplistic? (or) Are we going wrong somewhere? You guys have
> already achieved a lot in this space. Looking forward to your advice.
>
> Best,
> Sarnath
>
> On Thu, Nov 19, 2015 at 9:11 PM, Luke Han <[email protected]> wrote:
>
> > Hi Sarnath,
> > You are welcome, I agree with you there are different approaches,
> > Would you mind to share more about your system design and why you
> > choose ES as storage (just curious), is there any public information I
> > could refer to?
> > And would live to have your result and please share any finding
> > which could help to improve this project:)
> >
> > Thanks.
> >
> > Luke
> >
> >
> >
> >
> > Best Regards!
> > ---------------------
> >
> > Luke Han
> >
> > On Thu, Nov 19, 2015 at 10:41 PM, Sarnath <[email protected]> wrote:
> >
> > > Hi Luke,
> > >
> > > My intention is to learn. That's all. As I said there are many ways to
> > > attack a problem. And, big data offers lot of choices..so, at least, to
> > > justify to myself, I would like to run this at scale and report
> findings.
> > > If nothing, it would at least make a good read over tea on Sunday
> > > afternoon.
> > > I will write back to this thread after doing some meaningful test.
> > > Thanks for all your time and your patience.
> > >
> > > Best,
> > > Sarnath
> > >
> >
>
