Is there some coprocessor-equivalent mechanism in maprdb? If yes, you can try to implement the logic into it. Otherwise, the performance might be unacceptable when there is bunch of data in each region.
I rememer in very early version, Kylin doesn't have coprocessor, all data be fetched to Kylin server and then do filtering and aggregation by Calcite. Later coprocessor was added and improved continiously. Now there is no option to disable it. You have to modify the code if don't want to trigger it. 2017-07-09 1:34 GMT+08:00 Nirav Patel <[email protected]>: > > Thanks Shao for some pointers. I would like some more advice on how I can > replace maprdb as datasource. > > Currently, you can create cubes on maprdb using hbase-storage module as > maprdb does support hbase client/admin APIs. Only problem is while querying > it throws an error due to Lack of support of Co-processor. > > My initial goal is to get past coprocessor error and return query result. > Then find a better way to implement that strategy - may be a separate > maprdb-storage module. Then look into how to improve performance without > coprocessors. Any advice on these approaches ? > > Thanks, > Nirav > > On Sat, Jul 8, 2017 at 7:54 AM, ShaoFeng Shi <[email protected]> > wrote: > >> The core modules like core-cube, core-storage are totally independent of >> HBase; while some others like engine-mr, engine-spark has dependencies on >> HBase. If you want to replace it, you need implement new cubing engine as >> well. Please also note that Kylin's metadata is persisted to HBase by >> default, you need have another implementation for ResourceStore. >> >> In a short, the plug-in architecture works (we have verfied that), while >> changing the storage is a complex task which take some time to be function >> complete and performance stable. >> >> >> >> >> >> >> 2017-07-08 5:19 GMT+08:00 Nirav Patel <[email protected]>: >> >>> Yes, MapR-DB doesn't support coprocessors. >>> >>> Here's the thing though - based on kylin plugin architecture it >>> shouldn't be a problem ideally. Aggregation as well as other DML/DDL >>> operation on datasources should be done transparently. i.e. using >>> kylin-hbase adapter written with calcite or something. It's upto the writer >>> of those adapters to implement aggregates however they want. i.e. either >>> using coprocessors, or in-memory on application server side or using spark. >>> http://kylin.apache.org/development/plugin_arch.html >>> >>> I think for mapr-db we can leverage mapr-drill with secondary indexes >>> for faster filtering and drill does parallel aggregation as well. Other >>> option is to use mapr-spark with mapr-db which can does the same. >>> >>> Do you know how tightly hbase is coupled with other modules of kylin >>> source other then hbase-storage. >>> >>> >>> >>> >>> >>> >>> On Thu, Jul 6, 2017 at 6:53 PM, ShaoFeng Shi <[email protected]> >>> wrote: >>> >>>> Hi Nirav, >>>> >>>> I googled that "HBase coprocessors are not present in MapR DB", is >>>> this true? You know Kylin relies on HBase coprocessor to do filering and >>>> aggregation in each region local; If coprocessor is not available, the >>>> performance will be a problem. >>>> >>>> 2017-07-07 1:29 GMT+08:00 Nirav Patel <[email protected]>: >>>> >>>>> Hi, We are a mapr users. You will need to deploy separate Hbase >>>>> Cluster alongside your mapr Cluster or on top of it. You won't be able to >>>>> use Mapr-DB with Kylin. >>>>> I am looking into writing Kyling - MaprDB adapter. >>>>> >>>>> On Tue, Jul 4, 2017 at 8:23 AM, <[email protected]> wrote: >>>>> >>>>>> V5.2, >>>>>> >>>>>> I’m trying to figure out how to deploy Kylin on a MapR cluster, >>>>>> whether Kylin has to be installed on the same cluster of the MapR cluster >>>>>> >>>>>> >>>>>> >>>>>> *From:* Luke Han [mailto:[email protected]] >>>>>> *Sent:* Tuesday, July 04, 2017 10:04 PM >>>>>> *To:* user >>>>>> *Subject:* Re: Kylin for MapR >>>>>> >>>>>> >>>>>> >>>>>> Kylin support MapR, which MapR version you are using now? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Best Regards! >>>>>> --------------------- >>>>>> >>>>>> Luke Han >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Jul 4, 2017 at 4:19 PM, <[email protected]> wrote: >>>>>> >>>>>> Hi >>>>>> >>>>>> Does kylin support MapR version of the Hadoop? >>>>>> >>>>>> >>>>>> >>>>>> This e-mail (including any attachments) is private and confidential, >>>>>> may contain proprietary or privileged information and is intended for the >>>>>> named recipient(s) only. Unintended recipients are strictly prohibited >>>>>> from >>>>>> taking action on the basis of information in this e-mail and must contact >>>>>> the sender immediately, delete this e-mail (and all attachments) and >>>>>> destroy any hard copies. Nomura will not accept responsibility or >>>>>> liability >>>>>> for the accuracy or completeness of, or the presence of any virus or >>>>>> disabling code in, this e-mail. If verification is sought please request >>>>>> a >>>>>> hard copy. Any reference to the terms of executed transactions should be >>>>>> treated as preliminary only and subject to formal written confirmation by >>>>>> Nomura. Nomura reserves the right to retain, monitor and intercept e-mail >>>>>> communications through its networks (subject to and in accordance with >>>>>> applicable laws). No confidentiality or privilege is waived or lost by >>>>>> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" >>>>>> is >>>>>> a reference to any entity in the Nomura Holdings, Inc. group. Please read >>>>>> our Electronic Communications Legal Notice which forms part of this >>>>>> e-mail: >>>>>> http://www.Nomura.com/email_disclaimer.htm >>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.Nomura.com_email-5Fdisclaimer.htm&d=DwMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=nFqGYx5SOTBIei-Hwk8uz_JfhmSu2UlelQfdmGX2k5I&m=s6jR2dvhFFTZIuHhCE8WU8EYNQ7M5u-9SggFAoyn5-M&s=iXZr3lmnuJ4Xk2-qORnbYZQALda0jRLZ3kzGy816bco&e=> >>>>>> >>>>>> >>>>>> >>>>>> This e-mail (including any attachments) is private and confidential, >>>>>> may contain proprietary or privileged information and is intended for the >>>>>> named recipient(s) only. Unintended recipients are strictly prohibited >>>>>> from >>>>>> taking action on the basis of information in this e-mail and must contact >>>>>> the sender immediately, delete this e-mail (and all attachments) and >>>>>> destroy any hard copies. Nomura will not accept responsibility or >>>>>> liability >>>>>> for the accuracy or completeness of, or the presence of any virus or >>>>>> disabling code in, this e-mail. If verification is sought please request >>>>>> a >>>>>> hard copy. Any reference to the terms of executed transactions should be >>>>>> treated as preliminary only and subject to formal written confirmation by >>>>>> Nomura. Nomura reserves the right to retain, monitor and intercept e-mail >>>>>> communications through its networks (subject to and in accordance with >>>>>> applicable laws). No confidentiality or privilege is waived or lost by >>>>>> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" >>>>>> is >>>>>> a reference to any entity in the Nomura Holdings, Inc. group. Please read >>>>>> our Electronic Communications Legal Notice which forms part of this >>>>>> e-mail: >>>>>> http://www.Nomura.com/email_disclaimer.htm >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> [image: What's New with Xactly] >>>>> <http://www.xactlycorp.com/email-click/> >>>>> >>>>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] >>>>> <https://www.linkedin.com/company/xactly-corporation> [image: >>>>> Twitter] <https://twitter.com/Xactly> [image: Facebook] >>>>> <https://www.facebook.com/XactlyCorp> [image: YouTube] >>>>> <http://www.youtube.com/xactlycorporation> >>>> >>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> >>>> Shaofeng Shi 史少锋 >>>> >>>> >>> >>> >>> >>> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> >>> >>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] >>> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] >>> <https://twitter.com/Xactly> [image: Facebook] >>> <https://www.facebook.com/XactlyCorp> [image: YouTube] >>> <http://www.youtube.com/xactlycorporation> >>> >> >> >> >> -- >> Best regards, >> >> Shaofeng Shi 史少锋 >> >> > > > > [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> > > <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] > <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] > <https://twitter.com/Xactly> [image: Facebook] > <https://www.facebook.com/XactlyCorp> [image: YouTube] > <http://www.youtube.com/xactlycorporation> > -- Best regards, Shaofeng Shi 史少锋
