Collect practices of running Kylin in AWS/Azure

[ShaoFeng Shi]

Hello Kyliner,

Recently a couple users are asking for the practices about running Kylin in
AWS, together with cloud services like EMR, S3; We can forsee that more and
more people will move to cloud in next couple years. If you have a success
story and like to share, please raise your hand. We can add topics about
cloud deployments in Kylin website.

Thanks.

-- 
Best regards,

Shaofeng Shi 史少锋

Re: Kylin Config as external Node

[Patricio Huichulef]

Thank Li to confirm to me this information, where Can I find the
documentation about this configuration?, because  I have an internal Lab,
i'm configuring a VM (VMWare) as Kylin Node to access the  Cloudera Nodes.

Can you show me what config files I need to change to get this?

Thanks in advance
PEHC

On Tue, Jul 25, 2017 at 1:16 PM, Li Yang  wrote:

> Yes, Kylin can be installed on any Hadoop client node. That includes edge
> node, or external node with necessary Hadoop libs and configurations.
>
> The prerequisite of Kylin can verified by commands like `hadoop`, `hive`,
> `hbase`. As long as these command lines work, Kylin should be able to work.
>
> Cheers
> Yang
>
> On Sat, Jul 22, 2017 at 1:04 AM, Patricio Huichulef 
> wrote:
>
>> Hello Folks:
>>I would like to configure a Node as Kylin Engine Server, but this node
>> should be separated from Cloudera Servers. It´s mean is an External node to
>> Data Lake, but with access to Data Lake,
>>
>>  How can I configure this Kylin server to connect to external Hive
>> and HBase servers?
>>
>> As Test I am creating a VM with Kylin 2.0.0 and We have a Lab with
>> three nodes with Cloudera 5.5.5.
>>
>> I would appreciate your information and help , thanks in advance,
>>
>> PEHC
>>
>> PD: I´m new with Kylin.
>>
>> --
>> Patricio Edo. Huichulef C.
>>
>>
>

Re: New blog "Improving Spark Cubing in Kylin 2.0"

[Li Yang]

Bravo Kaisen!
赞 Kaisen!

On Sat, Jul 22, 2017 at 12:32 PM, ShaoFeng Shi 
wrote:

> Hello Kyliner,
>
> The blog "Improving Spark Cubing in Kylin 2.0" from committer Kangkai Sen
> is published in Kylin website today. For the users who are interested about
> Spark + Kylin, it can help:
>
> https://kylin.apache.org/blog/2017/07/21/Improving-Spark-Cubing/
>
> The original Chinese version is in Kaisen's personal blog:
> http://blog.bcmeng.com/
>
> Thanks Kaisen's sharing!
>
> If anyone want to contribute Kylin tech blog, please contact me.
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>

Re: How to make a Sizing Server for Kylin

[Li Yang]

Sharing will be appreciated.  :-)

On Sat, Jul 22, 2017 at 2:20 AM, Patricio Huichulef 
wrote:

> Therefore, we need to build a methodology to make an appropriate sizing
> for any solution. OK Let´s start to work on this.
>
> Thanks Li
>
> On Fri, Jul 21, 2017 at 3:36 AM, Li Yang  wrote:
>
>> For me, build a small chunk of data and assess linearly for the full data
>> set is still the best approach.
>>
>> On Tue, Jul 18, 2017 at 12:43 AM, Patricio Eduardo Huichulef Carvajal <
>> phuichu...@gmail.com> wrote:
>>
>>> Hello Folks:
>>>
>>> I would like to know if somebody has a methodology to make a Sizing
>>> Server for Kylin OLAP Engine.
>>>
>>> Is there a process to size properly one or multiple server, to
>>> design a Kylin Architecture? For example, for cluster, what is the
>>> recommendation of Hardware Sizing for each node?, What are the parameters
>>> to make a properly size ? (Txs, Data Volume, Cube Size, amount of queries,
>>> etc.
>>>
>>> Thanks in advance
>>>
>>> PEHC
>>>
>>>
>>
>
>
> --
> Patricio Edo. Huichulef C.
> IT Trusted Advisor
>
> MIL Technology
> Ramon Sotomayor 2937
> Depto. 302, Providencia
> Santiago, 6650788 Chile
>
> Work: 562 4740959 <(562)%20474-0959>
> Mobile: 569 98858814
> Email: patricio.huichu...@vtr.net
> IM: phuichulef (AIM)
>  http://www.linkedin.com/in/patriciohuichulef
> My Blog: http://phuichulef.blogspot.com
>

Re: 答复: Hbase on S3

[ShaoFeng Shi]

Sorry my initial statements is something misleading; Adding the
"hbase.zookeeper.quorum"
to kylin_job_conf.xml will make Kylin works with EMR to finish the cubing.
While the default FS of EMR is still be the HDFS (with local disks), not S3

To use S3 as the data store, you need 1) configure kylin working dir as
something like s3://your-bucket/kylin ; 2) use a HBase version which
support S3 as storage.





2017-05-31 12:33 GMT+08:00 iain wright :

> It looks like Hbase can use S3 via EMRFS , it must be a fairly new
> feature, it was asked on the Hbase list for years and wasn't possible before
>
> http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hbase-s3.html
>
> Sent from my iPhone
>
> On May 30, 2017, at 6:48 PM, ShaoFeng Shi  wrote:
>
> Zhuoran, what's the problem you encountered? Some Kylin users already runs
> on AWS EMR or Azure HDInsight. No blocking issue we see. If you got issue,
> please share more information.
>
> 2017-05-31 9:41 GMT+08:00 吕卓然 :
>
>> I do not think so
>>
>>
>>
>> *发件人:* Tokayer, Jason M. [mailto:jason.toka...@capitalone.com]
>> *发送时间:* 2017年5月30日 21:30
>> *收件人:* user@kylin.apache.org
>> *主题:* Hbase on S3
>>
>>
>>
>> Does Kylin support storing data in S3 rather than HDFS?
>>
>>
>> --
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Re: Define Model and Cube Programmatically

[Li Yang]

There is no public stable RESTful API to create cube programmatically.

However there is one API that Kylin GUI uses to create cube, and in the
debug console of modern browser, advanced user can observe how this API
works. But note, it is not a stable API and is subject to change from
version to version. User must not rely on it to build anything of
production quality.

On Thu, Jul 20, 2017 at 2:28 PM,  wrote:

> Hi All
>
> Is there a way of defining models and cubes programmatically?
>
>
>
> This e-mail (including any attachments) is private and confidential, may
> contain proprietary or privileged information and is intended for the named
> recipient(s) only. Unintended recipients are strictly prohibited from
> taking action on the basis of information in this e-mail and must contact
> the sender immediately, delete this e-mail (and all attachments) and
> destroy any hard copies. Nomura will not accept responsibility or liability
> for the accuracy or completeness of, or the presence of any virus or
> disabling code in, this e-mail. If verification is sought please request a
> hard copy. Any reference to the terms of executed transactions should be
> treated as preliminary only and subject to formal written confirmation by
> Nomura. Nomura reserves the right to retain, monitor and intercept e-mail
> communications through its networks (subject to and in accordance with
> applicable laws). No confidentiality or privilege is waived or lost by
> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is
> a reference to any entity in the Nomura Holdings, Inc. group. Please read
> our Electronic Communications Legal Notice which forms part of this e-mail:
> http://www.Nomura.com/email_disclaimer.htm
>