Re:Re: use timestamp for cube partition

[崔苗]

well,the timestamp column was bigint such as 1507547479434 in hive table, when 
I define the endtime to build the cube ,I found the timestamp 1507547479434 was 
converted to '20171009' and the log show that kylin loaded data from hive with 
condition "WHERE (USER_REG.REG_TIME  20171009)",so the Intermediate Flat 
Hive Table was null. I want to know could kylin derive other time values like 
“year_start”, “day_start” from the bigint timestamp in hive as it does in kafka 
table? or we must change the bigint timestamp into data format such as 
"2017-10-09" in hive?
At 2017-10-09 22:04:56, ShaoFeng Shi <shaofeng...@apache.org> wrote:
>Hi Miao,
>
>What's the error as you said: "kylin failed to load data from hive tables"?
>
>In my opinion, it is not recommended to use timestamp as the partition
>column, since it is too fine granularity. Usually, the cube is partitioned
>by day/week/month; in some cases, it is by the hour; In streaming case, it
>might partition by the minute; But no case by timestamp. I put some
>comments about this in this document:
>https://kylin.apache.org/docs21/tutorial/cube_streaming.html
>
>2017-10-09 14:27 GMT+08:00 崔苗 <cuim...@danale.com>:
>
>> Hi,
>> we want to use tables in kafka as fact tables and tables in MySql as
>> lookup tables,so we put all the tables into hive and want to join them as
>> cubes.
>>
>> the time column in fact table was timestamp, so does kylin2.1 support
>> timestamp for cube partition?
>> I find this :https://issues.apache.org/jira/browse/KYLIN-633 ,
>>
>> it seems kylin already supprt Timestamp for cube partition,but when we
>> define timestamp as partition , kylin failed to load data from hive tables.
>>
>>
>> thanks in advanced for your reply.
>>
>>
>>
>>
>>
>
>
>-- 
>Best regards,
>
>Shaofeng Shi 史少锋

Re: Re: kylin configuration

[Billy Liu]

Yes. Kylin should be deployed at the edge node, not the server node.

2017-10-09 18:35 GMT+08:00 崔苗 :

> so what about hive,hbase and spark ? just looks for library too ?
> 在 2017-10-09 18:28:08，Billy Liu  写道：
> >Kylin looks for Kafka library from the classpath, not the Kafka broker
> >servers. You could deploy kafka into any other servers.
> >
> >2017-10-09 16:30 GMT+08:00 崔苗 :
> >
> >> Hi,
> >>
> >>
> >> we have some problems in kylin installation , we want to know how does
> >> kylin find the hbase, hive or kafka environment , how to configure
> kylin if
> >> these servers such as kafka are installed in other cluster, not the same
> >> machine with kylin.
> >>
> >>
> >> thanks in advanced for your reply.
> >>
> >>
> >>
> >>
> >>
>
>
>

Re: use timestamp for cube partition

[ShaoFeng Shi]

Hi Miao,

What's the error as you said: "kylin failed to load data from hive tables"?

In my opinion, it is not recommended to use timestamp as the partition
column, since it is too fine granularity. Usually, the cube is partitioned
by day/week/month; in some cases, it is by the hour; In streaming case, it
might partition by the minute; But no case by timestamp. I put some
comments about this in this document:
https://kylin.apache.org/docs21/tutorial/cube_streaming.html

2017-10-09 14:27 GMT+08:00 崔苗 :

> Hi,
> we want to use tables in kafka as fact tables and tables in MySql as
> lookup tables,so we put all the tables into hive and want to join them as
> cubes.
>
> the time column in fact table was timestamp, so does kylin2.1 support
> timestamp for cube partition?
> I find this :https://issues.apache.org/jira/browse/KYLIN-633 ,
>
> it seems kylin already supprt Timestamp for cube partition,but when we
> define timestamp as partition , kylin failed to load data from hive tables.
>
>
> thanks in advanced for your reply.
>
>
>
>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Re: Kylin 2.2.0 release soon

[ShaoFeng Shi]

Dong, the link doesn't work at my side; it always shows the KYLIN-2915.

I created a filter to list the unresolved issues which set v2.2.0 as fixed
version, most of them are reported by Yanghong and in progress:

https://issues.apache.org/jira/issues/?filter=-1=fixVersion%20%3D%20%27v2.2.0%27%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20key%20DESC%2C%20updatedDate%20DESC

2017-10-09 20:41 GMT+08:00 Dong Li :

> Hello community,
>
> The release branch "2.2.x" is out. You can build a package off it and test.
> http://kylin.apache.org/development/howto_package.html
>
> @committers and contributors, please help to check and update your JIRA
> status.
> For completed issues, please ensure the status to be Resolved.
> For unplanned issues, please reset the "fix version" to the real versions.
> https://issues.apache.org/jira/browse/KYLIN-2915?filter=
> 12333925=project%20%3D%20KYLIN%20AND%20status%20in%
> 20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%
> 20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%20v2.2.0%
> 20ORDER%20BY%20createdDate%20DESC
>
>
> I'll be the Release Manager for this release. If no show stopper is
> reported, a formal release will be in 2 weeks.
>
> Thanks,
> Dong Li
>
> 2017-10-08 14:58 GMT+08:00 Li Yang :
>
> > My major concern is https://issues.apache.org/jira/browse/KYLIN-2918
> >
> > Table ACL is half done at the moment. So either we release the 2.2.0
> with a
> > clear warning of the half done feature. Or we postpone the release till
> > Table ACL is fully done.
> >
> > Any suggestion?
> >
> > Yang
> >
> > On Wed, Sep 27, 2017 at 5:33 PM, Billy Liu  wrote:
> >
> > > Expecting to see the new release.
> > >
> > > 2017-09-25 22:20 GMT+08:00 ShaoFeng Shi :
> > >
> > > > Cool, when do you plan to kick off the release?
> > > >
> > > > 2017-09-25 21:49 GMT+08:00 Li Yang :
> > > >
> > > > > Anything you want to squeeze into the release, it's time.
> > > > >
> > > > >
> > > > > Cheers
> > > > > Yang
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > Shaofeng Shi 史少锋
> > > >
> > >
> >
>



-- 
Best regards,

Shaofeng Shi 史少锋

Re: Kylin 2.2.0 release soon

[Dong Li]

Hello community,

The release branch "2.2.x" is out. You can build a package off it and test.
http://kylin.apache.org/development/howto_package.html

@committers and contributors, please help to check and update your JIRA
status.
For completed issues, please ensure the status to be Resolved.
For unplanned issues, please reset the "fix version" to the real versions.
https://issues.apache.org/jira/browse/KYLIN-2915?filter=12333925=project%20%3D%20KYLIN%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%20v2.2.0%20ORDER%20BY%20createdDate%20DESC


I'll be the Release Manager for this release. If no show stopper is
reported, a formal release will be in 2 weeks.

Thanks,
Dong Li

2017-10-08 14:58 GMT+08:00 Li Yang :

> My major concern is https://issues.apache.org/jira/browse/KYLIN-2918
>
> Table ACL is half done at the moment. So either we release the 2.2.0 with a
> clear warning of the half done feature. Or we postpone the release till
> Table ACL is fully done.
>
> Any suggestion?
>
> Yang
>
> On Wed, Sep 27, 2017 at 5:33 PM, Billy Liu  wrote:
>
> > Expecting to see the new release.
> >
> > 2017-09-25 22:20 GMT+08:00 ShaoFeng Shi :
> >
> > > Cool, when do you plan to kick off the release?
> > >
> > > 2017-09-25 21:49 GMT+08:00 Li Yang :
> > >
> > > > Anything you want to squeeze into the release, it's time.
> > > >
> > > >
> > > > Cheers
> > > > Yang
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > >
> >
>

Re:Re: kylin configuration

[崔苗]

so what about hive,hbase and spark ? just looks for library too ?
在 2017-10-09 18:28:08，Billy Liu  写道：
>Kylin looks for Kafka library from the classpath, not the Kafka broker
>servers. You could deploy kafka into any other servers.
>
>2017-10-09 16:30 GMT+08:00 崔苗 :
>
>> Hi,
>>
>>
>> we have some problems in kylin installation , we want to know how does
>> kylin find the hbase, hive or kafka environment , how to configure kylin if
>> these servers such as kafka are installed in other cluster, not the same
>> machine with kylin.
>>
>>
>> thanks in advanced for your reply.
>>
>>
>>
>>
>>

Re: kylin configuration

[Billy Liu]

Kylin looks for Kafka library from the classpath, not the Kafka broker
servers. You could deploy kafka into any other servers.

2017-10-09 16:30 GMT+08:00 崔苗 :

> Hi,
>
>
> we have some problems in kylin installation , we want to know how does
> kylin find the hbase, hive or kafka environment , how to configure kylin if
> these servers such as kafka are installed in other cluster, not the same
> machine with kylin.
>
>
> thanks in advanced for your reply.
>
>
>
>
>

kylin configuration

[崔苗]

Hi,


we have some problems in kylin installation , we want to know how does kylin 
find the hbase, hive or kafka environment , how to configure kylin if these 
servers such as kafka are installed in other cluster, not the same machine with 
kylin.


thanks in advanced for your reply.

use timestamp for cube partition

[崔苗]

Hi,
we want to use tables in kafka as fact tables and tables in MySql as lookup 
tables,so we put all the tables into hive and want to join them as cubes.

the time column in fact table was timestamp, so does kylin2.1 support timestamp 
for cube partition?
I find this :https://issues.apache.org/jira/browse/KYLIN-633 , 

it seems kylin already supprt Timestamp for cube partition,but when we define 
timestamp as partition , kylin failed to load data from hive tables.


thanks in advanced for your reply.