Re: Can kylin intermediate tables in hive be deleted ?

2016-12-13 Thread Alberto Ramón 
You will need execute cleanUp Storage
<http://kylin.apache.org/docs/howto/howto_cleanup_storage.html>

Can be made with Kylin Online? Yes

In this mailList
<http://apache-kylin.74782.x6.nabble.com/Can-not-Clean-Up-td6554.html#a6556>you
will find extra info


2016-12-13 10:14 GMT+01:00 Luke_Selina <huangzhendon...@gmail.com>:

> <http://apache-kylin.74782.x6.nabble.com/file/n6617/intermediate.png>
>
> Hi all, just as the pic shows, can these intermediate hive tables can be
> deleted manually?
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Can-kylin-intermediate-tables-in-hive-be-deleted-tp6617.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>

Can kylin intermediate tables in hive be deleted ?

2016-12-13 Thread Luke_Selina 
<http://apache-kylin.74782.x6.nabble.com/file/n6617/intermediate.png> 

Hi all, just as the pic shows, can these intermediate hive tables can be
deleted manually?

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Can-kylin-intermediate-tables-in-hive-be-deleted-tp6617.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: kylin intermediate tables in Hive

2016-07-24 Thread hongbin ma 
it might have something to do with the bug described in
https://issues.apache.org/jira/browse/KYLIN-1828, I'll cross check

On Tue, Jun 21, 2016 at 10:36 PM, ShaoFeng Shi 
wrote:

> Hi Jie, would you mind to report a JIRA with this problem you found? If you
> can attache the json of this job and a couple of screen shot, that would be
> great for analysis. Thank you!
>
> 2016-06-21 16:49 GMT+08:00 Jie Tao :
>
> > actually the jobs have been disgarded. Maybe it is a bug that the status
> > of the job is still "ERROR". I have a look of the jobs  at the Kylin Web
> UI
> > and found that the job was not marked with black colors like other
> > discarded jobs althouth the "action" button only has one choice
> > "diagonose". My Kylin is 1.5.2.1.
> >
> > Cheers,
> >
> > Jie
> >
> >
> > Am 17.06.2016 um 11:05 schrieb ShaoFeng Shi:
> >
> >> by default the web UI only shows the jobs in LAST ONE WEEK, pls have a
> >> check.
> >>
> >> 2016-06-17 16:58 GMT+08:00 Jie Tao :
> >>
> >> actually I discarded all jobs and I do not see any ERROR job in the
> >>> Monitor view of Kylin UI.
> >>>
> >>> Where can I see these error jobs?
> >>>
> >>> Jie
> >>>
> >>>
> >>> Am 17.06.2016 um 10:31 schrieb ShaoFeng Shi:
> >>>
> >>> Hi Jie,
> 
>  If a job is "ERROR", the intermediate hive table of it will not be
>  dropped,
>  as "ERROR" is not a final state; User can resume an "Error" job at any
>  time, so Kylin skipped to cleanup for that.
> 
>  If you discard these error jobs, and re-run the cleanup, the
>  intermediate
>  hive table will be dropped.
> 
>  The message here is not clear, will change the wording...
> 
>  2016-06-17 15:48 GMT+08:00 Jie Tao :
> 
>  You are correct, the intermediate tables are left by fail-building. I
> do
> 
> > clean up storage based on the linked guide. Intermediate data in HDFS
> > and
> > Hbase are deleted, but the intermediate tables in Hive not. The
> command
> > shows the tables but do not drop them. I donot have a lookup table
> but
> > my
> > fact table is a view.
> >
> > As I run the cleanup command,
> > kylin_intermediate_logout_full_cube_1970010100_2015100100
> > kylin_intermediate_logout_full_cube_1970010100_20160529010500
> > kylin_intermediate_logout_full_cube_1970010100_2016060800
> > kylin_intermediate_logout_full_cube_1970010100_20160608010500
> > kylin_intermediate_logout_full_cube_1970010100_20160609010500
> > kylin_intermediate_logout_full_cube_1970010100_2016061500
> > kylin_intermediate_logout_full_cube_1970010100_2016062600
> > kylin_intermediate_logout_full_cube_1970010100_20160626042000
> > kylin_intermediate_test_cube_1970010100_20151201010500
> > kylin_intermediate_test_cube_1970010100_20151231234000
> > kylin_intermediate_test_cube_1970010100_20160302063000
> > kylin_intermediate_test_cube_1970010100_2016062600
> > kylin_intermediate_test_cube_1970010100_20160626042000
> > kylin_intermediate_test_cube_1970010100_20160704082000
> > Time taken: 0.189 seconds, Fetched: 14 row(s)
> > 2016-06-17 09:37:12,645 INFO  [main StorageCleanupJob:262]: Remove
> > intermediate hive table with job id
> > 493fd20b-3074-403e-9963-fe4fb7ff7c65
> > with job status ERROR
> > 2016-06-17 09:37:12,648 INFO  [main StorageCleanupJob:262]: Remove
> > intermediate hive table with job id
> > 8a377e30-e3ba-4fe2-be12-e7d412afec5e
> > with job status ERROR
> >
> > Best regards,
> >
> > Jie
> >
> >
> > Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi:
> >
> > BTW, are you using a view as lookup table?
> >
> >> 2016-06-17 15:15 GMT+08:00 ShaoFeng Shi :
> >>
> >> This is common; If you have a job failed in between, and you discard
> >> that
> >>
> >> job, the "Garbage collection" step will not be executed, so the
> >>> garbages
> >>> will be left there.
> >>>
> >>> This is why we still recommend user to run offline cleanup every
> some
> >>> period; It is not perfert, but be good for most scenarios:
> >>> https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
> >>>
> >>> 2016-06-17 15:00 GMT+08:00 Li Yang :
> >>>
> >>> Woo... something new to me. Anybody knows?
> >>>
> >>> On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao 
>  wrote:
> 
>  Kylin actually drops useless intermediate tables after cube
>  building,
>  but
> 
>  I still see one "kylin_intermediate_cubename_searchdata" table for
> 
> > each
> > cube building in Hive. Are these tables still usefull for Kylin?
> I
> > use
> > Kylin

Re: kylin intermediate tables in Hive

2016-06-21 Thread ShaoFeng Shi 
Hi Jie, would you mind to report a JIRA with this problem you found? If you
can attache the json of this job and a couple of screen shot, that would be
great for analysis. Thank you!

2016-06-21 16:49 GMT+08:00 Jie Tao :

> actually the jobs have been disgarded. Maybe it is a bug that the status
> of the job is still "ERROR". I have a look of the jobs  at the Kylin Web UI
> and found that the job was not marked with black colors like other
> discarded jobs althouth the "action" button only has one choice
> "diagonose". My Kylin is 1.5.2.1.
>
> Cheers,
>
> Jie
>
>
> Am 17.06.2016 um 11:05 schrieb ShaoFeng Shi:
>
>> by default the web UI only shows the jobs in LAST ONE WEEK, pls have a
>> check.
>>
>> 2016-06-17 16:58 GMT+08:00 Jie Tao :
>>
>> actually I discarded all jobs and I do not see any ERROR job in the
>>> Monitor view of Kylin UI.
>>>
>>> Where can I see these error jobs?
>>>
>>> Jie
>>>
>>>
>>> Am 17.06.2016 um 10:31 schrieb ShaoFeng Shi:
>>>
>>> Hi Jie,

 If a job is "ERROR", the intermediate hive table of it will not be
 dropped,
 as "ERROR" is not a final state; User can resume an "Error" job at any
 time, so Kylin skipped to cleanup for that.

 If you discard these error jobs, and re-run the cleanup, the
 intermediate
 hive table will be dropped.

 The message here is not clear, will change the wording...

 2016-06-17 15:48 GMT+08:00 Jie Tao :

 You are correct, the intermediate tables are left by fail-building. I do

> clean up storage based on the linked guide. Intermediate data in HDFS
> and
> Hbase are deleted, but the intermediate tables in Hive not. The command
> shows the tables but do not drop them. I donot have a lookup table but
> my
> fact table is a view.
>
> As I run the cleanup command,
> kylin_intermediate_logout_full_cube_1970010100_2015100100
> kylin_intermediate_logout_full_cube_1970010100_20160529010500
> kylin_intermediate_logout_full_cube_1970010100_2016060800
> kylin_intermediate_logout_full_cube_1970010100_20160608010500
> kylin_intermediate_logout_full_cube_1970010100_20160609010500
> kylin_intermediate_logout_full_cube_1970010100_2016061500
> kylin_intermediate_logout_full_cube_1970010100_2016062600
> kylin_intermediate_logout_full_cube_1970010100_20160626042000
> kylin_intermediate_test_cube_1970010100_20151201010500
> kylin_intermediate_test_cube_1970010100_20151231234000
> kylin_intermediate_test_cube_1970010100_20160302063000
> kylin_intermediate_test_cube_1970010100_2016062600
> kylin_intermediate_test_cube_1970010100_20160626042000
> kylin_intermediate_test_cube_1970010100_20160704082000
> Time taken: 0.189 seconds, Fetched: 14 row(s)
> 2016-06-17 09:37:12,645 INFO  [main StorageCleanupJob:262]: Remove
> intermediate hive table with job id
> 493fd20b-3074-403e-9963-fe4fb7ff7c65
> with job status ERROR
> 2016-06-17 09:37:12,648 INFO  [main StorageCleanupJob:262]: Remove
> intermediate hive table with job id
> 8a377e30-e3ba-4fe2-be12-e7d412afec5e
> with job status ERROR
>
> Best regards,
>
> Jie
>
>
> Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi:
>
> BTW, are you using a view as lookup table?
>
>> 2016-06-17 15:15 GMT+08:00 ShaoFeng Shi :
>>
>> This is common; If you have a job failed in between, and you discard
>> that
>>
>> job, the "Garbage collection" step will not be executed, so the
>>> garbages
>>> will be left there.
>>>
>>> This is why we still recommend user to run offline cleanup every some
>>> period; It is not perfert, but be good for most scenarios:
>>> https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
>>>
>>> 2016-06-17 15:00 GMT+08:00 Li Yang :
>>>
>>> Woo... something new to me. Anybody knows?
>>>
>>> On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao 
 wrote:

 Kylin actually drops useless intermediate tables after cube
 building,
 but

 I still see one "kylin_intermediate_cubename_searchdata" table for

> each
> cube building in Hive. Are these tables still usefull for Kylin? I
> use
> Kylin 1.5.2.1.
>
> Cheers,
>
> Jie
>
>
> --
>
 Best regards,
>>>
>>> Shaofeng Shi
>>>
>>>
>>>
>>>
>>>
>>
>


-- 
Best regards,

Shaofeng Shi

Re: kylin intermediate tables in Hive

2016-06-21 Thread Jie Tao 
actually the jobs have been disgarded. Maybe it is a bug that the status 
of the job is still "ERROR". I have a look of the jobs  at the Kylin Web 
UI and found that the job was not marked with black colors like other 
discarded jobs althouth the "action" button only has one choice 
"diagonose". My Kylin is 1.5.2.1.


Cheers,

Jie

Am 17.06.2016 um 11:05 schrieb ShaoFeng Shi:

by default the web UI only shows the jobs in LAST ONE WEEK, pls have a
check.

2016-06-17 16:58 GMT+08:00 Jie Tao :


actually I discarded all jobs and I do not see any ERROR job in the
Monitor view of Kylin UI.

Where can I see these error jobs?

Jie


Am 17.06.2016 um 10:31 schrieb ShaoFeng Shi:


Hi Jie,

If a job is "ERROR", the intermediate hive table of it will not be
dropped,
as "ERROR" is not a final state; User can resume an "Error" job at any
time, so Kylin skipped to cleanup for that.

If you discard these error jobs, and re-run the cleanup, the intermediate
hive table will be dropped.

The message here is not clear, will change the wording...

2016-06-17 15:48 GMT+08:00 Jie Tao :

You are correct, the intermediate tables are left by fail-building. I do

clean up storage based on the linked guide. Intermediate data in HDFS and
Hbase are deleted, but the intermediate tables in Hive not. The command
shows the tables but do not drop them. I donot have a lookup table but my
fact table is a view.

As I run the cleanup command,
kylin_intermediate_logout_full_cube_1970010100_2015100100
kylin_intermediate_logout_full_cube_1970010100_20160529010500
kylin_intermediate_logout_full_cube_1970010100_2016060800
kylin_intermediate_logout_full_cube_1970010100_20160608010500
kylin_intermediate_logout_full_cube_1970010100_20160609010500
kylin_intermediate_logout_full_cube_1970010100_2016061500
kylin_intermediate_logout_full_cube_1970010100_2016062600
kylin_intermediate_logout_full_cube_1970010100_20160626042000
kylin_intermediate_test_cube_1970010100_20151201010500
kylin_intermediate_test_cube_1970010100_20151231234000
kylin_intermediate_test_cube_1970010100_20160302063000
kylin_intermediate_test_cube_1970010100_2016062600
kylin_intermediate_test_cube_1970010100_20160626042000
kylin_intermediate_test_cube_1970010100_20160704082000
Time taken: 0.189 seconds, Fetched: 14 row(s)
2016-06-17 09:37:12,645 INFO  [main StorageCleanupJob:262]: Remove
intermediate hive table with job id 493fd20b-3074-403e-9963-fe4fb7ff7c65
with job status ERROR
2016-06-17 09:37:12,648 INFO  [main StorageCleanupJob:262]: Remove
intermediate hive table with job id 8a377e30-e3ba-4fe2-be12-e7d412afec5e
with job status ERROR

Best regards,

Jie


Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi:

BTW, are you using a view as lookup table?

2016-06-17 15:15 GMT+08:00 ShaoFeng Shi :

This is common; If you have a job failed in between, and you discard
that


job, the "Garbage collection" step will not be executed, so the
garbages
will be left there.

This is why we still recommend user to run offline cleanup every some
period; It is not perfert, but be good for most scenarios:
https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html

2016-06-17 15:00 GMT+08:00 Li Yang :

Woo... something new to me. Anybody knows?


On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao 
wrote:

Kylin actually drops useless intermediate tables after cube building,
but

I still see one "kylin_intermediate_cubename_searchdata" table for

each
cube building in Hive. Are these tables still usefull for Kylin? I
use
Kylin 1.5.2.1.

Cheers,

Jie


--

Best regards,

Shaofeng Shi

Re: kylin intermediate tables in Hive

2016-06-17 Thread ShaoFeng Shi 
by default the web UI only shows the jobs in LAST ONE WEEK, pls have a
check.

2016-06-17 16:58 GMT+08:00 Jie Tao :

> actually I discarded all jobs and I do not see any ERROR job in the
> Monitor view of Kylin UI.
>
> Where can I see these error jobs?
>
> Jie
>
>
> Am 17.06.2016 um 10:31 schrieb ShaoFeng Shi:
>
>> Hi Jie,
>>
>> If a job is "ERROR", the intermediate hive table of it will not be
>> dropped,
>> as "ERROR" is not a final state; User can resume an "Error" job at any
>> time, so Kylin skipped to cleanup for that.
>>
>> If you discard these error jobs, and re-run the cleanup, the intermediate
>> hive table will be dropped.
>>
>> The message here is not clear, will change the wording...
>>
>> 2016-06-17 15:48 GMT+08:00 Jie Tao :
>>
>> You are correct, the intermediate tables are left by fail-building. I do
>>> clean up storage based on the linked guide. Intermediate data in HDFS and
>>> Hbase are deleted, but the intermediate tables in Hive not. The command
>>> shows the tables but do not drop them. I donot have a lookup table but my
>>> fact table is a view.
>>>
>>> As I run the cleanup command,
>>> kylin_intermediate_logout_full_cube_1970010100_2015100100
>>> kylin_intermediate_logout_full_cube_1970010100_20160529010500
>>> kylin_intermediate_logout_full_cube_1970010100_2016060800
>>> kylin_intermediate_logout_full_cube_1970010100_20160608010500
>>> kylin_intermediate_logout_full_cube_1970010100_20160609010500
>>> kylin_intermediate_logout_full_cube_1970010100_2016061500
>>> kylin_intermediate_logout_full_cube_1970010100_2016062600
>>> kylin_intermediate_logout_full_cube_1970010100_20160626042000
>>> kylin_intermediate_test_cube_1970010100_20151201010500
>>> kylin_intermediate_test_cube_1970010100_20151231234000
>>> kylin_intermediate_test_cube_1970010100_20160302063000
>>> kylin_intermediate_test_cube_1970010100_2016062600
>>> kylin_intermediate_test_cube_1970010100_20160626042000
>>> kylin_intermediate_test_cube_1970010100_20160704082000
>>> Time taken: 0.189 seconds, Fetched: 14 row(s)
>>> 2016-06-17 09:37:12,645 INFO  [main StorageCleanupJob:262]: Remove
>>> intermediate hive table with job id 493fd20b-3074-403e-9963-fe4fb7ff7c65
>>> with job status ERROR
>>> 2016-06-17 09:37:12,648 INFO  [main StorageCleanupJob:262]: Remove
>>> intermediate hive table with job id 8a377e30-e3ba-4fe2-be12-e7d412afec5e
>>> with job status ERROR
>>>
>>> Best regards,
>>>
>>> Jie
>>>
>>>
>>> Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi:
>>>
>>> BTW, are you using a view as lookup table?

 2016-06-17 15:15 GMT+08:00 ShaoFeng Shi :

 This is common; If you have a job failed in between, and you discard
 that

> job, the "Garbage collection" step will not be executed, so the
> garbages
> will be left there.
>
> This is why we still recommend user to run offline cleanup every some
> period; It is not perfert, but be good for most scenarios:
> https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
>
> 2016-06-17 15:00 GMT+08:00 Li Yang :
>
> Woo... something new to me. Anybody knows?
>
>> On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao 
>> wrote:
>>
>> Kylin actually drops useless intermediate tables after cube building,
>> but
>>
>> I still see one "kylin_intermediate_cubename_searchdata" table for
>>> each
>>> cube building in Hive. Are these tables still usefull for Kylin? I
>>> use
>>> Kylin 1.5.2.1.
>>>
>>> Cheers,
>>>
>>> Jie
>>>
>>>
>>> --
> Best regards,
>
> Shaofeng Shi
>
>
>
>
>>
>


-- 
Best regards,

Shaofeng Shi

Re: kylin intermediate tables in Hive

2016-06-17 Thread Jie Tao 
actually I discarded all jobs and I do not see any ERROR job in the 
Monitor view of Kylin UI.


Where can I see these error jobs?

Jie

Am 17.06.2016 um 10:31 schrieb ShaoFeng Shi:

Hi Jie,

If a job is "ERROR", the intermediate hive table of it will not be dropped,
as "ERROR" is not a final state; User can resume an "Error" job at any
time, so Kylin skipped to cleanup for that.

If you discard these error jobs, and re-run the cleanup, the intermediate
hive table will be dropped.

The message here is not clear, will change the wording...

2016-06-17 15:48 GMT+08:00 Jie Tao :


You are correct, the intermediate tables are left by fail-building. I do
clean up storage based on the linked guide. Intermediate data in HDFS and
Hbase are deleted, but the intermediate tables in Hive not. The command
shows the tables but do not drop them. I donot have a lookup table but my
fact table is a view.

As I run the cleanup command,
kylin_intermediate_logout_full_cube_1970010100_2015100100
kylin_intermediate_logout_full_cube_1970010100_20160529010500
kylin_intermediate_logout_full_cube_1970010100_2016060800
kylin_intermediate_logout_full_cube_1970010100_20160608010500
kylin_intermediate_logout_full_cube_1970010100_20160609010500
kylin_intermediate_logout_full_cube_1970010100_2016061500
kylin_intermediate_logout_full_cube_1970010100_2016062600
kylin_intermediate_logout_full_cube_1970010100_20160626042000
kylin_intermediate_test_cube_1970010100_20151201010500
kylin_intermediate_test_cube_1970010100_20151231234000
kylin_intermediate_test_cube_1970010100_20160302063000
kylin_intermediate_test_cube_1970010100_2016062600
kylin_intermediate_test_cube_1970010100_20160626042000
kylin_intermediate_test_cube_1970010100_20160704082000
Time taken: 0.189 seconds, Fetched: 14 row(s)
2016-06-17 09:37:12,645 INFO  [main StorageCleanupJob:262]: Remove
intermediate hive table with job id 493fd20b-3074-403e-9963-fe4fb7ff7c65
with job status ERROR
2016-06-17 09:37:12,648 INFO  [main StorageCleanupJob:262]: Remove
intermediate hive table with job id 8a377e30-e3ba-4fe2-be12-e7d412afec5e
with job status ERROR

Best regards,

Jie


Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi:


BTW, are you using a view as lookup table?

2016-06-17 15:15 GMT+08:00 ShaoFeng Shi :

This is common; If you have a job failed in between, and you discard that

job, the "Garbage collection" step will not be executed, so the garbages
will be left there.

This is why we still recommend user to run offline cleanup every some
period; It is not perfert, but be good for most scenarios:
https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html

2016-06-17 15:00 GMT+08:00 Li Yang :

Woo... something new to me. Anybody knows?

On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao  wrote:

Kylin actually drops useless intermediate tables after cube building,
but


I still see one "kylin_intermediate_cubename_searchdata" table for each
cube building in Hive. Are these tables still usefull for Kylin? I use
Kylin 1.5.2.1.

Cheers,

Jie



--
Best regards,

Shaofeng Shi

Re: kylin intermediate tables in Hive

2016-06-17 Thread ShaoFeng Shi 
Hi Jie,

If a job is "ERROR", the intermediate hive table of it will not be dropped,
as "ERROR" is not a final state; User can resume an "Error" job at any
time, so Kylin skipped to cleanup for that.

If you discard these error jobs, and re-run the cleanup, the intermediate
hive table will be dropped.

The message here is not clear, will change the wording...

2016-06-17 15:48 GMT+08:00 Jie Tao :

> You are correct, the intermediate tables are left by fail-building. I do
> clean up storage based on the linked guide. Intermediate data in HDFS and
> Hbase are deleted, but the intermediate tables in Hive not. The command
> shows the tables but do not drop them. I donot have a lookup table but my
> fact table is a view.
>
> As I run the cleanup command,
> kylin_intermediate_logout_full_cube_1970010100_2015100100
> kylin_intermediate_logout_full_cube_1970010100_20160529010500
> kylin_intermediate_logout_full_cube_1970010100_2016060800
> kylin_intermediate_logout_full_cube_1970010100_20160608010500
> kylin_intermediate_logout_full_cube_1970010100_20160609010500
> kylin_intermediate_logout_full_cube_1970010100_2016061500
> kylin_intermediate_logout_full_cube_1970010100_2016062600
> kylin_intermediate_logout_full_cube_1970010100_20160626042000
> kylin_intermediate_test_cube_1970010100_20151201010500
> kylin_intermediate_test_cube_1970010100_20151231234000
> kylin_intermediate_test_cube_1970010100_20160302063000
> kylin_intermediate_test_cube_1970010100_2016062600
> kylin_intermediate_test_cube_1970010100_20160626042000
> kylin_intermediate_test_cube_1970010100_20160704082000
> Time taken: 0.189 seconds, Fetched: 14 row(s)
> 2016-06-17 09:37:12,645 INFO  [main StorageCleanupJob:262]: Remove
> intermediate hive table with job id 493fd20b-3074-403e-9963-fe4fb7ff7c65
> with job status ERROR
> 2016-06-17 09:37:12,648 INFO  [main StorageCleanupJob:262]: Remove
> intermediate hive table with job id 8a377e30-e3ba-4fe2-be12-e7d412afec5e
> with job status ERROR
>
> Best regards,
>
> Jie
>
>
> Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi:
>
>> BTW, are you using a view as lookup table?
>>
>> 2016-06-17 15:15 GMT+08:00 ShaoFeng Shi :
>>
>> This is common; If you have a job failed in between, and you discard that
>>> job, the "Garbage collection" step will not be executed, so the garbages
>>> will be left there.
>>>
>>> This is why we still recommend user to run offline cleanup every some
>>> period; It is not perfert, but be good for most scenarios:
>>> https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
>>>
>>> 2016-06-17 15:00 GMT+08:00 Li Yang :
>>>
>>> Woo... something new to me. Anybody knows?

 On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao  wrote:

 Kylin actually drops useless intermediate tables after cube building,
>
 but

> I still see one "kylin_intermediate_cubename_searchdata" table for each
> cube building in Hive. Are these tables still usefull for Kylin? I use
> Kylin 1.5.2.1.
>
> Cheers,
>
> Jie
>
>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi
>>>
>>>
>>>
>>
>


-- 
Best regards,

Shaofeng Shi

Re: kylin intermediate tables in Hive

2016-06-17 Thread Jie Tao 
You are correct, the intermediate tables are left by fail-building. I do 
clean up storage based on the linked guide. Intermediate data in HDFS 
and Hbase are deleted, but the intermediate tables in Hive not. The 
command shows the tables but do not drop them. I donot have a lookup 
table but my fact table is a view.


As I run the cleanup command,
kylin_intermediate_logout_full_cube_1970010100_2015100100
kylin_intermediate_logout_full_cube_1970010100_20160529010500
kylin_intermediate_logout_full_cube_1970010100_2016060800
kylin_intermediate_logout_full_cube_1970010100_20160608010500
kylin_intermediate_logout_full_cube_1970010100_20160609010500
kylin_intermediate_logout_full_cube_1970010100_2016061500
kylin_intermediate_logout_full_cube_1970010100_2016062600
kylin_intermediate_logout_full_cube_1970010100_20160626042000
kylin_intermediate_test_cube_1970010100_20151201010500
kylin_intermediate_test_cube_1970010100_20151231234000
kylin_intermediate_test_cube_1970010100_20160302063000
kylin_intermediate_test_cube_1970010100_2016062600
kylin_intermediate_test_cube_1970010100_20160626042000
kylin_intermediate_test_cube_1970010100_20160704082000
Time taken: 0.189 seconds, Fetched: 14 row(s)
2016-06-17 09:37:12,645 INFO  [main StorageCleanupJob:262]: Remove 
intermediate hive table with job id 493fd20b-3074-403e-9963-fe4fb7ff7c65 
with job status ERROR
2016-06-17 09:37:12,648 INFO  [main StorageCleanupJob:262]: Remove 
intermediate hive table with job id 8a377e30-e3ba-4fe2-be12-e7d412afec5e 
with job status ERROR


Best regards,

Jie

Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi:

BTW, are you using a view as lookup table?

2016-06-17 15:15 GMT+08:00 ShaoFeng Shi :


This is common; If you have a job failed in between, and you discard that
job, the "Garbage collection" step will not be executed, so the garbages
will be left there.

This is why we still recommend user to run offline cleanup every some
period; It is not perfert, but be good for most scenarios:
https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html

2016-06-17 15:00 GMT+08:00 Li Yang :


Woo... something new to me. Anybody knows?

On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao  wrote:


Kylin actually drops useless intermediate tables after cube building,

but

I still see one "kylin_intermediate_cubename_searchdata" table for each
cube building in Hive. Are these tables still usefull for Kylin? I use
Kylin 1.5.2.1.

Cheers,

Jie




--
Best regards,

Shaofeng Shi

Re: kylin intermediate tables in Hive

2016-06-17 Thread ShaoFeng Shi 
BTW, are you using a view as lookup table?

2016-06-17 15:15 GMT+08:00 ShaoFeng Shi :

> This is common; If you have a job failed in between, and you discard that
> job, the "Garbage collection" step will not be executed, so the garbages
> will be left there.
>
> This is why we still recommend user to run offline cleanup every some
> period; It is not perfert, but be good for most scenarios:
> https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
>
> 2016-06-17 15:00 GMT+08:00 Li Yang :
>
>> Woo... something new to me. Anybody knows?
>>
>> On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao  wrote:
>>
>> > Kylin actually drops useless intermediate tables after cube building,
>> but
>> > I still see one "kylin_intermediate_cubename_searchdata" table for each
>> > cube building in Hive. Are these tables still usefull for Kylin? I use
>> > Kylin 1.5.2.1.
>> >
>> > Cheers,
>> >
>> > Jie
>> >
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
>


-- 
Best regards,

Shaofeng Shi

Re: kylin intermediate tables in Hive

2016-06-17 Thread ShaoFeng Shi 
This is common; If you have a job failed in between, and you discard that
job, the "Garbage collection" step will not be executed, so the garbages
will be left there.

This is why we still recommend user to run offline cleanup every some
period; It is not perfert, but be good for most scenarios:
https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html

2016-06-17 15:00 GMT+08:00 Li Yang :

> Woo... something new to me. Anybody knows?
>
> On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao  wrote:
>
> > Kylin actually drops useless intermediate tables after cube building, but
> > I still see one "kylin_intermediate_cubename_searchdata" table for each
> > cube building in Hive. Are these tables still usefull for Kylin? I use
> > Kylin 1.5.2.1.
> >
> > Cheers,
> >
> > Jie
> >
>



-- 
Best regards,

Shaofeng Shi

Re: kylin intermediate tables in Hive

2016-06-17 Thread Li Yang 
Woo... something new to me. Anybody knows?

On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao  wrote:

> Kylin actually drops useless intermediate tables after cube building, but
> I still see one "kylin_intermediate_cubename_searchdata" table for each
> cube building in Hive. Are these tables still usefull for Kylin? I use
> Kylin 1.5.2.1.
>
> Cheers,
>
> Jie
>