[
https://issues.apache.org/jira/browse/KYLIN-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533465#comment-16533465
]
Temple Zhou edited comment on KYLIN-3430 at 7/5/18 10:03 AM:
-------------------------------------------------------------
Hi Kaisen,
Emm... I check the versions dir in my HDFS, and I found that there were more
than 3 versions dir exist despite of the default value of
"kylin.dictionary.append-max-versions" is 3.
{noformat}
kylin@kylin5:~$ hdfs dfs -ls
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx
Found 9 items
drwxr-xr-x - kylin supergroup 0 2017-11-03 17:26
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx/version_1509701213681
drwxr-xr-x - kylin supergroup 0 2017-11-03 17:30
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx/version_1509701420683
drwxr-xr-x - kylin supergroup 0 2017-11-03 17:44
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx/version_1509702256099
drwxr-xr-x - kylin supergroup 0 2017-11-04 08:16
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx/version_1509754562354
drwxr-xr-x - kylin supergroup 0 2017-11-05 08:16
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx/version_1509840996984
drwxr-xr-x - kylin supergroup 0 2017-11-06 08:16
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx/version_1509927413190
drwxr-xr-x - kylin supergroup 0 2017-11-06 10:34
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx/version_1509935653127
drwxr-xr-x - kylin supergroup 0 2017-11-06 14:07
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx/version_1509948477575
drwxr-xr-x - kylin supergroup 0 2017-11-06 15:26
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/xxx.xxx/xxx/version_1509953184643
{noformat}
In addition, even the cube is disabled or removed, the GD versions dir still
exist in my HDFS... :( :( :(
was (Author: temple.zhou):
Hi Kaisen,
Emm... I check the versions dir in my HDFS, and I found that there were more
than 3 versions dir exist despite of the default value of
"kylin.dictionary.append-max-versions" is 3.
{noformat}
kylin@kylin5:~$ hdfs dfs -du
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID
45634188 136902564
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID/version_1509701213681
53439651 160318953
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID/version_1509701420683
53439651 160318953
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID/version_1509702256099
53527132 160581396
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID/version_1509754562354
53588834 160766502
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID/version_1509840996984
53655766 160967298
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID/version_1509927413190
53655766 160967298
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID/version_1509935653127
53655766 160967298
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID/version_1509948477575
53655766 160967298
/tmp/kylin-data/kylin_metadata/resources/GlobalDict/dict/ADL.APP_USER_FRESHNESS_DI/DXYID/version_1509953184643
{noformat}
In addition, even the cube is disabled or removed, the GD versions dir still
exist in my HDFS... :( :( :(
> Global Dictionary Cleanup
> -------------------------
>
> Key: KYLIN-3430
> URL: https://issues.apache.org/jira/browse/KYLIN-3430
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
> Affects Versions: v2.1.0, v2.2.0, v2.3.0, v2.3.1, v2.4.0
> Reporter: Temple Zhou
> Assignee: Temple Zhou
> Priority: Major
> Attachments: KYLIN-3430.master.001.patch
>
>
> I had run "{{./bin/metastore.sh clean --delete true" to cleanup my Kylin
> metadata, but, after that, the Global Dictionary still exists in my HDFS and
> the size of directory "/kylin_metadata/resources/GlobalDict/dict" hasn't
> shrunk.}}
>
> {{BTW: I'm very sure that there are redundant Global Dictionaries.}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
