[
https://issues.apache.org/jira/browse/KYLIN-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3368:
--------------------------------
Description:
If using Spark as cube engine, Kylin will dump metadata to HDFS in
"/kylin/kylin_metadata/metadata/"; As time goes on, many files were left there.
They should be deleted when the job is finished.
hadoop fs -ls /kylin/kylin_metadata/metadata
Found 583 items
drwxr-xr-x - root hdfs 0 2018-02-11 16:20
/kylin/kylin_metadata/metadata/007d8b6c-5db1-478f-a373-7fd57231494b
drwxr-xr-x - root hdfs 0 2018-02-21 08:24
/kylin/kylin_metadata/metadata/00f93335-0d85-4c19-953c-5e4660f4d58e
drwxr-xr-x - root hdfs 0 2018-03-21 10:17
/kylin/kylin_metadata/metadata/011d3c7b-e707-42ca-b355-a0037820159b
drwxr-xr-x - root hdfs 0 2018-04-07 00:41
/kylin/kylin_metadata/metadata/0263789f-969a-485f-aa6c-df4394a45df8
drwxr-xr-x - root hdfs 0 2018-02-23 00:24
/kylin/kylin_metadata/metadata/02977825-92cd-475d-8a3f-0f75255a9ba9
drwxr-xr-x - root hdfs 0 2018-02-15 08:24
/kylin/kylin_metadata/metadata/02e058fd-ac6a-451f-99b4-f0b03ad6e3a5
drwxr-xr-x - root hdfs 0 2018-03-15 10:21
/kylin/kylin_metadata/metadata/0337c6c9-aebb-4ae0-8485-9905283fc818
drwxr-xr-x - root hdfs 0 2018-02-23 12:22
/kylin/kylin_metadata/metadata/03d7f34f-10dc-4329-9dde-f646cc040ba9
drwxr-xr-x - root hdfs 0 2018-04-10 12:20
/kylin/kylin_metadata/metadata/03fa7654-7b14-4bda-aed8-64ac9d08081a
drwxr-xr-x - root hdfs 0 2018-03-30 00:23
/kylin/kylin_metadata/metadata/0406e5bb-112f-4c54-bdf8-7b1e34652522
drwxr-xr-x - root hdfs 0 2018-03-24 00:33
/kylin/kylin_metadata/metadata/0426907d-541f-4f23-9db7-9a965f5b6739
drwxr-xr-x - root hdfs 0 2018-02-07 15:03
/kylin/kylin_metadata/metadata/04771302-2090-4793-956d-bf5ba9360d9b
drwxr-xr-x - root hdfs 0 2018-03-30 00:20
/kylin/kylin_metadata/metadata/0482b928-6b09-48d1-909f-68c66f71a827
drwxr-xr-x - root hdfs 0 2018-03-04 16:36
/kylin/kylin_metadata/metadata/04943bbd-8425-4304-b9c8-cc13a55e8ac1
drwxr-xr-x - root hdfs 0 2018-03-05 08:35
/kylin/kylin_metadata/metadata/04ee2c3d-bd07-4e7a-90db-3d87d088ff7d
drwxr-xr-x - root hdfs 0 2018-03-02 16:24
/kylin/kylin_metadata/metadata/05808a9d-2e55-4dc3-8c28-5b662068dda8
drwxr-xr-x - root hdfs 0 2018-03-04 08:35
/kylin/kylin_metadata/metadata/05c63d7c-4ea5-48fc-897d-6db9d78ffaf7
drwxr-xr-x - root hdfs 0 2018-03-01 01:02
/kylin/kylin_metadata/metadata/05ea3568-e85f-452a-b44d-4657145a724a
was:
If using Spark as cube engine, Kylin will dump metadata to HDFS in
"/kylin/kylin_metadata/metadata/"; As time goes on, many files were left there.
They should be deleted when the job is finished.
> "/kylin/kylin_metadata/metadata/" has many gargage for spark cubing
> -------------------------------------------------------------------
>
> Key: KYLIN-3368
> URL: https://issues.apache.org/jira/browse/KYLIN-3368
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
> Affects Versions: v2.2.0, v2.3.0
> Reporter: Shaofeng SHI
> Priority: Major
>
> If using Spark as cube engine, Kylin will dump metadata to HDFS in
> "/kylin/kylin_metadata/metadata/"; As time goes on, many files were left
> there.
>
> They should be deleted when the job is finished.
>
> hadoop fs -ls /kylin/kylin_metadata/metadata
> Found 583 items
> drwxr-xr-x - root hdfs 0 2018-02-11 16:20
> /kylin/kylin_metadata/metadata/007d8b6c-5db1-478f-a373-7fd57231494b
> drwxr-xr-x - root hdfs 0 2018-02-21 08:24
> /kylin/kylin_metadata/metadata/00f93335-0d85-4c19-953c-5e4660f4d58e
> drwxr-xr-x - root hdfs 0 2018-03-21 10:17
> /kylin/kylin_metadata/metadata/011d3c7b-e707-42ca-b355-a0037820159b
> drwxr-xr-x - root hdfs 0 2018-04-07 00:41
> /kylin/kylin_metadata/metadata/0263789f-969a-485f-aa6c-df4394a45df8
> drwxr-xr-x - root hdfs 0 2018-02-23 00:24
> /kylin/kylin_metadata/metadata/02977825-92cd-475d-8a3f-0f75255a9ba9
> drwxr-xr-x - root hdfs 0 2018-02-15 08:24
> /kylin/kylin_metadata/metadata/02e058fd-ac6a-451f-99b4-f0b03ad6e3a5
> drwxr-xr-x - root hdfs 0 2018-03-15 10:21
> /kylin/kylin_metadata/metadata/0337c6c9-aebb-4ae0-8485-9905283fc818
> drwxr-xr-x - root hdfs 0 2018-02-23 12:22
> /kylin/kylin_metadata/metadata/03d7f34f-10dc-4329-9dde-f646cc040ba9
> drwxr-xr-x - root hdfs 0 2018-04-10 12:20
> /kylin/kylin_metadata/metadata/03fa7654-7b14-4bda-aed8-64ac9d08081a
> drwxr-xr-x - root hdfs 0 2018-03-30 00:23
> /kylin/kylin_metadata/metadata/0406e5bb-112f-4c54-bdf8-7b1e34652522
> drwxr-xr-x - root hdfs 0 2018-03-24 00:33
> /kylin/kylin_metadata/metadata/0426907d-541f-4f23-9db7-9a965f5b6739
> drwxr-xr-x - root hdfs 0 2018-02-07 15:03
> /kylin/kylin_metadata/metadata/04771302-2090-4793-956d-bf5ba9360d9b
> drwxr-xr-x - root hdfs 0 2018-03-30 00:20
> /kylin/kylin_metadata/metadata/0482b928-6b09-48d1-909f-68c66f71a827
> drwxr-xr-x - root hdfs 0 2018-03-04 16:36
> /kylin/kylin_metadata/metadata/04943bbd-8425-4304-b9c8-cc13a55e8ac1
> drwxr-xr-x - root hdfs 0 2018-03-05 08:35
> /kylin/kylin_metadata/metadata/04ee2c3d-bd07-4e7a-90db-3d87d088ff7d
> drwxr-xr-x - root hdfs 0 2018-03-02 16:24
> /kylin/kylin_metadata/metadata/05808a9d-2e55-4dc3-8c28-5b662068dda8
> drwxr-xr-x - root hdfs 0 2018-03-04 08:35
> /kylin/kylin_metadata/metadata/05c63d7c-4ea5-48fc-897d-6db9d78ffaf7
> drwxr-xr-x - root hdfs 0 2018-03-01 01:02
> /kylin/kylin_metadata/metadata/05ea3568-e85f-452a-b44d-4657145a724a
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
