[
https://issues.apache.org/jira/browse/KYLIN-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mukvin updated KYLIN-5179:
--------------------------
Summary: Kylin4 can not delete invalid segments on HDFS by
StorageCleanupJob (was: Kylin4无法通过StorageCleanupJob移除HDFS中失效的segment)
> Kylin4 can not delete invalid segments on HDFS by StorageCleanupJob
> -------------------------------------------------------------------
>
> Key: KYLIN-5179
> URL: https://issues.apache.org/jira/browse/KYLIN-5179
> Project: Kylin
> Issue Type: Bug
> Components: Storage - Parquet
> Affects Versions: v4.0.1
> Reporter: Sunbuck
> Assignee: mukvin
> Priority: Major
> Fix For: v4.0.1
>
> Attachments: image-2022-04-22-16-20-14-714.png,
> image-2022-04-22-16-22-09-804.png, image-2022-04-22-16-30-27-879.png,
> image-2022-04-22-16-31-18-621.png
>
>
> 在测试环境中通过KylinWeb中对cube进行数据构建后，想通过kylin.sh
> org.apache.kylin.tool.StorageCleanupJob进行失效segment进行清理，目前有效的cube为0RI的parquet
> storage：
> !image-2022-04-22-16-20-14-714.png!
> 在hdfs中查看该cube的segment情况：
> !image-2022-04-22-16-22-09-804.png!
> 其中MV6和C54的segment是我已经在Web中进行delete segment操作后的遗留垃圾数据，在按照Kylin
> Wiki（[https://cwiki.apache.org/confluence/display/KYLIN/How+to+clean+up+storage+in+Kylin+4]）中的说明中执行
> kylin.sh org.apache.kylin.tool.StorageCleanupJob时，报错如下：
>
> {code:java}
> 2022-04-22 16:26:13,328 INFO [main] persistence.JDBCResourceStore:115 :
> Table [kylin_metadata] already exists
> 2022-04-22 16:26:13,332 INFO [main] persistence.JDBCResourceStore:115 :
> Table [kylin_metadata_log] already exists
> 2022-04-22 16:26:13,561 INFO [main] common.KylinConfig:493 : Creating new
> manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
> Running org.apache.kylin.rest.job.StorageCleanupJob
> 2022-04-22 16:26:13,571 INFO [main] job.StorageCleanupJob:143 : options: ''
> 2022-04-22 16:26:13,571 INFO [main] job.StorageCleanupJob:160 :
> ===================================================================
> delete : false; cleanupTableSnapshot : true; cleanupGlobalDict : true;
> cleanupJobTmp : false; cleanBeforeDate : Fri Apr 15 16:26:13 CST 2022.
> 2022-04-22 16:26:13,574 INFO [main] common.KylinConfig:493 : Creating new
> manager instance of class org.apache.kylin.metadata.project.ProjectManager
> 2022-04-22 16:26:13,576 INFO [main] project.ProjectManager:81 : Initializing
> ProjectManager with metadata url
> kylin_metadata@jdbc,url=jdbc:mysql://10.62.0.245:3306/kylin4,username=xxx,password=xxx,maxActive=10,maxIdle=10
> 2022-04-22 16:26:13,602 INFO [main] common.KylinConfig:493 : Creating new
> manager instance of class org.apache.kylin.cube.CubeManager
> 2022-04-22 16:26:13,608 INFO [main] cube.CubeManager:122 : Initializing
> CubeManager with config
> kylin_metadata@jdbc,url=jdbc:mysql://10.62.0.245:3306/kylin4,username=xxx,password=xxx,maxActive=10,maxIdle=10
> 2022-04-22 16:26:13,651 INFO [main] common.KylinConfig:493 : Creating new
> manager instance of class org.apache.kylin.cube.CubeDescManager
> 2022-04-22 16:26:13,652 INFO [main] cube.CubeDescManager:91 : Initializing
> CubeDescManager with config
> kylin_metadata@jdbc,url=jdbc:mysql://10.62.0.245:3306/kylin4,username=xxx,password=xxx,maxActive=10,maxIdle=10
> 2022-04-22 16:26:13,705 INFO [main] common.KylinConfig:493 : Creating new
> manager instance of class org.apache.kylin.metadata.model.DataModelManager
> 2022-04-22 16:26:13,709 INFO [main] common.KylinConfig:493 : Creating new
> manager instance of class org.apache.kylin.metadata.TableMetadataManager
> 2022-04-22 16:26:13,866 INFO [main] job.StorageCleanupJob:174 : Start to
> clean up unreferenced projects and cubes ...
> 2022-04-22 16:26:13,867 INFO [main] job.StorageCleanupJob:200 : Start to
> clean up no unreferenced segments ...
> 2022-04-22 16:26:13,867 WARN [main] job.StorageCleanupJob:222 : Cube path
> doesn't exist! The path is
> file:/kylin4/kylin_metadata/storage/parquet/mobile_core_analysis_cube
> 2022-04-22 16:26:13,868 WARN [main] job.StorageCleanupJob:222 : Cube path
> doesn't exist! The path is
> file:/kylin4/kylin_metadata/storage/parquet/storage_cube{code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
