[ https://issues.apache.org/jira/browse/KYLIN-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liyang resolved KYLIN-2708. --------------------------- Resolution: Duplicate Fix Version/s: v2.1.0 Dup with KYLIN-2648. Thanks for reporting! > cube merge operations can not execute success > --------------------------------------------- > > Key: KYLIN-2708 > URL: https://issues.apache.org/jira/browse/KYLIN-2708 > Project: Kylin > Issue Type: Bug > Components: General > Affects Versions: v2.0.0 > Reporter: catchen > Fix For: v2.1.0 > > > By looking at the log, the cube merge process contains operations of > Collection Garbage on HDFS , but it will throw ERROR at this step every time。 > in kylin.properties file: > kylin.storage.hbase.cluster-fs=hdfs://hacluster2/hbase > in core-site.xml file:fs.defaultFS=hdfs://hacluster > our's HBase deploy in hbase cluster,Hadoop deploy in Hadoop cluster ,they are > respective > the detail log like this:------------------- > 2017-07-04 06:01:13,795 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:78 : Drop HDFS path on FileSystem: > hdfs://hacluster > 2017-07-04 06:01:13,799 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:90 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-70750aee-2dbe-4eb8-bc8a-553f2696fce8 not > exists. > 2017-07-04 06:01:13,802 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:90 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-b138cc1f-3709-4959-b7ed-96782770cb8c not > exists. > 2017-07-04 06:01:13,805 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:87 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-2fae97ee-3ad9-4d15-87ea-07db83971f64 is > dropped. > 2017-07-04 06:01:13,809 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:87 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-22859c63-a80a-4287-9449-f57f730748fc is > dropped. > 2017-07-04 06:01:13,813 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:87 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-fe23f867-6922-4eee-b4a5-fc639d3baaf8 is > dropped. > 2017-07-04 06:01:13,817 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:87 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-f4671734-74e0-42a3-a3eb-0b848a180495 is > dropped. > 2017-07-04 06:01:13,821 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:87 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-fbd89367-d323-45de-8254-f228ea8873ef is > dropped. > 2017-07-04 06:01:13,824 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:87 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-dc945b55-86f4-4c22-860a-3b6a0e11c172 is > dropped. > 2017-07-04 06:01:13,828 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:87 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-aa723e9b-d76d-44a8-92f4-2429b576f945 is > dropped. > 2017-07-04 06:01:13,832 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:87 : HDFS path > hdfs:///kylin/kylin_metadata/kylin-9c41f48c-42e8-414f-acb3-5815f43176ef is > dropped. > 2017-07-04 06:01:13,833 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > steps.HDFSPathGarbageCollectionStep:78 : Drop HDFS path on FileSystem: > hdfs://hacluster2 > 2017-07-04 06:01:13,834 ERROR [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > execution.AbstractExecutable:126 : error running Executable: > HDFSPathGarbageCollectionStep{id=43bdda4d-be69-46c3-9458-87d075145b3c-08, > name=Garbage Collection on HDFS, state=RUNNING} > 2017-07-04 06:01:13,835 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > dao.ExecutableDao:217 : updating job output, id: > 43bdda4d-be69-46c3-9458-87d075145b3c-08 > 2017-07-04 06:01:13,837 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > dao.ExecutableDao:217 : updating job output, id: > 43bdda4d-be69-46c3-9458-87d075145b3c-08 > 2017-07-04 06:01:13,839 INFO [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > execution.ExecutableManager:389 : job > id:43bdda4d-be69-46c3-9458-87d075145b3c-08 from RUNNING to ERROR > 2017-07-04 06:01:13,839 ERROR [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > execution.AbstractExecutable:126 : error running Executable: > CubingJob{id=43bdda4d-be69-46c3-9458-87d075145b3c, > name=cube_total_active_user - 20170606000000_20170704000000 - MERGE - > GMT+08:00 2017-07-04 05:58:05, state=RUNNING} > 2017-07-04 06:01:13,841 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > dao.ExecutableDao:217 : updating job output, id: > 43bdda4d-be69-46c3-9458-87d075145b3c > 2017-07-04 06:01:13,843 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > dao.ExecutableDao:217 : updating job output, id: > 43bdda4d-be69-46c3-9458-87d075145b3c > 2017-07-04 06:01:13,844 INFO [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > execution.ExecutableManager:389 : job id:43bdda4d-be69-46c3-9458-87d075145b3c > from RUNNING to ERROR > 2017-07-04 06:01:13,844 WARN [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281] > execution.AbstractExecutable:258 : no need to send email, user list is empty > 2017-07-04 06:01:13,844 ERROR [pool-9-thread-9] > threadpool.DefaultScheduler:146 : ExecuteException > job:43bdda4d-be69-46c3-9458-87d075145b3c > org.apache.kylin.job.exception.ExecuteException: > org.apache.kylin.job.exception.ExecuteException: > java.lang.IllegalArgumentException: Wrong FS: > hdfs:/kylin/kylin_metadata/kylin-70750aee-2dbe-4eb8-bc8a-553f2696fce8, > expected: hdfs://hacluster2 > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.kylin.job.exception.ExecuteException: > java.lang.IllegalArgumentException: Wrong FS: > hdfs:/kylin/kylin_metadata/kylin-70750aee-2dbe-4eb8-bc8a-553f2696fce8, > expected: hdfs://hacluster2 > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124) > ... 4 more > Caused by: java.lang.IllegalArgumentException: Wrong FS: > hdfs:/kylin/kylin_metadata/kylin-70750aee-2dbe-4eb8-bc8a-553f2696fce8, > expected: hdfs://hacluster2 > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:101) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398) > at > org.apache.kylin.storage.hbase.steps.HDFSPathGarbageCollectionStep.dropHdfsPathOnCluster(HDFSPathGarbageCollectionStep.java:85) > at > org.apache.kylin.storage.hbase.steps.HDFSPathGarbageCollectionStep.doWork(HDFSPathGarbageCollectionStep.java:65) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124) > ... 6 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)