Thanks, @ShaoFeng,Thats really great help, we really appreciate 
that.Thanks,ke...@exponential.com Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: ShaoFeng Shi <shaofeng...@apache.org> 
Date: 30/10/2018  12:02 pm  (GMT+05:30) To: dev <dev@kylin.apache.org> Subject: 
Re: Merge Job in inconsistent state Kylin document is updated now, please 
refresh your web 
browser:https://kylin.apache.org/docs/howto/howto_backup_metadata.htmlShaoFeng 
Shi <shaofeng...@apache.org> 于2018年10月30日周二 上午11:35写道:> Hi Ketan,>> You're 
welcome. JIRA is created for this issue:> 
https://issues.apache.org/jira/browse/KYLIN-3652>> kdcool6932 
<kdcool6...@yahoo.com.invalid> 于2018年10月30日周二 上午10:49写道:>>> Thanks, 
ShaoFeng,That's just what we need.I will use 'to-add' as>> reference, along 
with the online 'howto', that explains it>> all.Thanks,ke...@exponential.com 
Sent from my Samsung Galaxy smartphone.>> -------- Original message 
--------From: ShaoFeng Shi <>> shaofeng...@apache.org> Date: 30/10/2018  7:49 
am  (GMT+05:30) To: dev <>> dev@kylin.apache.org> Subject: Re: Merge Job in 
inconsistent state I'm>> updating the document for modifying metadata in a 
selective way, butthe>> jenkins has a problem today. Here is the to-add part 
inhttps://>> kylin.apache.org/docs/howto/howto_backup_metadata.html:##>> 
<http://kylin.apache.org/docs/howto/howto_backup_metadata.html:#%23>>> Restore 
metadata selectively (Recommended)If only changes a couple of>> metadata files, 
the administrator can just pickthese files to restore,>> without having to 
cover all the metadata. Comparedto the full recovery,>> this approach is more 
efficient, safer, so it isrecommended.Create a new>> empty directory, and then 
create subdirectories in itaccording to the>> location of the metadata files to 
restore; for example, torestore a Cube>> instance, you should create a "cube" 
subdirectory:{% highlight Groff markup>> %}mkdir /path/to/restore_newmkdir 
/path/to/restore_new/cube{% endhighlight>> %}Copy the metadata file to be 
restored to this new directory:{% highlight>> Groff markup %}cp>> 
meta_backups/meta_2016_06_10_20_24_50/cube/kylin_sales_cube.json/path/to/restore_new/cube/{%>>
 endhighlight %}At this point, you can modify/fix the metadata>> 
manually.Restore from this directory:{% highlight Groff markup %}cd>> 
$KYLIN_HOME./bin/metastore.sh restore /path/to/restore_new{% endhighlight>> 
%}Only the files in the folder will be uploaded to Kylin>> metastore.Similarly, 
after the recovery is finished, click Reload Metadata>> button onthe Web UI to 
flush the cache.kdcool6932>> <kdcool6...@yahoo.com.invalid> 于2018年10月29日周一 
下午7:54写道:> Thanks guys,,>> really appreciate the prompt response.@ShaoFeng,Yes 
we have> the data in>> hive(or we can load that if needed). And we will be 
rebuilding> those>> segments. Do we have any mail thread or document or blog to 
refer to> for>> manully editing and restoring metadata. That would actually be 
great> help,>> as we often get into situations like this as we don't want to 
take> risk>> for 3 plus years (120+TB) data in Hbase for Kylin.Again,, 
really>>> appreciate the help provided.Thanks,Ketan Sent from my Samsung 
Galaxy>>> smartphone.> -------- Original message --------From: ShaoFeng Shi 
<>>> shaofeng...@apache.org> Date: 29/10/2018  12:36 pm  (GMT+05:30) To: dev>> 
<> dev@kylin.apache.org> Subject: Re: Merge Job in inconsistent state It>> is> 
a known issue; the auto-merge was triggered on each segment>> change.Maybe,> 
Kylin should not trigger the auto-merge on>> canceling/deleting> 
ajob/segment?But you can keep that error job/segment,>> it won't impact on> the 
query. Theonly thing is an error job there.Do you>> know the root cause> of  
"No input paths specified in job"? Did youdelete>> some folders from> HDFS?If 
you have the source data in Hive, you can>> rebuild those segments;> Youcan 
take a backup of the metadata, and then>> dump metadata in local> disk,copy 
that cube's json to a clean folder, edit>> it to delete these> segment,and then 
restore metadata from the clean folder>> (same structure,> 
e.g/cube/yourcube.json). After restored, build the>> segments for the> 
missingdate range.Chao Long <wayn...@qq.com>>> 于2018年10月29日周一 下午2:12写道:> Hi> 
Ketan,>    As this merge job is an>> automatically triggered job, so it start> 
again> when you discard it. If>> you don't want this job to be triggered> 
again, you> can remove the "Auto>> Merge" related configuration on the cube> 
design page> until the problem is>> resolved or fixed(if it's a bug).>>>> This 
is the merging>> Segment[20181005080000_20181012170000]. And the>> exception 
occurred during>> the merging job, so it's in an incorrect> state.>    
Segment:>> 20181005080000_20181012170000 //**This segment Table> was> deleted 
from>> Hbase (somehow, we don’t have the reason)**//>    Start> Time: 
2018-10-05>> 08:00:00>    End Time: 2018-10-12 17:00:00>    Source> Count: 0>   
 HBase>> Table: KYLIN_CFLY2CKMCU>    Region Count: 3>    Size:> less than 1 
MB>>>>> To identify the root cause, you may provide more log> around the 
error>>> message.>>>>>> ------------------ 原始邮件> ------------------> 发件人: 
"ketan>> dikshit"<kdcool6...@yahoo.com.INVALID>;>> 发送时间: 2018年10月28日(星期天) 
凌晨0:25>>> 收件人: "dev"<dev@kylin.apache.org>;>> 主题:> Merge Job in inconsistent>> 
state>>>> Hi Team,> We are using Kylin 2.3.1, And> in the merge Job(which>> 
gets automatically> triggered), we are getting this> error in Merge Cuboid>> 
Data Step;>> java.io.IOException: No input paths> specified in job>>> at>>>> 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:239)>>>>
 at>>>> 
org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)>>>>
 at>>>> 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387)>>>>
 at org.apache.kylin.engine.mr>>>> 
.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:622)>>>> 
at>>>> 
org.apache.kylin.storage.hbase.steps.HBaseMROutput2Transition$HBaseMergeMROutputFormat.configureJobOutput(HBaseMROutput2Transition.java:166)>>>>
 at org.apache.kylin.engine.mr>>>> 
.steps.MergeCuboidJob.run(MergeCuboidJob.java:82)>         at>>> 
org.apache.kylin.engine.mr>>>> 
.common.MapReduceExecutable.doWork(MapReduceExecutable.java:130)>> at>>>> 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)>>>>
 at>>>> 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)>>>>
 at>>>> 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)>>>>
 at>>>> 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)>>>>
 at>>>> 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)>>>>
 at>>>> 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)>>>>
 at java.lang.Thread.run(Thread.java:745)>> result code:2>> As it is a>> merge> 
job, whenever I discard this job,, it again starts> automatically.>>> So we> 
need a way(if there is one), of permanently discarding this job>>> somehow.>>> 
Also in case this can be done by changing the metadata for the>> cube>> 
segment, do let me know how it can be done(even if I have to>> delete/remove>> 
the segments for these below days)> We are open for all>> options (except> 
dropping the cube as it has more than> 1 year data, and is>> crucial for> 
organisation). We are stuck on this for> some days now, help>> would really> be 
appreciated.>> Merge Thresholds:> 8 (Hours)> 1 (Days)> 7>> (Days)> 15> (Days)>> 
Merge Job, (start and end):> MERGE CUBE - XXXX ->>> 
20181005080000_20181012170000>> Here are list of segments;> Segment:>>> 
20181005080000_20181006080000> Start Time: 2018-10-05 08:00:00> End Time:>>> 
2018-10-06 08:00:00> Source Count: 14899048> HBase Table:>> KYLIN_R1MUK56K71>> 
Region Count: 1> Size: 860 MB>> Segment:>> 20181005080000_20181012170000> 
//**This segment Table was deleted> from>> Hbase (somehow, we don’t have the> 
reason)**//> Start Time: 2018-10-05>> 08:00:00> End Time: 2018-10-12> 17:00:00> 
Source Count: 0> HBase Table:>> KYLIN_CFLY2CKMCU> Region Count: 3>> Size: less 
than 1 MB>> Segment:>> 20181006080000_20181008000000> Start Time:> 2018-10-06 
08:00:00> End Time:>> 2018-10-08 00:00:00> Source Count: 24455686>> HBase 
Table:>> KYLIN_0KH6PHTEM2> Region Count: 1> Size: 1.0498 GB>> Segment:>>> 
20181008000000_20181009000000> Start Time: 2018-10-08 00:00:00> End Time:>>> 
2018-10-09 00:00:00> Source Count: 14882090> HBase Table:>> KYLIN_V1CC4LDSIR>> 
Region Count: 1> Size: 598 MB>> Segment:>> 20181009000000_20181010000000>> 
Start Time: 2018-10-09 00:00:00> End Time:>> 2018-10-10 00:00:00> Source> 
Count: 16245847> HBase Table:>> KYLIN_4A44K2VJEU> Region Count: 1> Size: 628> 
MB>> Segment:>> 20181010000000_20181010080000> Start Time: 2018-10-10> 
00:00:00> End Time:>> 2018-10-10 08:00:00> Source Count: 5213022> HBase> Table: 
KYLIN_EHO316VC7M>>> Region Count: 1> Size: 397 MB>> Segment:> 
20181010080000_20181010090000>>> Start Time: 2018-10-10 08:00:00> End Time:> 
2018-10-10 09:00:00> Source>> Count: 865722> HBase Table: KYLIN_I9LEJ2JDZ8>> 
Region Count: 1> Size: 181>> MB>> Segment: 20181010090000_20181010100000>> 
Start Time: 2018-10-10>> 09:00:00> End Time: 2018-10-10 10:00:00> Source> 
Count: 859127> HBase>> Table: KYLIN_9IBX3W4UNL> Region Count: 1> Size: 180> 
MB>> Segment:>> 20181010100000_20181010110000> Start Time: 2018-10-10> 
10:00:00> End Time:>> 2018-10-10 11:00:00> Source Count: 855752> HBase Table:> 
KYLIN_HRDJ16B3O8>>> Region Count: 1> Size: 179 MB>> Segment:> 
20181010110000_20181010120000>>> Start Time: 2018-10-10 11:00:00> End Time:> 
2018-10-10 12:00:00> Source>> Count: 849363> HBase Table: KYLIN_6BFHFA5LU1>> 
Region Count: 1> Size: 178>> MB>> Segment: 20181010120000_20181010130000>> 
Start Time: 2018-10-10>> 12:00:00> End Time: 2018-10-10 13:00:00> Source> 
Count: 851162> HBase>> Table: KYLIN_H41KZXUIRN> Region Count: 1> Size: 177> 
MB>> Segment:>> 20181010130000_20181010140000> Start Time: 2018-10-10> 
13:00:00> End Time:>> 2018-10-10 14:00:00> Source Count: 836481> HBase Table:> 
KYLIN_8RXPI7T0PA>>> Region Count: 1> Size: 173 MB>> Segment:> 
20181010140000_20181010150000>>> Start Time: 2018-10-10 14:00:00> End Time:> 
2018-10-10 15:00:00> Source>> Count: 780337> HBase Table: KYLIN_7L3WHR3ZQY>> 
Region Count: 1> Size: 164>> MB>> Segment: 20181010150000_20181010160000>> 
Start Time: 2018-10-10>> 15:00:00> End Time: 2018-10-10 16:00:00> Source> 
Count: 723669> HBase>> Table: KYLIN_RM0ICHV5EP> Region Count: 1> Size: 155> 
MB>> Segment:>> 20181010160000_20181011170000> Start Time: 2018-10-10> 
16:00:00> End Time:>> 2018-10-11 17:00:00> Source Count: 17476745> HBase> 
Table:>> KYLIN_Y8ZOSIWNJP> Region Count: 1> Size: 941 MB>> Segment:>>> 
20181011170000_20181012170000> Start Time: 2018-10-11 17:00:00> End Time:>>> 
2018-10-12 17:00:00> Source Count: 15485276> HBase Table:>> KYLIN_RWTQZFY6J4>> 
Region Count: 1> Size: 887 MB>> Thanks,>>> Ketan@Exponential-- Best> 
regards,Shaofeng Shi 史少锋-- Best>> regards,Shaofeng Shi 史少锋>>>> --> Best 
regards,>> Shaofeng Shi 史少锋>>-- Best regards,Shaofeng Shi 史少锋

Reply via email to