Thanks, ShaoFeng,That's just what we need.I will use 'to-add' as reference, 
along with the online 'howto', that explains it 
all.Thanks, Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: ShaoFeng Shi <> 
Date: 30/10/2018  7:49 am  (GMT+05:30) To: dev <> Subject: 
Re: Merge Job in inconsistent state I'm updating the document for modifying 
metadata in a selective way, butthe jenkins has a problem today. Here is the 
to-add part in 
Restore metadata selectively (Recommended)If only changes a couple of metadata 
files, the administrator can just pickthese files to restore, without having to 
cover all the metadata. Comparedto the full recovery, this approach is more 
efficient, safer, so it isrecommended.Create a new empty directory, and then 
create subdirectories in itaccording to the location of the metadata files to 
restore; for example, torestore a Cube instance, you should create a "cube" 
subdirectory:{% highlight Groff markup %}mkdir /path/to/restore_newmkdir 
/path/to/restore_new/cube{% endhighlight %}Copy the metadata file to be 
restored to this new directory:{% highlight Groff markup %}cp 
 endhighlight %}At this point, you can modify/fix the metadata manually.Restore 
from this directory:{% highlight Groff markup %}cd 
$KYLIN_HOME./bin/ restore /path/to/restore_new{% endhighlight 
%}Only the files in the folder will be uploaded to Kylin metastore.Similarly, 
after the recovery is finished, click Reload Metadata button onthe Web UI to 
flush the cache.kdcool6932 <> 于2018年10月29日周一 
下午7:54写道:> Thanks guys,, really appreciate the prompt response.@ShaoFeng,Yes we 
have> the data in hive(or we can load that if needed). And we will be 
rebuilding> those segments. Do we have any mail thread or document or blog to 
refer to> for manully editing and restoring metadata. That would actually be 
great> help, as we often get into situations like this as we don't want to 
take> risk for 3 plus years (120+TB) data in Hbase for Kylin.Again,, really> 
appreciate the help provided.Thanks,Ketan Sent from my Samsung Galaxy> 
smartphone.> -------- Original message --------From: ShaoFeng Shi <>> Date: 29/10/2018  12:36 pm  (GMT+05:30) To: dev <>> Subject: Re: Merge Job in inconsistent state It is> a 
known issue; the auto-merge was triggered on each segment change.Maybe,> Kylin 
should not trigger the auto-merge on canceling/deleting> ajob/segment?But you 
can keep that error job/segment, it won't impact on> the query. Theonly thing 
is an error job there.Do you know the root cause> of  "No input paths specified 
in job"? Did youdelete some folders from> HDFS?If you have the source data in 
Hive, you can rebuild those segments;> Youcan take a backup of the metadata, 
and then dump metadata in local> disk,copy that cube's json to a clean folder, 
edit it to delete these> segment,and then restore metadata from the clean 
folder (same structure,> e.g/cube/yourcube.json). After restored, build the 
segments for the> missingdate range.Chao Long <> 于2018年10月29日周一 
下午2:12写道:> Hi> Ketan,>    As this merge job is an automatically triggered job, 
so it start> again> when you discard it. If you don't want this job to be 
triggered> again, you> can remove the "Auto Merge" related configuration on the 
cube> design page> until the problem is resolved or fixed(if it's a bug).>>>> 
This is the merging Segment[20181005080000_20181012170000]. And the>> exception 
occurred during the merging job, so it's in an incorrect> state.>    Segment: 
20181005080000_20181012170000 //**This segment Table> was> deleted from Hbase 
(somehow, we don’t have the reason)**//>    Start> Time: 2018-10-05 08:00:00>   
 End Time: 2018-10-12 17:00:00>    Source> Count: 0>    HBase Table: 
KYLIN_CFLY2CKMCU>    Region Count: 3>    Size:> less than 1 MB>>>    To 
identify the root cause, you may provide more log> around the error> 
message.>>>>>> ------------------ 原始邮件> ------------------> 发件人: "ketan 
dikshit"<>;>> 发送时间: 2018年10月28日(星期天) 凌晨0:25> 收件人: 
"dev"<>;>> 主题:> Merge Job in inconsistent state>>>> Hi 
Team,> We are using Kylin 2.3.1, And> in the merge Job(which gets 
automatically> triggered), we are getting this> error in Merge Cuboid Data 
Step;>> No input paths> specified in job>         at>> 
.common.AbstractHadoopJob.getTotalMapInputMB(>> at>>$HBaseMergeMROutputFormat.configureJobOutput(>>
 at>>>         at>>> 
.common.MapReduceExecutable.doWork(>> at>> 
 at>> result code:2>> As it is a merge> 
job, whenever I discard this job,, it again starts> automatically.> So we> need 
a way(if there is one), of permanently discarding this job> somehow.>>> Also in 
case this can be done by changing the metadata for the cube>> segment, do let 
me know how it can be done(even if I have to delete/remove>> the segments for 
these below days)> We are open for all options (except> dropping the cube as it 
has more than> 1 year data, and is crucial for> organisation). We are stuck on 
this for> some days now, help would really> be appreciated.>> Merge 
Thresholds:> 8 (Hours)> 1 (Days)> 7 (Days)> 15> (Days)>> Merge Job, (start and 
end):> MERGE CUBE - XXXX -> 20181005080000_20181012170000>> Here are list of 
segments;> Segment:> 20181005080000_20181006080000> Start Time: 2018-10-05 
08:00:00> End Time:> 2018-10-06 08:00:00> Source Count: 14899048> HBase Table: 
KYLIN_R1MUK56K71>> Region Count: 1> Size: 860 MB>> Segment: 
20181005080000_20181012170000> //**This segment Table was deleted> from Hbase 
(somehow, we don’t have the> reason)**//> Start Time: 2018-10-05 08:00:00> End 
Time: 2018-10-12> 17:00:00> Source Count: 0> HBase Table: KYLIN_CFLY2CKMCU> 
Region Count: 3>> Size: less than 1 MB>> Segment: 
20181006080000_20181008000000> Start Time:> 2018-10-06 08:00:00> End Time: 
2018-10-08 00:00:00> Source Count: 24455686>> HBase Table: KYLIN_0KH6PHTEM2> 
Region Count: 1> Size: 1.0498 GB>> Segment:> 20181008000000_20181009000000> 
Start Time: 2018-10-08 00:00:00> End Time:> 2018-10-09 00:00:00> Source Count: 
14882090> HBase Table: KYLIN_V1CC4LDSIR>> Region Count: 1> Size: 598 MB>> 
Segment: 20181009000000_20181010000000>> Start Time: 2018-10-09 00:00:00> End 
Time: 2018-10-10 00:00:00> Source> Count: 16245847> HBase Table: 
KYLIN_4A44K2VJEU> Region Count: 1> Size: 628> MB>> Segment: 
20181010000000_20181010080000> Start Time: 2018-10-10> 00:00:00> End Time: 
2018-10-10 08:00:00> Source Count: 5213022> HBase> Table: KYLIN_EHO316VC7M> 
Region Count: 1> Size: 397 MB>> Segment:> 20181010080000_20181010090000> Start 
Time: 2018-10-10 08:00:00> End Time:> 2018-10-10 09:00:00> Source Count: 
865722> HBase Table: KYLIN_I9LEJ2JDZ8>> Region Count: 1> Size: 181 MB>> 
Segment: 20181010090000_20181010100000>> Start Time: 2018-10-10 09:00:00> End 
Time: 2018-10-10 10:00:00> Source> Count: 859127> HBase Table: 
KYLIN_9IBX3W4UNL> Region Count: 1> Size: 180> MB>> Segment: 
20181010100000_20181010110000> Start Time: 2018-10-10> 10:00:00> End Time: 
2018-10-10 11:00:00> Source Count: 855752> HBase Table:> KYLIN_HRDJ16B3O8> 
Region Count: 1> Size: 179 MB>> Segment:> 20181010110000_20181010120000> Start 
Time: 2018-10-10 11:00:00> End Time:> 2018-10-10 12:00:00> Source Count: 
849363> HBase Table: KYLIN_6BFHFA5LU1>> Region Count: 1> Size: 178 MB>> 
Segment: 20181010120000_20181010130000>> Start Time: 2018-10-10 12:00:00> End 
Time: 2018-10-10 13:00:00> Source> Count: 851162> HBase Table: 
KYLIN_H41KZXUIRN> Region Count: 1> Size: 177> MB>> Segment: 
20181010130000_20181010140000> Start Time: 2018-10-10> 13:00:00> End Time: 
2018-10-10 14:00:00> Source Count: 836481> HBase Table:> KYLIN_8RXPI7T0PA> 
Region Count: 1> Size: 173 MB>> Segment:> 20181010140000_20181010150000> Start 
Time: 2018-10-10 14:00:00> End Time:> 2018-10-10 15:00:00> Source Count: 
780337> HBase Table: KYLIN_7L3WHR3ZQY>> Region Count: 1> Size: 164 MB>> 
Segment: 20181010150000_20181010160000>> Start Time: 2018-10-10 15:00:00> End 
Time: 2018-10-10 16:00:00> Source> Count: 723669> HBase Table: 
KYLIN_RM0ICHV5EP> Region Count: 1> Size: 155> MB>> Segment: 
20181010160000_20181011170000> Start Time: 2018-10-10> 16:00:00> End Time: 
2018-10-11 17:00:00> Source Count: 17476745> HBase> Table: KYLIN_Y8ZOSIWNJP> 
Region Count: 1> Size: 941 MB>> Segment:> 20181011170000_20181012170000> Start 
Time: 2018-10-11 17:00:00> End Time:> 2018-10-12 17:00:00> Source Count: 
15485276> HBase Table: KYLIN_RWTQZFY6J4>> Region Count: 1> Size: 887 MB>> 
Thanks,> Ketan@Exponential-- Best> regards,Shaofeng Shi 史少锋-- Best 
regards,Shaofeng Shi 史少锋

Reply via email to