Hi Jakob, did you create JIRA for this already? Tez itself should not the problem but just some statistics information Kylin can't parse as you figure out already.
Please let me know the JIRA number and we will enhance that to avoid such issue again. Thanks. Luke Best Regards! --------------------- Luke Han 2015-03-03 2:38 GMT+08:00 Jakob Stengård <[email protected]>: > I looked at the source code, but the fix wasn't obvious to me. > > Maybe, if i have time, i'll look into it further. > > /J > > > On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu <[email protected]> > wrote: > > > Thanks Jakob, > > > > Would be fantastic if you could dig in and submit a patch… > > > > > > On 3/2/15, 10:08 AM, "Jakob Stengård" <[email protected]> wrote: > > > > >I'll file a bug in jira. > > >For your information i also happen to know that tez boosted map-reduce > > >doesn't work either, but there, the problem seems to have to do with the > > >map reduce jobs them selves failing. > > > > > >In this case however, from what i've read on git hub about the "Can't > get > > >cube source record size", > > >error", it could possibly be a problem with that tez doesn't output a > > >string on the form: > > >"HDFS Read: (\d+) HDFS Write" > > > > > >However, it could be something else, it could also be my logging > > >configuration for tez. > > > > > >/J > > > > > > > > > > > >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <[email protected] > > > > >wrote: > > > > > >> Did not read your logs fully. The culprit seems to be > > >> > > >> > > >> 1. [QuartzScheduler_Worker-7]:[2015-03-02 > > >> > > > >>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti > > >>on > > >> (JobFlowListener.java:312)] - Can't get cube source record size. > > >> > > >> source_records_size may have to be computed differently when using Tez > > >>as > > >> the Hive engine. I will let Yang/Luke answer this when they come back > > >> onlineŠ > > >> > > >> > > >> > > >> > > >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <[email protected]> wrote: > > >> > > >> >Jakob, > > >> > > > >> >Could you go ahead and file a JIRA with the stack trace at > > >> >https://issues.apache.org/jira/browse/KYLIN/ > > >> > > > >> >Also include the App Master and Map/Reduce Task logs for the Tasks > that > > >> >failed. > > >> > > > >> >Regards > > >> >Seshu > > >> > > > >> >On 3/2/15, 8:12 AM, "Jakob Stengård" <[email protected]> > wrote: > > >> > > > >> >>Correction: That, isn't the last job, it's step 9. > > >> >> > > >> >>/J > > >> >> > > >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård > > >><[email protected] > > >> > > > >> >>wrote: > > >> >> > > >> >>> Well, this only happens with > > >> >>> <property> > > >> >>> <name>hive.execution.engine</name> > > >> >>> <value>tez</value> > > >> >>> </property> > > >> >>> > > >> >>> If i use mr, it works fine. > > >> >>> > > >> >>> > > >> >>> From what i can see form the job log, none of the previous > > >>map-reduce > > >> >>>jobs > > >> >>> fail, but the build fails at the last step. > > >> >>> Log from the last mr job for example: > > >> >>> http://pastebin.com/jNWSRKLi > > >> >>> > > >> >>> > > >> >>> Is tez supposed to work? Because in that case, maybe it's just my > > >> >>>setup. > > >> >>> > > >> >>> /Jakob > > >> >>> > > >> >>> > > >> >>> > > >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh < > > >> >>> [email protected]> wrote: > > >> >>> > > >> >>>> This doesn't seems to be complete kylin.log. Do you have any > > >>exception > > >> >>>> related to MR counter in map reduce logs? If yes then this needs > > >>to be > > >> >>>> checked because kylin will need the MR counter to calculate the > > >>cube > > >> >>>>size. > > >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård > > >> >>>><[email protected]> > > >> >>>> wrote: > > >> >>>> > > >> >>>> > It breaks on the last step: "Load HFile to HBase Table": > > >> >>>> > > > >> >>>> > Kylin.log: > > >> >>>> > http://pastebin.com/3kAQmJ3u > > >> >>>> > > > >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu > > >> >>>><[email protected] > > >> >>>> > > > >> >>>> > wrote: > > >> >>>> > > > >> >>>> > > Jakob, > > >> >>>> > > > > >> >>>> > > Could you please send out the complete stack trace with Tez > > >> >>>>enabled? > > >> >>>> > > > > >> >>>> > > Regards > > >> >>>> > > Seshu > > >> >>>> > > > > >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" < > [email protected] > > > > > >> >>>> wrote: > > >> >>>> > > > > >> >>>> > > >Hi > > >> >>>> > > >Will kylin suport hive+tez in v0.7.x? > > >> >>>> > > > > > >> >>>> > > >I've tried using tez with hive, but currently the cube build > > >> >>>>fails > > >> >>>> with > > >> >>>> > > >"Can't get cube source record size", because kylin expects > > >> >>>>certain > > >> >>>> > console > > >> >>>> > > >output from map-reduce, and tez doesn't print this. > > >> >>>> > > > > > >> >>>> > > >BG. > > >> >>>> > > >Jakob Stengard > > >> >>>> > > > > >> >>>> > > > > >> >>>> > > > >> >>>> > > >> >>> > > >> >>> > > >> > > > >> > > >> > > > > >
