I tried installing 0.7.1, but i get "The 'partition_date_column' format is
invalid: , it should be {db}.{table}.{column}."
when trying to set up a cube.Dunno if it's my tables that it doesn't like or if it's some deprecated settings from 0.6.5 that causes this. Tried deleting all Hbase tables, but i still get the error after restarting Kylin and re-syncing the tables. /J On Tue, Mar 3, 2015 at 9:24 AM, Li Yang <[email protected]> wrote: > I briefly looked into the code and seems the issue has been fixed in 0.7.x > (the current inverted-index branch) already. Basically, it does not rely on > counters of the hive step any more. > > However I don't have the env to verify this. @Jakob, may try this latest > version http://kylin.incubator.apache.org/download/ > > On Tue, Mar 3, 2015 at 6:59 AM, Jakob Stengård <[email protected]> > wrote: > > > Yeah, i know that. > > Here is the jira bug i filed: > > https://issues.apache.org/jira/browse/KYLIN-619 > > > > On Mon, Mar 2, 2015 at 11:47 PM, Luke Han <[email protected]> wrote: > > > > > Hi Jakob, did you create JIRA for this already? > > > > > > Tez itself should not the problem but just some statistics information > > > Kylin can't parse as you figure out already. > > > > > > Please let me know the JIRA number and we will enhance that to avoid > such > > > issue again. > > > > > > Thanks. > > > Luke > > > > > > > > > Best Regards! > > > --------------------- > > > > > > Luke Han > > > > > > 2015-03-03 2:38 GMT+08:00 Jakob Stengård <[email protected]>: > > > > > > > I looked at the source code, but the fix wasn't obvious to me. > > > > > > > > Maybe, if i have time, i'll look into it further. > > > > > > > > /J > > > > > > > > > > > > On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu < > > [email protected]> > > > > wrote: > > > > > > > > > Thanks Jakob, > > > > > > > > > > Would be fantastic if you could dig in and submit a patch… > > > > > > > > > > > > > > > On 3/2/15, 10:08 AM, "Jakob Stengård" <[email protected]> > > > wrote: > > > > > > > > > > >I'll file a bug in jira. > > > > > >For your information i also happen to know that tez boosted > > map-reduce > > > > > >doesn't work either, but there, the problem seems to have to do > with > > > the > > > > > >map reduce jobs them selves failing. > > > > > > > > > > > >In this case however, from what i've read on git hub about the > > "Can't > > > > get > > > > > >cube source record size", > > > > > >error", it could possibly be a problem with that tez doesn't > output > > a > > > > > >string on the form: > > > > > >"HDFS Read: (\d+) HDFS Write" > > > > > > > > > > > >However, it could be something else, it could also be my logging > > > > > >configuration for tez. > > > > > > > > > > > >/J > > > > > > > > > > > > > > > > > > > > > > > >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu < > > > [email protected] > > > > > > > > > > >wrote: > > > > > > > > > > > >> Did not read your logs fully. The culprit seems to be > > > > > >> > > > > > >> > > > > > >> 1. [QuartzScheduler_Worker-7]:[2015-03-02 > > > > > >> > > > > > > > > > > > > > > > >>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti > > > > > >>on > > > > > >> (JobFlowListener.java:312)] - Can't get cube source record size. > > > > > >> > > > > > >> source_records_size may have to be computed differently when > using > > > Tez > > > > > >>as > > > > > >> the Hive engine. I will let Yang/Luke answer this when they come > > > back > > > > > >> onlineŠ > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <[email protected]> > > > wrote: > > > > > >> > > > > > >> >Jakob, > > > > > >> > > > > > > >> >Could you go ahead and file a JIRA with the stack trace at > > > > > >> >https://issues.apache.org/jira/browse/KYLIN/ > > > > > >> > > > > > > >> >Also include the App Master and Map/Reduce Task logs for the > > Tasks > > > > that > > > > > >> >failed. > > > > > >> > > > > > > >> >Regards > > > > > >> >Seshu > > > > > >> > > > > > > >> >On 3/2/15, 8:12 AM, "Jakob Stengård" <[email protected] > > > > > > wrote: > > > > > >> > > > > > > >> >>Correction: That, isn't the last job, it's step 9. > > > > > >> >> > > > > > >> >>/J > > > > > >> >> > > > > > >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård > > > > > >><[email protected] > > > > > >> > > > > > > >> >>wrote: > > > > > >> >> > > > > > >> >>> Well, this only happens with > > > > > >> >>> <property> > > > > > >> >>> <name>hive.execution.engine</name> > > > > > >> >>> <value>tez</value> > > > > > >> >>> </property> > > > > > >> >>> > > > > > >> >>> If i use mr, it works fine. > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> From what i can see form the job log, none of the previous > > > > > >>map-reduce > > > > > >> >>>jobs > > > > > >> >>> fail, but the build fails at the last step. > > > > > >> >>> Log from the last mr job for example: > > > > > >> >>> http://pastebin.com/jNWSRKLi > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> Is tez supposed to work? Because in that case, maybe it's > just > > > my > > > > > >> >>>setup. > > > > > >> >>> > > > > > >> >>> /Jakob > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh < > > > > > >> >>> [email protected]> wrote: > > > > > >> >>> > > > > > >> >>>> This doesn't seems to be complete kylin.log. Do you have > any > > > > > >>exception > > > > > >> >>>> related to MR counter in map reduce logs? If yes then this > > > needs > > > > > >>to be > > > > > >> >>>> checked because kylin will need the MR counter to calculate > > the > > > > > >>cube > > > > > >> >>>>size. > > > > > >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård > > > > > >> >>>><[email protected]> > > > > > >> >>>> wrote: > > > > > >> >>>> > > > > > >> >>>> > It breaks on the last step: "Load HFile to HBase Table": > > > > > >> >>>> > > > > > > >> >>>> > Kylin.log: > > > > > >> >>>> > http://pastebin.com/3kAQmJ3u > > > > > >> >>>> > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu > > > > > >> >>>><[email protected] > > > > > >> >>>> > > > > > > >> >>>> > wrote: > > > > > >> >>>> > > > > > > >> >>>> > > Jakob, > > > > > >> >>>> > > > > > > > >> >>>> > > Could you please send out the complete stack trace with > > Tez > > > > > >> >>>>enabled? > > > > > >> >>>> > > > > > > > >> >>>> > > Regards > > > > > >> >>>> > > Seshu > > > > > >> >>>> > > > > > > > >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" < > > > > [email protected] > > > > > > > > > > > >> >>>> wrote: > > > > > >> >>>> > > > > > > > >> >>>> > > >Hi > > > > > >> >>>> > > >Will kylin suport hive+tez in v0.7.x? > > > > > >> >>>> > > > > > > > > >> >>>> > > >I've tried using tez with hive, but currently the cube > > > build > > > > > >> >>>>fails > > > > > >> >>>> with > > > > > >> >>>> > > >"Can't get cube source record size", because kylin > > expects > > > > > >> >>>>certain > > > > > >> >>>> > console > > > > > >> >>>> > > >output from map-reduce, and tez doesn't print this. > > > > > >> >>>> > > > > > > > > >> >>>> > > >BG. > > > > > >> >>>> > > >Jakob Stengard > > > > > >> >>>> > > > > > > > >> >>>> > > > > > > > >> >>>> > > > > > > >> >>>> > > > > > >> >>> > > > > > >> >>> > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > -- Med vänlig hälsning (Best Regards) *Jakob Stengård*
