From 0.7.1 Kylin support tables from non-default database, it strictly require the table name in cube definition be <db_name>.<table_name>; If your table is in default database, please append “default.” ahead of the table name.
I assume you’re upgrading from 0.6.x to 0.7.1; The metadata or say cubes created in 0.6.x couldn’t be parsed correctly in 0.7.1; We suggest you re-create the cubes; You can drop the old metadata table “kylin_metadata_qa" in hbase or rename it to another name, so Kylin will not load the old data anymore; Kylin will automatically create the htable if it doesn’t exist; On 3/4/15, 5:07 PM, "Jakob Stengård" <[email protected]> wrote: >I tried installing 0.7.1, but i get "The 'partition_date_column' format is >invalid: , it should be {db}.{table}.{column}." >when trying to set up a cube. > >Dunno if it's my tables that it doesn't like or if it's some deprecated >settings from 0.6.5 that causes this. >Tried deleting all Hbase tables, but i still get the error after >restarting >Kylin and re-syncing the tables. > >/J > >On Tue, Mar 3, 2015 at 9:24 AM, Li Yang <[email protected]> wrote: > >> I briefly looked into the code and seems the issue has been fixed in >>0.7.x >> (the current inverted-index branch) already. Basically, it does not >>rely on >> counters of the hive step any more. >> >> However I don't have the env to verify this. @Jakob, may try this latest >> version http://kylin.incubator.apache.org/download/ >> >> On Tue, Mar 3, 2015 at 6:59 AM, Jakob Stengård >><[email protected]> >> wrote: >> >> > Yeah, i know that. >> > Here is the jira bug i filed: >> > https://issues.apache.org/jira/browse/KYLIN-619 >> > >> > On Mon, Mar 2, 2015 at 11:47 PM, Luke Han <[email protected]> wrote: >> > >> > > Hi Jakob, did you create JIRA for this already? >> > > >> > > Tez itself should not the problem but just some statistics >>information >> > > Kylin can't parse as you figure out already. >> > > >> > > Please let me know the JIRA number and we will enhance that to avoid >> such >> > > issue again. >> > > >> > > Thanks. >> > > Luke >> > > >> > > >> > > Best Regards! >> > > --------------------- >> > > >> > > Luke Han >> > > >> > > 2015-03-03 2:38 GMT+08:00 Jakob Stengård <[email protected]>: >> > > >> > > > I looked at the source code, but the fix wasn't obvious to me. >> > > > >> > > > Maybe, if i have time, i'll look into it further. >> > > > >> > > > /J >> > > > >> > > > >> > > > On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu < >> > [email protected]> >> > > > wrote: >> > > > >> > > > > Thanks Jakob, >> > > > > >> > > > > Would be fantastic if you could dig in and submit a patch… >> > > > > >> > > > > >> > > > > On 3/2/15, 10:08 AM, "Jakob Stengård" <[email protected]> >> > > wrote: >> > > > > >> > > > > >I'll file a bug in jira. >> > > > > >For your information i also happen to know that tez boosted >> > map-reduce >> > > > > >doesn't work either, but there, the problem seems to have to do >> with >> > > the >> > > > > >map reduce jobs them selves failing. >> > > > > > >> > > > > >In this case however, from what i've read on git hub about the >> > "Can't >> > > > get >> > > > > >cube source record size", >> > > > > >error", it could possibly be a problem with that tez doesn't >> output >> > a >> > > > > >string on the form: >> > > > > >"HDFS Read: (\d+) HDFS Write" >> > > > > > >> > > > > >However, it could be something else, it could also be my >>logging >> > > > > >configuration for tez. >> > > > > > >> > > > > >/J >> > > > > > >> > > > > > >> > > > > > >> > > > > >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu < >> > > [email protected] >> > > > > >> > > > > >wrote: >> > > > > > >> > > > > >> Did not read your logs fully. The culprit seems to be >> > > > > >> >> > > > > >> >> > > > > >> 1. [QuartzScheduler_Worker-7]:[2015-03-02 >> > > > > >> >> > > > > >> > > > >> > > >> > >> >>>>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcep >>>>ti >> > > > > >>on >> > > > > >> (JobFlowListener.java:312)] - Can't get cube source record >>size. >> > > > > >> >> > > > > >> source_records_size may have to be computed differently when >> using >> > > Tez >> > > > > >>as >> > > > > >> the Hive engine. I will let Yang/Luke answer this when they >>come >> > > back >> > > > > >> onlineŠ >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" >><[email protected]> >> > > wrote: >> > > > > >> >> > > > > >> >Jakob, >> > > > > >> > >> > > > > >> >Could you go ahead and file a JIRA with the stack trace at >> > > > > >> >https://issues.apache.org/jira/browse/KYLIN/ >> > > > > >> > >> > > > > >> >Also include the App Master and Map/Reduce Task logs for the >> > Tasks >> > > > that >> > > > > >> >failed. >> > > > > >> > >> > > > > >> >Regards >> > > > > >> >Seshu >> > > > > >> > >> > > > > >> >On 3/2/15, 8:12 AM, "Jakob Stengård" >><[email protected] >> > >> > > > wrote: >> > > > > >> > >> > > > > >> >>Correction: That, isn't the last job, it's step 9. >> > > > > >> >> >> > > > > >> >>/J >> > > > > >> >> >> > > > > >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård >> > > > > >><[email protected] >> > > > > >> > >> > > > > >> >>wrote: >> > > > > >> >> >> > > > > >> >>> Well, this only happens with >> > > > > >> >>> <property> >> > > > > >> >>> <name>hive.execution.engine</name> >> > > > > >> >>> <value>tez</value> >> > > > > >> >>> </property> >> > > > > >> >>> >> > > > > >> >>> If i use mr, it works fine. >> > > > > >> >>> >> > > > > >> >>> >> > > > > >> >>> From what i can see form the job log, none of the >>previous >> > > > > >>map-reduce >> > > > > >> >>>jobs >> > > > > >> >>> fail, but the build fails at the last step. >> > > > > >> >>> Log from the last mr job for example: >> > > > > >> >>> http://pastebin.com/jNWSRKLi >> > > > > >> >>> >> > > > > >> >>> >> > > > > >> >>> Is tez supposed to work? Because in that case, maybe it's >> just >> > > my >> > > > > >> >>>setup. >> > > > > >> >>> >> > > > > >> >>> /Jakob >> > > > > >> >>> >> > > > > >> >>> >> > > > > >> >>> >> > > > > >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh < >> > > > > >> >>> [email protected]> wrote: >> > > > > >> >>> >> > > > > >> >>>> This doesn't seems to be complete kylin.log. Do you have >> any >> > > > > >>exception >> > > > > >> >>>> related to MR counter in map reduce logs? If yes then >>this >> > > needs >> > > > > >>to be >> > > > > >> >>>> checked because kylin will need the MR counter to >>calculate >> > the >> > > > > >>cube >> > > > > >> >>>>size. >> > > > > >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård >> > > > > >> >>>><[email protected]> >> > > > > >> >>>> wrote: >> > > > > >> >>>> >> > > > > >> >>>> > It breaks on the last step: "Load HFile to HBase >>Table": >> > > > > >> >>>> > >> > > > > >> >>>> > Kylin.log: >> > > > > >> >>>> > http://pastebin.com/3kAQmJ3u >> > > > > >> >>>> > >> > > > > >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu >> > > > > >> >>>><[email protected] >> > > > > >> >>>> > >> > > > > >> >>>> > wrote: >> > > > > >> >>>> > >> > > > > >> >>>> > > Jakob, >> > > > > >> >>>> > > >> > > > > >> >>>> > > Could you please send out the complete stack trace >>with >> > Tez >> > > > > >> >>>>enabled? >> > > > > >> >>>> > > >> > > > > >> >>>> > > Regards >> > > > > >> >>>> > > Seshu >> > > > > >> >>>> > > >> > > > > >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" < >> > > > [email protected] >> > > > > > >> > > > > >> >>>> wrote: >> > > > > >> >>>> > > >> > > > > >> >>>> > > >Hi >> > > > > >> >>>> > > >Will kylin suport hive+tez in v0.7.x? >> > > > > >> >>>> > > > >> > > > > >> >>>> > > >I've tried using tez with hive, but currently the >>cube >> > > build >> > > > > >> >>>>fails >> > > > > >> >>>> with >> > > > > >> >>>> > > >"Can't get cube source record size", because kylin >> > expects >> > > > > >> >>>>certain >> > > > > >> >>>> > console >> > > > > >> >>>> > > >output from map-reduce, and tez doesn't print this. >> > > > > >> >>>> > > > >> > > > > >> >>>> > > >BG. >> > > > > >> >>>> > > >Jakob Stengard >> > > > > >> >>>> > > >> > > > > >> >>>> > > >> > > > > >> >>>> > >> > > > > >> >>>> >> > > > > >> >>> >> > > > > >> >>> >> > > > > >> > >> > > > > >> >> > > > > >> >> > > > > >> > > > > >> > > > >> > > >> > >> > > > >-- > >Med vänlig hälsning (Best Regards) >*Jakob Stengård*
