Hi hongbin, Thanks for pointing out!
Have you ported it to 0.7-staging, anything I can help? Best, Dayue > 在 2015年8月4日，下午12:18，hongbin ma <[email protected]> 写道： > > hi, dayue, > > what you described is in short re-generate the binary package and deploy > the new one, this is not our typical development style. For developers we > suggest following > http://kylin.incubator.apache.org/docs/development/dev_env.html to setup a > dev env and run tests after code modification. > > The issue I reported yesterday is a bug raised when there is no fact column > uses dictionary.(test cube test_kylin_cube_with_slr_empty as an example, > all of the fact table columns are FK, which will in term uses dictionary on > lookup table's PK, thus bypassing fact table column values collecting.) > > when factDictColRowKeyIndexes is empty, the follow code will somehow fail > to set property into job's conf.(Maybe MR automatically skips empty > strings) > > > *jobConf.set(BatchConstants.CFG_FACT_DICT_COLUMN_ROWKEY_INDEXES, > joiner.join(factDictColRowKeyIndexes)); * > > so we just need to double check the property at mapper/reducer side > as 07db22536657dfb278dfd29f616789560b208948 did. > > > On Tue, Aug 4, 2015 at 8:33 AM, Dayue Gao <[email protected]> wrote: > >> Hi Hongbin, >> >> Not sure how your CI is configured, but the NPE problem indeed occurred >> when I tested the patch using the following workflow (took me quite a long >> time to figure it out). Let's say my kylin binary is deployed under >> $DIR/kylin-mt-v0.7.2.1 >> >> 1. stop the old instance. "$DIR/kylin-mt-v0.7.2.1/bin/kylin.sh stop" >> 2. generate a new tarball using './script/package.sh', put into $DIR >> 3. "replace" the binary. "cd $DIR; tar xzf kylin-mt-v0.7.2.1.tar.gz" >> 4. start the new instance and build the cube >> >> The problem is >> $DIR/kylin-mt-v0.7.2.1/tomcat/webapp/kylin/WEB-INF/lib/kylin-job-mt-v0.7.2.1.jar >> is still the old one, kylin web use this jar to submit job (without new >> configuration params) while reduce task use lib/kylin-job-mt-v0.7.2.1.jar >> (the new one) to read these two params, which results in NPE. >> >> The fix is to remove $DIR/kylin-mt-v0.7.2.1/tomcat/webapp/kylin before >> step 2. >> >> Best, >> Dayue >> >>> 在 2015年8月3日，下午9:40，hongbin ma <[email protected]> 写道： >>> >>> After I applied the patch on 0.7 staging (commit >>> 388488be80d24ecb4d3209726c90a1b829a18087), CI failed to due >>> to FactDistinctColumnsJob fail. The error message is >>> https://gist.github.com/binmahone/9a6d8bdff37d94f59dea. >>> >>> I checked the job's configuration at job history page, seems the >>> property "fact.dict.column.rowkey.indexes" and "fact.dict.column.names" >>> not successfully set into job's configuration. >>> >>> I'm looking into the issue, any comments is welcomed. >>> >>> On Mon, Aug 3, 2015 at 9:32 PM, Dayue Gao (JIRA) <[email protected]> >> wrote: >>> >>>> >>>> [ >>>> >> https://issues.apache.org/jira/browse/KYLIN-921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14651856#comment-14651856 >>>> ] >>>> >>>> Dayue Gao commented on KYLIN-921: >>>> --------------------------------- >>>> >>>> Thanks Hongbin, Shaofeng! Glad to know it has been fixed in 0.7.3 :-) >>>> >>>> >>>>> Dimension with all nulls cause BuildDimensionDictionary failed due to >>>> FileNotFoundException >>>>> >>>> >> ------------------------------------------------------------------------------------------- >>>>> >>>>> Key: KYLIN-921 >>>>> URL: https://issues.apache.org/jira/browse/KYLIN-921 >>>>> Project: Kylin >>>>> Issue Type: Bug >>>>> Components: Job Engine >>>>> Affects Versions: v0.7.2 >>>>> Reporter: Dayue Gao >>>>> Assignee: ZhouQianhao >>>>> Fix For: v0.7.3 >>>>> >>>>> Attachments: KYLIN-921.patch >>>>> >>>>> >>>>> From mailing list >>>>> ---------------------- >>>>> {noformat} >>>>> I am building a cube with some lookup table in between and getting >>>>> exception at third step of cube build i.e Build Dimension Dictionary >> with >>>>> exception saying >>>>> java.io.FileNotFoundException: File does not exist: >>>>> >>>> >> /tmp/kylin-5a2ea405-24a2-45ed-958e-2a7fddd8cc97/sc_o2s_metrics_verified123455/fact_distinct_columns/SC >>>>> at >>>>> >>>> >> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093) >>>>> at >>>>> >>>> >> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) >>>>> at >>>>> >>>> >> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) >>>>> at >>>>> >>>> >> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) >>>>> at >> org.apache.kylin.dict.lookup.FileTable.getSignature(FileTable.java:62) >>>>> at >>>>> >>>> >> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:164) >>>>> at >>>> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:154) >>>>> at >>>>> >>>> >> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:53) >>>>> at >>>>> >>>> >> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42) >>>>> at >>>>> >>>> >> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:53) >>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>>>> at >>>>> >>>> >> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) >>>>> at >>>>> >>>> >> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) >>>>> at >>>>> >>>> >> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) >>>>> at >>>>> >>>> >> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) >>>>> at >>>>> >>>> >> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132) >>>>> at >>>>> >>>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> >>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> {noformat} >>>>> The problem is that FactDistinctColumnsMapper's map method skips null >>>> values. As a result, if all values of dimension 'x' are null, >>>> FactDistinctColumnsReducer will not create file for 'x', thereafter the >>>> following job throws FileNotFoundException. >>>>> {code:title=FactDistinctColumnsMapper.java|borderStyle=solid} >>>>> public void map(KEYIN key, HCatRecord record, Context context) throws >>>> IOException, InterruptedException { >>>>> try { >>>>> // code ommited ... >>>>> for (int i : factDictCols) { >>>>> outputKey.set((short) i); >>>>> fieldSchema = schema.get(flatTableIndexes[i]); >>>>> Object fieldValue = record.get(fieldSchema.getName(), >>>> schema); >>>>> // NULL VALUE IS SKIPPED >>>>> if (fieldValue == null) >>>>> continue; >>>>> // code ommited ... >>>>> } >>>>> } catch (Exception ex) { >>>>> handleErrorRecord(record, ex); >>>>> } >>>>> } >>>>> {code} >>>> >>>> >>>> >>>> -- >>>> This message was sent by Atlassian JIRA >>>> (v6.3.4#6332) >>>> >>> >>> >>> >>> -- >>> Regards, >>> >>> *Bin Mahone | 马洪宾* >>> Apache Kylin: http://kylin.io >>> Github: https://github.com/binmahone >> >> >> > > > -- > Regards, > > *Bin Mahone | 马洪宾* > Apache Kylin: http://kylin.io > Github: https://github.com/binmahone
