[
https://issues.apache.org/jira/browse/KYLIN-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-1985.
---------------------------------
Resolution: Fixed
> SnapshotTable should only keep the columns described in tableDesc
> -----------------------------------------------------------------
>
> Key: KYLIN-1985
> URL: https://issues.apache.org/jira/browse/KYLIN-1985
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v1.5.3
> Reporter: zhengdong
> Assignee: zhengdong
> Fix For: v1.5.4
>
> Attachments: KYLIN-1985.patch
>
>
> we suffered from a strange problem that we got a
> java.lang.ArrayIndexOutOfBoundsException when build of refresh a cube,
> exception stack like this :
> java.lang.IllegalStateException: Failed to load lookup table DIM_TABLE_NAME
> from snapshot
> /table_snapshot/dim_table_name/5a78a522-6f85-4650-b47d-6a5f5806b7f7.snapshot
> at
> org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:621)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:61)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
> at
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:127)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 19
> at
> org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:85)
> at
> org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:34)
> at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:76)
> at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:67)
> at
> org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79)
> at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:55)
> at
> org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65)
> at
> org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:619)
> ... 13 more
> and a simple exception when queried by a lookup table dimension
>
> ERROR [http-bio-7070-exec-7] controller.QueryController:209 : Exception when
> execute sql
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:143)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:186)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:366)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:278)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:121)
> at
> org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke(<generated>)
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
> at
> org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$315e2079.query(<generated>)
> at
> org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 19
> at
> org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:85)
> at
> org.apache.kylin.dict.lookup.LookupStringTable.convertRow(LookupStringTable.java:34)
> at
> org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:76)
> at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:67)
> at
> org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79)
> at
> org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:55)
> at
> org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65)
> at
> org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:619)
> Though the exception message, we found that one lookup table had been
> changed in hive (add columns) and not been synchronized with kylin. However,
> the cause of this problem is too subtle and not easily found.
> As for SnapshotTable, only checking 'row.length <= maxIndex' in
> takeSnapshot method to detect 'Bad hive table row ' is not enough. And only
> encode and store the data columns described in tableDesc could be better
> since these columns not in tableDesc are not used in any cube definition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
