[
https://issues.apache.org/jira/browse/KYLIN-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952599#comment-16952599
]
wangrupeng edited comment on KYLIN-4194 at 10/21/19 1:52 AM:
-------------------------------------------------------------
In KYLIN-4012, using CacheBuilder from Guava to init objects--LoadingCache<T,
Integer> valueToIdCache. And it get config using
KylinConfig.getInstanceFromEnv(). But spark job will throw an Exception because
it will use CacheDictionary.enableCache() and spark cluater cannot get kylin
configuration through KylinConfig.getInstanceFromEnv();
!image-2019-10-16-16-13-00-190.png|width=741,height=358!
was (Author: wangrupeng):
在KYLIN-4012中，改用了Guava包中的CacheBuilder工具来初始化对象 LoadingCache<T, Integer>
valueToIdCache 。并通过KylinConfig.getInstanceFromEnv()方法来获取配置项。在Spark
job中构建字典的步骤会调用CacheDictionary.enableCache方法，由于spark任务需要的kylin
metadata是通过提交spark任务之前将medata复制到hdfs中获取的，最终运行在yarn集群中，所以KylinConfig.getInstanceFromEnv()无法获取kylin
config。
!image-2019-10-16-16-13-00-190.png|width=741,height=358!
> Throw KylinConfigCannotInitException at STEP "Extract Fact Table Distinct
> Columns" with spark
> ---------------------------------------------------------------------------------------------
>
> Key: KYLIN-4194
> URL: https://issues.apache.org/jira/browse/KYLIN-4194
> Project: Kylin
> Issue Type: Bug
> Affects Versions: v3.0.0-beta
> Reporter: wangrupeng
> Assignee: wangrupeng
> Priority: Major
> Attachments: image-2019-10-16-16-13-00-190.png, kylin_spark.log
>
>
> When I use spark to extract fact table distinct columns values, kylin throw
> an exception,
> org.apache.kylin.common.KylinConfigCannotInitException: Didn't find
> KYLIN_CONF or KYLIN_HOME, please set one of them,
> see more details in [^kylin_spark.log] ^^ . I tried cdh5.7 and hdp2.4, both
> platforms got this problem.
> It seems like an environment variables issue, but when I set KYLIN_CONF or
> KYLIN_HOME on each yarn cluster node, it still not work.
> As we can see from the log, this exception occurs when TrieDictionaryBuilder
> try to build Dictionary<String> objects, but cannot find KylinConfig using
> KylinConfig.getInstanceFromEnv() method.
> My solution is to add code in spark job by reset
> KylinConfig.THREAD_ENV_INSTANCE
> from hdfs metadata as the spark job's metadata is stored in hdfs.And finally
> it works fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
