[
https://issues.apache.org/jira/browse/KYLIN-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15527849#comment-15527849
]
liyang commented on KYLIN-1839:
-------------------------------
We still need to divide the patch into two parts. On one hand, the HDFS
classpath improvement is good (it may further consider FS like s3, mapr etc).
On the other hand, the cache part is not always good. While the cache is good
for FengYu's case, it could cause problem for other users who set libs at cube
level. And code is not on performance critical path, caching here won't help
overall system performance.
Suggest the cache be removed, and the classpath improvement patch can apply.
> improvement set classpath before submitting mr job
> --------------------------------------------------
>
> Key: KYLIN-1839
> URL: https://issues.apache.org/jira/browse/KYLIN-1839
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
> Affects Versions: v1.5.2
> Reporter: fengYu
> Assignee: fengYu
> Attachments:
> 0001-KYLIN-1839-support-extend-lib-from-HDFS-and-cache-tm.patch
>
>
> in setClasspath, kylin will alway find hive jars from hive dependency using
> regex, however, this will not change in one process lifetime, so I cache the
> location of tmpjars and tmpfiles.
> What is more, support extends user lib setting to hdfs path rather than only
> support local filesystem which will cause upload jars every time if
> DistributedCache do not exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
