[
https://issues.apache.org/jira/browse/KYLIN-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381978#comment-17381978
]
Gabor Arki edited comment on KYLIN-5022 at 7/16/21, 10:44 AM:
--------------------------------------------------------------
The root cause seems to be that Kylin is creating and configuring a unique jar
per HBase table thus HBase region servers are downloading this 5.5M jar for
each table separately. In our case, the ~11000 tables result in 50+GB of space
needed on our HBase region servers.
To make this issue worse, it seems over time HBase is starting to delete these
jars (maybe when a table is cleaned up, maybe it does so occasionally anyway).
But given the HBase region server process is continuing to run, the disk space
occupied by these deleted jars is not freed up either unless the region server
is shut down. Only then are these deleted files released and removed from the
disk.
was (Author: arkigabor):
The root cause seems to be that Kylin is creating and configuring a unique jar
per HBase table thus HBase region servers are downloading this 5.5M jar for
each table separately. In our case, the ~11000 tables result in 50+GB of space
needed on our HBase region servers.
To make this issue worse, it seems over time HBase is starting to delete these
jars (maybe when a table is cleaned up, maybe it does so occasionally anyway).
But given the HBase region server process is continuing to run, the disk space
occupied by these deleted jars is not freed up unless the region server is shut
down. Only then are these deleted files released and removed from the disk.
> kylin升级新版本-/mnt/tmp/hbase-hbase/local/jars/tmp产生大量的kylin-coprocessor文件
> ----------------------------------------------------------------------
>
> Key: KYLIN-5022
> URL: https://issues.apache.org/jira/browse/KYLIN-5022
> Project: Kylin
> Issue Type: Bug
> Components: Metadata, Storage - HBase
> Affects Versions: v3.1.1
> Reporter: star_dev
> Priority: Major
> Attachments: Capture.PNG, 屏幕快照1.png, 屏幕快照2.png, 日志.log
>
>
> kylin版本从3.0.2更新到3.1.1，还是用原来的元数据。
> 发现在EMR的core节点中有大量的kylin-coprocessor文件生成，见附件屏幕快照1，占用了大量的空间，导致hdfs文件系统可用空间变少。路径为/mnt/tmp/hbase-hbase/local/jars/tmp
> 查询官方文档 [http://kylin.apache.org/docs/howto/howto_update_coprocessor.html]
> 执行如下命令仍然不好用，日志信息见附件
> -------------------------
>
> {{$KYLIN_HOME/bin/kylin.sh
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI default all}}
> {{-------------------------}}
> 同时发现kylin元数据
> kylin_metadata/coprocessor/下有大量的kylin-coprocessor-3.1.1-*.jar文件，见附件屏幕快照2
>
> 是什么原因导致的这种现象？
> 如何才能在/mnt/tmp/hbase-hbase/local/jars/tmp路径下不再产生大量的文件？
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
