[
https://issues.apache.org/jira/browse/KYLIN-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900042#comment-16900042
]
ASF GitHub Bot commented on KYLIN-4107:
---------------------------------------
nichunen commented on pull request #776: KYLIN-4107 Fix bug of
StorageCleanupJob fails to delete Hive tables with "Argument list too long"
error
URL: https://github.com/apache/kylin/pull/776
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> StorageCleanupJob fails to delete Hive tables with "Argument list too long"
> error
> ---------------------------------------------------------------------------------
>
> Key: KYLIN-4107
> URL: https://issues.apache.org/jira/browse/KYLIN-4107
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
> Affects Versions: v2.6.2
> Environment: CentOS 7.6, HDP 2.6.5, Kylin 2.6.3
> Reporter: Vsevolod Ostapenko
> Assignee: weibin0516
> Priority: Major
> Fix For: v3.0.0-beta
>
>
> On a system with multiple Kylin developers that experiment with cube design
> and (re)build/drop cube segments often intermediate Hive tables and HBase
> left over tables accumulate very quickly.
> After a certain point storage cleanup cannot be executed using suggested
> method:
> {{${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete
> true}}
> Apparently, storage cleanup job creates a single shell command to drop all
> Hive tables, which fails to execute because command line is just too long.
> For example:
> {quote}
> 2019-07-23 17:47:31,611 ERROR [main] job.StorageCleanupJob:377 : Error during
> deleting Hive tables
> java.io.IOException: Cannot run program "/bin/bash": error=7, Argument list
> too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at
> org.apache.kylin.common.util.CliCommandExecutor.runNativeCommand(CliCommandExecutor.java:133)
> at
> org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:89)
> at
> org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:83)
> at
> org.apache.kylin.rest.job.StorageCleanupJob.deleteHiveTables(StorageCleanupJob.java:409)
> at
> org.apache.kylin.rest.job.StorageCleanupJob.cleanUnusedIntermediateHiveTableInternal(StorageCleanupJob.java:375)
> at
> org.apache.kylin.rest.job.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:278)
> at
> org.apache.kylin.rest.job.StorageCleanupJob.cleanup(StorageCleanupJob.java:151)
> at
> org.apache.kylin.rest.job.StorageCleanupJob.execute(StorageCleanupJob.java:145)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> at org.apache.kylin.tool.StorageCleanupJob.main(StorageCleanupJob.java:27)
> Caused by: java.io.IOException: error=7, Argument list too long
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 10 more
> {quote}
> Instead of composing one long command, storage cleanup need to generate a
> script and feed that into beeline or hive CLI.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
