[
https://issues.apache.org/jira/browse/IMPALA-9165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979673#comment-16979673
]
ASF subversion and git services commented on IMPALA-9165:
---------------------------------------------------------
Commit 6dd2cfa9e0aa709e97e710e8327291021205210e in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6dd2cfa ]
IMPALA-9165: Hack to remove MasterProcWALs directory in kill-hbase.sh
Some jobs have been hanging in the testdata/bin/create-hbase.sh
script. Logs during the hang show the HBase Master is stuck
unitialized:
Master startup cannot progress, in holding-pattern until region onlined.
...
ERROR master.HMaster: Master failed to complete initialization after 900000ms.
Anecdotally, the HBase Master doesn't have this problem if we remove
the /hbase/MasterProcWALs directory in kill-hbase.sh. This patch
does exactly that. It is a hack, and we should update this code
once we know what is going on.
Testing:
- test-with-docker.py fails without this patch and passes with it
- Hand testing on my minicluster shows that this allows HBase to
restart and be consistently usable
Change-Id: Icef3d30e6b539a175e03f63fcdbfb2d4608c08fa
Reviewed-on: http://gerrit.cloudera.org:8080/14757
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Precommit jobs getting stuck in testdata/bin/create-hbase.sh
> ------------------------------------------------------------
>
> Key: IMPALA-9165
> URL: https://issues.apache.org/jira/browse/IMPALA-9165
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.4.0
> Reporter: Joe McDonnell
> Priority: Blocker
> Labels: broken-build
>
> Several of our precommit jobs have been getting stuck and cancelled with this
> log:
> {noformat}
> 01:54:19 /home/ubuntu/Impala/testdata/target
> 01:54:19 SUCCESS, data generated into /home/ubuntu/Impala/testdata/target
> 01:54:20 Executing: create-load-data.sh
> 01:54:20 Generating HBase data (logging to
> /home/ubuntu/Impala/logs/data_loading/create-hbase.log)...
> 11:23:26 FATAL: Unable to delete script file
> /tmp/jenkins4199210326564796706.sh
> 11:23:26 java.lang.InterruptedException
> 11:23:26 at java.lang.Object.wait(Native Method)
> 11:23:26 at hudson.remoting.Request.call(Request.java:177)
> 11:23:26 at hudson.remoting.Channel.call(Channel.java:956)
> 11:23:26 at hudson.FilePath.act(FilePath.java:1070)
> 11:23:26 at hudson.FilePath.act(FilePath.java:1059)
> 11:23:26 at hudson.FilePath.delete(FilePath.java:1540)
> 11:23:26 at
> hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:123)
> 11:23:26 at
> hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
> 11:23:26 at
> hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
> 11:23:26 at
> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
> 11:23:26 at hudson.model.Build$BuildExecution.build(Build.java:206)
> 11:23:26 at hudson.model.Build$BuildExecution.doRun(Build.java:163)
> 11:23:26 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
> 11:23:26 at hudson.model.Run.execute(Run.java:1818)
> 11:23:26 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 11:23:26 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 11:23:26 at hudson.model.Executor.run(Executor.java:429){noformat}
> This indicates that the job is getting stuck inside
> testdata/bin/create-hbase.sh. The logs are not available, because the job
> gets cancelled.
> Some example jobs with this problem:
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8847/]
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8762/]
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8850/]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
