[
https://issues.apache.org/jira/browse/IMPALA-9165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980323#comment-16980323
]
Csaba Ringhofer commented on IMPALA-9165:
-----------------------------------------
[~joemcdonnell]
I saw this issue reoccuring in a recent build:
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/1683
I found a suspicious error in HBase's zookeeper logs:
{code}
19/11/22 10:04:01 INFO server.NIOServerCnxnFactory: binding to port
0.0.0.0/0.0.0.0:2181
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:90)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:119)
at
org.apache.hadoop.hbase.zookeeper.HQuorumPeer.runZKServer(HQuorumPeer.java:94)
at
org.apache.hadoop.hbase.zookeeper.HQuorumPeer.main(HQuorumPeer.java:78)
{code}
see
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/1683/artifact/Impala/logs_static/logs/cluster/hbase/hbase-ubuntu-zookeeper-ip-172-31-26-202.out/*view*/
I also saw this in another failed run:
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/1681/artifact/Impala/logs_static/logs/cluster/hbase/hbase-ubuntu-zookeeper-ip-172-31-37-124.out/*view*/
But I didn't see it in healthy runs:
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/1682/artifact/Impala/logs_static/logs/cluster/hbase/hbase-ubuntu-zookeeper-ip-172-31-26-202.out/*view*/
In this case listening on the same port 2181 was successful.
> Precommit jobs getting stuck in testdata/bin/create-hbase.sh
> ------------------------------------------------------------
>
> Key: IMPALA-9165
> URL: https://issues.apache.org/jira/browse/IMPALA-9165
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.4.0
> Reporter: Joe McDonnell
> Priority: Blocker
> Labels: broken-build
>
> Several of our precommit jobs have been getting stuck and cancelled with this
> log:
> {noformat}
> 01:54:19 /home/ubuntu/Impala/testdata/target
> 01:54:19 SUCCESS, data generated into /home/ubuntu/Impala/testdata/target
> 01:54:20 Executing: create-load-data.sh
> 01:54:20 Generating HBase data (logging to
> /home/ubuntu/Impala/logs/data_loading/create-hbase.log)...
> 11:23:26 FATAL: Unable to delete script file
> /tmp/jenkins4199210326564796706.sh
> 11:23:26 java.lang.InterruptedException
> 11:23:26 at java.lang.Object.wait(Native Method)
> 11:23:26 at hudson.remoting.Request.call(Request.java:177)
> 11:23:26 at hudson.remoting.Channel.call(Channel.java:956)
> 11:23:26 at hudson.FilePath.act(FilePath.java:1070)
> 11:23:26 at hudson.FilePath.act(FilePath.java:1059)
> 11:23:26 at hudson.FilePath.delete(FilePath.java:1540)
> 11:23:26 at
> hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:123)
> 11:23:26 at
> hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
> 11:23:26 at
> hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
> 11:23:26 at
> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
> 11:23:26 at hudson.model.Build$BuildExecution.build(Build.java:206)
> 11:23:26 at hudson.model.Build$BuildExecution.doRun(Build.java:163)
> 11:23:26 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
> 11:23:26 at hudson.model.Run.execute(Run.java:1818)
> 11:23:26 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 11:23:26 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 11:23:26 at hudson.model.Executor.run(Executor.java:429){noformat}
> This indicates that the job is getting stuck inside
> testdata/bin/create-hbase.sh. The logs are not available, because the job
> gets cancelled.
> Some example jobs with this problem:
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8847/]
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8762/]
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8850/]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
