[ https://issues.apache.org/jira/browse/IMPALA-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628286#comment-16628286 ]
ASF subversion and git services commented on IMPALA-7624: --------------------------------------------------------- Commit 91673fee607b552f142c6ab2aad0e96efa9e0f80 in impala's branch refs/heads/master from [~philip] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=91673fe ] IMPALA-7624: Workaround docker/kernel bug causing test-with-docker to sometimes hang. I've observed that builds of test-with-docker that have "suite parallelism" sometimes hang when the Docker containers are being created. (The implementation had multiple threads calling "docker create" simultaneously.) Trolling the mailing lists, it's maybe a bug in Docker or the kernel. I've never caught it live enough to strace it. A hopeful workaround is to serialize the docker create calls, which is easy and harmless, given that "docker create" is usually pretty quick (subsecond) and the overall run time here is hours+. With this change, I was able to run test-with-docker with --suite-concurrency=6 on a c5.9xlarge in AWS, with a total runtime of 1h35m. The hangs are intermittent and cause, in the typical case, inconsistency in runtimes because less parallelism happens when one of the "docker create" calls hang. (I've seen them resume after one of the other containers finishes.) We'll find out with time whether this stabilizes it or has no effect. Change-Id: I3e44db7a6ce08a42d6fe574d7348332578cd9e51 Reviewed-on: http://gerrit.cloudera.org:8080/11481 Reviewed-by: Philip Zeyliger <phi...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > test-with-docker sometimes hangs creating docker containers > ----------------------------------------------------------- > > Key: IMPALA-7624 > URL: https://issues.apache.org/jira/browse/IMPALA-7624 > Project: IMPALA > Issue Type: Task > Reporter: Philip Zeyliger > Priority: Major > > I've seen the test-with-docker executions hang, or sort of hang, in threads > doing {{docker create}}. I think this is ultimately a Docker or kernel bug, > but we can work around it by serializing our "docker create" invocations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org