marcoabreu opened a new issue #11999: Corrupt docker tar in CI URL: https://github.com/apache/incubator-mxnet/issues/11999 During docker pull, the CI fails to download the docker layers: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1321/pipeline ``` build.py: 2018-08-02 07:58:49,180 3 out of 3 tries to build the docker image. build.py: 2018-08-02 07:58:49,180 Running command: 'docker build -f docker/Dockerfile.build.centos7_gpu --build-arg USER_ID=1001 --build-arg GROUP_ID=1001 --cache-from mxnetci/build.centos7_gpu -t mxnetci/build.centos7_gpu docker' Sending build context to Docker daemon 150kB Step 1/14 : FROM nvidia/cuda:9.1-cudnn7-devel-centos7 9.1-cudnn7-devel-centos7: Pulling from nvidia/cuda 7dc0dca2b151: Already exists 635f539bce28: Pulling fs layer b530d160452d: Pulling fs layer 708d351e5650: Pulling fs layer ebed533092e0: Pulling fs layer 0966b9ded8e5: Pulling fs layer d883c1c4a803: Pulling fs layer b103f26d08e3: Pulling fs layer ebed533092e0: Waiting d883c1c4a803: Waiting 635f539bce28: Verifying Checksum b530d160452d: Verifying Checksum b530d160452d: Download complete 635f539bce28: Pull complete b530d160452d: Pull complete 708d351e5650: Download complete 708d351e5650: Pull complete ebed533092e0: Verifying Checksum ebed533092e0: Download complete ebed533092e0: Pull complete 0966b9ded8e5: Verifying Checksum 0966b9ded8e5: Download complete d883c1c4a803: Download complete b103f26d08e3: Verifying Checksum b103f26d08e3: Download complete failed to register layer: Error processing tar file(exit status 1: unpigz: abort: corrupted gzip stream -- crc32 mismatch: <stdin> ): build.py: 2018-08-02 07:58:58,027 Failed to build docker image --- Logging error --- Traceback (most recent call last): File "/usr/lib/python3.5/logging/__init__.py", line 980, in emit msg = self.format(record) File "/usr/lib/python3.5/logging/__init__.py", line 830, in format return fmt.format(record) File "/usr/lib/python3.5/logging/__init__.py", line 567, in format record.message = record.getMessage() File "/usr/lib/python3.5/logging/__init__.py", line 330, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "ci/build.py", line 408, in <module> File "ci/build.py", line 329, in main File "ci/build.py", line 117, in build_docker Message: 'Exception during build of docker image' Arguments: (CalledProcessError(1, ['docker', 'build', '-f', 'docker/Dockerfile.build.centos7_gpu', '--build-arg', 'USER_ID=1001', '--build-arg', 'GROUP_ID=1001', '--cache-from', 'mxnetci/build.centos7_gpu', '-t', 'mxnetci/build.centos7_gpu', 'docker']),) build.py: 2018-08-02 07:58:58,028 Failed to build the docker image, aborting... script returned exit code 1 ``` This could be an error on our side, but I rather think that our slaves had a problematic internet connection. If you have a look at the pasted CI run, you will see that there have been other download corruptions: ``` Cloning into '3rdparty/googletest'... fatal: did not receive expected object 662a8ab6d0910bf16d3b8e673c8152b758d65cd4 fatal: index-pack failed fatal: clone of 'https://github.com/google/googletest.git' into submodule path '3rdparty/googletest' failed script returned exit code 128 ``` ``` Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://github.com/apache/incubator-mxnet.git > git init /home/jenkins_slave/workspace/build-centos7-gpu # timeout=10 Fetching upstream changes from https://github.com/apache/incubator-mxnet.git > git --version # timeout=10 using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https://github.com/apache/incubator-mxnet.git +refs/heads/master:refs/remotes/origin/master ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git fetch --no-tags --progress https://github.com/apache/incubator-mxnet.git +refs/heads/master:refs/remotes/origin/master" returned status code 128: stdout: stderr: remote: Counting objects: 66937, done. remote: Compressing objects: 4% (1/23) remote: Compressing objects: 8% (2/23) remote: Compressing objects: 13% (3/23) remote: Compressing objects: 17% (4/23) remote: Compressing objects: 21% (5/23) remote: Compressing objects: 26% (6/23) remote: Compressing objects: 30% (7/23) remote: Compressing objects: 34% (8/23) remote: Compressing objects: 39% (9/23) remote: Compressing objects: 43% (10/23) remote: Compressing objects: 47% (11/23) remote: Compressing objects: 52% (12/23) remote: Compressing objects: 56% (13/23) remote: Compressing objects: 60% (14/23) remote: Compressing objects: 65% (15/23) remote: Compressing objects: 69% (16/23) remote: Compressing objects: 73% (17/23) remote: Compressing objects: 78% (18/23) remote: Compressing objects: 82% (19/23) remote: Compressing objects: 86% (20/23) remote: Compressing objects: 91% (21/23) remote: Compressing objects: 95% (22/23) remote: Compressing objects: 100% (23/23) remote: Compressing objects: 100% (23/23), done. Receiving objects: 0% (1/66937) Receiving objects: 1% (670/66937) Receiving objects: 2% (1339/66937) Receiving objects: 3% (2009/66937) Receiving objects: 4% (2678/66937) Receiving objects: 5% (3347/66937) Receiving objects: 6% (4017/66937) Receiving objects: 7% (4686/66937) Receiving objects: 8% (5355/66937) Receiving objects: 9% (6025/66937) Receiving objects: 10% (6694/66937) Receiving objects: 11% (7364/66937) Receiving objects: 12% (8033/66937) Receiving objects: 13% (8702/66937) error: RPC failed; curl 56 GnuTLS recv error (-24): Decryption has failed. fatal: The remote end hung up unexpectedly fatal: early EOF fatal: index-pack failed at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1721) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:614) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) at java.lang.Thread.run(Thread.java:748) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-172-31-13-44.us-west-2.compute.internal/172.31.13.44:33806 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:955) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) at sun.reflect.GeneratedMethodAccessor602.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) at com.sun.proxy.$Proxy94.execute(Unknown Source) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1146) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Error cloning remote repo 'origin' ``` I imagine that we can't do anything here and that it's simply a bad slave.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
