marcoabreu commented on a change in pull request #13818: [CI] Prevent timeouts when rebuilding containers with docker. URL: https://github.com/apache/incubator-mxnet/pull/13818#discussion_r246929246
########## File path: ci/docker_cache.py ########## @@ -37,7 +37,7 @@ DOCKERHUB_LOGIN_NUM_RETRIES = 5 DOCKERHUB_RETRY_SECONDS = 5 DOCKER_CACHE_NUM_RETRIES = 3 -DOCKER_CACHE_TIMEOUT_MINS = 5 +DOCKER_CACHE_TIMEOUT_MINS = 15 Review comment: Did you monitor the network and ebs metrics during that execution? Jose and I have made the experience that we have a huge download spike at the beginning (which is good), followed up by a super long hang which then results in the timeout. The problem there is that hang and increasing the timeout won't solve it but just delay the failure. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
