[
https://issues.apache.org/jira/browse/BEAM-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773266#comment-16773266
]
Valentyn Tymofieiev commented on BEAM-6706:
-------------------------------------------
Error messages like "GET /v1.27/ ... json returned error: No such image:
gcr.io/cloud-dataflow/v1beta3/... " are a common Dataflow red herring, they
often appear when an image that is referenced in the log is not yet cached in
local Docker repository in Dataflow worker VM, and was not yet pulled from
external repository (gcr.io). However, in most cases, after seeing this error,
Dataflow worker will fetch the image from GCR and pipeline resumes.
It will be a common and, unfortunately, misleading message that users may see
when migrating to a new Beam SDK, since it takes some time for Dataflow workers
to pick up container images used by most recent Beam SDK. However in most cases
this is not an error.
To confirm that this is indeed a red herring and not a permanent error we can
run a docker command to pull this image ourselves:
{noformat}
$ docker pull gcr.io/cloud-dataflow/v1beta3/beam-java-batch:beam-2.10.0
...
Digest: sha256:ca623baad176a04dcdfd77e7524f1b15f0ab75b415351617f11bac6dffb49230
Status: Downloaded newer image for
gcr.io/cloud-dataflow/v1beta3/beam-java-batch:beam-2.10.0
{noformat}
In rare cases such as when network on Dataflow workers is restricted or there
is GCR.io outage, this error will cause the pipeline to fail. However in most
cases, pipeline fails for some other reason.
> User reports trouble downloading 2.10.0 Dataflow worker image
> -------------------------------------------------------------
>
> Key: BEAM-6706
> URL: https://issues.apache.org/jira/browse/BEAM-6706
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Reporter: Kenneth Knowles
> Assignee: Kenneth Knowles
> Priority: Blocker
>
> DataFlow however is throwing all sorts of errors. For example:
> * Handler for GET
> /v1.27/images/gcr.io/cloud-dataflow/v1beta3/beam-java-batch:beam-2.10.0/json
> returned error: No such image:
> gcr.io/cloud-dataflow/v1beta3/beam-java-batch:beam-2.10.0"
> * while reading 'google-dockercfg' metadata: http status code: 404 while
> fetching url
> http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg";
> * Error syncing pod..."
> The job gets stuck after starting a worker and after an hour or so it gives
> up with a failure. 2.9.0 runs fine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
