Bad Elasticsearch queries

[Alex Wauck]

When I click the "View archive" link for a pod's logs, I get a Kibana page
with a query like this:

kubernetes_pod_name: some-pod-name && kubernetes_namespace_name:
random-namespace

Am I missing something, or should it instead be this:

kubernetes_pod_name: "some-pod-name" && kubernetes_namespace_name:
"random-namespace"

Seems like a bug to me.  I noticed this after clicking the "View archive"
link for a build and getting a lot of log messages from random other pods
in other namespaces.  I guess the current way works fine if you don't have
hyphens in any names anywhere.

-- 

Alex Wauck // DevOps Engineer

*E X O S I T E*
*www.exosite.com *

Making Machines More Human.
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Problem authenticating to private docker registry

[Tony Saxon]

[root@os-node1 ~]# docker pull
docker-lab.example.com:5000/testwebapp@sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
Trying to pull repository docker-lab.example.com:5000/testwebapp ...
manifest unknown: manifest unknown



On Wed, Aug 10, 2016 at 2:07 PM, Andy Goldstein  wrote:

> Tony, can you show the output when you try to manually 'docker pull'?
>
> On Wed, Aug 10, 2016 at 2:04 PM, Cesar Wong  wrote:
>
>> Hmm, I didn't know the issue existed between 1.10 and 1.12 as well.
>>
>> Andy, what would you recommend?
>>
>>
>> On Aug 10, 2016, at 1:58 PM, Tony Saxon  wrote:
>>
>> Ok, maybe that is the issue. I can not do the docker pull referencing the
>> sha256 hash on the node.
>>
>> The docker version running on the node is docker 1.10.3, and the docker
>> version on the machine that pushed the image is 1.12.0. Is there a
>> potential workaround for this, or do I need to get the docker version
>> updated on the nodes? For reference, I installed the openshift platform
>> using the ansible advanced installation referenced in the documentation.
>>
>> On Wed, Aug 10, 2016 at 1:46 PM, Cesar Wong  wrote:
>>
>>> Tony,
>>>
>>> The only other time that I've seen the manifest not found error was when
>>> there was a version mismatch between the Docker version that pushed the
>>> image vs the version that was consuming the image (ie. images pushed with
>>> Docker 1.9 and pulled with Docker 1.10). Are you able to pull the image
>>> spec directly from your node using the Docker cli?
>>>
>>> $ docker pull docker-lab.example.com:5000/testwebapp@sha256:9799a25cd
>>> 6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
>>>
>>> On Aug 10, 2016, at 1:02 PM, Tony Saxon  wrote:
>>>
>>> I'm not sure if this has anything to do with it, but I looked at the
>>> details of the imagestream that I imported and see that it has this as the
>>> docker image reference:
>>>
>>> status:
>>>   dockerImageRepository: 172.30.11.167:5000/testwebapp/testwebapp
>>>   tags:
>>>   - items:
>>> - created: 2016-08-10T13:26:01Z
>>>   dockerImageReference: docker-lab.example.com:5000/te
>>> stwebapp@sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb2
>>> 2f687a5b8a3ed2bf9ec3
>>>   generation: 1
>>>   image: sha256:9799a25cd6fd7f7908bad74
>>> 0fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
>>> tag: latest
>>>
>>> I also see these errors show up on the docker registry when I try to
>>> deploy the app:
>>>
>>> time="2016-08-10T16:58:26Z" level=warning msg="error authorizing
>>> context: basic authentication challenge for realm \"Registry Realm\":
>>> invalid authorization credential" go.version=go1.6.3 http.request.host="
>>> docker-lab.evolveip.net:5000" 
>>> http.request.id=ecce6c57-6273-42d6-b7a9-441877c0338f
>>> http.request.method=GET http.request.remoteaddr="192.168.122.156:35858"
>>> http.request.uri="/v2/" http.request.useragent="docker/1.10.3
>>> go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64
>>> os/linux arch/amd64" instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6
>>> version=v2.5.0
>>> 192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET /v2/ HTTP/1.1" 401
>>> 87 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported
>>> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
>>> time="2016-08-10T16:58:26Z" level=error msg="response completed with
>>> error" auth.user.name=maven err.code="manifest unknown"
>>> err.detail="unknown manifest name=testwebapp revision=sha256:9799a25cd6fd7f
>>> 7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>>> err.message="manifest unknown" go.version=go1.6.3 http.request.host="
>>> docker-lab.evolveip.net:5000" 
>>> http.request.id=b994a477-6beb-4908-8589-c051b9048e87
>>> http.request.method=GET http.request.remoteaddr="192.168.122.156:35860"
>>> http.request.uri="/v2/testwebapp/manifests/sha256:9799a25cd6
>>> fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>>> http.request.useragent="docker/1.10.3 go/go1.4.2
>>> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64
>>> os/linux arch/amd64" http.response.contenttype="application/json;
>>> charset=utf-8" http.response.duration=6.04215ms
>>> http.response.status=404 http.response.written=186 instance.id
>>> =f0d70491-6e34-44eb-a51c-3b13eae8daa6 vars.name=testwebapp
>>> vars.reference="sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>>> version=v2.5.0
>>> 192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET
>>> /v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c
>>> 85823e38aa22afb22f687a5b8a3ed2bf9ec3 HTTP/1.1" 404 186 ""
>>> "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported
>>> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
>>>
>>> So it looks like the manifest isn't found, or am I misunderstanding that?
>>>
>>> The imagestream was imported by simply:
>>>
>>> [root@os-master ~]# oc import-image testwebapp --confirm 

Re: Problem authenticating to private docker registry

[Andy Goldstein]

Tony, can you show the output when you try to manually 'docker pull'?

On Wed, Aug 10, 2016 at 2:04 PM, Cesar Wong  wrote:

> Hmm, I didn't know the issue existed between 1.10 and 1.12 as well.
>
> Andy, what would you recommend?
>
>
> On Aug 10, 2016, at 1:58 PM, Tony Saxon  wrote:
>
> Ok, maybe that is the issue. I can not do the docker pull referencing the
> sha256 hash on the node.
>
> The docker version running on the node is docker 1.10.3, and the docker
> version on the machine that pushed the image is 1.12.0. Is there a
> potential workaround for this, or do I need to get the docker version
> updated on the nodes? For reference, I installed the openshift platform
> using the ansible advanced installation referenced in the documentation.
>
> On Wed, Aug 10, 2016 at 1:46 PM, Cesar Wong  wrote:
>
>> Tony,
>>
>> The only other time that I've seen the manifest not found error was when
>> there was a version mismatch between the Docker version that pushed the
>> image vs the version that was consuming the image (ie. images pushed with
>> Docker 1.9 and pulled with Docker 1.10). Are you able to pull the image
>> spec directly from your node using the Docker cli?
>>
>> $ docker pull docker-lab.example.com:5000/testwebapp@sha256:9799a25cd
>> 6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
>>
>> On Aug 10, 2016, at 1:02 PM, Tony Saxon  wrote:
>>
>> I'm not sure if this has anything to do with it, but I looked at the
>> details of the imagestream that I imported and see that it has this as the
>> docker image reference:
>>
>> status:
>>   dockerImageRepository: 172.30.11.167:5000/testwebapp/testwebapp
>>   tags:
>>   - items:
>> - created: 2016-08-10T13:26:01Z
>>   dockerImageReference: docker-lab.example.com:5000/te
>> stwebapp@sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb2
>> 2f687a5b8a3ed2bf9ec3
>>   generation: 1
>>   image: sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8
>> a3ed2bf9ec3
>> tag: latest
>>
>> I also see these errors show up on the docker registry when I try to
>> deploy the app:
>>
>> time="2016-08-10T16:58:26Z" level=warning msg="error authorizing context:
>> basic authentication challenge for realm \"Registry Realm\": invalid
>> authorization credential" go.version=go1.6.3 http.request.host="
>> docker-lab.evolveip.net:5000" 
>> http.request.id=ecce6c57-6273-42d6-b7a9-441877c0338f
>> http.request.method=GET http.request.remoteaddr="192.168.122.156:35858"
>> http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2
>> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64
>> os/linux arch/amd64" instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6
>> version=v2.5.0
>> 192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET /v2/ HTTP/1.1" 401
>> 87 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported
>> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
>> time="2016-08-10T16:58:26Z" level=error msg="response completed with
>> error" auth.user.name=maven err.code="manifest unknown"
>> err.detail="unknown manifest name=testwebapp revision=sha256:9799a25cd6fd7f
>> 7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>> err.message="manifest unknown" go.version=go1.6.3 http.request.host="
>> docker-lab.evolveip.net:5000" 
>> http.request.id=b994a477-6beb-4908-8589-c051b9048e87
>> http.request.method=GET http.request.remoteaddr="192.168.122.156:35860"
>> http.request.uri="/v2/testwebapp/manifests/sha256:9799a25cd6
>> fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>> http.request.useragent="docker/1.10.3 go/go1.4.2
>> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64
>> os/linux arch/amd64" http.response.contenttype="application/json;
>> charset=utf-8" http.response.duration=6.04215ms http.response.status=404
>> http.response.written=186 instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6
>> vars.name=testwebapp vars.reference="sha256:9799a25
>> cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3" version=v2.5.0
>> 192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET
>> /v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c
>> 85823e38aa22afb22f687a5b8a3ed2bf9ec3 HTTP/1.1" 404 186 "" "docker/1.10.3
>> go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64
>> os/linux arch/amd64"
>>
>> So it looks like the manifest isn't found, or am I misunderstanding that?
>>
>> The imagestream was imported by simply:
>>
>> [root@os-master ~]# oc import-image testwebapp --confirm --from=
>> docker-lab.example.com:5000/testwebapp:latest
>> The import completed successfully.
>>
>> Name:   testwebapp
>> Created:Less than a second ago
>> Labels: 
>> Annotations:openshift.io/image.dockerRepos
>> itoryCheck=2016-08-10T17:01:46Z
>> Docker Pull Spec:   172.30.11.167:5000/testwebapp/testwebapp
>>
>> Tag Spec  

Re: Problem authenticating to private docker registry

[Cesar Wong]

Hmm, I didn't know the issue existed between 1.10 and 1.12 as well. 

Andy, what would you recommend?

> On Aug 10, 2016, at 1:58 PM, Tony Saxon  wrote:
> 
> Ok, maybe that is the issue. I can not do the docker pull referencing the 
> sha256 hash on the node.
> 
> The docker version running on the node is docker 1.10.3, and the docker 
> version on the machine that pushed the image is 1.12.0. Is there a potential 
> workaround for this, or do I need to get the docker version updated on the 
> nodes? For reference, I installed the openshift platform using the ansible 
> advanced installation referenced in the documentation.
> 
> On Wed, Aug 10, 2016 at 1:46 PM, Cesar Wong  > wrote:
> Tony,
> 
> The only other time that I've seen the manifest not found error was when 
> there was a version mismatch between the Docker version that pushed the image 
> vs the version that was consuming the image (ie. images pushed with Docker 
> 1.9 and pulled with Docker 1.10). Are you able to pull the image spec 
> directly from your node using the Docker cli?
> 
> $ docker pull 
> docker-lab.example.com:5000/testwebapp@sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
>  
> 
> 
>> On Aug 10, 2016, at 1:02 PM, Tony Saxon > > wrote:
>> 
>> I'm not sure if this has anything to do with it, but I looked at the details 
>> of the imagestream that I imported and see that it has this as the docker 
>> image reference:
>> 
>> status:
>>   dockerImageRepository: 172.30.11.167:5000/testwebapp/testwebapp 
>> 
>>   tags:
>>   - items:
>> - created: 2016-08-10T13:26:01Z
>>   dockerImageReference: 
>> docker-lab.example.com:5000/testwebapp@sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
>>  
>> 
>>   generation: 1
>>   image: 
>> sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
>> tag: latest
>> 
>> I also see these errors show up on the docker registry when I try to deploy 
>> the app:
>> 
>> time="2016-08-10T16:58:26Z" level=warning msg="error authorizing context: 
>> basic authentication challenge for realm \"Registry Realm\": invalid 
>> authorization credential" go.version=go1.6.3 
>> http.request.host="docker-lab.evolveip.net:5000 
>> " http.request.id 
>> =ecce6c57-6273-42d6-b7a9-441877c0338f 
>> http.request.method=GET http.request.remoteaddr="192.168.122.156:35858 
>> " http.request.uri="/v2/" 
>> http.request.useragent="docker/1.10.3 go/go1.4.2 
>> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux 
>> arch/amd64" instance.id 
>> =f0d70491-6e34-44eb-a51c-3b13eae8daa6 version=v2.5.0
>> 192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET /v2/ HTTP/1.1" 401 87 
>> "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported 
>> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
>> time="2016-08-10T16:58:26Z" level=error msg="response completed with error" 
>> auth.user.name =maven err.code="manifest unknown" 
>> err.detail="unknown manifest name=testwebapp 
>> revision=sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>>  err.message="manifest unknown" go.version=go1.6.3 
>> http.request.host="docker-lab.evolveip.net:5000 
>> " http.request.id 
>> =b994a477-6beb-4908-8589-c051b9048e87 
>> http.request.method=GET http.request.remoteaddr="192.168.122.156:35860 
>> " 
>> http.request.uri="/v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>>  http.request.useragent="docker/1.10.3 go/go1.4.2 
>> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux 
>> arch/amd64" http.response.contenttype="application/json; charset=utf-8" 
>> http.response.duration=6.04215ms http.response.status=404 
>> http.response.written=186 instance.id 
>> =f0d70491-6e34-44eb-a51c-3b13eae8daa6 vars.name 
>> =testwebapp 
>> vars.reference="sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>>  version=v2.5.0
>> 192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET 
>> /v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
>>  HTTP/1.1" 404 186 "" "docker/1.10.3 go/go1.4.2 
>> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux 
>> arch/amd64"
>> 
>> So it looks like the manifest isn't found, or am I 

Re: Problem authenticating to private docker registry

[Cesar Wong]

Tony,

The only other time that I've seen the manifest not found error was when there 
was a version mismatch between the Docker version that pushed the image vs the 
version that was consuming the image (ie. images pushed with Docker 1.9 and 
pulled with Docker 1.10). Are you able to pull the image spec directly from 
your node using the Docker cli?

$ docker pull 
docker-lab.example.com:5000/testwebapp@sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
 

> On Aug 10, 2016, at 1:02 PM, Tony Saxon  wrote:
> 
> I'm not sure if this has anything to do with it, but I looked at the details 
> of the imagestream that I imported and see that it has this as the docker 
> image reference:
> 
> status:
>   dockerImageRepository: 172.30.11.167:5000/testwebapp/testwebapp 
> 
>   tags:
>   - items:
> - created: 2016-08-10T13:26:01Z
>   dockerImageReference: 
> docker-lab.example.com:5000/testwebapp@sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
>  
> 
>   generation: 1
>   image: 
> sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
> tag: latest
> 
> I also see these errors show up on the docker registry when I try to deploy 
> the app:
> 
> time="2016-08-10T16:58:26Z" level=warning msg="error authorizing context: 
> basic authentication challenge for realm \"Registry Realm\": invalid 
> authorization credential" go.version=go1.6.3 
> http.request.host="docker-lab.evolveip.net:5000 
> " http.request.id 
> =ecce6c57-6273-42d6-b7a9-441877c0338f 
> http.request.method=GET http.request.remoteaddr="192.168.122.156:35858 
> " http.request.uri="/v2/" 
> http.request.useragent="docker/1.10.3 go/go1.4.2 
> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux 
> arch/amd64" instance.id 
> =f0d70491-6e34-44eb-a51c-3b13eae8daa6 version=v2.5.0
> 192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET /v2/ HTTP/1.1" 401 87 
> "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported 
> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
> time="2016-08-10T16:58:26Z" level=error msg="response completed with error" 
> auth.user.name =maven err.code="manifest unknown" 
> err.detail="unknown manifest name=testwebapp 
> revision=sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>  err.message="manifest unknown" go.version=go1.6.3 
> http.request.host="docker-lab.evolveip.net:5000 
> " http.request.id 
> =b994a477-6beb-4908-8589-c051b9048e87 
> http.request.method=GET http.request.remoteaddr="192.168.122.156:35860 
> " 
> http.request.uri="/v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>  http.request.useragent="docker/1.10.3 go/go1.4.2 
> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux 
> arch/amd64" http.response.contenttype="application/json; charset=utf-8" 
> http.response.duration=6.04215ms http.response.status=404 
> http.response.written=186 instance.id 
> =f0d70491-6e34-44eb-a51c-3b13eae8daa6 vars.name 
> =testwebapp 
> vars.reference="sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>  version=v2.5.0
> 192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET 
> /v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
>  HTTP/1.1" 404 186 "" "docker/1.10.3 go/go1.4.2 
> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux 
> arch/amd64"
> 
> So it looks like the manifest isn't found, or am I misunderstanding that?
> 
> The imagestream was imported by simply:
> 
> [root@os-master ~]# oc import-image testwebapp --confirm 
> --from=docker-lab.example.com:5000/testwebapp:latest 
>  
> The import completed successfully.
>  
> Name:   testwebapp
> Created:Less than a second ago
> Labels: 
> Annotations:
> openshift.io/image.dockerRepositoryCheck=2016-08-10T17:01:46Z 
> 
> Docker Pull Spec:   172.30.11.167:5000/testwebapp/testwebapp 
> 
>  
> Tag SpecCreated 
> PullSpecImage
> latest  

Re: Problem authenticating to private docker registry

[Tony Saxon]

I'm not sure if this has anything to do with it, but I looked at the
details of the imagestream that I imported and see that it has this as the
docker image reference:

status:
  dockerImageRepository: 172.30.11.167:5000/testwebapp/testwebapp
  tags:
  - items:
- created: 2016-08-10T13:26:01Z
  dockerImageReference:
docker-lab.example.com:5000/testwebapp@sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
  generation: 1
  image:
sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
tag: latest

I also see these errors show up on the docker registry when I try to deploy
the app:

time="2016-08-10T16:58:26Z" level=warning msg="error authorizing context:
basic authentication challenge for realm \"Registry Realm\": invalid
authorization credential" go.version=go1.6.3 http.request.host="
docker-lab.evolveip.net:5000"
http.request.id=ecce6c57-6273-42d6-b7a9-441877c0338f
http.request.method=GET http.request.remoteaddr="192.168.122.156:35858"
http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2
git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux
arch/amd64" instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6 version=v2.5.0
192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET /v2/ HTTP/1.1" 401 87
"" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported
kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
time="2016-08-10T16:58:26Z" level=error msg="response completed with error"
auth.user.name=maven err.code="manifest unknown" err.detail="unknown
manifest name=testwebapp
revision=sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
err.message="manifest unknown" go.version=go1.6.3 http.request.host="
docker-lab.evolveip.net:5000"
http.request.id=b994a477-6beb-4908-8589-c051b9048e87
http.request.method=GET http.request.remoteaddr="192.168.122.156:35860"
http.request.uri="/v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
http.request.useragent="docker/1.10.3 go/go1.4.2
git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux
arch/amd64" http.response.contenttype="application/json; charset=utf-8"
http.response.duration=6.04215ms http.response.status=404
http.response.written=186 instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6
vars.name=testwebapp
vars.reference="sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
version=v2.5.0
192.168.122.156 - - [10/Aug/2016:16:58:26 +] "GET
/v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3
HTTP/1.1" 404 186 "" "docker/1.10.3 go/go1.4.2
git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux
arch/amd64"

So it looks like the manifest isn't found, or am I misunderstanding that?

The imagestream was imported by simply:

[root@os-master ~]# oc import-image testwebapp --confirm --from=
docker-lab.example.com:5000/testwebapp:latest
The import completed successfully.

Name:   testwebapp
Created:Less than a second ago
Labels: 
Annotations:
openshift.io/image.dockerRepositoryCheck=2016-08-10T17:01:46Z
Docker Pull Spec:   172.30.11.167:5000/testwebapp/testwebapp

Tag SpecCreated
PullSpec
Image
latest  docker-lab.example.com:5000/testwebapp:latest  1 seconds ago
docker-lab.example.com:5000/testwebapp@sha256:9799a25cd6fd7f...



On Wed, Aug 10, 2016 at 11:39 AM, Tony Saxon  wrote:

> Not a problem, can you point me in the right direction for how to get
> those logs?
>
> For the first one, I know I can set the USER option in the dockerfile that
> builds the docker image. Does it matter what user I set that as? I'm
> assuming that the user needs to exist on the nodes...
>
> On Wed, Aug 10, 2016 at 11:20 AM, Maciej Szulik 
> wrote:
>
>> There are two problems here:
>> 1. WARNING: Image "testwebapp" runs as the 'root' user which may not be
>> permitted by your cluster administrator
>> unless your user is allowed to run privileged containers you should
>> fix that.
>> 2. Error from server: container "testwebapp" in pod "testwebapp-1-1x7ex"
>> is waiting to start: image can't be pulled
>>
>> First it would be good to get rid of no 1. for the latter I'd need to see
>> master logs with loglevel=8 if it's not a problem.
>>
>>
>>
>> On Wed, Aug 10, 2016 at 5:14 PM, Tony Saxon  wrote:
>>
>>> It does not work:
>>>
>>> [root@os-master ~]# oc new-app testwebapp
>>> --> Found image 59826fe (4 days old) in image stream testwebapp under
>>> tag "latest" for "testwebapp"
>>>
>>> * This image will be deployed in deployment config "testwebapp"
>>> * The image does not expose any ports - if you want to load balance
>>> or send traffic to this component
>>>   you will need to create a service with 'expose dc/testwebapp
>>> --port=[port]' later
>>> * 

Re: Problem authenticating to private docker registry

[Tony Saxon]

Not a problem, can you point me in the right direction for how to get those
logs?

For the first one, I know I can set the USER option in the dockerfile that
builds the docker image. Does it matter what user I set that as? I'm
assuming that the user needs to exist on the nodes...

On Wed, Aug 10, 2016 at 11:20 AM, Maciej Szulik  wrote:

> There are two problems here:
> 1. WARNING: Image "testwebapp" runs as the 'root' user which may not be
> permitted by your cluster administrator
> unless your user is allowed to run privileged containers you should
> fix that.
> 2. Error from server: container "testwebapp" in pod "testwebapp-1-1x7ex"
> is waiting to start: image can't be pulled
>
> First it would be good to get rid of no 1. for the latter I'd need to see
> master logs with loglevel=8 if it's not a problem.
>
>
>
> On Wed, Aug 10, 2016 at 5:14 PM, Tony Saxon  wrote:
>
>> It does not work:
>>
>> [root@os-master ~]# oc new-app testwebapp
>> --> Found image 59826fe (4 days old) in image stream testwebapp under tag
>> "latest" for "testwebapp"
>>
>> * This image will be deployed in deployment config "testwebapp"
>> * The image does not expose any ports - if you want to load balance
>> or send traffic to this component
>>   you will need to create a service with 'expose dc/testwebapp
>> --port=[port]' later
>> * WARNING: Image "testwebapp" runs as the 'root' user which may not
>> be permitted by your cluster administrator
>>
>> --> Creating resources with label app=testwebapp ...
>> deploymentconfig "testwebapp" created
>> --> Success
>> Run 'oc status' to view your app.
>> [root@os-master ~]# oc status
>> In project testwebapp on server https://os-master.libvirt:8443
>>
>> dc/testwebapp deploys istag/testwebapp:latest
>>   deployment #1 pending 5 seconds ago
>>
>> 1 warning identified, use 'oc status -v' to see details.
>> [root@os-master ~]# oc get pods
>> NAME  READY STATUS  RESTARTS   AGE
>> testwebapp-1-1x7ex0/1   ContainerCreating   0  3s
>> testwebapp-1-deploy   1/1   Running 0  9s
>> [root@os-master ~]# oc get pods
>> NAME  READY STATUS RESTARTS   AGE
>> testwebapp-1-1x7ex0/1   ErrImagePull   0  6s
>> testwebapp-1-deploy   1/1   Running0  12s
>> [root@os-master ~]# oc logs testwebapp-1-1x7ex
>> Error from server: container "testwebapp" in pod "testwebapp-1-1x7ex" is
>> waiting to start: image can't be pulled
>> [root@os-master ~]#
>>
>> On Wed, Aug 10, 2016 at 10:58 AM, Maciej Szulik 
>> wrote:
>>
>>> Yeah, that looks ok, did that work? If not, what error did you get?
>>>
>>>
>>>
>>> On Wed, Aug 10, 2016 at 4:19 PM, Tony Saxon 
>>> wrote:
>>>
 I've done that:

 [root@os-master ~]# oc get secrets
 NAME   TYPE
 DATA  AGE
 builder-dockercfg-7bjookubernetes.io/dockercfg
 1 1d
 builder-token-gi9o9kubernetes.io/service-account-token
 3 1d
 builder-token-wf31ukubernetes.io/service-account-token
 3 1d
 default-dockercfg-pfotakubernetes.io/dockercfg
 1 1d
 default-token-vbcmckubernetes.io/service-account-token
 3 1d
 default-token-xffu5kubernetes.io/service-account-token
 3 1d
 deployer-dockercfg-lfiuw   kubernetes.io/dockercfg
 1 1d
 deployer-token-9euo2   kubernetes.io/service-account-token
 3 1d
 deployer-token-mq6vw   kubernetes.io/service-account-token
 3 1d
 docker-lab kubernetes.io/dockercfg
 1 19h

 [root@os-master ~]# oc describe sa default
 Name:   default
 Namespace:  testwebapp
 Labels: 

 Image pull secrets: default-dockercfg-pfota
 docker-lab

 Mountable secrets:  default-token-xffu5
 default-dockercfg-pfota

 Tokens: default-token-vbcmc
 default-token-xffu5


 One thing to note is that the "link" command didn't work. I had to use
 'oc secrets add default docker-lab --for=pull'


>>> Can you open an issue it's not working? Either docs should be updated or
>>> code fixed.
>>>
>>>



>>> Does it need to be added to a different service account? Here are the
 accounts that currently exist:

 [root@os-master ~]# oc get sa
 NAME   SECRETS   AGE
 builder2 1d
 default2 1d
 deployer   2 1d

 On Wed, Aug 10, 2016 at 10:09 AM, Maciej Szulik 
 wrote:

>
>
> On Wed, Aug 10, 2016 at 3:59 PM, Cesar Wong  wrote:
>
>> Hi Tony,
>>
>> Sorry for 

Second master-api fails to start

[Daniel Barker]

I'm installing OpenShift for the first time on internal servers that were
provisioned and then configured by me. I had to host the repos internally
on our Artifactory server as the internal cluster has no direct internet
access. I followed all of the steps here:
https://docs.openshift.org/latest/install_config/install/prerequisites.html
and here:
https://docs.openshift.org/latest/install_config/install/advanced_install.html.
I'm installing it using openshift-ansible at
commit ac4ef6bb65c24890592b6b35b11be82f4ddc6deb. These systems are RHEL7.

My configuration file looks like below with minor edits to remove internal
information:
# Create an OSOv3 group that contains the master, nodes, etcd, and lb
groups.
# The lb group lets Ansible configure HAProxy as the load balancing
solution.
# Comment lb out if your load balancer is pre-configured.
[OSOv3:children]
masters
nodes
etcd
lb

# Set variables common for all OSOv3 hosts
[OSOv3:vars]
ansible_ssh_user=username
ansible_sudo=true
deployment_type=origin

# Uncomment the following to enable htpasswd authentication; defaults to
# DenyAllPasswordIdentityProvider.
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login':
'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider',
'filename': '/etc/origin/master/htpasswd'}]

# Native high availability cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
openshift_master_cluster_method=native
openshift_master_cluster_hostname=oshiftmst01.example.com
openshift_master_cluster_public_hostname=oshiftmst01.example.com

# host group for masters
[masters]
oshiftmst01.example.com
oshiftmst02.example.com
oshiftmst03.example.com

# host group for etcd
[etcd]
oshiftmst01.example.com
oshiftmst02.example.com
oshiftmst03.example.com

# Specify load balancer host
[lb]
oshiftinf01.example.com

# host group for nodes, includes region info
[nodes]
oshiftmst0[1:3].example.com openshift_node_labels="{'region': 'infra',
'zone': 'default'}"
oshiftslv01.example.com
oshiftslv02.example.com
oshiftslv03.example.com
oshiftinf01.example.com

Everything is installed correctly on the mst01 host and it starts
correctly. When I get to the mst02 host, it fails like this:
TASK [openshift_master : Start and enable master api]
**
fatal: [oshiftmst02.example.com]: FAILED! => {"changed": false, "failed":
true, "msg": "Job for origin-master-api.service failed because a timeout
was exceeded. See \"systemctl status origin-master-api.service\" and
\"journalctl -xe\" for details.\n"}

When I try to manually start it, I get this:
[username@oshiftmst02 ~]$ sudo systemctl start origin-master-api.service
Job for origin-master-api.service failed because a timeout was exceeded.
See "systemctl status origin-master-api.service" and "journalctl -xe" for
details.

When I look closer:
Aug 09 15:48:47 oshiftmst02 systemd[1]: Starting Atomic OpenShift Master
API...
Aug 09 15:48:47 oshiftmst02 atomic-openshift-master-api[15969]: I0809
15:48:47.573484   15969 start_api.go:102] Using a listen address override "
0.0.0.0:8443"
Aug 09 15:48:47 oshiftmst02 atomic-openshift-master-api[15969]: W0809
15:48:47.576872   15969 start_master.go:270] assetConfig.loggingPublicURL:
Invalid value: "": required to view aggregated container
Aug 09 15:48:47 oshiftmst02 atomic-openshift-master-api[15969]: W0809
15:48:47.576910   15969 start_master.go:270] assetConfig.metricsPublicURL:
Invalid value: "": required to view cluster metrics in t
Aug 09 15:48:47 oshiftmst02 atomic-openshift-master-api[15969]: I0809
15:48:47.587135   15969 plugins.go:71] No cloud provider specified.
Aug 09 15:48:47 oshiftmst02 atomic-openshift-master-api[15969]: I0809
15:48:47.588409   15969 genericapiserver.go:81] Adding storage destination
for group
Aug 09 15:48:47 oshiftmst02 atomic-openshift-master-api[15969]: I0809
15:48:47.588443   15969 genericapiserver.go:81] Adding storage destination
for group extensions
Aug 09 15:48:47 oshiftmst02 atomic-openshift-master-api[15969]: I0809
15:48:47.588472   15969 start_master.go:383] Starting master on 0.0.0.0:8443
(v1.2.1)
Aug 09 15:48:47 oshiftmst02 atomic-openshift-master-api[15969]: I0809
15:48:47.588482   15969 start_master.go:384] Public master address is
https://oshiftmst01.example.com:8443
Aug 09 15:48:47 oshiftmst02 atomic-openshift-master-api[15969]: I0809
15:48:47.588511   15969 start_master.go:388] Using images from
"openshift/origin-:v1.2.1"
Aug 09 15:50:01 oshiftmst02 kernel: SELinux: initialized (dev tmpfs, type
tmpfs), uses transition SIDs
Aug 09 15:50:01 oshiftmst02 systemd[1]: Created slice user-0.slice.
Aug 09 15:50:01 oshiftmst02 systemd[1]: Starting user-0.slice.
Aug 09 15:50:01 oshiftmst02 systemd[1]: Started Session 10668 of user 

Re: Problem authenticating to private docker registry

[Cesar Wong]

Maciej,

In this case, Tony is trying to connect to the OpenShift registry, so the 
secret should exist; the dockercfg secret for the project's default service 
account. 

Tony, two things that may be your issue:

1) You're using a route for your registry (docker-lab.example.net 
). The dockercfg secret will likely only have 
an entry for the ip address of the registry and not the route. (Maciej, maybe 
you know of a way to get the secrets to include an entry for the host of the 
route). Otherwise, you're better off specifying the service ip when invoking 
new-app.

You can check what hosts are included in the dockercfg secret by doing 'oc 
describe secret/default-dockercfg-' where  is whatever suffix is used 
in your project.

2) The image ref that you're using in your new-app invocation doesn't include a 
namespace. All images on the OpenShift registry will have a namespace and name 
like:
 [registry-host]:[port]/projectname/testwebapp:latest. Make sure you have the 
full spec for the image (from 'oc get is').

> On Aug 10, 2016, at 5:44 AM, Maciej Szulik  wrote:
> 
> You need to follow the docs here: 
> https://docs.openshift.org/latest/dev_guide/managing_images.html#private-registries
>  
> 
> to setup the secret in the same project your ImageStream is created and then 
> re-import the image. 
> During import proper secrets will be picked automatically based on the urls 
> of the registry and your image metadata
> should be downloaded to the server. This will handle the import part, now for 
> actually using an image from private
> registry you need to follow this: 
> https://docs.openshift.org/latest/dev_guide/managing_images.html#allowing-pods-to-reference-images-from-other-secured-registries
>  
> 
> 
> Hope that helps,
> Maciej
> 
> On Tue, Aug 9, 2016 at 4:00 PM, Tony Saxon  > wrote:
> I'm not sure what I'm missing here. I have a private docker registry that is 
> set up securely and uses authentication. I followed the docs at 
> https://docs.openshift.org/latest/dev_guide/managing_images.html#using-image-pull-secrets
>  
> 
>  to create the secret with the username and password to authenticate with the 
> docker registry. I verified that I can manually login to the docker registry 
> from the master and the nodes. However, when I go to deploy a new app based 
> on an image from the docker registry it seem to be failing to authenticate. 
> The command that I'm running to create the new app:
> 
> oc new-app docker-lab.example.net:5000/testwebapp:latest 
> 
> 
> It creates the imagestream and attempts to deploy the pod. I get the 
> following in the logs on the pod:
> 
> # oc logs testwebapp-1-us1wu
> Error from server: container "testwebapp" in pod "testwebapp-1-us1wu" is 
> waiting to start: image can't be pulled
> 
> The logs on the docker registry show:
> 
> time="2016-08-09T13:54:45Z" level=warning msg="error authorizing context: 
> basic authentication challenge for realm \"Registry Realm\": invalid 
> authorization credential" go.version=go1.6.3 
> http.request.host="docker-lab.example.net:5000 
> " http.request.id 
> =f5aeb8b9-ce4e-41b7-86a8-76e8c520bd22 
> http.request.method=GET http.request.remoteaddr="192.168.122.158:54436 
> " http.request.uri="/v2/" 
> http.request.useragent="docker/1.10.3 go/go1.4.2 
> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux 
> arch/amd64" instance.id 
> =f0d70491-6e34-44eb-a51c-3b13eae8daa6 version=v2.5.0
> 192.168.122.158 - - [09/Aug/2016:13:54:45 +] "GET /v2/ HTTP/1.1" 401 87 
> "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported 
> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
> time="2016-08-09T13:54:45Z" level=error msg="response completed with error" 
> auth.user.name =tsaxon err.code="manifest unknown" 
> err.detail="unknown manifest name=testwebapp 
> revision=sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>  err.message="manifest unknown" go.version=go1.6.3 
> http.request.host="docker-lab.example.net:5000 
> " http.request.id 
> =130a9014-7c19-48f7-bef3-2b8cfe0470a0 
> http.request.method=GET http.request.remoteaddr="192.168.122.158:54438 
> " 
> http.request.uri="/v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
>  

Re: Problem authenticating to private docker registry

[Maciej Szulik]

You need to follow the docs here:
https://docs.openshift.org/latest/dev_guide/managing_images.html#private-registries
to setup the secret in the same project your ImageStream is created and
then re-import the image.
During import proper secrets will be picked automatically based on the urls
of the registry and your image metadata
should be downloaded to the server. This will handle the import part, now
for actually using an image from private
registry you need to follow this:
https://docs.openshift.org/latest/dev_guide/managing_images.html#allowing-pods-to-reference-images-from-other-secured-registries

Hope that helps,
Maciej

On Tue, Aug 9, 2016 at 4:00 PM, Tony Saxon  wrote:

> I'm not sure what I'm missing here. I have a private docker registry that
> is set up securely and uses authentication. I followed the docs at
> https://docs.openshift.org/latest/dev_guide/managing_
> images.html#using-image-pull-secrets to create the secret with the
> username and password to authenticate with the docker registry. I verified
> that I can manually login to the docker registry from the master and the
> nodes. However, when I go to deploy a new app based on an image from the
> docker registry it seem to be failing to authenticate. The command that I'm
> running to create the new app:
>
> oc new-app docker-lab.example.net:5000/testwebapp:latest
>
> It creates the imagestream and attempts to deploy the pod. I get the
> following in the logs on the pod:
>
> # oc logs testwebapp-1-us1wu
> Error from server: container "testwebapp" in pod "testwebapp-1-us1wu" is
> waiting to start: image can't be pulled
>
> The logs on the docker registry show:
>
> time="2016-08-09T13:54:45Z" level=warning msg="error authorizing context:
> basic authentication challenge for realm \"Registry Realm\": invalid
> authorization credential" go.version=go1.6.3 http.request.host="
> docker-lab.example.net:5000" 
> http.request.id=f5aeb8b9-ce4e-41b7-86a8-76e8c520bd22
> http.request.method=GET http.request.remoteaddr="192.168.122.158:54436"
> http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2
> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux
> arch/amd64" instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6
> version=v2.5.0
> 192.168.122.158 - - [09/Aug/2016:13:54:45 +] "GET /v2/ HTTP/1.1" 401
> 87 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported
> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
> time="2016-08-09T13:54:45Z" level=error msg="response completed with
> error" auth.user.name=tsaxon err.code="manifest unknown"
> err.detail="unknown manifest name=testwebapp revision=sha256:
> 9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
> err.message="manifest unknown" go.version=go1.6.3 http.request.host="
> docker-lab.example.net:5000" 
> http.request.id=130a9014-7c19-48f7-bef3-2b8cfe0470a0
> http.request.method=GET http.request.remoteaddr="192.168.122.158:54438"
> http.request.uri="/v2/testwebapp/manifests/sha256:
> 9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
> http.request.useragent="docker/1.10.3 go/go1.4.2
> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux
> arch/amd64" http.response.contenttype="application/json; charset=utf-8"
> http.response.duration=6.174905ms http.response.status=404
> http.response.written=186 instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6
> vars.name=testwebapp vars.reference="sha256:9799a25cd6fd7f7908bad740fc0c85
> 823e38aa22afb22f687a5b8a3ed2bf9ec3" version=v2.5.0
> 192.168.122.158 - - [09/Aug/2016:13:54:45 +] "GET
> /v2/testwebapp/manifests/sha256:9799a25cd6fd7f7908bad740fc0c85
> 823e38aa22afb22f687a5b8a3ed2bf9ec3 HTTP/1.1" 404 186 "" "docker/1.10.3
> go/go1.4.2 git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64
> os/linux arch/amd64"
> time="2016-08-09T13:54:45Z" level=warning msg="error authorizing context:
> basic authentication challenge for realm \"Registry Realm\": invalid
> authorization credential" go.version=go1.6.3 http.request.host="
> docker-lab.example.net:5000" 
> http.request.id=0185e07b-f1c1-48e6-91ea-dede2339f087
> http.request.method=GET http.request.remoteaddr="192.168.122.158:54440"
> http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.4.2
> git-commit/9419b24-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux
> arch/amd64" instance.id=f0d70491-6e34-44eb-a51c-3b13eae8daa6
> version=v2.5.0
> 192.168.122.158 - - [09/Aug/2016:13:54:45 +] "GET /v2/ HTTP/1.1" 401
> 87 "" "docker/1.10.3 go/go1.4.2 git-commit/9419b24-unsupported
> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
> time="2016-08-09T13:54:46Z" level=error msg="response completed with
> error" auth.user.name=tsaxon err.code="manifest unknown"
> err.detail="unknown manifest name=testwebapp revision=sha256:
> 9799a25cd6fd7f7908bad740fc0c85823e38aa22afb22f687a5b8a3ed2bf9ec3"
> err.message="manifest unknown" go.version=go1.6.3