Re: openshift/api and openshift/client-go are authoritative

[Simo Sorce]

On Tue, 2017-12-05 at 16:25 -0500, Ben Parees wrote:
> On Tue, Dec 5, 2017 at 4:13 PM, Simo Sorce  wrote:
> 
> > On Tue, 2017-11-28 at 13:58 -0500, David Eads wrote:
> > > As of https://github.com/openshift/origin/pull/17477, h
> > > ttps://github.com/openshift/api and https://github.com/openshift/
> > 
> > client-go are
> > > the authoritative source of the OpenShift API types and the OpenShift
> > > external clients.  The external types and go client are no longer present
> > > in https://github.com/openshift/origin.  This makes it possible to
> > 
> > interact
> > > with an OpenShift cluster without trying to vendor openshift/origin and
> > 
> > it
> > > changes the way that API changes are merged.
> > > 
> > > To make an API change in 3.8+:
> > > 
> > >1. open a pull with the external types to openshift/api and get it
> > >reviewed, approved, and merged
> > >2. bump the vendored dependencies in openshift/client-go, regenerate
> > >(make generate build), and get it merged.
> > >3. bump the vendored dependencies in openshift/origin
> > >(hack/update-deps.sh), update your internal types, and start serving
> > 
> > your
> > >new API.
> > > 
> > > For forks, you will have to merge into the appropriate branches of the
> > > various repositories.
> > 
> > SoI have to deal with this now.
> > 
> > Looking at the api repository I see you opened commits with titles like
> > this:
> > 
> > UPSTREAM: openshift/origin: missed tags
> > 
> > I do not understand, if the api repository is authoritative, then this
> > repository is upstream, not openshift/origin
> > 
> > Is there a convention we need to follow for the vendoring of api into
> > client-go and origin as far as naming commits ? Care to provide an
> > example/template ?
> > 
> 
> "bump(*)"
> 
> I got to be the guinea pig on this, so you can see my prs here:
> 
> changed the api:
> https://github.com/openshift/api/pull/10
> 
> bumped the client-go deps:
> https://github.com/openshift/client-go/pull/12
> 
> trying to bump the origin deps so i can actually make the change i care
> about:
> https://github.com/openshift/origin/pull/17314/commits/fc47ea3baba0631ba84e82a80a98c10670dabbd8
> 

I can't think of anything worse than "bump(*)" (or any other fixed
strings) to make it so that we have to excavate hashes in the glide
files to correlate changes between trees.

Can we use a convention where we at least reference the PR in the
API/client-go trees ?

Simo.

-- 
Simo Sorce
Sr. Principal Software Engineer
Red Hat, Inc

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: openshift/api and openshift/client-go are authoritative

[Ben Parees]

On Tue, Dec 5, 2017 at 4:13 PM, Simo Sorce  wrote:

> On Tue, 2017-11-28 at 13:58 -0500, David Eads wrote:
> > As of https://github.com/openshift/origin/pull/17477, h
> > ttps://github.com/openshift/api and https://github.com/openshift/
> client-go are
> > the authoritative source of the OpenShift API types and the OpenShift
> > external clients.  The external types and go client are no longer present
> > in https://github.com/openshift/origin.  This makes it possible to
> interact
> > with an OpenShift cluster without trying to vendor openshift/origin and
> it
> > changes the way that API changes are merged.
> >
> > To make an API change in 3.8+:
> >
> >1. open a pull with the external types to openshift/api and get it
> >reviewed, approved, and merged
> >2. bump the vendored dependencies in openshift/client-go, regenerate
> >(make generate build), and get it merged.
> >3. bump the vendored dependencies in openshift/origin
> >(hack/update-deps.sh), update your internal types, and start serving
> your
> >new API.
> >
> > For forks, you will have to merge into the appropriate branches of the
> > various repositories.
>
> SoI have to deal with this now.
>
> Looking at the api repository I see you opened commits with titles like
> this:
>
> UPSTREAM: openshift/origin: missed tags
>
> I do not understand, if the api repository is authoritative, then this
> repository is upstream, not openshift/origin
>
> Is there a convention we need to follow for the vendoring of api into
> client-go and origin as far as naming commits ? Care to provide an
> example/template ?
>

"bump(*)"

I got to be the guinea pig on this, so you can see my prs here:

changed the api:
https://github.com/openshift/api/pull/10

bumped the client-go deps:
https://github.com/openshift/client-go/pull/12

trying to bump the origin deps so i can actually make the change i care
about:
https://github.com/openshift/origin/pull/17314/commits/fc47ea3baba0631ba84e82a80a98c10670dabbd8




>
> Simo.
>
> --
> Simo Sorce
> Sr. Principal Software Engineer
> Red Hat, Inc
>
> ___
> dev mailing list
> dev@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>



-- 
Ben Parees | OpenShift
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: openshift/api and openshift/client-go are authoritative

[Simo Sorce]

On Tue, 2017-11-28 at 13:58 -0500, David Eads wrote:
> As of https://github.com/openshift/origin/pull/17477, h
> ttps://github.com/openshift/api and https://github.com/openshift/client-go are
> the authoritative source of the OpenShift API types and the OpenShift
> external clients.  The external types and go client are no longer present
> in https://github.com/openshift/origin.  This makes it possible to interact
> with an OpenShift cluster without trying to vendor openshift/origin and it
> changes the way that API changes are merged.
> 
> To make an API change in 3.8+:
> 
>1. open a pull with the external types to openshift/api and get it
>reviewed, approved, and merged
>2. bump the vendored dependencies in openshift/client-go, regenerate
>(make generate build), and get it merged.
>3. bump the vendored dependencies in openshift/origin
>(hack/update-deps.sh), update your internal types, and start serving your
>new API.
> 
> For forks, you will have to merge into the appropriate branches of the
> various repositories.

SoI have to deal with this now.

Looking at the api repository I see you opened commits with titles like
this:

UPSTREAM: openshift/origin: missed tags

I do not understand, if the api repository is authoritative, then this
repository is upstream, not openshift/origin

Is there a convention we need to follow for the vendoring of api into
client-go and origin as far as naming commits ? Care to provide an
example/template ?

Simo.

-- 
Simo Sorce
Sr. Principal Software Engineer
Red Hat, Inc

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Docker builds in an OpenShift Jenkins slave?

[Ben Parees]

On Tue, Dec 5, 2017 at 11:54 AM, Alan Christie <
achris...@informaticsmatters.com> wrote:

> Thanks again, Ben.
>
> I run into two problems with your _alternative_ suggestion … it looked
> really promising because at least you have access to the pod configuration
> (in the Jenkins "Configure System->Cloud->Kubernetes”), which is cool, but
> I encounter the following in the Jenkins log as it attempts to launch my
> slave Pod...
>
> * Invalid value: true: Privileged containers are not allowed*
>
> That’s annoying, especially as there’s a checkbox for it. But also…
>
>
> * Invalid value: "hostPath": hostPath volumes are not allowed to be used*
>

You'll have to take that up w/ Kubernetes plugin maintainers:
https://plugins.jenkins.io/kubernetes

Hence my recommendation you use DOCKER_HOST instead.



>
>
> On 5 Dec 2017, at 15:48, Ben Parees  wrote:
>
>
>
> On Tue, Dec 5, 2017 at 10:36 AM, Alan Christie  informaticsmatters.com> wrote:
>
>> Thanks Ben. It does seem sensible to use build strategies but prior to a
>> wholesale migration to OpenShift, and for existing workflows that may
>> contain docker and docker-compose commands is there any reasonable option
>> other than a an external (cloud/proprietary/dedicated) docker-enabled
>> slave? I can, for example, just have a Docker slave available (outside the
>> OpenShift cluster) but that’s not ideal.
>>
>> Is there an _unsafe_ route I might be able to use now?
>>
>
> use DOCKER_HOST env variable and point to a host w/ a public docker.
>
> The alternative is to try to use a hostpath volume definition in your
> slave pod template but then you also need to run the slave pod as
> privileged.
>
>
>
>
>> I understand the issues around sharing a docker.sock but it seems to be
>> an acceptable strategy for many. And, for a controlled environment, just
>> mounting docker.sock is a rather neat (quick-n-dirty) solution.
>>
>> It may be that, was you say there’s no sensible route down the
>> OpenShift/CICD road other than build strategies. It’s just that for
>> existing/legacy projects not having docker.sock is quite a hill to climb.
>>
>> Thanks for your advice though, that has been gratefully received.
>>
>> Alan.
>>
>> On 5 Dec 2017, at 13:41, Ben Parees  wrote:
>>
>>
>>
>>
>>
>> On Dec 5, 2017 07:57, "Alan Christie" 
>> wrote:
>>
>> I’m using Jenkins from the CI/CD catalogue and am able to spin up slaves
>> and use an `ImageStream` to identify my own slave image. That’s useful, but
>> what I want to be able to do is build and run Docker images, primarily for
>> unit/functional test purposes. The _sticking point_, it seems, is the
>> ability to mount the host's `docker.sock`, without this I’m unable to run
>> any Docker commands in my Docker containers.
>>
>> Q. Is there a way to mount the Jenkins/OpenShift host’s
>> /var/run/docker.sock in my slave so that I can run Docker commands?
>>
>>
>> Not safely. (mounting the host docker socket is giving out root access to
>> your host).
>>
>> You could use a remote docker host with a certificate for access I
>> believe. (that's still handing out root access on the docker host but at
>> least it's a little protected)
>>
>> If not, what is the recommended/best practice for
>> building/running/pushing Docker images from a slave agent?
>>
>>
>> Define docker build strategies in openshift and trigger them from your
>> jenkins job.
>>
>>
>> Alan
>>
>> ___
>> dev mailing list
>> dev@lists.openshift.redhat.com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>>
>>
>>
>>
>
>
> --
> Ben Parees | OpenShift
>
>
>


-- 
Ben Parees | OpenShift
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Docker builds in an OpenShift Jenkins slave?

[Alan Christie]

Thanks again, Ben.

I run into two problems with your _alternative_ suggestion … it looked really 
promising because at least you have access to the pod configuration (in the 
Jenkins "Configure System->Cloud->Kubernetes”), which is cool, but I encounter 
the following in the Jenkins log as it attempts to launch my slave Pod...

Invalid value: true: Privileged containers are not allowed

That’s annoying, especially as there’s a checkbox for it. But also…

Invalid value: "hostPath": hostPath volumes are not allowed to be used


> On 5 Dec 2017, at 15:48, Ben Parees  wrote:
> 
> 
> 
> On Tue, Dec 5, 2017 at 10:36 AM, Alan Christie 
> mailto:achris...@informaticsmatters.com>> 
> wrote:
> Thanks Ben. It does seem sensible to use build strategies but prior to a 
> wholesale migration to OpenShift, and for existing workflows that may contain 
> docker and docker-compose commands is there any reasonable option other than 
> a an external (cloud/proprietary/dedicated) docker-enabled slave? I can, for 
> example, just have a Docker slave available (outside the OpenShift cluster) 
> but that’s not ideal.
> 
> Is there an _unsafe_ route I might be able to use now?
> 
> use DOCKER_HOST env variable and point to a host w/ a public docker.
> 
> The alternative is to try to use a hostpath volume definition in your slave 
> pod template but then you also need to run the slave pod as privileged.
> 
> 
> 
> 
> I understand the issues around sharing a docker.sock but it seems to be an 
> acceptable strategy for many. And, for a controlled environment, just 
> mounting docker.sock is a rather neat (quick-n-dirty) solution.
> 
> It may be that, was you say there’s no sensible route down the OpenShift/CICD 
> road other than build strategies. It’s just that for existing/legacy projects 
> not having docker.sock is quite a hill to climb.
> 
> Thanks for your advice though, that has been gratefully received.
> 
> Alan.
> 
>> On 5 Dec 2017, at 13:41, Ben Parees > > wrote:
>> 
>> 
>> 
>> 
>> 
>> On Dec 5, 2017 07:57, "Alan Christie" > > wrote:
>> I’m using Jenkins from the CI/CD catalogue and am able to spin up slaves and 
>> use an `ImageStream` to identify my own slave image. That’s useful, but what 
>> I want to be able to do is build and run Docker images, primarily for 
>> unit/functional test purposes. The _sticking point_, it seems, is the 
>> ability to mount the host's `docker.sock`, without this I’m unable to run 
>> any Docker commands in my Docker containers.
>> 
>> Q. Is there a way to mount the Jenkins/OpenShift host’s /var/run/docker.sock 
>> in my slave so that I can run Docker commands?
>> 
>> Not safely. (mounting the host docker socket is giving out root access to 
>> your host). 
>> 
>> You could use a remote docker host with a certificate for access I believe. 
>> (that's still handing out root access on the docker host but at least it's a 
>> little protected) 
>> 
>> If not, what is the recommended/best practice for building/running/pushing 
>> Docker images from a slave agent?
>> 
>> Define docker build strategies in openshift and trigger them from your 
>> jenkins job. 
>> 
>> 
>> Alan
>> 
>> ___
>> dev mailing list
>> dev@lists.openshift.redhat.com 
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev 
>> 
>> 
> 
> 
> 
> 
> -- 
> Ben Parees | OpenShift

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [aos-devel] optimizing go guru

[Luke Meyer]

If the answer is just "go guru is dog slow, use something else in that
case" then that seems like a useful thing to note in the README :) Along
with what people actually use in development. Seems like a number of tools
rely on guru but everyone complains about how slow it is on large projects.
So that's one more vote for VS...

On Tue, Dec 5, 2017 at 10:53 AM, Dan Mace  wrote:

>
>
> On Tue, Dec 5, 2017 at 10:43 AM, Luke Meyer  wrote:
>
>> In the context of the vim-go plugin. However behavior seems much the same
>> if I run the same command at the command line (I pulled it out of ps -ef).
>>
>> On Tue, Dec 5, 2017 at 10:40 AM, Sebastian Jug  wrote:
>>
>>> Are you using guru in some sort of editor/IDE or just standalone?
>>>
>>> On Dec 5, 2017 9:40 AM, "Luke Meyer"  wrote:
>>>


 On Tue, Dec 5, 2017 at 9:36 AM, Sebastian Jug  wrote:

> Sounds like you have got auto compile still on?
>
>
 What does this mean in the context of go guru? Is there an env var to
 set, an option to add, a config file to change to control this behavior?


>>>
>>
> ​The same query:
>
> guru -scope github.com/openshift/origin/cmd/oc whicherrs
> ./pkg/oc/admin/diagnostics/diagnostics.go:#7624
>
> was taking long enough for me (go1.8.3 darwin/amd64) that I killed it.
> It's hard to say without doing a deeper profile of that guru command. Even
> with your relatively narrow pointer analysis scope ​it seems really slow,
> but then again it's hard to gauge exactly how narrow that scope is without
> looking at a full import dependency graph...
>
> Guru has always been really slow for lots of useful pointer analysis
> queries, so I'm not entirely surprised. This is why vscode-go uses a
> variety of more optimized special purpose tools for most analysis[1].
>
> [1] https://github.com/Microsoft/vscode-go/blob/
> master/src/goInstallTools.ts#L21
>
> --
>
> Dan Mace
>
> Principal Software Engineer, OpenShift
>
> Red Hat
>
> dm...@redhat.com
>
>
>
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [aos-devel] optimizing go guru

[Luke Meyer]

On Tue, Dec 5, 2017 at 10:51 AM, Clayton Coleman 
wrote:

> Openshift and Kubernetes are massive go projects - over 3 million lines of
> code (last I checked).  Initial compile can take a few minutes for these
> tools.  Things to check:
>
> 1. Go 1.9 uses less memory when compiling
> 2. Be sure you are reusing your go compiled artifacts dir between multiple
> tools (sometimes that is GOPATH/pkg, but openshift explicitly only compiles
> temp packages into _output/local/pkgdir for reasons)
>


So if I make clean all and then run my guru command, won't that be reusing
compiled artifacts? Is there some config that controls this? I don't think
I've customized anything.

It does seem to speed up a little bit after the first run but then it's
still pretty slow.



> 3. Get faster laptop :)
>
> On Dec 5, 2017, at 9:44 AM, Luke Meyer  wrote:
>
> In the context of the vim-go plugin. However behavior seems much the same
> if I run the same command at the command line (I pulled it out of ps -ef).
>
> On Tue, Dec 5, 2017 at 10:40 AM, Sebastian Jug  wrote:
>
>> Are you using guru in some sort of editor/IDE or just standalone?
>>
>> On Dec 5, 2017 9:40 AM, "Luke Meyer"  wrote:
>>
>>>
>>>
>>> On Tue, Dec 5, 2017 at 9:36 AM, Sebastian Jug  wrote:
>>>
 Sounds like you have got auto compile still on?


>>> What does this mean in the context of go guru? Is there an env var to
>>> set, an option to add, a config file to change to control this behavior?
>>>
>>>
>>
> ___
> dev mailing list
> dev@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>
>
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [aos-devel] optimizing go guru

[Dan Mace]

On Tue, Dec 5, 2017 at 10:43 AM, Luke Meyer  wrote:

> In the context of the vim-go plugin. However behavior seems much the same
> if I run the same command at the command line (I pulled it out of ps -ef).
>
> On Tue, Dec 5, 2017 at 10:40 AM, Sebastian Jug  wrote:
>
>> Are you using guru in some sort of editor/IDE or just standalone?
>>
>> On Dec 5, 2017 9:40 AM, "Luke Meyer"  wrote:
>>
>>>
>>>
>>> On Tue, Dec 5, 2017 at 9:36 AM, Sebastian Jug  wrote:
>>>
 Sounds like you have got auto compile still on?


>>> What does this mean in the context of go guru? Is there an env var to
>>> set, an option to add, a config file to change to control this behavior?
>>>
>>>
>>
>
​The same query:

guru -scope github.com/openshift/origin/cmd/oc whicherrs
./pkg/oc/admin/diagnostics/diagnostics.go:#7624

was taking long enough for me (go1.8.3 darwin/amd64) that I killed it. It's
hard to say without doing a deeper profile of that guru command. Even with
your relatively narrow pointer analysis scope ​it seems really slow, but
then again it's hard to gauge exactly how narrow that scope is without
looking at a full import dependency graph...

Guru has always been really slow for lots of useful pointer analysis
queries, so I'm not entirely surprised. This is why vscode-go uses a
variety of more optimized special purpose tools for most analysis[1].

[1]
https://github.com/Microsoft/vscode-go/blob/master/src/goInstallTools.ts#L21

-- 

Dan Mace

Principal Software Engineer, OpenShift

Red Hat

dm...@redhat.com
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [aos-devel] optimizing go guru

[Clayton Coleman]

Openshift and Kubernetes are massive go projects - over 3 million lines of
code (last I checked).  Initial compile can take a few minutes for these
tools.  Things to check:

1. Go 1.9 uses less memory when compiling
2. Be sure you are reusing your go compiled artifacts dir between multiple
tools (sometimes that is GOPATH/pkg, but openshift explicitly only compiles
temp packages into _output/local/pkgdir for reasons)
3. Get faster laptop :)

On Dec 5, 2017, at 9:44 AM, Luke Meyer  wrote:

In the context of the vim-go plugin. However behavior seems much the same
if I run the same command at the command line (I pulled it out of ps -ef).

On Tue, Dec 5, 2017 at 10:40 AM, Sebastian Jug  wrote:

> Are you using guru in some sort of editor/IDE or just standalone?
>
> On Dec 5, 2017 9:40 AM, "Luke Meyer"  wrote:
>
>>
>>
>> On Tue, Dec 5, 2017 at 9:36 AM, Sebastian Jug  wrote:
>>
>>> Sounds like you have got auto compile still on?
>>>
>>>
>> What does this mean in the context of go guru? Is there an env var to
>> set, an option to add, a config file to change to control this behavior?
>>
>>
>
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Docker builds in an OpenShift Jenkins slave?

[Ben Parees]

On Tue, Dec 5, 2017 at 10:36 AM, Alan Christie <
achris...@informaticsmatters.com> wrote:

> Thanks Ben. It does seem sensible to use build strategies but prior to a
> wholesale migration to OpenShift, and for existing workflows that may
> contain docker and docker-compose commands is there any reasonable option
> other than a an external (cloud/proprietary/dedicated) docker-enabled
> slave? I can, for example, just have a Docker slave available (outside the
> OpenShift cluster) but that’s not ideal.
>
> Is there an _unsafe_ route I might be able to use now?
>

use DOCKER_HOST env variable and point to a host w/ a public docker.

The alternative is to try to use a hostpath volume definition in your slave
pod template but then you also need to run the slave pod as privileged.




> I understand the issues around sharing a docker.sock but it seems to be an
> acceptable strategy for many. And, for a controlled environment, just
> mounting docker.sock is a rather neat (quick-n-dirty) solution.
>
> It may be that, was you say there’s no sensible route down the
> OpenShift/CICD road other than build strategies. It’s just that for
> existing/legacy projects not having docker.sock is quite a hill to climb.
>
> Thanks for your advice though, that has been gratefully received.
>
> Alan.
>
> On 5 Dec 2017, at 13:41, Ben Parees  wrote:
>
>
>
>
>
> On Dec 5, 2017 07:57, "Alan Christie" 
> wrote:
>
> I’m using Jenkins from the CI/CD catalogue and am able to spin up slaves
> and use an `ImageStream` to identify my own slave image. That’s useful, but
> what I want to be able to do is build and run Docker images, primarily for
> unit/functional test purposes. The _sticking point_, it seems, is the
> ability to mount the host's `docker.sock`, without this I’m unable to run
> any Docker commands in my Docker containers.
>
> Q. Is there a way to mount the Jenkins/OpenShift host’s
> /var/run/docker.sock in my slave so that I can run Docker commands?
>
>
> Not safely. (mounting the host docker socket is giving out root access to
> your host).
>
> You could use a remote docker host with a certificate for access I
> believe. (that's still handing out root access on the docker host but at
> least it's a little protected)
>
> If not, what is the recommended/best practice for building/running/pushing
> Docker images from a slave agent?
>
>
> Define docker build strategies in openshift and trigger them from your
> jenkins job.
>
>
> Alan
>
> ___
> dev mailing list
> dev@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>
>
>
>


-- 
Ben Parees | OpenShift
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [aos-devel] optimizing go guru

[Luke Meyer]

In the context of the vim-go plugin. However behavior seems much the same
if I run the same command at the command line (I pulled it out of ps -ef).

On Tue, Dec 5, 2017 at 10:40 AM, Sebastian Jug  wrote:

> Are you using guru in some sort of editor/IDE or just standalone?
>
> On Dec 5, 2017 9:40 AM, "Luke Meyer"  wrote:
>
>>
>>
>> On Tue, Dec 5, 2017 at 9:36 AM, Sebastian Jug  wrote:
>>
>>> Sounds like you have got auto compile still on?
>>>
>>>
>> What does this mean in the context of go guru? Is there an env var to
>> set, an option to add, a config file to change to control this behavior?
>>
>>
>
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Fwd: [aos-devel] optimizing go guru

[Luke Meyer]

-- Forwarded message --
From: Luke Meyer 
Date: Tue, Dec 5, 2017 at 10:39 AM
Subject: Re: [aos-devel] optimizing go guru
To: Sebastian Jug 
Cc: dev 




On Tue, Dec 5, 2017 at 9:36 AM, Sebastian Jug  wrote:

> Sounds like you have got auto compile still on?
>
>
What does this mean in the context of go guru? Is there an env var to set,
an option to add, a config file to change to control this behavior?
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [aos-devel] optimizing go guru

[Luke Meyer]

In this case I'm also running into
https://github.com/openshift/origin/issues/17588 but perhaps it's all
related.

[origin] $ git describe
v3.9.0-alpha.0-11-ga5c80373e4
[origin] $ go version
go version go1.8.5 linux/amd64
[origin] $ echo $GOPATH
/home/lmeyer/go
[origin] $ time /home/lmeyer/go/bin/guru -scope
github.com/openshift/origin/cmd/oc whicherrs /home/lmeyer/go/src/
github.com/openshift/origin/pkg/oc/admin/diagnostics/diagnostics.go:#7624

/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/mtrmac/gpgme/data.go:4:11:
fatal error: gpgme.h: No such file or directory
 // #include 
   ^
compilation terminated.
cgo failed: [go tool cgo -objdir
/tmp/github.com_openshift_origin_vendor_github.com_mtrmac_gpgme_C519117980
-- -D_FILE_OFFSET_BITS=64 -I
/tmp/github.com_openshift_origin_vendor_github.com_mtrmac_gpgme_C519117980
data.go gpgme.go]: exit status 1
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:16:16:
Context not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:66:44:
Context not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:97:20:
NewDataBytes not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:101:14:
invalid operation: m.ctx (variable of type *invalid type) has no field or
method Import
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:106:20:
cannot range over res.Imports (invalid operand)
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:122:14:
invalid operation: m.ctx (variable of type *invalid type) has no field or
method GetKey
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:126:20:
NewDataBytes not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:131:18:
NewDataWriter not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:135:11:
invalid operation: m.ctx (variable of type *invalid type) has no field or
method Sign
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:135:25:
Key not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:135:61:
SigModeNormal not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:144:21:
NewDataWriter not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:148:34:
NewDataBytes not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:152:18:
invalid operation: m.ctx (variable of type *invalid type) has no field or
method Verify
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:161:42:
ValidityNever not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:67:14:
New not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:71:27:
ProtocolOpenPGP not declared by package gpgme
/home/lmeyer/go/src/
github.com/openshift/origin/vendor/github.com/containers/image/signature/mechanism_gpgme.go:75:28:
ProtocolOpenPGP not declared by package gpgme
guru: couldn't load packages due to errors:
github.com/openshift/origin/vendor/github.com/containers/image/signature,
github.com/openshift/origin/vendor/github.com/mtrmac/gpgme

real 0m10.041s
user 1m0.303s
sys 0m7.648s



On Tue, Dec 5, 2017 at 9:36 AM, Dan Mace  wrote:

>
>
> On Tue, Dec 5, 2017 at 9:31 AM, Luke Meyer  wrote:
>
>> I must be doing something wrong. Whenever go guru is fired off against
>> the origin codebase (for example, with godef or callstack or whicherrs) it
>> takes several seconds (or more) to do anything, sucking up GB of RAM and
>> all CPUs. I imagine it must be compiling the world, which is rather large.
>> Perhaps I am using the wrong scope. For instance, when working with oc, I
>> set scope to github.com/openshift/origin/cmd/oc. Should it be something
>> else? The guru docs are not as clear as they could be on exactly what
>> impact this has.
>>
>> How do you configure your development environment to do code analysis
>> efficiently against origin? T

Re: Docker builds in an OpenShift Jenkins slave?

[Alan Christie]

Thanks Ben. It does seem sensible to use build strategies but prior to a 
wholesale migration to OpenShift, and for existing workflows that may contain 
docker and docker-compose commands is there any reasonable option other than a 
an external (cloud/proprietary/dedicated) docker-enabled slave? I can, for 
example, just have a Docker slave available (outside the OpenShift cluster) but 
that’s not ideal.

Is there an _unsafe_ route I might be able to use now?

I understand the issues around sharing a docker.sock but it seems to be an 
acceptable strategy for many. And, for a controlled environment, just mounting 
docker.sock is a rather neat (quick-n-dirty) solution.

It may be that, was you say there’s no sensible route down the OpenShift/CICD 
road other than build strategies. It’s just that for existing/legacy projects 
not having docker.sock is quite a hill to climb.

Thanks for your advice though, that has been gratefully received.

Alan.

> On 5 Dec 2017, at 13:41, Ben Parees  wrote:
> 
> 
> 
> 
> 
> On Dec 5, 2017 07:57, "Alan Christie"  > wrote:
> I’m using Jenkins from the CI/CD catalogue and am able to spin up slaves and 
> use an `ImageStream` to identify my own slave image. That’s useful, but what 
> I want to be able to do is build and run Docker images, primarily for 
> unit/functional test purposes. The _sticking point_, it seems, is the ability 
> to mount the host's `docker.sock`, without this I’m unable to run any Docker 
> commands in my Docker containers.
> 
> Q. Is there a way to mount the Jenkins/OpenShift host’s /var/run/docker.sock 
> in my slave so that I can run Docker commands?
> 
> Not safely. (mounting the host docker socket is giving out root access to 
> your host). 
> 
> You could use a remote docker host with a certificate for access I believe. 
> (that's still handing out root access on the docker host but at least it's a 
> little protected) 
> 
> If not, what is the recommended/best practice for building/running/pushing 
> Docker images from a slave agent?
> 
> Define docker build strategies in openshift and trigger them from your 
> jenkins job. 
> 
> 
> Alan
> 
> ___
> dev mailing list
> dev@lists.openshift.redhat.com 
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev 
> 
> 

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

optimizing go guru

[Luke Meyer]

I must be doing something wrong. Whenever go guru is fired off against the
origin codebase (for example, with godef or callstack or whicherrs) it
takes several seconds (or more) to do anything, sucking up GB of RAM and
all CPUs. I imagine it must be compiling the world, which is rather large.
Perhaps I am using the wrong scope. For instance, when working with oc, I
set scope to github.com/openshift/origin/cmd/oc. Should it be something
else? The guru docs are not as clear as they could be on exactly what
impact this has.

How do you configure your development environment to do code analysis
efficiently against origin? The defaults don't seem to work too well, so if
there are any tips, it would be nice to have them in our dev readme(s).
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Docker builds in an OpenShift Jenkins slave?

[Ben Parees]

On Dec 5, 2017 07:57, "Alan Christie" 
wrote:

I’m using Jenkins from the CI/CD catalogue and am able to spin up slaves
and use an `ImageStream` to identify my own slave image. That’s useful, but
what I want to be able to do is build and run Docker images, primarily for
unit/functional test purposes. The _sticking point_, it seems, is the
ability to mount the host's `docker.sock`, without this I’m unable to run
any Docker commands in my Docker containers.

Q. Is there a way to mount the Jenkins/OpenShift host’s
/var/run/docker.sock in my slave so that I can run Docker commands?


Not safely. (mounting the host docker socket is giving out root access to
your host).

You could use a remote docker host with a certificate for access I believe.
(that's still handing out root access on the docker host but at least it's
a little protected)

If not, what is the recommended/best practice for building/running/pushing
Docker images from a slave agent?


Define docker build strategies in openshift and trigger them from your
jenkins job.


Alan

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Docker builds in an OpenShift Jenkins slave?

[Alan Christie]

I’m using Jenkins from the CI/CD catalogue and am able to spin up slaves and 
use an `ImageStream` to identify my own slave image. That’s useful, but what I 
want to be able to do is build and run Docker images, primarily for 
unit/functional test purposes. The _sticking point_, it seems, is the ability 
to mount the host's `docker.sock`, without this I’m unable to run any Docker 
commands in my Docker containers.

Q. Is there a way to mount the Jenkins/OpenShift host’s /var/run/docker.sock in 
my slave so that I can run Docker commands? If not, what is the 
recommended/best practice for building/running/pushing Docker images from a 
slave agent?

Alan

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev