Re: Seeking Advice On Exporting Image Streams

2017-08-09 Thread Clayton Coleman 
I don't know what that us to do with templates though - what "parameter"
applies?  The registry external name?

On Aug 9, 2017, at 1:22 PM, Cesar Wong  wrote:

The purpose of the endpoint is to parameterize an aspect of the resources
passed to it. We are initially implementing image references, but the idea
is to allow parameterizing other things such as environment variables,
names, etc. This is orthogonal to transforming the resources for export
which ‘GET’ with the export parameter needs to handle.

On Aug 9, 2017, at 1:20 PM, Clayton Coleman  wrote:

Why do we need parameters? Which parameters are we adding?

On Aug 9, 2017, at 12:21 PM, Cesar Wong  wrote:

Hi Devan,

You can see my branch here:
https://github.com/csrwng/origin/tree/parameterize_template
(last 5 commits)

Hopefully should be a PR soon. The REST endpoint should be functional, the
CLI still needs work, but basically the idea is to have the reverse of the
‘oc process’ command, where the input is a list of resources and out comes
a template with parameters.

On Aug 9, 2017, at 11:40 AM, Devan Goodwin  wrote:

On Wed, Aug 9, 2017 at 11:44 AM, Cesar Wong  wrote:

Hi Devan,

This past iteration I started work on this same problem [1]

https://trello.com/c/I2ZJxS94/998-5-improve-oc-export-to-parameterize-containerapppromotion

The problem is broad and the way I decided to break it up is to consider the
export and parameterize operations independently. The export should be
handled by the resource’s strategy as you mentioned in the Kube issue you
opened. The parameterization part can be a follow up to the export. Here’s
an initial document describing it:

https://docs.google.com/a/redhat.com/document/d/15SLkhXRovY1dLbxpWFy_Wfq3I6xMznsOAnopTYrXw_A/edit?usp=sharing


Thanks that was a good read, will keep an eye on this document.

Does anything exist yet for your parameterization code? Curious what
it looks like and if it's something we could re-use yet, what the
inputs and outputs are, etc.


On the export side, I think we need to decide whether there is different
“types” of export that can happen which should affect the logic of the
resource strategy. For example, does a deployment config look different if
you’re exporting it for use in a different namespace vs a different cluster.
If this is the case, then right now is probably a good time to drive that
change to the upstream API as David suggested.


Is anyone working on a proposal for this export logic upstream? I am
wondering if I should try to put one together if I can find the time.
The general idea (as I understand it) would be to migrate the
currently quite broken export=true param to something strategy based,
and interpret "true" to mean a strategy that matches what we do today.
The references in code I've seen indicate that the current intention
is to strip anything the user cannot specify themselves.




On Aug 9, 2017, at 10:27 AM, Ben Parees  wrote:



On Wed, Aug 9, 2017 at 10:00 AM, Devan Goodwin  wrote:


On Wed, Aug 9, 2017 at 9:58 AM, Ben Parees  wrote:



On Wed, Aug 9, 2017 at 8:49 AM, Devan Goodwin 
wrote:


We are working on a more robust project export/import process (into a
new namespace, possibly a new cluster, etc) and have a question on how
to handle image streams.

Our first test was with "oc new-app
https://github.com/openshift/ruby-hello-world.git;, this results in an
image stream like the following:

$ oc get is ruby-hello-world -o yaml
apiVersion: v1
kind: ImageStream
metadata:
 annotations:
   openshift.io/generated-by: OpenShiftNewApp
 creationTimestamp: 2017-08-08T12:01:22Z
 generation: 1
 labels:
   app: ruby-hello-world
 name: ruby-hello-world
 namespace: project1
 resourceVersion: "183991"
 selfLink: /oapi/v1/namespaces/project1/imagestreams/ruby-hello-world
 uid: 4bd229be-7c31-11e7-badf-989096de63cb
spec:
 lookupPolicy:
   local: false
status:
 dockerImageRepository: 172.30.1.1:5000/project1/ruby-hello-world
 tags:
 - items:
   - created: 2017-08-08T12:02:04Z
 dockerImageReference:


172.30.1.1:5000/project1/ruby-hello-world@sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
 generation: 1
 image:
sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
   tag: latest


If we link up with the kubernetes resource exporting by adding
--export:

$ oc get is ruby-hello-world -o yaml --export
apiVersion: v1
kind: ImageStream
metadata:
 annotations:
   openshift.io/generated-by: OpenShiftNewApp
 creationTimestamp: null
 generation: 1
 labels:
   app: ruby-hello-world
 name: ruby-hello-world
 namespace: default
 selfLink: /oapi/v1/namespaces/default/imagestreams/ruby-hello-world
spec:
 lookupPolicy:
   local: false
status:
 dockerImageRepository: 172.30.1.1:5000/default/ruby-hello-world


This leads to an initial question, what stripped

Re: Seeking Advice On Exporting Image Streams

2017-08-09 Thread Devan Goodwin 
On Wed, Aug 9, 2017 at 2:20 PM, Clayton Coleman  wrote:
> Why do we need parameters? Which parameters are we adding?

For our use case, the idea was detect image refs that point to the
internal registry and the old project name during archival/export,
parameterize, and provide the integrated registry IP and new project
name as params during unarchival/import.

>
> On Aug 9, 2017, at 12:21 PM, Cesar Wong  wrote:
>
> Hi Devan,
>
> You can see my branch here:
> https://github.com/csrwng/origin/tree/parameterize_template
> (last 5 commits)
>
> Hopefully should be a PR soon. The REST endpoint should be functional, the
> CLI still needs work, but basically the idea is to have the reverse of the
> ‘oc process’ command, where the input is a list of resources and out comes a
> template with parameters.
>
> On Aug 9, 2017, at 11:40 AM, Devan Goodwin  wrote:
>
> On Wed, Aug 9, 2017 at 11:44 AM, Cesar Wong  wrote:
>
> Hi Devan,
>
> This past iteration I started work on this same problem [1]
>
> https://trello.com/c/I2ZJxS94/998-5-improve-oc-export-to-parameterize-containerapppromotion
>
> The problem is broad and the way I decided to break it up is to consider the
> export and parameterize operations independently. The export should be
> handled by the resource’s strategy as you mentioned in the Kube issue you
> opened. The parameterization part can be a follow up to the export. Here’s
> an initial document describing it:
>
> https://docs.google.com/a/redhat.com/document/d/15SLkhXRovY1dLbxpWFy_Wfq3I6xMznsOAnopTYrXw_A/edit?usp=sharing
>
>
> Thanks that was a good read, will keep an eye on this document.
>
> Does anything exist yet for your parameterization code? Curious what
> it looks like and if it's something we could re-use yet, what the
> inputs and outputs are, etc.
>
>
> On the export side, I think we need to decide whether there is different
> “types” of export that can happen which should affect the logic of the
> resource strategy. For example, does a deployment config look different if
> you’re exporting it for use in a different namespace vs a different cluster.
> If this is the case, then right now is probably a good time to drive that
> change to the upstream API as David suggested.
>
>
> Is anyone working on a proposal for this export logic upstream? I am
> wondering if I should try to put one together if I can find the time.
> The general idea (as I understand it) would be to migrate the
> currently quite broken export=true param to something strategy based,
> and interpret "true" to mean a strategy that matches what we do today.
> The references in code I've seen indicate that the current intention
> is to strip anything the user cannot specify themselves.
>
>
>
>
> On Aug 9, 2017, at 10:27 AM, Ben Parees  wrote:
>
>
>
> On Wed, Aug 9, 2017 at 10:00 AM, Devan Goodwin  wrote:
>
>
> On Wed, Aug 9, 2017 at 9:58 AM, Ben Parees  wrote:
>
>
>
> On Wed, Aug 9, 2017 at 8:49 AM, Devan Goodwin 
> wrote:
>
>
> We are working on a more robust project export/import process (into a
> new namespace, possibly a new cluster, etc) and have a question on how
> to handle image streams.
>
> Our first test was with "oc new-app
> https://github.com/openshift/ruby-hello-world.git;, this results in an
> image stream like the following:
>
> $ oc get is ruby-hello-world -o yaml
> apiVersion: v1
> kind: ImageStream
> metadata:
>  annotations:
>openshift.io/generated-by: OpenShiftNewApp
>  creationTimestamp: 2017-08-08T12:01:22Z
>  generation: 1
>  labels:
>app: ruby-hello-world
>  name: ruby-hello-world
>  namespace: project1
>  resourceVersion: "183991"
>  selfLink: /oapi/v1/namespaces/project1/imagestreams/ruby-hello-world
>  uid: 4bd229be-7c31-11e7-badf-989096de63cb
> spec:
>  lookupPolicy:
>local: false
> status:
>  dockerImageRepository: 172.30.1.1:5000/project1/ruby-hello-world
>  tags:
>  - items:
>- created: 2017-08-08T12:02:04Z
>  dockerImageReference:
>
>
> 172.30.1.1:5000/project1/ruby-hello-world@sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>  generation: 1
>  image:
> sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>tag: latest
>
>
> If we link up with the kubernetes resource exporting by adding
> --export:
>
> $ oc get is ruby-hello-world -o yaml --export
> apiVersion: v1
> kind: ImageStream
> metadata:
>  annotations:
>openshift.io/generated-by: OpenShiftNewApp
>  creationTimestamp: null
>  generation: 1
>  labels:
>app: ruby-hello-world
>  name: ruby-hello-world
>  namespace: default
>  selfLink: /oapi/v1/namespaces/default/imagestreams/ruby-hello-world
> spec:
>  lookupPolicy:
>local: false
> status:
>  dockerImageRepository: 172.30.1.1:5000/default/ruby-hello-world
>
>
> This leads to an initial question, what stripped the status tags? I
> would have expected

Re: Seeking Advice On Exporting Image Streams

2017-08-09 Thread Clayton Coleman 
Why do we need parameters? Which parameters are we adding?

On Aug 9, 2017, at 12:21 PM, Cesar Wong  wrote:

Hi Devan,

You can see my branch here:
https://github.com/csrwng/origin/tree/parameterize_template
(last 5 commits)

Hopefully should be a PR soon. The REST endpoint should be functional, the
CLI still needs work, but basically the idea is to have the reverse of the
‘oc process’ command, where the input is a list of resources and out comes
a template with parameters.

On Aug 9, 2017, at 11:40 AM, Devan Goodwin  wrote:

On Wed, Aug 9, 2017 at 11:44 AM, Cesar Wong  wrote:

Hi Devan,

This past iteration I started work on this same problem [1]

https://trello.com/c/I2ZJxS94/998-5-improve-oc-export-to-parameterize-containerapppromotion

The problem is broad and the way I decided to break it up is to consider the
export and parameterize operations independently. The export should be
handled by the resource’s strategy as you mentioned in the Kube issue you
opened. The parameterization part can be a follow up to the export. Here’s
an initial document describing it:

https://docs.google.com/a/redhat.com/document/d/15SLkhXRovY1dLbxpWFy_Wfq3I6xMznsOAnopTYrXw_A/edit?usp=sharing


Thanks that was a good read, will keep an eye on this document.

Does anything exist yet for your parameterization code? Curious what
it looks like and if it's something we could re-use yet, what the
inputs and outputs are, etc.


On the export side, I think we need to decide whether there is different
“types” of export that can happen which should affect the logic of the
resource strategy. For example, does a deployment config look different if
you’re exporting it for use in a different namespace vs a different cluster.
If this is the case, then right now is probably a good time to drive that
change to the upstream API as David suggested.


Is anyone working on a proposal for this export logic upstream? I am
wondering if I should try to put one together if I can find the time.
The general idea (as I understand it) would be to migrate the
currently quite broken export=true param to something strategy based,
and interpret "true" to mean a strategy that matches what we do today.
The references in code I've seen indicate that the current intention
is to strip anything the user cannot specify themselves.




On Aug 9, 2017, at 10:27 AM, Ben Parees  wrote:



On Wed, Aug 9, 2017 at 10:00 AM, Devan Goodwin  wrote:


On Wed, Aug 9, 2017 at 9:58 AM, Ben Parees  wrote:



On Wed, Aug 9, 2017 at 8:49 AM, Devan Goodwin 
wrote:


We are working on a more robust project export/import process (into a
new namespace, possibly a new cluster, etc) and have a question on how
to handle image streams.

Our first test was with "oc new-app
https://github.com/openshift/ruby-hello-world.git;, this results in an
image stream like the following:

$ oc get is ruby-hello-world -o yaml
apiVersion: v1
kind: ImageStream
metadata:
 annotations:
   openshift.io/generated-by: OpenShiftNewApp
 creationTimestamp: 2017-08-08T12:01:22Z
 generation: 1
 labels:
   app: ruby-hello-world
 name: ruby-hello-world
 namespace: project1
 resourceVersion: "183991"
 selfLink: /oapi/v1/namespaces/project1/imagestreams/ruby-hello-world
 uid: 4bd229be-7c31-11e7-badf-989096de63cb
spec:
 lookupPolicy:
   local: false
status:
 dockerImageRepository: 172.30.1.1:5000/project1/ruby-hello-world
 tags:
 - items:
   - created: 2017-08-08T12:02:04Z
 dockerImageReference:


172.30.1.1:5000/project1/ruby-hello-world@sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
 generation: 1
 image:
sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
   tag: latest


If we link up with the kubernetes resource exporting by adding
--export:

$ oc get is ruby-hello-world -o yaml --export
apiVersion: v1
kind: ImageStream
metadata:
 annotations:
   openshift.io/generated-by: OpenShiftNewApp
 creationTimestamp: null
 generation: 1
 labels:
   app: ruby-hello-world
 name: ruby-hello-world
 namespace: default
 selfLink: /oapi/v1/namespaces/default/imagestreams/ruby-hello-world
spec:
 lookupPolicy:
   local: false
status:
 dockerImageRepository: 172.30.1.1:5000/default/ruby-hello-world


This leads to an initial question, what stripped the status tags? I
would have expected this code to live in the image stream strategy:


https://github.com/openshift/origin/blob/master/pkg/image/registry/imagestream/strategy.go
but this does not satisfy RESTExportStrategy, I wasn't able to
determine where this is happening.

The dockerImageRepository in status remains, but weirdly flips from
"project1" to "default" when doing an export. Should this remain in an
exported IS at all? And if so is there any reason why it would flip
from project1 to default?

Our real problem however picks up in the deployment config after
import, in here we end

Re: Seeking Advice On Exporting Image Streams

2017-08-09 Thread Cesar Wong 
Hi Devan,

You can see my branch here:
https://github.com/csrwng/origin/tree/parameterize_template 

(last 5 commits)

Hopefully should be a PR soon. The REST endpoint should be functional, the CLI 
still needs work, but basically the idea is to have the reverse of the ‘oc 
process’ command, where the input is a list of resources and out comes a 
template with parameters.

> On Aug 9, 2017, at 11:40 AM, Devan Goodwin  wrote:
> 
> On Wed, Aug 9, 2017 at 11:44 AM, Cesar Wong  wrote:
>> Hi Devan,
>> 
>> This past iteration I started work on this same problem [1]
>> 
>> https://trello.com/c/I2ZJxS94/998-5-improve-oc-export-to-parameterize-containerapppromotion
>> 
>> The problem is broad and the way I decided to break it up is to consider the
>> export and parameterize operations independently. The export should be
>> handled by the resource’s strategy as you mentioned in the Kube issue you
>> opened. The parameterization part can be a follow up to the export. Here’s
>> an initial document describing it:
>> 
>> https://docs.google.com/a/redhat.com/document/d/15SLkhXRovY1dLbxpWFy_Wfq3I6xMznsOAnopTYrXw_A/edit?usp=sharing
> 
> Thanks that was a good read, will keep an eye on this document.
> 
> Does anything exist yet for your parameterization code? Curious what
> it looks like and if it's something we could re-use yet, what the
> inputs and outputs are, etc.
> 
>> 
>> On the export side, I think we need to decide whether there is different
>> “types” of export that can happen which should affect the logic of the
>> resource strategy. For example, does a deployment config look different if
>> you’re exporting it for use in a different namespace vs a different cluster.
>> If this is the case, then right now is probably a good time to drive that
>> change to the upstream API as David suggested.
> 
> Is anyone working on a proposal for this export logic upstream? I am
> wondering if I should try to put one together if I can find the time.
> The general idea (as I understand it) would be to migrate the
> currently quite broken export=true param to something strategy based,
> and interpret "true" to mean a strategy that matches what we do today.
> The references in code I've seen indicate that the current intention
> is to strip anything the user cannot specify themselves.
> 
> 
> 
>> 
>> On Aug 9, 2017, at 10:27 AM, Ben Parees  wrote:
>> 
>> 
>> 
>> On Wed, Aug 9, 2017 at 10:00 AM, Devan Goodwin  wrote:
>>> 
>>> On Wed, Aug 9, 2017 at 9:58 AM, Ben Parees  wrote:
 
 
 On Wed, Aug 9, 2017 at 8:49 AM, Devan Goodwin 
 wrote:
> 
> We are working on a more robust project export/import process (into a
> new namespace, possibly a new cluster, etc) and have a question on how
> to handle image streams.
> 
> Our first test was with "oc new-app
> https://github.com/openshift/ruby-hello-world.git;, this results in an
> image stream like the following:
> 
> $ oc get is ruby-hello-world -o yaml
> apiVersion: v1
> kind: ImageStream
> metadata:
>  annotations:
>openshift.io/generated-by: OpenShiftNewApp
>  creationTimestamp: 2017-08-08T12:01:22Z
>  generation: 1
>  labels:
>app: ruby-hello-world
>  name: ruby-hello-world
>  namespace: project1
>  resourceVersion: "183991"
>  selfLink: /oapi/v1/namespaces/project1/imagestreams/ruby-hello-world
>  uid: 4bd229be-7c31-11e7-badf-989096de63cb
> spec:
>  lookupPolicy:
>local: false
> status:
>  dockerImageRepository: 172.30.1.1:5000/project1/ruby-hello-world
>  tags:
>  - items:
>- created: 2017-08-08T12:02:04Z
>  dockerImageReference:
> 
> 
> 172.30.1.1:5000/project1/ruby-hello-world@sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>  generation: 1
>  image:
> sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>tag: latest
> 
> 
> If we link up with the kubernetes resource exporting by adding
> --export:
> 
> $ oc get is ruby-hello-world -o yaml --export
> apiVersion: v1
> kind: ImageStream
> metadata:
>  annotations:
>openshift.io/generated-by: OpenShiftNewApp
>  creationTimestamp: null
>  generation: 1
>  labels:
>app: ruby-hello-world
>  name: ruby-hello-world
>  namespace: default
>  selfLink: /oapi/v1/namespaces/default/imagestreams/ruby-hello-world
> spec:
>  lookupPolicy:
>local: false
> status:
>  dockerImageRepository: 172.30.1.1:5000/default/ruby-hello-world
> 
> 
> This leads to an initial question, what stripped the status tags? I
> would have expected this code to live in the image stream strategy:
> 
> 
>

Re: Seeking Advice On Exporting Image Streams

2017-08-09 Thread Devan Goodwin 
On Wed, Aug 9, 2017 at 11:44 AM, Cesar Wong  wrote:
> Hi Devan,
>
> This past iteration I started work on this same problem [1]
>
> https://trello.com/c/I2ZJxS94/998-5-improve-oc-export-to-parameterize-containerapppromotion
>
> The problem is broad and the way I decided to break it up is to consider the
> export and parameterize operations independently. The export should be
> handled by the resource’s strategy as you mentioned in the Kube issue you
> opened. The parameterization part can be a follow up to the export. Here’s
> an initial document describing it:
>
> https://docs.google.com/a/redhat.com/document/d/15SLkhXRovY1dLbxpWFy_Wfq3I6xMznsOAnopTYrXw_A/edit?usp=sharing

Thanks that was a good read, will keep an eye on this document.

Does anything exist yet for your parameterization code? Curious what
it looks like and if it's something we could re-use yet, what the
inputs and outputs are, etc.

>
> On the export side, I think we need to decide whether there is different
> “types” of export that can happen which should affect the logic of the
> resource strategy. For example, does a deployment config look different if
> you’re exporting it for use in a different namespace vs a different cluster.
> If this is the case, then right now is probably a good time to drive that
> change to the upstream API as David suggested.

Is anyone working on a proposal for this export logic upstream? I am
wondering if I should try to put one together if I can find the time.
The general idea (as I understand it) would be to migrate the
currently quite broken export=true param to something strategy based,
and interpret "true" to mean a strategy that matches what we do today.
The references in code I've seen indicate that the current intention
is to strip anything the user cannot specify themselves.



>
> On Aug 9, 2017, at 10:27 AM, Ben Parees  wrote:
>
>
>
> On Wed, Aug 9, 2017 at 10:00 AM, Devan Goodwin  wrote:
>>
>> On Wed, Aug 9, 2017 at 9:58 AM, Ben Parees  wrote:
>> >
>> >
>> > On Wed, Aug 9, 2017 at 8:49 AM, Devan Goodwin 
>> > wrote:
>> >>
>> >> We are working on a more robust project export/import process (into a
>> >> new namespace, possibly a new cluster, etc) and have a question on how
>> >> to handle image streams.
>> >>
>> >> Our first test was with "oc new-app
>> >> https://github.com/openshift/ruby-hello-world.git;, this results in an
>> >> image stream like the following:
>> >>
>> >> $ oc get is ruby-hello-world -o yaml
>> >> apiVersion: v1
>> >> kind: ImageStream
>> >> metadata:
>> >>   annotations:
>> >> openshift.io/generated-by: OpenShiftNewApp
>> >>   creationTimestamp: 2017-08-08T12:01:22Z
>> >>   generation: 1
>> >>   labels:
>> >> app: ruby-hello-world
>> >>   name: ruby-hello-world
>> >>   namespace: project1
>> >>   resourceVersion: "183991"
>> >>   selfLink: /oapi/v1/namespaces/project1/imagestreams/ruby-hello-world
>> >>   uid: 4bd229be-7c31-11e7-badf-989096de63cb
>> >> spec:
>> >>   lookupPolicy:
>> >> local: false
>> >> status:
>> >>   dockerImageRepository: 172.30.1.1:5000/project1/ruby-hello-world
>> >>   tags:
>> >>   - items:
>> >> - created: 2017-08-08T12:02:04Z
>> >>   dockerImageReference:
>> >>
>> >>
>> >> 172.30.1.1:5000/project1/ruby-hello-world@sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>> >>   generation: 1
>> >>   image:
>> >> sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>> >> tag: latest
>> >>
>> >>
>> >> If we link up with the kubernetes resource exporting by adding
>> >> --export:
>> >>
>> >> $ oc get is ruby-hello-world -o yaml --export
>> >> apiVersion: v1
>> >> kind: ImageStream
>> >> metadata:
>> >>   annotations:
>> >> openshift.io/generated-by: OpenShiftNewApp
>> >>   creationTimestamp: null
>> >>   generation: 1
>> >>   labels:
>> >> app: ruby-hello-world
>> >>   name: ruby-hello-world
>> >>   namespace: default
>> >>   selfLink: /oapi/v1/namespaces/default/imagestreams/ruby-hello-world
>> >> spec:
>> >>   lookupPolicy:
>> >> local: false
>> >> status:
>> >>   dockerImageRepository: 172.30.1.1:5000/default/ruby-hello-world
>> >>
>> >>
>> >> This leads to an initial question, what stripped the status tags? I
>> >> would have expected this code to live in the image stream strategy:
>> >>
>> >>
>> >> https://github.com/openshift/origin/blob/master/pkg/image/registry/imagestream/strategy.go
>> >> but this does not satisfy RESTExportStrategy, I wasn't able to
>> >> determine where this is happening.
>> >>
>> >> The dockerImageRepository in status remains, but weirdly flips from
>> >> "project1" to "default" when doing an export. Should this remain in an
>> >> exported IS at all? And if so is there any reason why it would flip
>> >> from project1 to default?
>> >>
>> >> Our real problem however picks up in the deployment config after
>> >> import, in here we end up with the

Re: Seeking Advice On Exporting Image Streams

2017-08-09 Thread Cesar Wong 
Hi Devan,

This past iteration I started work on this same problem [1]

https://trello.com/c/I2ZJxS94/998-5-improve-oc-export-to-parameterize-containerapppromotion
 


The problem is broad and the way I decided to break it up is to consider the 
export and parameterize operations independently. The export should be handled 
by the resource’s strategy as you mentioned in the Kube issue you opened. The 
parameterization part can be a follow up to the export. Here’s an initial 
document describing it:

https://docs.google.com/a/redhat.com/document/d/15SLkhXRovY1dLbxpWFy_Wfq3I6xMznsOAnopTYrXw_A/edit?usp=sharing
 


On the export side, I think we need to decide whether there is different 
“types” of export that can happen which should affect the logic of the resource 
strategy. For example, does a deployment config look different if you’re 
exporting it for use in a different namespace vs a different cluster. If this 
is the case, then right now is probably a good time to drive that change to the 
upstream API as David suggested.

> On Aug 9, 2017, at 10:27 AM, Ben Parees  wrote:
> 
> 
> 
> On Wed, Aug 9, 2017 at 10:00 AM, Devan Goodwin  > wrote:
> On Wed, Aug 9, 2017 at 9:58 AM, Ben Parees  > wrote:
> >
> >
> > On Wed, Aug 9, 2017 at 8:49 AM, Devan Goodwin  > > wrote:
> >>
> >> We are working on a more robust project export/import process (into a
> >> new namespace, possibly a new cluster, etc) and have a question on how
> >> to handle image streams.
> >>
> >> Our first test was with "oc new-app
> >> https://github.com/openshift/ruby-hello-world.git 
> >> ", this results in an
> >> image stream like the following:
> >>
> >> $ oc get is ruby-hello-world -o yaml
> >> apiVersion: v1
> >> kind: ImageStream
> >> metadata:
> >>   annotations:
> >> openshift.io/generated-by : 
> >> OpenShiftNewApp
> >>   creationTimestamp: 2017-08-08T12:01:22Z
> >>   generation: 1
> >>   labels:
> >> app: ruby-hello-world
> >>   name: ruby-hello-world
> >>   namespace: project1
> >>   resourceVersion: "183991"
> >>   selfLink: /oapi/v1/namespaces/project1/imagestreams/ruby-hello-world
> >>   uid: 4bd229be-7c31-11e7-badf-989096de63cb
> >> spec:
> >>   lookupPolicy:
> >> local: false
> >> status:
> >>   dockerImageRepository: 172.30.1.1:5000/project1/ruby-hello-world 
> >> 
> >>   tags:
> >>   - items:
> >> - created: 2017-08-08T12:02:04Z
> >>   dockerImageReference:
> >>
> >> 172.30.1.1:5000/project1/ruby-hello-world@sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
> >>  
> >> 
> >>   generation: 1
> >>   image:
> >> sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
> >> tag: latest
> >>
> >>
> >> If we link up with the kubernetes resource exporting by adding --export:
> >>
> >> $ oc get is ruby-hello-world -o yaml --export
> >> apiVersion: v1
> >> kind: ImageStream
> >> metadata:
> >>   annotations:
> >> openshift.io/generated-by : 
> >> OpenShiftNewApp
> >>   creationTimestamp: null
> >>   generation: 1
> >>   labels:
> >> app: ruby-hello-world
> >>   name: ruby-hello-world
> >>   namespace: default
> >>   selfLink: /oapi/v1/namespaces/default/imagestreams/ruby-hello-world
> >> spec:
> >>   lookupPolicy:
> >> local: false
> >> status:
> >>   dockerImageRepository: 172.30.1.1:5000/default/ruby-hello-world 
> >> 
> >>
> >>
> >> This leads to an initial question, what stripped the status tags? I
> >> would have expected this code to live in the image stream strategy:
> >>
> >> https://github.com/openshift/origin/blob/master/pkg/image/registry/imagestream/strategy.go
> >>  
> >> 
> >> but this does not satisfy RESTExportStrategy, I wasn't able to
> >> determine where this is happening.
> >>
> >> The dockerImageRepository in status remains, but weirdly flips from
> >> "project1" to "default" when doing an export. Should this remain in an
> >> exported IS at all? And if so is there any reason why it would flip
> >> from project1 to default?
> >>
> >> Our real problem however picks up in the deployment config after
> >> import, in here we end up with the following (partial) DC:
> >>
> >> apiVersion: v1
> >> kind: DeploymentConfig
> >> metadata:
> >>

Re: Seeking Advice On Exporting Image Streams

2017-08-09 Thread Devan Goodwin 
On Wed, Aug 9, 2017 at 9:58 AM, Ben Parees  wrote:
>
>
> On Wed, Aug 9, 2017 at 8:49 AM, Devan Goodwin  wrote:
>>
>> We are working on a more robust project export/import process (into a
>> new namespace, possibly a new cluster, etc) and have a question on how
>> to handle image streams.
>>
>> Our first test was with "oc new-app
>> https://github.com/openshift/ruby-hello-world.git;, this results in an
>> image stream like the following:
>>
>> $ oc get is ruby-hello-world -o yaml
>> apiVersion: v1
>> kind: ImageStream
>> metadata:
>>   annotations:
>> openshift.io/generated-by: OpenShiftNewApp
>>   creationTimestamp: 2017-08-08T12:01:22Z
>>   generation: 1
>>   labels:
>> app: ruby-hello-world
>>   name: ruby-hello-world
>>   namespace: project1
>>   resourceVersion: "183991"
>>   selfLink: /oapi/v1/namespaces/project1/imagestreams/ruby-hello-world
>>   uid: 4bd229be-7c31-11e7-badf-989096de63cb
>> spec:
>>   lookupPolicy:
>> local: false
>> status:
>>   dockerImageRepository: 172.30.1.1:5000/project1/ruby-hello-world
>>   tags:
>>   - items:
>> - created: 2017-08-08T12:02:04Z
>>   dockerImageReference:
>>
>> 172.30.1.1:5000/project1/ruby-hello-world@sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>>   generation: 1
>>   image:
>> sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>> tag: latest
>>
>>
>> If we link up with the kubernetes resource exporting by adding --export:
>>
>> $ oc get is ruby-hello-world -o yaml --export
>> apiVersion: v1
>> kind: ImageStream
>> metadata:
>>   annotations:
>> openshift.io/generated-by: OpenShiftNewApp
>>   creationTimestamp: null
>>   generation: 1
>>   labels:
>> app: ruby-hello-world
>>   name: ruby-hello-world
>>   namespace: default
>>   selfLink: /oapi/v1/namespaces/default/imagestreams/ruby-hello-world
>> spec:
>>   lookupPolicy:
>> local: false
>> status:
>>   dockerImageRepository: 172.30.1.1:5000/default/ruby-hello-world
>>
>>
>> This leads to an initial question, what stripped the status tags? I
>> would have expected this code to live in the image stream strategy:
>>
>> https://github.com/openshift/origin/blob/master/pkg/image/registry/imagestream/strategy.go
>> but this does not satisfy RESTExportStrategy, I wasn't able to
>> determine where this is happening.
>>
>> The dockerImageRepository in status remains, but weirdly flips from
>> "project1" to "default" when doing an export. Should this remain in an
>> exported IS at all? And if so is there any reason why it would flip
>> from project1 to default?
>>
>> Our real problem however picks up in the deployment config after
>> import, in here we end up with the following (partial) DC:
>>
>> apiVersion: v1
>> kind: DeploymentConfig
>> metadata:
>>   annotations:
>> openshift.io/generated-by: OpenShiftNewApp
>>   labels:
>> app: ruby-hello-world
>>   name: ruby-hello-world
>>   namespace: project2
>>   selfLink:
>> /oapi/v1/namespaces/project2/deploymentconfigs/ruby-hello-world
>> spec:
>>   template:
>> metadata:
>>   annotations:
>> openshift.io/generated-by: OpenShiftNewApp
>>   labels:
>> app: ruby-hello-world
>> deploymentconfig: ruby-hello-world
>> spec:
>>   containers:
>>   - image:
>> 172.30.1.1:5000/project1/ruby-hello-world@sha256:8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>> imagePullPolicy: Always
>> name: ruby-hello-world
>>
>> So our deployment config still refers to a very specific image and
>> points to the old project. Is there any logic we could apply safely to
>> address this?
>>
>> It feels like this should boil down to something like
>> "ruby-hello-world@sha256:HASH", could we watch for
>> $REGISTRY_IP:PORT/projectname/ during export and strip that leading
>> portion out? What would be the risks in doing so?
>
>
> Adding Cesar since he was recently looking at some of the export logic you
> have questions about and he's also very interested in this subject since
> he's working on a related piece of functionality.  That said:
>
> if you've got an imagechangetrigger in the DC you should be able to strip
> the entire image field (it should be repopulated from the ICT imagestream
> reference during deployment).  However:

Ok good, so during export we can iterate the image change triggers, if
we see one we can match up on containerName and strip container.image
for that name.

>
> 1) you still need to straighten out the ICT reference which is also going to
> be pointing to an imagestreamtag in the old project/cluster/whatever

Ok I think we can handle this explicitly. More below though.

> 2) if you don't have an ICT reference you do need to sort this out and
> stripping it the way you propose is definitely not a good idea...what's
> going to repopulate that w/ the right prefix/project in the new cluster?
> What if the image field was pointing to docker.io or some other

Re: Seeking Advice On Exporting Image Streams

2017-08-09 Thread Ben Parees 
On Wed, Aug 9, 2017 at 8:49 AM, Devan Goodwin  wrote:

> We are working on a more robust project export/import process (into a
> new namespace, possibly a new cluster, etc) and have a question on how
> to handle image streams.
>
> Our first test was with "oc new-app
> https://github.com/openshift/ruby-hello-world.git;, this results in an
> image stream like the following:
>
> $ oc get is ruby-hello-world -o yaml
> apiVersion: v1
> kind: ImageStream
> metadata:
>   annotations:
> openshift.io/generated-by: OpenShiftNewApp
>   creationTimestamp: 2017-08-08T12:01:22Z
>   generation: 1
>   labels:
> app: ruby-hello-world
>   name: ruby-hello-world
>   namespace: project1
>   resourceVersion: "183991"
>   selfLink: /oapi/v1/namespaces/project1/imagestreams/ruby-hello-world
>   uid: 4bd229be-7c31-11e7-badf-989096de63cb
> spec:
>   lookupPolicy:
> local: false
> status:
>   dockerImageRepository: 172.30.1.1:5000/project1/ruby-hello-world
>   tags:
>   - items:
> - created: 2017-08-08T12:02:04Z
>   dockerImageReference:
> 172.30.1.1:5000/project1/ruby-hello-world@sha256:
> 8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
>   generation: 1
>   image: sha256:8d0f81a13ec1b8f8fa4372d26075f0
> dd87578fba2ec120776133db71ce2c2074
> tag: latest
>
>
> If we link up with the kubernetes resource exporting by adding --export:
>
> $ oc get is ruby-hello-world -o yaml --export
> apiVersion: v1
> kind: ImageStream
> metadata:
>   annotations:
> openshift.io/generated-by: OpenShiftNewApp
>   creationTimestamp: null
>   generation: 1
>   labels:
> app: ruby-hello-world
>   name: ruby-hello-world
>   namespace: default
>   selfLink: /oapi/v1/namespaces/default/imagestreams/ruby-hello-world
> spec:
>   lookupPolicy:
> local: false
> status:
>   dockerImageRepository: 172.30.1.1:5000/default/ruby-hello-world
>
>
> This leads to an initial question, what stripped the status tags? I
> would have expected this code to live in the image stream strategy:
> https://github.com/openshift/origin/blob/master/pkg/image/
> registry/imagestream/strategy.go
> but this does not satisfy RESTExportStrategy, I wasn't able to
> determine where this is happening.
>
> The dockerImageRepository in status remains, but weirdly flips from
> "project1" to "default" when doing an export. Should this remain in an
> exported IS at all? And if so is there any reason why it would flip
> from project1 to default?
>
> Our real problem however picks up in the deployment config after
> import, in here we end up with the following (partial) DC:
>
> apiVersion: v1
> kind: DeploymentConfig
> metadata:
>   annotations:
> openshift.io/generated-by: OpenShiftNewApp
>   labels:
> app: ruby-hello-world
>   name: ruby-hello-world
>   namespace: project2
>   selfLink: /oapi/v1/namespaces/project2/deploymentconfigs/ruby-hello-
> world
> spec:
>   template:
> metadata:
>   annotations:
> openshift.io/generated-by: OpenShiftNewApp
>   labels:
> app: ruby-hello-world
> deploymentconfig: ruby-hello-world
> spec:
>   containers:
>   - image: 172.30.1.1:5000/project1/ruby-hello-world@sha256:
> 8d0f81a13ec1b8f8fa4372d26075f0dd87578fba2ec120776133db71ce2c2074
> imagePullPolicy: Always
> name: ruby-hello-world
>
> So our deployment config still refers to a very specific image and
> points to the old project. Is there any logic we could apply safely to
> address this?
>
> It feels like this should boil down to something like
> "ruby-hello-world@sha256:HASH", could we watch for
> $REGISTRY_IP:PORT/projectname/ during export and strip that leading
> portion out? What would be the risks in doing so?
>

Adding Cesar since he was recently looking at some of the export logic you
have questions about and he's also very interested in this subject since
he's working on a related piece of functionality.  That said:

if you've got an imagechangetrigger in the DC you should be able to strip
the entire image field (it should be repopulated from the ICT imagestream
reference during deployment).  However:

1) you still need to straighten out the ICT reference which is also going
to be pointing to an imagestreamtag in the old project/cluster/whatever
2) if you don't have an ICT reference you do need to sort this out and
stripping it the way you propose is definitely not a good idea...what's
going to repopulate that w/ the right prefix/project in the new cluster?
What if the image field was pointing to docker.io or some other external
registry?

In short, you're attempting to tackle a very complex problem where the
answer is frequently "it depends".  We wrote some documentation discussing
some of the considerations when exporting/importing resources between
clusters/projects:

https://docs.openshift.org/latest/dev_guide/application_lifecycle/promoting_applications.html



>
> All help appreciated, thanks.
>
> Devan
>



-- 
Ben Parees | OpenShift