Re: published containers overwrite locally built containers

[Valentyn Tymofieiev]

On Wed, Nov 6, 2019 at 3:28 PM Heejong Lee  wrote:

> I think that implicitly (and forcefully) pull the remote image is not good
> even in case of a bug fix. The better approach would be releasing a
> separate bug fix version. Implicitly pulling the updated version of the
> same container looks weird to me since it feels like releasing the jar
> artifact with the same version multiple times or publishing already
> published git branch again. However, I understand it's much easier to just
> update the container with the same tag than release another Beam version.
>

Fair point, if we need to release an update to Beam container image we
would likely have a bugfix release. However there are other use cases
(pre-release images, custom images) where a pull may result in a more
deterministic behavior as discussed in earlier messages.


>
> On Wed, Nov 6, 2019 at 8:05 AM Valentyn Tymofieiev 
> wrote:
>
>> I agree with the resolutions in the link Thomas mentioned [1]. Using
>> latest tag is not reliable, and a unique tag ID should be generated when
>> running tests on Jenkins against master branch.
>> I think pulling the latest image for the current tag is actually a
>> desired behavior, in case the external image was updated (due to a bug fix
>> for example). Our custom container documentation should reflect this
>> behavior.
>> Consider continuing the conversation in [1] to keep it in one place if
>> there are other suggestions/opinions.
>>
>> [1]
>> https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E
>>
>>
>> On Fri, Nov 1, 2019 at 5:14 PM Thomas Weise  wrote:
>>
>>> More here:
>>> https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E
>>>
>>>
>>> On Fri, Nov 1, 2019 at 10:56 AM Chamikara Jayalath 
>>> wrote:
>>>
 I think it makes sense to override published docker images with locally
 built versions when testing HEAD.

 Thanks,
 Cham

 On Thu, Oct 31, 2019 at 6:31 PM Heejong Lee  wrote:

> Hi, happy halloween!
>
> I'm looking into failing cross language post commit tests:
> https://issues.apache.org/jira/browse/BEAM-8534
> 
>
> After a few runs, I've found that published SDK harness containers
> overwrite locally built containers when docker pull happens. I can think 
> of
> two possible solutions here: 1) remove the published images with the 
> latest
> tag, so make the image with the latest tag available for testing and
> development purposes. 2) put serialVersionUID to the class printing out 
> the
> error.
>
> 2) doesn't sound like a fundamental solution if we're not going to
> attach serialVersionUID to all serializable classes. 1) might work but I'm
> not sure whether there's another use for the latest tag somewhere. Any
> better ideas?
>
> Thanks,
> Heejong
>

Re: published containers overwrite locally built containers

[Heejong Lee]

I think that implicitly (and forcefully) pull the remote image is not good
even in case of a bug fix. The better approach would be releasing a
separate bug fix version. Implicitly pulling the updated version of the
same container looks weird to me since it feels like releasing the jar
artifact with the same version multiple times or publishing already
published git branch again. However, I understand it's much easier to just
update the container with the same tag than release another Beam version.

On Wed, Nov 6, 2019 at 8:05 AM Valentyn Tymofieiev 
wrote:

> I agree with the resolutions in the link Thomas mentioned [1]. Using
> latest tag is not reliable, and a unique tag ID should be generated when
> running tests on Jenkins against master branch.
> I think pulling the latest image for the current tag is actually a desired
> behavior, in case the external image was updated (due to a bug fix for
> example). Our custom container documentation should reflect this behavior.
> Consider continuing the conversation in [1] to keep it in one place if
> there are other suggestions/opinions.
>
> [1]
> https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E
>
>
> On Fri, Nov 1, 2019 at 5:14 PM Thomas Weise  wrote:
>
>> More here:
>> https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E
>>
>>
>> On Fri, Nov 1, 2019 at 10:56 AM Chamikara Jayalath 
>> wrote:
>>
>>> I think it makes sense to override published docker images with locally
>>> built versions when testing HEAD.
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Thu, Oct 31, 2019 at 6:31 PM Heejong Lee  wrote:
>>>
 Hi, happy halloween!

 I'm looking into failing cross language post commit tests:
 https://issues.apache.org/jira/browse/BEAM-8534
 

 After a few runs, I've found that published SDK harness containers
 overwrite locally built containers when docker pull happens. I can think of
 two possible solutions here: 1) remove the published images with the latest
 tag, so make the image with the latest tag available for testing and
 development purposes. 2) put serialVersionUID to the class printing out the
 error.

 2) doesn't sound like a fundamental solution if we're not going to
 attach serialVersionUID to all serializable classes. 1) might work but I'm
 not sure whether there's another use for the latest tag somewhere. Any
 better ideas?

 Thanks,
 Heejong

>>>

Re: published containers overwrite locally built containers

[Valentyn Tymofieiev]

I agree with the resolutions in the link Thomas mentioned [1]. Using latest
tag is not reliable, and a unique tag ID should be generated when running
tests on Jenkins against master branch.
I think pulling the latest image for the current tag is actually a desired
behavior, in case the external image was updated (due to a bug fix for
example). Our custom container documentation should reflect this behavior.
Consider continuing the conversation in [1] to keep it in one place if
there are other suggestions/opinions.

[1]
https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E


On Fri, Nov 1, 2019 at 5:14 PM Thomas Weise  wrote:

> More here:
> https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E
>
>
> On Fri, Nov 1, 2019 at 10:56 AM Chamikara Jayalath 
> wrote:
>
>> I think it makes sense to override published docker images with locally
>> built versions when testing HEAD.
>>
>> Thanks,
>> Cham
>>
>> On Thu, Oct 31, 2019 at 6:31 PM Heejong Lee  wrote:
>>
>>> Hi, happy halloween!
>>>
>>> I'm looking into failing cross language post commit tests:
>>> https://issues.apache.org/jira/browse/BEAM-8534
>>> 
>>>
>>> After a few runs, I've found that published SDK harness containers
>>> overwrite locally built containers when docker pull happens. I can think of
>>> two possible solutions here: 1) remove the published images with the latest
>>> tag, so make the image with the latest tag available for testing and
>>> development purposes. 2) put serialVersionUID to the class printing out the
>>> error.
>>>
>>> 2) doesn't sound like a fundamental solution if we're not going to
>>> attach serialVersionUID to all serializable classes. 1) might work but I'm
>>> not sure whether there's another use for the latest tag somewhere. Any
>>> better ideas?
>>>
>>> Thanks,
>>> Heejong
>>>
>>

Re: published containers overwrite locally built containers

[Thomas Weise]

More here:
https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E


On Fri, Nov 1, 2019 at 10:56 AM Chamikara Jayalath 
wrote:

> I think it makes sense to override published docker images with locally
> built versions when testing HEAD.
>
> Thanks,
> Cham
>
> On Thu, Oct 31, 2019 at 6:31 PM Heejong Lee  wrote:
>
>> Hi, happy halloween!
>>
>> I'm looking into failing cross language post commit tests:
>> https://issues.apache.org/jira/browse/BEAM-8534
>> 
>>
>> After a few runs, I've found that published SDK harness containers
>> overwrite locally built containers when docker pull happens. I can think of
>> two possible solutions here: 1) remove the published images with the latest
>> tag, so make the image with the latest tag available for testing and
>> development purposes. 2) put serialVersionUID to the class printing out the
>> error.
>>
>> 2) doesn't sound like a fundamental solution if we're not going to attach
>> serialVersionUID to all serializable classes. 1) might work but I'm not
>> sure whether there's another use for the latest tag somewhere. Any better
>> ideas?
>>
>> Thanks,
>> Heejong
>>
>

***UNCHECKED*** Re: published containers overwrite locally built containers

[Kyle Weaver]

For additional context, this was discussed weeks ago on this list:
https://lists.apache.org/thread.html/932fe0bc838b92e80475b2bf862e6cec34fbd6ac0d4f3c9de5ac25e1@%3Cdev.beam.apache.org%3E


On Fri, Nov 1, 2019 at 10:56 AM Chamikara Jayalath 
wrote:

> I think it makes sense to override published docker images with locally
> built versions when testing HEAD.
>
> Thanks,
> Cham
>
> On Thu, Oct 31, 2019 at 6:31 PM Heejong Lee  wrote:
>
>> Hi, happy halloween!
>>
>> I'm looking into failing cross language post commit tests:
>> https://issues.apache.org/jira/browse/BEAM-8534
>> 
>>
>> After a few runs, I've found that published SDK harness containers
>> overwrite locally built containers when docker pull happens. I can think of
>> two possible solutions here: 1) remove the published images with the latest
>> tag, so make the image with the latest tag available for testing and
>> development purposes. 2) put serialVersionUID to the class printing out the
>> error.
>>
>> 2) doesn't sound like a fundamental solution if we're not going to attach
>> serialVersionUID to all serializable classes. 1) might work but I'm not
>> sure whether there's another use for the latest tag somewhere. Any better
>> ideas?
>>
>> Thanks,
>> Heejong
>>
>

***UNCHECKED*** Re: published containers overwrite locally built containers

[Heejong Lee]

Since 'docker run' automatically pulls when the image doesn't exist
locally, I think it's safe to remove explicit 'docker pull' before 'docker
run'. Without 'docker pull', we won't update the local image with the
remote image (for the same tag) but it shouldn't be a problem in prod that
the unique tag is assumed for each released version.

On Fri, Nov 1, 2019 at 10:56 AM Chamikara Jayalath 
wrote:

> I think it makes sense to override published docker images with locally
> built versions when testing HEAD.
>
> Thanks,
> Cham
>
> On Thu, Oct 31, 2019 at 6:31 PM Heejong Lee  wrote:
>
>> Hi, happy halloween!
>>
>> I'm looking into failing cross language post commit tests:
>> https://issues.apache.org/jira/browse/BEAM-8534
>> 
>>
>> After a few runs, I've found that published SDK harness containers
>> overwrite locally built containers when docker pull happens. I can think of
>> two possible solutions here: 1) remove the published images with the latest
>> tag, so make the image with the latest tag available for testing and
>> development purposes. 2) put serialVersionUID to the class printing out the
>> error.
>>
>> 2) doesn't sound like a fundamental solution if we're not going to attach
>> serialVersionUID to all serializable classes. 1) might work but I'm not
>> sure whether there's another use for the latest tag somewhere. Any better
>> ideas?
>>
>> Thanks,
>> Heejong
>>
>

Re: published containers overwrite locally built containers

[Chamikara Jayalath]

I think it makes sense to override published docker images with locally
built versions when testing HEAD.

Thanks,
Cham

On Thu, Oct 31, 2019 at 6:31 PM Heejong Lee  wrote:

> Hi, happy halloween!
>
> I'm looking into failing cross language post commit tests:
> https://issues.apache.org/jira/browse/BEAM-8534
> 
>
> After a few runs, I've found that published SDK harness containers
> overwrite locally built containers when docker pull happens. I can think of
> two possible solutions here: 1) remove the published images with the latest
> tag, so make the image with the latest tag available for testing and
> development purposes. 2) put serialVersionUID to the class printing out the
> error.
>
> 2) doesn't sound like a fundamental solution if we're not going to attach
> serialVersionUID to all serializable classes. 1) might work but I'm not
> sure whether there's another use for the latest tag somewhere. Any better
> ideas?
>
> Thanks,
> Heejong
>

published containers overwrite locally built containers

[Heejong Lee]

Hi, happy halloween!

I'm looking into failing cross language post commit tests:
https://issues.apache.org/jira/browse/BEAM-8534


After a few runs, I've found that published SDK harness containers
overwrite locally built containers when docker pull happens. I can think of
two possible solutions here: 1) remove the published images with the latest
tag, so make the image with the latest tag available for testing and
development purposes. 2) put serialVersionUID to the class printing out the
error.

2) doesn't sound like a fundamental solution if we're not going to attach
serialVersionUID to all serializable classes. 1) might work but I'm not
sure whether there's another use for the latest tag somewhere. Any better
ideas?

Thanks,
Heejong