Hi all, After some recent changes (e.g. [1]) we have a feasible container that we can use to test Python SDK on portability framework. Until now we were using Google provided container images for testing and for the released product. We can gradually move away from that (at least partially) for Python SDK.
I would like to propose building containers for testing purposes only and pushing them to gcr.io as part of jenkins jobs. I would like to clarify two points with the team first: 1. Use of GCR, I am proposing it for a few reasons: - Beam's jenkins workers run on GCP, and it would be easy to push them to gcr from there. - If we use another service (perhaps with a free tier for open source projects) we might be overusing it by pushing/pulling from our daily tests. - This is similar to how we stage some artifacts to GCS as part of the testing process. 2. Frequency of building and pushing containers a. We can run it at every PR, by integrating with python post commit tests. b. We can run it daily, by having a new Jenkins job. c. We can run it manually, by having a parameterized Jenkins job that can build and push a new container from a tag/commit. Given that we infrequently change container code, I would suggest choosing this option. What do you think about this? To be clear, this is just a proposal about the testing environment. I am not suggesting anything about the release artifacts. Thank you, Ahmet [1] https://github.com/apache/beam/pull/4286
