We finally have a fully green Travis CI build with all of the new Python 3.8+ updates. The PR is here: https://github.com/apache/incubator-heron/pull/3646
This PR started with Oliver's awesome updates he contributed back in December 2020. The PR is actually a merge of two different PRs (The other PR: https://github.com/apache/incubator-heron/pull/3642). There were a number of different changes needed to make it all work. For the items that were not directly tied to this PR, I submitted separate PRs as best I could. This was not always possible. Below is a summary of what is in this larger PR: - All of the awesome Python updates provided by Oliver - Tracker code was simplified with more modern Python libraries - Upgraded Travis image to Ubuntu 20.04 to get newer Python packages - Upgraded Python 3.8 as minimum version - Collapsed `--config` targets to `linux` and `darwin` (no more `ubuntu`, `centos` and `darwin` in bazel.rc config file) - Upgraded various Pex dependencies - Upgraded `rules_python` (0.0.2 -> 0.1.0) - Upgraded the `io_bazel_rules_docker` (0.14.4 -> 0.22.0). This was dependent on `rules_python` - Debian 9 docker image is now Debian 10 - Centos 7 docker image is now Centos 8 On Fri, Jan 21, 2022 at 7:08 PM Saad Ur Rahman <[email protected]> wrote: > I do not think there is an issue with consolidating `--config` to `linux` > and `darwin`. > > For the libraries on Linux systems, you need to link in the libraries > manually, at least in the older compilers. For instance, if you are using > the underlying POSIX threading API you will need to link in the pthreads > library. Removing these libraries from linking may break the build on > Linux. > > On Fri, Jan 21, 2022 at 6:58 PM Nicholas Nezis <[email protected]> > wrote: > > > For the 3 libraries, it seems they are already included in macOS's main > > system library. So keeping the Darwin/Linux split in bazel.rc still makes > > sense. > > > > On Fri, Jan 21, 2022 at 6:23 PM Nicholas Nezis <[email protected] > > > > wrote: > > > > > Thanks guys. Making good progress. > > > > > > Found an issue with this `ci.sh` script. > > > > > > https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/scripts/shutils/common.sh#L95-L108 > > > > > > The platform was not detected properly. The strings never match what > > > `python3 -mplatform` outputs. I've decided to simplify the whole flavor > > > thing in our Bazel build. From now on we'll have `linux` and `darwin` > as > > > `--config` options. We'll still have different target platforms as we > > build > > > docker images, but there have been no differences in config for years > > since > > > I've joined the project. So I'm going to simplify things. Changes going > > > into the same PR. > > > > > > These are the only 3 lines that differ between linux and darwin in the > > > bazel.rc file. Do we even need these flags? Should the flags also be on > > the > > > darwin build? > > > > > > https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/tools/bazel.rc#L32-L34 > > > > > > Hopefully this is also ok. My frustrations with old outdated stuff is > > > making me simplify things in hopes it will be more maintainable. > > > > > > On Fri, Jan 21, 2022 at 4:53 PM Ning Wang <[email protected]> > wrote: > > > > > >> nvm. you mentioned it already and I missed it. My bad. > > >> > > >> no issues. > > >> > > >> On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <[email protected]> > wrote: > > >> > > >> > Sounds good to me. Any reason for going to 3.8 instead of 3.9? > > >> > > > >> > On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <[email protected]> > > >> wrote: > > >> > > > >> >> No issues from me. > > >> >> > > >> >> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis < > > >> [email protected]> > > >> >> wrote: > > >> >> > > >> >> > Heron Devs, > > >> >> > > > >> >> > We are running into issues with the older Python 3.6 that seems > to > > be > > >> >> > resolved in more modern versions of Python. To move things > forward > > >> and > > >> >> get > > >> >> > our release working, I'm upgrading to Python 3.8 in our older > > Docker > > >> >> > container flavors. > > >> >> > > > >> >> > Much of the work is being done off of Oliver's previous PR to > > upgrade > > >> >> us to > > >> >> > Python 3.9 support. We updated various Pex dependencies (and > other > > >> >> > python dependencies such as PyYAML). Please review the latest > > >> changes in > > >> >> > the PR here: https://github.com/apache/incubator-heron/pull/3646 > > >> >> > > > >> >> > I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is > > >> >> everyone > > >> >> > ok with this change? > > >> >> > > > >> >> > Some of the key changes: > > >> >> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9) > > >> >> > Ubuntu 18.04 (Python 3.6 -> Python 3.8) > > >> >> > > > >> >> > I didn't really want to update Centos 7 to 8. But I don't think > > >> anyone > > >> >> is > > >> >> > using it. The Python 3.9 packages did not exist in the older > image > > >> >> flavor. > > >> >> > I eventually want to upgrade this to Rocky or Alma Linux flavor > or > > >> maybe > > >> >> > even RedHat's UBI image which could also help solve the OpenJDK > > issue > > >> >> (i.e. > > >> >> > ubi8/openjdk-11). But for now I'm trying to do the least amount > of > > >> >> changes > > >> >> > to get us over the release finish line. > > >> >> > > > >> >> > I've also cleaned up a lot of the documentation references to > older > > >> >> flavors > > >> >> > that no longer exist (ubuntu14.04 and debian9). > > >> >> > > > >> >> > > >> > > > >> > > > > > >
