Monumental work! On Mon, Jan 24, 2022 at 2:51 PM Josh Fischer <[email protected]> wrote:
> +1 Nick. I’ll try to check it later this weekS > > On Mon, Jan 24, 2022 at 1:42 PM Nicholas Nezis <[email protected]> > wrote: > > > We finally have a fully green Travis CI build with all of the new Python > > 3.8+ updates. The PR is here: > > https://github.com/apache/incubator-heron/pull/3646 > > > > This PR started with Oliver's awesome updates he contributed back in > > December 2020. The PR is actually a merge of two different PRs (The other > > PR: https://github.com/apache/incubator-heron/pull/3642). There were a > > number of different changes needed to make it all work. For the items > that > > were not directly tied to this PR, I submitted separate PRs as best I > > could. This was not always possible. Below is a summary of what is in > this > > larger PR: > > > > - All of the awesome Python updates provided by Oliver > > - Tracker code was simplified with more modern Python libraries > > - Upgraded Travis image to Ubuntu 20.04 to get newer Python packages > > - Upgraded Python 3.8 as minimum version > > - Collapsed `--config` targets to `linux` and `darwin` (no more > > `ubuntu`, `centos` and `darwin` in bazel.rc config file) > > - Upgraded various Pex dependencies > > - Upgraded `rules_python` (0.0.2 -> 0.1.0) > > - Upgraded the `io_bazel_rules_docker` (0.14.4 -> 0.22.0). This was > > dependent on `rules_python` > > - Debian 9 docker image is now Debian 10 > > - Centos 7 docker image is now Centos 8 > > > > > > On Fri, Jan 21, 2022 at 7:08 PM Saad Ur Rahman <[email protected]> > > wrote: > > > > > I do not think there is an issue with consolidating `--config` to > `linux` > > > and `darwin`. > > > > > > For the libraries on Linux systems, you need to link in the libraries > > > manually, at least in the older compilers. For instance, if you are > using > > > the underlying POSIX threading API you will need to link in the > pthreads > > > library. Removing these libraries from linking may break the build on > > > Linux. > > > > > > On Fri, Jan 21, 2022 at 6:58 PM Nicholas Nezis < > [email protected] > > > > > > wrote: > > > > > > > For the 3 libraries, it seems they are already included in macOS's > main > > > > system library. So keeping the Darwin/Linux split in bazel.rc still > > makes > > > > sense. > > > > > > > > On Fri, Jan 21, 2022 at 6:23 PM Nicholas Nezis < > > [email protected] > > > > > > > > wrote: > > > > > > > > > Thanks guys. Making good progress. > > > > > > > > > > Found an issue with this `ci.sh` script. > > > > > > > > > > > > > > > https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/scripts/shutils/common.sh#L95-L108 > > > > > > > > > > The platform was not detected properly. The strings never match > what > > > > > `python3 -mplatform` outputs. I've decided to simplify the whole > > flavor > > > > > thing in our Bazel build. From now on we'll have `linux` and > `darwin` > > > as > > > > > `--config` options. We'll still have different target platforms as > we > > > > build > > > > > docker images, but there have been no differences in config for > years > > > > since > > > > > I've joined the project. So I'm going to simplify things. Changes > > going > > > > > into the same PR. > > > > > > > > > > These are the only 3 lines that differ between linux and darwin in > > the > > > > > bazel.rc file. Do we even need these flags? Should the flags also > be > > on > > > > the > > > > > darwin build? > > > > > > > > > > > > > > > https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/tools/bazel.rc#L32-L34 > > > > > > > > > > Hopefully this is also ok. My frustrations with old outdated stuff > is > > > > > making me simplify things in hopes it will be more maintainable. > > > > > > > > > > On Fri, Jan 21, 2022 at 4:53 PM Ning Wang <[email protected]> > > > wrote: > > > > > > > > > >> nvm. you mentioned it already and I missed it. My bad. > > > > >> > > > > >> no issues. > > > > >> > > > > >> On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <[email protected]> > > > wrote: > > > > >> > > > > >> > Sounds good to me. Any reason for going to 3.8 instead of 3.9? > > > > >> > > > > > >> > On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer < > [email protected] > > > > > > > >> wrote: > > > > >> > > > > > >> >> No issues from me. > > > > >> >> > > > > >> >> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis < > > > > >> [email protected]> > > > > >> >> wrote: > > > > >> >> > > > > >> >> > Heron Devs, > > > > >> >> > > > > > >> >> > We are running into issues with the older Python 3.6 that > seems > > > to > > > > be > > > > >> >> > resolved in more modern versions of Python. To move things > > > forward > > > > >> and > > > > >> >> get > > > > >> >> > our release working, I'm upgrading to Python 3.8 in our older > > > > Docker > > > > >> >> > container flavors. > > > > >> >> > > > > > >> >> > Much of the work is being done off of Oliver's previous PR to > > > > upgrade > > > > >> >> us to > > > > >> >> > Python 3.9 support. We updated various Pex dependencies (and > > > other > > > > >> >> > python dependencies such as PyYAML). Please review the latest > > > > >> changes in > > > > >> >> > the PR here: > > https://github.com/apache/incubator-heron/pull/3646 > > > > >> >> > > > > > >> >> > I'm setting the minimum Python in `bazel_configure.sh` to > 3.8. > > Is > > > > >> >> everyone > > > > >> >> > ok with this change? > > > > >> >> > > > > > >> >> > Some of the key changes: > > > > >> >> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9) > > > > >> >> > Ubuntu 18.04 (Python 3.6 -> Python 3.8) > > > > >> >> > > > > > >> >> > I didn't really want to update Centos 7 to 8. But I don't > think > > > > >> anyone > > > > >> >> is > > > > >> >> > using it. The Python 3.9 packages did not exist in the older > > > image > > > > >> >> flavor. > > > > >> >> > I eventually want to upgrade this to Rocky or Alma Linux > flavor > > > or > > > > >> maybe > > > > >> >> > even RedHat's UBI image which could also help solve the > OpenJDK > > > > issue > > > > >> >> (i.e. > > > > >> >> > ubi8/openjdk-11). But for now I'm trying to do the least > amount > > > of > > > > >> >> changes > > > > >> >> > to get us over the release finish line. > > > > >> >> > > > > > >> >> > I've also cleaned up a lot of the documentation references to > > > older > > > > >> >> flavors > > > > >> >> > that no longer exist (ubuntu14.04 and debian9). > > > > >> >> > > > > > >> >> > > > > >> > > > > > >> > > > > > > > > > > > > > > > -- > Sent from A Mobile Device >
