hi Phillip, Thank you for these thoughts.
I am all for improving our build systems to be more readable/maintainable and easier to debug and develop. I see our problems are being different than improving the build systems for the Python and C++ libraries: we need to set up a build and packaging automation system for our build systems, whatever they may be. Whether we use CMake or Bazel, we still need a system that will * Either nightly, or on each commit, build all variants of all packages on all platforms * Upload the binary artifacts to a central location * Capture build logs and upload to a central location for later inspection * Notify Arrow community via e-mail in the event of any failure Moving to Bazel will make the actual packaging process better in some aspects, but it will _not_ help with the workflow automation and nightly validation. Currently, it requires a human in the loop and commit like https://github.com/apache/arrow-dist/commit/2e99e83c352775f1ccde096242c4bec783340369 which took up collectively multiple days of development time in the last week. We have some other structural problems, i.e. much of the code in apache/arrow-dist should probably be in apache/arrow, and in the form of templates so that we can _generate_ a build configuration given a set of pinned dependencies and build parameters (e.g. in the case of C++/Python, flags relating to linking and bundling of thirdparty dependencies like Boost) I think it's worth seriously considering the Bazel route, but we should not start building a new build system without coming up with a coherent and actionable plan to address the build and packaging automation problem that we have now. Ideally this automation system would be agnostic to the underlying build systems used by each component. If we move to a better build system in the future, then the build/packaging ("metabuild") system will have to do less work. - Wes On Sat, Mar 24, 2018 at 4:13 PM, Phillip Cloud <cpcl...@gmail.com> wrote: > I think we need to use a tool that can perform every single step of the > deployment process, end-to-end. Right now, cmake isn't cutting it IMO > because it lends itself quite heavily to copy pasting and oodles of bash > scripts that are indecipherable by anyone except the original author. > > With that in mind, here's what I think the requirements are for the next > generation of arrow's build and deployment system for native code (C, C++, > Python): > > For every platform (Linux, OS X, and Windows): > > 1. Build sources (these langauges are currently the most cumbersome at the > moment) > 2. Run all tests for each language (ideally integration tests as well, but > I'm not sure if that should be a hard requirement at the moment) > 3. Build the API documentation for each platform > 4. Build installable packages that we support: > * conda > * pip wheels > * deb > * rpm > > Additional requirements: > 1. Be able to run all of these steps on any platform with minimal > environment setup. > 2. The output of the doc build and package build steps **should be in a > release-ready state at all times**. If they are not then we should fail the > build > 3. Ideally, these run on every PR so that we can find out if a commit would > introduce a change that would break the release-ready status of arrow. > > These requirements indicate to me that a single, extensible, cross-platform > tool--as opposed to many tools that are tied together by a shell script--is > what we need. > > There are a few tools in this space that I'm aware of: > > 1. Bazel (out of Google) > 2. Buck (out of Facebook) > > I'm not sure what others are out there, but I'm sure there must be some. > > I don't really have a strong opinion on either Bazel or Buck, but I suspect > that since we follow Google's conventions in a few places integrating Bazel > into the arrow codebase would be less work. > > The main risk I see here is that it's possible that bazel isn't the right > tool. I'm not sure how to mitigate this risk other than to make sure that > our requirements can be met by it by scouring the Bazel docs. > > I do think that the fact that Bazel is extensible mitigates some risk here. > For example, we'd likely have to add rule for building conda packages and > pip wheels. > > I guess CMake is extensible too, but I don't think I've ever seen the > extensibility features of CMake as anything but a burden. Bazel's extension > language is a subset of Python and I would therefore expect it to be a lot > easier to use. > > I'm interested to hear others' experiences and opinions on similar > problems. Also, if I've missed anything in the requirements list, please > don't hestitate to respond. > > Let's fix our packaging! > > -Phillip > > On Fri, Mar 23, 2018 at 11:21 PM Holden Karau <hol...@pigscanfly.ca> wrote: > >> I know in Spark we’ve benefited by having some of the different language >> devs act as RMs and each time that language dev has ended up improving a >> bunch of how their components packaging has been done. Not to suggest we >> should just do what other projects do, but maybe an idea to consider? >> >> On Fri, Mar 23, 2018 at 12:59 PM Wes McKinney <wesmck...@gmail.com> wrote: >> >> > hi folks, >> > >> > So, I want to bring light to the problems we are having delivering >> > binary artifacts after Arrow releases. >> > >> > We have some amount of packaging automation implemented in >> > https://github.com/apache/arrow-dist using Travis CI and Appveyor to >> > upload packages to Bintray, a packaging hosting service. >> > >> > Unfortunately, we discovered a bunch of problems with these packaging >> > scripts after the release vote closed on Monday, and now 4 days later, >> > we still have been unable to post binaries to >> > https://pypi.python.org/pypi/pyarrow >> > >> > This is no one's fault, but it highlights structural problems with our >> > development process: >> > >> > * Why does producing packages after a release require error-prone manual >> > labor? >> > >> > * Why are we only finding out about packaging problem after a release >> > vote closes? >> > >> > * Why is setting up nightly binary builds a brittle and bespoke process? >> > >> > I hope all agree that: >> > >> > * Packaging should not be a hardship or require a lot of manual labor >> > >> > * Packaging problems on the master branch should be made known within >> > ~24 hours, so they can be remedied immediately >> > >> > * It should be straightforward to produce binary artifacts for all >> > supported platforms and programming languages >> > >> > Eventually, we should include some binary artifacts in our release >> > votes, but we are pretty far away from suitable automation to make >> > this possible. >> > >> > I don't know any easy solutions, but Apache Arrow has grown widely >> > used enough that I think it's worth our taking the time to plan and >> > execute some solutions to these problems, which I expect to pay >> > dividends in our community's productivity over time. >> > >> > Thanks, >> > Wes >> > >> -- >> Twitter: https://twitter.com/holdenkarau >>
