I am afraid that until we have a Jenkins driven Windows CI we wouldn't be able to modernize any of the build or release plumbing.
Best regards, iilyak On 2019/08/20 12:28:12, Garren Smith <gar...@apache.org> wrote: > I'm a big +1 on this. Initially, I know we would have some teething issues > getting this working. But I don't think we can stay on rebar2 forever > either. Adding in erlang dependancies is becoming increasingly more > difficult as they expect rebar3. > > Cheers > Garren > > On Fri, Aug 16, 2019 at 2:02 PM Ilya Khlopotov <iil...@apache.org> wrote: > > > > > > > On 2019/08/15 18:51:29, Joan Touzet <woh...@apache.org> wrote: > > > Hi Ilya, > > > > > > > > > On 2019-08-15 10:09, Ilya Khlopotov wrote: > > > > Hello, > > > > > > > > There is an https://github.com/apache/couchdb/issues/1428 issue about > > migrating to rebar3 or mix. I did an experiment to switch from rebar into > > mix and wanted to share the results. The code for experiment is here > > https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. > > Overall I am happy with the experiment. > > > > > > > > # Requirements > > > > > > > > 1. Should be able to compile dependencies which use either rebar or > > rebar3 > > > > 2. Should be able to lock dependencies to specific versions using > > references to commits > > > > 3. Should be able to call eunit test suite > > > > 4. Should be able to compile NIFs > > > > 5. Should be easy to install plugins > > > > 6. Should support dialyzer > > > > 7. Should support release managers > > > > 8. Support for raw dependencies > > > > 9. Produce source tarball suitable for offline compilation (ASF > > requirement) > > > > > > > > # How mix supports the requirements > > > > > > > > 1. Mix is able to figure out which build tool to use. There is also a > > possibility to specify build tool individually for a given dependency. > > > > `manager: mix | rebar | rebar3 | make`. There is also support for > > calling arbitrary shell command configured via `compile: <command>`. The > > manual step is required to install the needed tools `mix local.rebar > > --force`. > > > > However there are means to automate that step (we can use Makefile > > or define a `bootstrap` mix alias ) > > > > 2. Dependencies are locked via `mix.lock` file. > > > > > > I don't have any strong opinions one way or the other, but I did see > > > Garren asking about 'hex' instead of / in conjunction with 'mix' for > > > dependency management. Is this effort combined with his or independent? > > > I don't know the tools well enough. > > > > mix works very well with hex > > > > > > > > > 3. Eunit is supported via one of the mix plugins. Both plugins are > > very old but it should be easy to write our own (32 lines of code). > > > > - https://github.com/dantswain/mix_eunit > > > > - https://github.com/talentdeficit/mixunit > > > > 4. NIFs can be compiled using one of the following approaches: > > > > - specify `rebar` as build manager for a dependency (easiest) > > > > > > Are you considering staying with rebar2 here, or moving to rebar(3)? It > > > would be nice not to have to keep supporting (and shipping!) our fork of > > > rebar2. > > I considered to stay with rebar2 for first iteration. Could you remind me > > why we are forking rebar? > > I did check that our version has no differences from upstream. > > git remote -v > > origin https://github.com/apache/couchdb-rebar (fetch) > > origin https://github.com/apache/couchdb-rebar (push) > > upstream https://github.com/rebar/rebar (fetch) > > upstream https://github.com/rebar/rebar (push) > > > > git log --oneline origin/master | head > > 0c435f3 Update master with mainline rebar (v2) > > b6d3094 Merge pull request #620 from tuncer/travis-dialyze > > 8a2aca0 travis-ci: allow Dialyzer job to fail > > 16d5dfe travis-ci: update otp versions > > 4e372a3 travis-ci: enable 20.1 > > 73feb5f rebar_xref: ignore opaque type match Dialyzer warning > > b05882a rebar_cover: ignore opaque type match Dialyzer warning > > 903c89d rebar_utils: fix Dialyzer warning > > b36e72b Run Dialyzer on Travis-CI > > 4bd43fe Merge pull request #645 from Juliusan/log_fix > > > > git log --oneline upstream/master | head > > b6d3094 Merge pull request #620 from tuncer/travis-dialyze > > 8a2aca0 travis-ci: allow Dialyzer job to fail > > 16d5dfe travis-ci: update otp versions > > 4e372a3 travis-ci: enable 20.1 > > 73feb5f rebar_xref: ignore opaque type match Dialyzer warning > > b05882a rebar_cover: ignore opaque type match Dialyzer warning > > 903c89d rebar_utils: fix Dialyzer warning > > b36e72b Run Dialyzer on Travis-CI > > 4bd43fe Merge pull request #645 from Juliusan/log_fix > > 2276f85 Log out success message with newlines > > > > git diff origin/master upstream/master > > > > > > > > > - call a Makefile from mix.exs. See examples: > > > > - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9 > > > > - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6 > > > > - use https://github.com/davisp/erlang-native-compiler > > > > - use https://github.com/blt/port_compiler > > > > > > Paul left a comment about this on the roadmap issue just today. Can you > > > respond to his comment (it was related tot dependencies-of-dependencies, > > > i.e. jiffy) > > > > > > He also raised the concern that it may force us to use only the very > > > latest Erlang versions - can you comment on this? Does the current > > > hex/mix situation require a minimum version? What version is that? > > > > I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported > > on erlang 19 which is AFAIK a minimal erlang version CouchDB supports. > > > > > > 5. Installing plugins for mix is similar to rebar3. It is sufficient > > to just specify list of plugins in the deps in mix.exs > > > > 6. There are multiple dyalizer plugins to choose from > > > > - https://github.com/jeremyjh/dialyxir > > > > - https://github.com/gabrielgatu/mix-dialyzer > > > > > > I know this is a requirement for Cloudant, but today isn't for base > > > CouchDB. I would prefer not to introduce this requirement at the same > > > time as migrating build tool. > > > > The reason I mentioned it here is because we have 'make dialyze' command. > > > > > > 7. Multiple release managers are supported via mix plugins > > > > - distillery is supported via > > https://hexdocs.pm/distillery/Mix.Releases.Plugin.html > > > > - relx and reltools can be used via > > https://github.com/bitwalker/exrm (deprecated in favor of distillery) > > > > - we can also roll our own mix alias using > > https://github.com/okeuday/reltool_util > > > > - latest elixir versions have `release` command out of the box > > > > 8. Raw dependencies are supported via `compile: false` option > > > > 9. I think it is doable. In the worst case scenario we could just tar > > the content of `src` directory after calling `mix deps.get` > > > > > > > > # Warts of mix in the context of CouchDB project > > > > > > > > - CouchDB doesn't use standard elixir directories layout which caused > > include_lib directives to not work. To solve this issue absolute path to > > `<root>/src` need to be added into include path > > > > - for dependencies managed by mix it is done via `erlc_options` > > parameter > > > > - Unfortunately when we call out to rebar there is no way to modify > > arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS` > > environment variable. Which is not very elegant solution. > > > > > > This may also cause problems on Windows, where we have very specific > > > requirements for compiler options so as to ensure the correct libraries > > > are linked for couchjs and so on. > > > > > > Have you tried compiling on that platform? ;) (Please don't make me do > > > all the work here, again...) > > > > > > > - We need to create empty `rebar.config` file for erlang applications > > which do not have `rebar.config` (all applications in couchdb repo). This > > problem can be fixed by adding mix.exs to dependencies we have control over. > > > > - The mix is slow when we have to call out to rebar. This problem can > > be fixed by adding mix.exs to dependencies we have control over. > > > > > > How much slower? Can you post a comparison of `time make couch` between > > > master and your branch? > > > > master: > > > time bin/rebar get-deps update-deps > > real 0m37.514s > > user 0m1.960s > > sys 0m3.300s > > > time make couch > > real 0m49.677s > > user 0m46.910s > > sys 0m22.210s > > > > mix: > > > time mix do deps.get > > real 0m24.405s > > user 0m1.590s > > sys 0m1.270s > > > time mix deps.compile > > real 3m47.968s > > user 0m0.860s > > sys 0m0.640s > > > > > > # Roadmap > > > > > > > > The full conversion would take some time and if we decide to go with > > mix it makes sense to do it incrementally. Possible roadmap could be: > > > > - initial PR > > > > - offload dependency fetching to mix.exs and generate rebar.config > > so release machinery still works > > > > - Update Makefile targets to call mix > > > > - Choose or implement eunit plugin for mix > > > > - Support running tests individually (replacement for `make eunit > > apps= tests= suites=`) > > > > - Replace rebar plugins we use with alternative solutions > > > > - Figure out how to use dialyzer > > > > - Figure out how to use mix for preparing release > > > > > > Anything that lands on master must support release preparation. If your > > > incremental approach breaks `make dist`, you'll need to stage all of > > > your incremental changes on a feature branch and fix `make dist` before > > > merging to master. master needs to be releasable at all times. > > +1000 > > > > > > Best regards, > > > > ILYA (aka iilyak) > > > > > > -Joan "release engineer hat is now off" Touzet > > > > > > > > >
