Hi Ilya,
On 2019-08-15 10:09, Ilya Khlopotov wrote: > Hello, > > There is an https://github.com/apache/couchdb/issues/1428 issue about > migrating to rebar3 or mix. I did an experiment to switch from rebar into mix > and wanted to share the results. The code for experiment is here > https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. > Overall I am happy with the experiment. > > # Requirements > > 1. Should be able to compile dependencies which use either rebar or rebar3 > 2. Should be able to lock dependencies to specific versions using references > to commits > 3. Should be able to call eunit test suite > 4. Should be able to compile NIFs > 5. Should be easy to install plugins > 6. Should support dialyzer > 7. Should support release managers > 8. Support for raw dependencies > 9. Produce source tarball suitable for offline compilation (ASF requirement) > > # How mix supports the requirements > > 1. Mix is able to figure out which build tool to use. There is also a > possibility to specify build tool individually for a given dependency. > `manager: mix | rebar | rebar3 | make`. There is also support for calling > arbitrary shell command configured via `compile: <command>`. The manual step > is required to install the needed tools `mix local.rebar --force`. > However there are means to automate that step (we can use Makefile or > define a `bootstrap` mix alias ) > 2. Dependencies are locked via `mix.lock` file. I don't have any strong opinions one way or the other, but I did see Garren asking about 'hex' instead of / in conjunction with 'mix' for dependency management. Is this effort combined with his or independent? I don't know the tools well enough. > 3. Eunit is supported via one of the mix plugins. Both plugins are very old > but it should be easy to write our own (32 lines of code). > - https://github.com/dantswain/mix_eunit > - https://github.com/talentdeficit/mixunit > 4. NIFs can be compiled using one of the following approaches: > - specify `rebar` as build manager for a dependency (easiest) Are you considering staying with rebar2 here, or moving to rebar(3)? It would be nice not to have to keep supporting (and shipping!) our fork of rebar2. > - call a Makefile from mix.exs. See examples: > - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9 > - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6 > - use https://github.com/davisp/erlang-native-compiler > - use https://github.com/blt/port_compiler Paul left a comment about this on the roadmap issue just today. Can you respond to his comment (it was related tot dependencies-of-dependencies, i.e. jiffy) He also raised the concern that it may force us to use only the very latest Erlang versions - can you comment on this? Does the current hex/mix situation require a minimum version? What version is that? > 5. Installing plugins for mix is similar to rebar3. It is sufficient to just > specify list of plugins in the deps in mix.exs > 6. There are multiple dyalizer plugins to choose from > - https://github.com/jeremyjh/dialyxir > - https://github.com/gabrielgatu/mix-dialyzer I know this is a requirement for Cloudant, but today isn't for base CouchDB. I would prefer not to introduce this requirement at the same time as migrating build tool. > 7. Multiple release managers are supported via mix plugins > - distillery is supported via > https://hexdocs.pm/distillery/Mix.Releases.Plugin.html > - relx and reltools can be used via https://github.com/bitwalker/exrm > (deprecated in favor of distillery) > - we can also roll our own mix alias using > https://github.com/okeuday/reltool_util > - latest elixir versions have `release` command out of the box > 8. Raw dependencies are supported via `compile: false` option > 9. I think it is doable. In the worst case scenario we could just tar the > content of `src` directory after calling `mix deps.get` > > # Warts of mix in the context of CouchDB project > > - CouchDB doesn't use standard elixir directories layout which caused > include_lib directives to not work. To solve this issue absolute path to > `<root>/src` need to be added into include path > - for dependencies managed by mix it is done via `erlc_options` parameter > - Unfortunately when we call out to rebar there is no way to modify > arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS` > environment variable. Which is not very elegant solution. This may also cause problems on Windows, where we have very specific requirements for compiler options so as to ensure the correct libraries are linked for couchjs and so on. Have you tried compiling on that platform? ;) (Please don't make me do all the work here, again...) > - We need to create empty `rebar.config` file for erlang applications which > do not have `rebar.config` (all applications in couchdb repo). This problem > can be fixed by adding mix.exs to dependencies we have control over. > - The mix is slow when we have to call out to rebar. This problem can be > fixed by adding mix.exs to dependencies we have control over. How much slower? Can you post a comparison of `time make couch` between master and your branch? > # Roadmap > > The full conversion would take some time and if we decide to go with mix it > makes sense to do it incrementally. Possible roadmap could be: > - initial PR > - offload dependency fetching to mix.exs and generate rebar.config so > release machinery still works > - Update Makefile targets to call mix > - Choose or implement eunit plugin for mix > - Support running tests individually (replacement for `make eunit apps= > tests= suites=`) > - Replace rebar plugins we use with alternative solutions > - Figure out how to use dialyzer > - Figure out how to use mix for preparing release Anything that lands on master must support release preparation. If your incremental approach breaks `make dist`, you'll need to stage all of your incremental changes on a feature branch and fix `make dist` before merging to master. master needs to be releasable at all times. > Best regards, > ILYA (aka iilyak) -Joan "release engineer hat is now off" Touzet
