Re: [PROPOSAL] Gradually replace rebar with mix
Jenkins driven Windows CI is in the cards. We now have Azure credits we can put towards this, or perhaps one of the workers IBM is donating can run this. It'll have to be a "snowflake" machine build, but I believe it can happen. But you should be able to run a Windows VM using VirtualBox? And we can get you a Windows license if that's an issue. It's kinda late to be solving it in CI... -Joan On 2019-08-20 8:56, Ilya Khlopotov wrote: > I am afraid that until we have a Jenkins driven Windows CI we wouldn't be > able to modernize any of the build or release plumbing. > > Best regards, > iilyak > > On 2019/08/20 12:28:12, Garren Smith wrote: >> I'm a big +1 on this. Initially, I know we would have some teething issues >> getting this working. But I don't think we can stay on rebar2 forever >> either. Adding in erlang dependancies is becoming increasingly more >> difficult as they expect rebar3. >> >> Cheers >> Garren >> >> On Fri, Aug 16, 2019 at 2:02 PM Ilya Khlopotov wrote: >> >>> >>> >>> On 2019/08/15 18:51:29, Joan Touzet wrote: Hi Ilya, On 2019-08-15 10:09, Ilya Khlopotov wrote: > Hello, > > There is an https://github.com/apache/couchdb/issues/1428 issue about >>> migrating to rebar3 or mix. I did an experiment to switch from rebar into >>> mix and wanted to share the results. The code for experiment is here >>> https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. >>> Overall I am happy with the experiment. > > # Requirements > > 1. Should be able to compile dependencies which use either rebar or >>> rebar3 > 2. Should be able to lock dependencies to specific versions using >>> references to commits > 3. Should be able to call eunit test suite > 4. Should be able to compile NIFs > 5. Should be easy to install plugins > 6. Should support dialyzer > 7. Should support release managers > 8. Support for raw dependencies > 9. Produce source tarball suitable for offline compilation (ASF >>> requirement) > > # How mix supports the requirements > > 1. Mix is able to figure out which build tool to use. There is also a >>> possibility to specify build tool individually for a given dependency. > `manager: mix | rebar | rebar3 | make`. There is also support for >>> calling arbitrary shell command configured via `compile: `. The >>> manual step is required to install the needed tools `mix local.rebar >>> --force`. >However there are means to automate that step (we can use Makefile >>> or define a `bootstrap` mix alias ) > 2. Dependencies are locked via `mix.lock` file. I don't have any strong opinions one way or the other, but I did see Garren asking about 'hex' instead of / in conjunction with 'mix' for dependency management. Is this effort combined with his or independent? I don't know the tools well enough. >>> >>> mix works very well with hex >>> > 3. Eunit is supported via one of the mix plugins. Both plugins are >>> very old but it should be easy to write our own (32 lines of code). > - https://github.com/dantswain/mix_eunit > - https://github.com/talentdeficit/mixunit > 4. NIFs can be compiled using one of the following approaches: > - specify `rebar` as build manager for a dependency (easiest) Are you considering staying with rebar2 here, or moving to rebar(3)? It would be nice not to have to keep supporting (and shipping!) our fork of rebar2. >>> I considered to stay with rebar2 for first iteration. Could you remind me >>> why we are forking rebar? >>> I did check that our version has no differences from upstream. >>> git remote -v >>> origin https://github.com/apache/couchdb-rebar (fetch) >>> origin https://github.com/apache/couchdb-rebar (push) >>> upstreamhttps://github.com/rebar/rebar (fetch) >>> upstreamhttps://github.com/rebar/rebar (push) >>> >>> git log --oneline origin/master | head >>> 0c435f3 Update master with mainline rebar (v2) >>> b6d3094 Merge pull request #620 from tuncer/travis-dialyze >>> 8a2aca0 travis-ci: allow Dialyzer job to fail >>> 16d5dfe travis-ci: update otp versions >>> 4e372a3 travis-ci: enable 20.1 >>> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning >>> b05882a rebar_cover: ignore opaque type match Dialyzer warning >>> 903c89d rebar_utils: fix Dialyzer warning >>> b36e72b Run Dialyzer on Travis-CI >>> 4bd43fe Merge pull request #645 from Juliusan/log_fix >>> >>> git log --oneline upstream/master | head >>> b6d3094 Merge pull request #620 from tuncer/travis-dialyze >>> 8a2aca0 travis-ci: allow Dialyzer job to fail >>> 16d5dfe travis-ci: update otp versions >>> 4e372a3 travis-ci: enable 20.1 >>> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning >>> b05882a rebar_cover: ignore opaque type match Dialyzer warning >>> 903c89d rebar_utils: fix Dialyzer warning >>> b36e72b Run Dialyzer on Travis-CI >>> 4bd43fe Merge pull
Re: [PROPOSAL] Gradually replace rebar with mix
I'm a big +1 on this. Initially, I know we would have some teething issues getting this working. But I don't think we can stay on rebar2 forever either. Adding in erlang dependancies is becoming increasingly more difficult as they expect rebar3. Cheers Garren On Fri, Aug 16, 2019 at 2:02 PM Ilya Khlopotov wrote: > > > On 2019/08/15 18:51:29, Joan Touzet wrote: > > Hi Ilya, > > > > > > On 2019-08-15 10:09, Ilya Khlopotov wrote: > > > Hello, > > > > > > There is an https://github.com/apache/couchdb/issues/1428 issue about > migrating to rebar3 or mix. I did an experiment to switch from rebar into > mix and wanted to share the results. The code for experiment is here > https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. > Overall I am happy with the experiment. > > > > > > # Requirements > > > > > > 1. Should be able to compile dependencies which use either rebar or > rebar3 > > > 2. Should be able to lock dependencies to specific versions using > references to commits > > > 3. Should be able to call eunit test suite > > > 4. Should be able to compile NIFs > > > 5. Should be easy to install plugins > > > 6. Should support dialyzer > > > 7. Should support release managers > > > 8. Support for raw dependencies > > > 9. Produce source tarball suitable for offline compilation (ASF > requirement) > > > > > > # How mix supports the requirements > > > > > > 1. Mix is able to figure out which build tool to use. There is also a > possibility to specify build tool individually for a given dependency. > > > `manager: mix | rebar | rebar3 | make`. There is also support for > calling arbitrary shell command configured via `compile: `. The > manual step is required to install the needed tools `mix local.rebar > --force`. > > >However there are means to automate that step (we can use Makefile > or define a `bootstrap` mix alias ) > > > 2. Dependencies are locked via `mix.lock` file. > > > > I don't have any strong opinions one way or the other, but I did see > > Garren asking about 'hex' instead of / in conjunction with 'mix' for > > dependency management. Is this effort combined with his or independent? > > I don't know the tools well enough. > > mix works very well with hex > > > > > > 3. Eunit is supported via one of the mix plugins. Both plugins are > very old but it should be easy to write our own (32 lines of code). > > > - https://github.com/dantswain/mix_eunit > > > - https://github.com/talentdeficit/mixunit > > > 4. NIFs can be compiled using one of the following approaches: > > > - specify `rebar` as build manager for a dependency (easiest) > > > > Are you considering staying with rebar2 here, or moving to rebar(3)? It > > would be nice not to have to keep supporting (and shipping!) our fork of > > rebar2. > I considered to stay with rebar2 for first iteration. Could you remind me > why we are forking rebar? > I did check that our version has no differences from upstream. > git remote -v > origin https://github.com/apache/couchdb-rebar (fetch) > origin https://github.com/apache/couchdb-rebar (push) > upstreamhttps://github.com/rebar/rebar (fetch) > upstreamhttps://github.com/rebar/rebar (push) > > git log --oneline origin/master | head > 0c435f3 Update master with mainline rebar (v2) > b6d3094 Merge pull request #620 from tuncer/travis-dialyze > 8a2aca0 travis-ci: allow Dialyzer job to fail > 16d5dfe travis-ci: update otp versions > 4e372a3 travis-ci: enable 20.1 > 73feb5f rebar_xref: ignore opaque type match Dialyzer warning > b05882a rebar_cover: ignore opaque type match Dialyzer warning > 903c89d rebar_utils: fix Dialyzer warning > b36e72b Run Dialyzer on Travis-CI > 4bd43fe Merge pull request #645 from Juliusan/log_fix > > git log --oneline upstream/master | head > b6d3094 Merge pull request #620 from tuncer/travis-dialyze > 8a2aca0 travis-ci: allow Dialyzer job to fail > 16d5dfe travis-ci: update otp versions > 4e372a3 travis-ci: enable 20.1 > 73feb5f rebar_xref: ignore opaque type match Dialyzer warning > b05882a rebar_cover: ignore opaque type match Dialyzer warning > 903c89d rebar_utils: fix Dialyzer warning > b36e72b Run Dialyzer on Travis-CI > 4bd43fe Merge pull request #645 from Juliusan/log_fix > 2276f85 Log out success message with newlines > > git diff origin/master upstream/master > > > > > > - call a Makefile from mix.exs. See examples: > > > - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9 > > > - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6 > > > - use https://github.com/davisp/erlang-native-compiler > > > - use https://github.com/blt/port_compiler > > > > Paul left a comment about this on the roadmap issue just today. Can you > > respond to his comment (it was related tot dependencies-of-dependencies, > > i.e. jiffy) > > > > He also raised the concern that it may force us to use only the very > > latest Erlang versions - can you comment on this? Does the current > > hex/mix situation require a
Re: [PROPOSAL] Gradually replace rebar with mix
On 2019/08/15 18:51:29, Joan Touzet wrote: > Hi Ilya, > > > On 2019-08-15 10:09, Ilya Khlopotov wrote: > > Hello, > > > > There is an https://github.com/apache/couchdb/issues/1428 issue about > > migrating to rebar3 or mix. I did an experiment to switch from rebar into > > mix and wanted to share the results. The code for experiment is here > > https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. > > Overall I am happy with the experiment. > > > > # Requirements > > > > 1. Should be able to compile dependencies which use either rebar or rebar3 > > 2. Should be able to lock dependencies to specific versions using > > references to commits > > 3. Should be able to call eunit test suite > > 4. Should be able to compile NIFs > > 5. Should be easy to install plugins > > 6. Should support dialyzer > > 7. Should support release managers > > 8. Support for raw dependencies > > 9. Produce source tarball suitable for offline compilation (ASF requirement) > > > > # How mix supports the requirements > > > > 1. Mix is able to figure out which build tool to use. There is also a > > possibility to specify build tool individually for a given dependency. > > `manager: mix | rebar | rebar3 | make`. There is also support for calling > > arbitrary shell command configured via `compile: `. The manual > > step is required to install the needed tools `mix local.rebar --force`. > >However there are means to automate that step (we can use Makefile or > > define a `bootstrap` mix alias ) > > 2. Dependencies are locked via `mix.lock` file. > > I don't have any strong opinions one way or the other, but I did see > Garren asking about 'hex' instead of / in conjunction with 'mix' for > dependency management. Is this effort combined with his or independent? > I don't know the tools well enough. mix works very well with hex > > > 3. Eunit is supported via one of the mix plugins. Both plugins are very old > > but it should be easy to write our own (32 lines of code). > > - https://github.com/dantswain/mix_eunit > > - https://github.com/talentdeficit/mixunit > > 4. NIFs can be compiled using one of the following approaches: > > - specify `rebar` as build manager for a dependency (easiest) > > Are you considering staying with rebar2 here, or moving to rebar(3)? It > would be nice not to have to keep supporting (and shipping!) our fork of > rebar2. I considered to stay with rebar2 for first iteration. Could you remind me why we are forking rebar? I did check that our version has no differences from upstream. git remote -v origin https://github.com/apache/couchdb-rebar (fetch) origin https://github.com/apache/couchdb-rebar (push) upstreamhttps://github.com/rebar/rebar (fetch) upstreamhttps://github.com/rebar/rebar (push) git log --oneline origin/master | head 0c435f3 Update master with mainline rebar (v2) b6d3094 Merge pull request #620 from tuncer/travis-dialyze 8a2aca0 travis-ci: allow Dialyzer job to fail 16d5dfe travis-ci: update otp versions 4e372a3 travis-ci: enable 20.1 73feb5f rebar_xref: ignore opaque type match Dialyzer warning b05882a rebar_cover: ignore opaque type match Dialyzer warning 903c89d rebar_utils: fix Dialyzer warning b36e72b Run Dialyzer on Travis-CI 4bd43fe Merge pull request #645 from Juliusan/log_fix git log --oneline upstream/master | head b6d3094 Merge pull request #620 from tuncer/travis-dialyze 8a2aca0 travis-ci: allow Dialyzer job to fail 16d5dfe travis-ci: update otp versions 4e372a3 travis-ci: enable 20.1 73feb5f rebar_xref: ignore opaque type match Dialyzer warning b05882a rebar_cover: ignore opaque type match Dialyzer warning 903c89d rebar_utils: fix Dialyzer warning b36e72b Run Dialyzer on Travis-CI 4bd43fe Merge pull request #645 from Juliusan/log_fix 2276f85 Log out success message with newlines git diff origin/master upstream/master > > > - call a Makefile from mix.exs. See examples: > > - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9 > > - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6 > > - use https://github.com/davisp/erlang-native-compiler > > - use https://github.com/blt/port_compiler > > Paul left a comment about this on the roadmap issue just today. Can you > respond to his comment (it was related tot dependencies-of-dependencies, > i.e. jiffy) > > He also raised the concern that it may force us to use only the very > latest Erlang versions - can you comment on this? Does the current > hex/mix situation require a minimum version? What version is that? I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported on erlang 19 which is AFAIK a minimal erlang version CouchDB supports. > > 5. Installing plugins for mix is similar to rebar3. It is sufficient to > > just specify list of plugins in the deps in mix.exs > > 6. There are multiple dyalizer plugins to choose from > > - https://github.com/jeremyjh/dialyxir > > -
Re: [PROPOSAL] Gradually replace rebar with mix
Hi Ilya, On 2019-08-15 10:09, Ilya Khlopotov wrote: > Hello, > > There is an https://github.com/apache/couchdb/issues/1428 issue about > migrating to rebar3 or mix. I did an experiment to switch from rebar into mix > and wanted to share the results. The code for experiment is here > https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix. > Overall I am happy with the experiment. > > # Requirements > > 1. Should be able to compile dependencies which use either rebar or rebar3 > 2. Should be able to lock dependencies to specific versions using references > to commits > 3. Should be able to call eunit test suite > 4. Should be able to compile NIFs > 5. Should be easy to install plugins > 6. Should support dialyzer > 7. Should support release managers > 8. Support for raw dependencies > 9. Produce source tarball suitable for offline compilation (ASF requirement) > > # How mix supports the requirements > > 1. Mix is able to figure out which build tool to use. There is also a > possibility to specify build tool individually for a given dependency. > `manager: mix | rebar | rebar3 | make`. There is also support for calling > arbitrary shell command configured via `compile: `. The manual step > is required to install the needed tools `mix local.rebar --force`. >However there are means to automate that step (we can use Makefile or > define a `bootstrap` mix alias ) > 2. Dependencies are locked via `mix.lock` file. I don't have any strong opinions one way or the other, but I did see Garren asking about 'hex' instead of / in conjunction with 'mix' for dependency management. Is this effort combined with his or independent? I don't know the tools well enough. > 3. Eunit is supported via one of the mix plugins. Both plugins are very old > but it should be easy to write our own (32 lines of code). > - https://github.com/dantswain/mix_eunit > - https://github.com/talentdeficit/mixunit > 4. NIFs can be compiled using one of the following approaches: > - specify `rebar` as build manager for a dependency (easiest) Are you considering staying with rebar2 here, or moving to rebar(3)? It would be nice not to have to keep supporting (and shipping!) our fork of rebar2. > - call a Makefile from mix.exs. See examples: > - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9 > - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6 > - use https://github.com/davisp/erlang-native-compiler > - use https://github.com/blt/port_compiler Paul left a comment about this on the roadmap issue just today. Can you respond to his comment (it was related tot dependencies-of-dependencies, i.e. jiffy) He also raised the concern that it may force us to use only the very latest Erlang versions - can you comment on this? Does the current hex/mix situation require a minimum version? What version is that? > 5. Installing plugins for mix is similar to rebar3. It is sufficient to just > specify list of plugins in the deps in mix.exs > 6. There are multiple dyalizer plugins to choose from > - https://github.com/jeremyjh/dialyxir > - https://github.com/gabrielgatu/mix-dialyzer I know this is a requirement for Cloudant, but today isn't for base CouchDB. I would prefer not to introduce this requirement at the same time as migrating build tool. > 7. Multiple release managers are supported via mix plugins > - distillery is supported via > https://hexdocs.pm/distillery/Mix.Releases.Plugin.html > - relx and reltools can be used via https://github.com/bitwalker/exrm > (deprecated in favor of distillery) > - we can also roll our own mix alias using > https://github.com/okeuday/reltool_util > - latest elixir versions have `release` command out of the box > 8. Raw dependencies are supported via `compile: false` option > 9. I think it is doable. In the worst case scenario we could just tar the > content of `src` directory after calling `mix deps.get` > > # Warts of mix in the context of CouchDB project > > - CouchDB doesn't use standard elixir directories layout which caused > include_lib directives to not work. To solve this issue absolute path to > `/src` need to be added into include path > - for dependencies managed by mix it is done via `erlc_options` parameter > - Unfortunately when we call out to rebar there is no way to modify > arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS` > environment variable. Which is not very elegant solution. This may also cause problems on Windows, where we have very specific requirements for compiler options so as to ensure the correct libraries are linked for couchjs and so on. Have you tried compiling on that platform? ;) (Please don't make me do all the work here, again...) > - We need to create empty `rebar.config` file for erlang applications which > do not have `rebar.config` (all applications in couchdb repo). This problem > can be fixed by adding mix.exs to dependencies we have
