I'd like us to instead consider flattening to a single repository. I've found no value and only pain from the multiple repositories approach (43 in total!).
The contention is that multiple repositories enforces application boundaries (low coupling / high cohesion) but I've not felt that in reality. We don't, and couldn't meaningfully, release any of our components separately, and, as Ilya makes clear, many enhancements require changes to multiple repositories, and we break this into multiple commits, losing the ability to look at an enhancement in toto. If what Ilya is proposing is the solution, I think it's the solution to a problem we should not have. B. > On 12 Apr 2016, at 16:22, Ilya Khlopotov <[email protected]> wrote: > > > > Dear community, > > > There is a problem with contributors workflow which renders our CI system > useless. As you might know couchdb project consists of multiple > repositories. Most of the time changes cross the repositories boundaries. > When this happens the push to any of the repositories causes CI failures. > CI fails since it uses the old version of dependencies from main repository > of the project. Here is what we can do about it. > > # Proposal > > Let's use multiple files for dependency management. > > - deps.json - serves the same purpose as dependencies list from current > rebar.config.script > - proposed.deps.json - here we specify list of PRs we want to commit > atomically > - override.deps.json - local file outside of version control which we > consult in order to include development tools specific to contributor (code > reloader, debugger, tracer, profiler, binpp, ...) > > Bellow is the example of a content of these files: > > ## deps.json > { > "src/b64url": [ > "https://github.com/apache/couchdb-b64url";, > "6895652d80f95cdf04efb14625abed868998f174" > ], > "src/cassim": [ > "https://github.com/apache/couchdb-cassim";, > "9bbfe82125284fa7cb3317079e8bc1dc876a07bf" > ], > "src/chttpd": [ > "https://github.com/apache/couchdb-chttpd";, > "54e8f6147486d9afc5245e0143d15a4dd1185654" > ], > "src/meck": [ > "https://github.com/apache/couchdb-meck";, > "tree/0.8.2" > ], > .... > } > > ## proposed.deps.json > { > "src/couch": "https://github.com/apache/couchdb-couch/pull/124";, > "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"; > "src/couch_tests": [ > "https://github.com/apache/couchdb-erlang-tests";, > "tree/branch" > ], > } > > # Interface > > I propose to write a simple CLI tool to work with this structure. Bellow is > a list of commands which we need to support (for minimal version) > > ## Adding new dependency > > git propose add https://github.com/apache/couchdb-foo > a2d5ad2eedc960248b806f61df0a1009462bdb46 > git propose add https://github.com/apache/couchdb-bar tree/branch_name > > ## Adding new PR to the change set > > git propose add https://github.com/apache/couchdb-config/pull/4 > > ## Checking out right dependencies > > git propose checkout > > ## Checking out release > > git propose checkout --release # this would ignore proposed.deps.json if it > exists > > ## Merge the change > > This command would do the following: > - Parse proposed.deps.json > - Retrieve merge commit sha for every PR (exit if dependency is not merged > yet) > - Update dependencies in deps.json with correct merge commit sha > - remove proposed.deps.json > > # Workflow > > export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv` > git checkout -b feature-ZZZ > cd src/X && hack dependency X > cd ../.. > cd src/Y && hack dependency Y > issue PRs for X and Y > cd ../.. > git propose add https://github.com/apache/couchdb-X/pull/4 > git propose add https://github.com/apache/couchdb-Y/pull/49 > git add proposed.deps.json > git commit -m "Commit feature {something} which does {a thing} and can be > tested as {procedure}" > git push origin feature-ZZZ > ^ this would trigger our CI > CI would do > git propose checkout && ./configure && make check > > # Pros and Cons > > ## Pros > > - Changes are merged atomically > - CI runs against expected versions of deps > - Enables git bisect > - Reduce tasks that needs to be done by ASF committer (no need to update > rebar.config.script manually) > - Simplifies testing of PRs by reviewers > - Simplifies rebar.config since rebar is not used for managing deps > > ## Cons > > - some github.com specifics (concept of PRs and access to github API to get > info about PR) > - we need to have github.com as one of the remotes > - we trigger CI only on push to main repository > > # Implementation > > We write a git-propose script in python and place it in ./bin. We add ./bin > into either GIT_EXEC_PATH or PATH. You always can call the script directly > (as ./bin/git-propose) if you don't like amending > your environment. > > # Later improvements > > - We can issue PRs from the tool itself > - We can merge from the tool itself > - We can implement support for multiple remotes (asf, github, private) > - We can implement support for multiple git transports (for the first > version all repositories in *.deps.json files would use https://) > > Sincerely, > ILYA KHLOPOTOV
