Re: On dependency management and CI issues associated with it

[Joan Touzet]

Garren, correct me if I'm wrong but Fauxton depends on a large number
of JS dependencies that we don't keep copies of, correct? Or is it just
for the build process?

-Joan

- Original Message -
> From: "Alexander Shorin" 
> To: dev@couchdb.apache.org
> Sent: Wednesday, April 13, 2016 2:08:20 PM
> Subject: Re: On dependency management and CI issues associated with it
> 
> On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson 
> wrote:
> > It's a thread derail but this notion that we're being "fairly rude"
> > needs resolving. It might be lost to history now but we got here,
> > I think, with the best intentions of ensuring all the code that
> > appears in couchdb can be traced back to code hosted at asf. Is it
> > a concrete requirement? I honestly forget but I thought so.
> 
> Yes, that's the answer why. If one day mochiweb owner will decide to
> drop his github repo, we shouldn't be leave with broken builds. See
> leftpad story as well. Initially, that requirement seems redundant,
> but recent npm drama showed that it has a huge point. Also there are
> some legal bits about.
> 
> --
> ,,,^..^,,,
> 

Re: On dependency management and CI issues associated with it

[Adam Kocoloski]

> On Apr 13, 2016, at 2:08 PM, Alexander Shorin  wrote:
> 
> On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson  wrote:
>> It's a thread derail but this notion that we're being "fairly rude" needs 
>> resolving. It might be lost to history now but we got here, I think, with 
>> the best intentions of ensuring all the code that appears in couchdb can be 
>> traced back to code hosted at asf. Is it a concrete requirement? I honestly 
>> forget but I thought so.
> 
> Yes, that's the answer why. If one day mochiweb owner will decide to
> drop his github repo, we shouldn't be leave with broken builds. See
> leftpad story as well. Initially, that requirement seems redundant,
> but recent npm drama showed that it has a huge point. Also there are
> some legal bits about.
> 
> --
> ,,,^..^,,,

All good points. I’m fine running off a fork but would like to see Apache give 
the author credit in the GitHub network.

Adam

Re: On dependency management and CI issues associated with it

[Alexander Shorin]

On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson  wrote:
> It's a thread derail but this notion that we're being "fairly rude" needs 
> resolving. It might be lost to history now but we got here, I think, with the 
> best intentions of ensuring all the code that appears in couchdb can be 
> traced back to code hosted at asf. Is it a concrete requirement? I honestly 
> forget but I thought so.

Yes, that's the answer why. If one day mochiweb owner will decide to
drop his github repo, we shouldn't be leave with broken builds. See
leftpad story as well. Initially, that requirement seems redundant,
but recent npm drama showed that it has a huge point. Also there are
some legal bits about.

--
,,,^..^,,,

Re: On dependency management and CI issues associated with it

[Paul Davis]

I wouldn't quite go so far as to call it rude but it is at least
annoying that we're not part of the usptream network on GitHub.
Unfortunately, between ASF rules and regulations and the GitHub
integration there's not a whole lot we can do. Granted given that
we're not developing downstream in those repos I'm not sure its a big
deal to worry about. Any ASF committer that wants to make an upstream
change can fork the upstream repo, patch and PR, and then our
downstream mirrors can be updated.

On Wed, Apr 13, 2016 at 12:39 PM, Robert Newson  wrote:
> I'd exclude third party repos, sure.
>
> It's a thread derail but this notion that we're being "fairly rude" needs 
> resolving. It might be lost to history now but we got here, I think, with the 
> best intentions of ensuring all the code that appears in couchdb can be 
> traced back to code hosted at asf. Is it a concrete requirement? I honestly 
> forget but I thought so.
>
> If it's not a requirement then we could point to non asf hosted repositories, 
> if we accept that things we've used in the past could vanish at any time.
>
> A halfway might be a true fork hosted on GitHub? I think even we're not safer 
> as the original project could be pulled and take the forks with it. This has 
> happened. It's obviously unlikely.
>
>> On 13 Apr 2016, at 17:44, Adam Kocoloski  wrote:
>>
>>
>>> On Apr 13, 2016, at 12:30 PM, Alexander Shorin  wrote:
>>>
>>> Hi Paul!
>>>
>>> Thanks for great input!
>>>
 On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis  
 wrote:
 If anyone has a strong objection to a monolithic Erlang repo I'd like
 to hear it. Otherwise I may work up a lengthier and more thorough
 proposal for dev@ to consider consolidating all of these repositories
 for sanity and profit.
>>>
>>> It's hard to object against that since this actually solves a lot of
>>> problems solution of which will require more work to do and still will
>>> leave a place for mistakes or require quite specific toolchain to
>>> work.
>>>
>>> Making our current repos design work right will require even more work to 
>>> apply.
>>>
>>> So, for point of time/resources/usability there is no much choice.
>>>
>>> I think folding the "Erlang repos developed by ASF" list will solve
>>> most part of the problems. However, I think it worth to keep these
>>> apps in own repos:
>>> - rexi
>>> - b64url
>>> - config
>>> - snappy
>>> - khash
>>> - ets-lru
>>> - twig (why we still need it?)
>>>
>>> As they could be reused outside and they shouldn't involve any
>>> dependencies with other couch modules by design. Everyone else may
>>> stand on where they are.
>>>
>>> P.S. I'm not sure if git-subtree will not introduce more new problems
>>> as it's quite tricky thing to live with it.
>>>
>>> --
>>> ,,,^..^,,,
>>
>> +1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just 
>> starting to get better support for package managers and the like, and it’d 
>> be a shame to miss out on opportunities for adoption of and contribution to 
>> general-purpose libraries like config, rexi, khash or ets-lru by bundling 
>> them into a repo that doesn’t look or feel like an idiomatic Erlang repo. 
>> And at any rate I certainly hope no one is proposing to copy/paste other 
>> upstream deps into one repo — the current practice of hosting a fork that 
>> doesn’t get recognized as a fork in GitHub is already fairly rude behavior 
>> on our part.
>>
>> I’ll agree that some of the repo creation has gotten a bit out of control. I 
>> believe _some_ of the friction is healthy; I think some of my contributions 
>> over the years were better precisely because the repo structure made bad 
>> designs harder to implement. The friction in the CI / CD pipeline and 
>> overall dependency management is not healthy, though.
>>
>> Moving more code back into one repo puts more responsibility on the devs to 
>> architect new enhancements in the right way without the repo structure there 
>> to enforce a few of the best practices. That’s OK so long as we understand 
>> the tradeoff.
>>
>> Adam
>>
>

Re: On dependency management and CI issues associated with it

[Robert Newson]

As for the separation we have enforcing good practices, I don't buy it. 

I don't think it will be difficult to prevent the kind of coupling you (and I) 
would find troubling.  It might even be easier to see if a single commit 
touches multiple src/ subdirectories that might be missed when reviewing 
separate pr's. At least, I'm not sure I could slide a credit card between them. 

> On 13 Apr 2016, at 17:44, Adam Kocoloski  wrote:
> 
> 
>> On Apr 13, 2016, at 12:30 PM, Alexander Shorin  wrote:
>> 
>> Hi Paul!
>> 
>> Thanks for great input!
>> 
>>> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis  
>>> wrote:
>>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>>> to hear it. Otherwise I may work up a lengthier and more thorough
>>> proposal for dev@ to consider consolidating all of these repositories
>>> for sanity and profit.
>> 
>> It's hard to object against that since this actually solves a lot of
>> problems solution of which will require more work to do and still will
>> leave a place for mistakes or require quite specific toolchain to
>> work.
>> 
>> Making our current repos design work right will require even more work to 
>> apply.
>> 
>> So, for point of time/resources/usability there is no much choice.
>> 
>> I think folding the "Erlang repos developed by ASF" list will solve
>> most part of the problems. However, I think it worth to keep these
>> apps in own repos:
>> - rexi
>> - b64url
>> - config
>> - snappy
>> - khash
>> - ets-lru
>> - twig (why we still need it?)
>> 
>> As they could be reused outside and they shouldn't involve any
>> dependencies with other couch modules by design. Everyone else may
>> stand on where they are.
>> 
>> P.S. I'm not sure if git-subtree will not introduce more new problems
>> as it's quite tricky thing to live with it.
>> 
>> --
>> ,,,^..^,,,
> 
> +1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just 
> starting to get better support for package managers and the like, and it’d be 
> a shame to miss out on opportunities for adoption of and contribution to 
> general-purpose libraries like config, rexi, khash or ets-lru by bundling 
> them into a repo that doesn’t look or feel like an idiomatic Erlang repo. And 
> at any rate I certainly hope no one is proposing to copy/paste other upstream 
> deps into one repo — the current practice of hosting a fork that doesn’t get 
> recognized as a fork in GitHub is already fairly rude behavior on our part.
> 
> I’ll agree that some of the repo creation has gotten a bit out of control. I 
> believe _some_ of the friction is healthy; I think some of my contributions 
> over the years were better precisely because the repo structure made bad 
> designs harder to implement. The friction in the CI / CD pipeline and overall 
> dependency management is not healthy, though.
> 
> Moving more code back into one repo puts more responsibility on the devs to 
> architect new enhancements in the right way without the repo structure there 
> to enforce a few of the best practices. That’s OK so long as we understand 
> the tradeoff.
> 
> Adam
> 

Re: On dependency management and CI issues associated with it

[Robert Newson]

I'd exclude third party repos, sure. 

It's a thread derail but this notion that we're being "fairly rude" needs 
resolving. It might be lost to history now but we got here, I think, with the 
best intentions of ensuring all the code that appears in couchdb can be traced 
back to code hosted at asf. Is it a concrete requirement? I honestly forget but 
I thought so.

If it's not a requirement then we could point to non asf hosted repositories, 
if we accept that things we've used in the past could vanish at any time. 

A halfway might be a true fork hosted on GitHub? I think even we're not safer 
as the original project could be pulled and take the forks with it. This has 
happened. It's obviously unlikely. 

> On 13 Apr 2016, at 17:44, Adam Kocoloski  wrote:
> 
> 
>> On Apr 13, 2016, at 12:30 PM, Alexander Shorin  wrote:
>> 
>> Hi Paul!
>> 
>> Thanks for great input!
>> 
>>> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis  
>>> wrote:
>>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>>> to hear it. Otherwise I may work up a lengthier and more thorough
>>> proposal for dev@ to consider consolidating all of these repositories
>>> for sanity and profit.
>> 
>> It's hard to object against that since this actually solves a lot of
>> problems solution of which will require more work to do and still will
>> leave a place for mistakes or require quite specific toolchain to
>> work.
>> 
>> Making our current repos design work right will require even more work to 
>> apply.
>> 
>> So, for point of time/resources/usability there is no much choice.
>> 
>> I think folding the "Erlang repos developed by ASF" list will solve
>> most part of the problems. However, I think it worth to keep these
>> apps in own repos:
>> - rexi
>> - b64url
>> - config
>> - snappy
>> - khash
>> - ets-lru
>> - twig (why we still need it?)
>> 
>> As they could be reused outside and they shouldn't involve any
>> dependencies with other couch modules by design. Everyone else may
>> stand on where they are.
>> 
>> P.S. I'm not sure if git-subtree will not introduce more new problems
>> as it's quite tricky thing to live with it.
>> 
>> --
>> ,,,^..^,,,
> 
> +1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just 
> starting to get better support for package managers and the like, and it’d be 
> a shame to miss out on opportunities for adoption of and contribution to 
> general-purpose libraries like config, rexi, khash or ets-lru by bundling 
> them into a repo that doesn’t look or feel like an idiomatic Erlang repo. And 
> at any rate I certainly hope no one is proposing to copy/paste other upstream 
> deps into one repo — the current practice of hosting a fork that doesn’t get 
> recognized as a fork in GitHub is already fairly rude behavior on our part.
> 
> I’ll agree that some of the repo creation has gotten a bit out of control. I 
> believe _some_ of the friction is healthy; I think some of my contributions 
> over the years were better precisely because the repo structure made bad 
> designs harder to implement. The friction in the CI / CD pipeline and overall 
> dependency management is not healthy, though.
> 
> Moving more code back into one repo puts more responsibility on the devs to 
> architect new enhancements in the right way without the repo structure there 
> to enforce a few of the best practices. That’s OK so long as we understand 
> the tradeoff.
> 
> Adam
> 

Re: On dependency management and CI issues associated with it

[Nick Vatamaniuc]

No experience using subtrees, but remember Rust switched to use those:

https://github.com/rust-lang/rust/tree/master/src

jemalloc, llvm and few others are subtrees.

PR with some discussion: https://github.com/rust-lang/rust/pull/26042

-Nick



On Wed, Apr 13, 2016 at 12:38 PM, Paul Davis
 wrote:
> Does anyone have any experience with git subtree on this list? I was
> under the impression that as long as you ensured that it was a strict
> copy of upstream it was fairly simple.
>
> For your list of repos to keep separate, those sound fine to me
> regardless of subtree. They're all fairly stable at this point and I
> agree at least plausibly usable outside CouchDB.
>
> On Wed, Apr 13, 2016 at 11:30 AM, Alexander Shorin  wrote:
>> Hi Paul!
>>
>> Thanks for great input!
>>
>> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis  
>> wrote:
>>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>>> to hear it. Otherwise I may work up a lengthier and more thorough
>>> proposal for dev@ to consider consolidating all of these repositories
>>> for sanity and profit.
>>
>> It's hard to object against that since this actually solves a lot of
>> problems solution of which will require more work to do and still will
>> leave a place for mistakes or require quite specific toolchain to
>> work.
>>
>> Making our current repos design work right will require even more work to 
>> apply.
>>
>> So, for point of time/resources/usability there is no much choice.
>>
>> I think folding the "Erlang repos developed by ASF" list will solve
>> most part of the problems. However, I think it worth to keep these
>> apps in own repos:
>> - rexi
>> - b64url
>> - config
>> - snappy
>> - khash
>> - ets-lru
>> - twig (why we still need it?)
>>
>> As they could be reused outside and they shouldn't involve any
>> dependencies with other couch modules by design. Everyone else may
>> stand on where they are.
>>
>> P.S. I'm not sure if git-subtree will not introduce more new problems
>> as it's quite tricky thing to live with it.
>>
>> --
>> ,,,^..^,,,

Re: On dependency management and CI issues associated with it

[Adam Kocoloski]

> On Apr 13, 2016, at 12:30 PM, Alexander Shorin  wrote:
> 
> Hi Paul!
> 
> Thanks for great input!
> 
> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis  
> wrote:
>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>> to hear it. Otherwise I may work up a lengthier and more thorough
>> proposal for dev@ to consider consolidating all of these repositories
>> for sanity and profit.
> 
> It's hard to object against that since this actually solves a lot of
> problems solution of which will require more work to do and still will
> leave a place for mistakes or require quite specific toolchain to
> work.
> 
> Making our current repos design work right will require even more work to 
> apply.
> 
> So, for point of time/resources/usability there is no much choice.
> 
> I think folding the "Erlang repos developed by ASF" list will solve
> most part of the problems. However, I think it worth to keep these
> apps in own repos:
> - rexi
> - b64url
> - config
> - snappy
> - khash
> - ets-lru
> - twig (why we still need it?)
> 
> As they could be reused outside and they shouldn't involve any
> dependencies with other couch modules by design. Everyone else may
> stand on where they are.
> 
> P.S. I'm not sure if git-subtree will not introduce more new problems
> as it's quite tricky thing to live with it.
> 
> --
> ,,,^..^,,,

+1 Alex. I disagree that “one Erlang repo” is the goal. Erlang is just starting 
to get better support for package managers and the like, and it’d be a shame to 
miss out on opportunities for adoption of and contribution to general-purpose 
libraries like config, rexi, khash or ets-lru by bundling them into a repo that 
doesn’t look or feel like an idiomatic Erlang repo. And at any rate I certainly 
hope no one is proposing to copy/paste other upstream deps into one repo — the 
current practice of hosting a fork that doesn’t get recognized as a fork in 
GitHub is already fairly rude behavior on our part.

I’ll agree that some of the repo creation has gotten a bit out of control. I 
believe _some_ of the friction is healthy; I think some of my contributions 
over the years were better precisely because the repo structure made bad 
designs harder to implement. The friction in the CI / CD pipeline and overall 
dependency management is not healthy, though.

Moving more code back into one repo puts more responsibility on the devs to 
architect new enhancements in the right way without the repo structure there to 
enforce a few of the best practices. That’s OK so long as we understand the 
tradeoff.

Adam

Re: On dependency management and CI issues associated with it

[Paul Davis]

Does anyone have any experience with git subtree on this list? I was
under the impression that as long as you ensured that it was a strict
copy of upstream it was fairly simple.

For your list of repos to keep separate, those sound fine to me
regardless of subtree. They're all fairly stable at this point and I
agree at least plausibly usable outside CouchDB.

On Wed, Apr 13, 2016 at 11:30 AM, Alexander Shorin  wrote:
> Hi Paul!
>
> Thanks for great input!
>
> On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis  
> wrote:
>> If anyone has a strong objection to a monolithic Erlang repo I'd like
>> to hear it. Otherwise I may work up a lengthier and more thorough
>> proposal for dev@ to consider consolidating all of these repositories
>> for sanity and profit.
>
> It's hard to object against that since this actually solves a lot of
> problems solution of which will require more work to do and still will
> leave a place for mistakes or require quite specific toolchain to
> work.
>
> Making our current repos design work right will require even more work to 
> apply.
>
> So, for point of time/resources/usability there is no much choice.
>
> I think folding the "Erlang repos developed by ASF" list will solve
> most part of the problems. However, I think it worth to keep these
> apps in own repos:
> - rexi
> - b64url
> - config
> - snappy
> - khash
> - ets-lru
> - twig (why we still need it?)
>
> As they could be reused outside and they shouldn't involve any
> dependencies with other couch modules by design. Everyone else may
> stand on where they are.
>
> P.S. I'm not sure if git-subtree will not introduce more new problems
> as it's quite tricky thing to live with it.
>
> --
> ,,,^..^,,,

Re: On dependency management and CI issues associated with it

[Alexander Shorin]

Hi Paul!

Thanks for great input!

On Wed, Apr 13, 2016 at 7:11 PM, Paul Davis  wrote:
> If anyone has a strong objection to a monolithic Erlang repo I'd like
> to hear it. Otherwise I may work up a lengthier and more thorough
> proposal for dev@ to consider consolidating all of these repositories
> for sanity and profit.

It's hard to object against that since this actually solves a lot of
problems solution of which will require more work to do and still will
leave a place for mistakes or require quite specific toolchain to
work.

Making our current repos design work right will require even more work to apply.

So, for point of time/resources/usability there is no much choice.

I think folding the "Erlang repos developed by ASF" list will solve
most part of the problems. However, I think it worth to keep these
apps in own repos:
- rexi
- b64url
- config
- snappy
- khash
- ets-lru
- twig (why we still need it?)

As they could be reused outside and they shouldn't involve any
dependencies with other couch modules by design. Everyone else may
stand on where they are.

P.S. I'm not sure if git-subtree will not introduce more new problems
as it's quite tricky thing to live with it.

--
,,,^..^,,,

Re: On dependency management and CI issues associated with it

[Robert Newson]

Keeping fauxton in a separate repo makes sense. It has a different release 
cycle. It's genuinely decoupled. Getting all the Erlang into one repo is really 
the goal. 

With couch_epi as a core application, anyone can extend and customise couchdb 
by adding another dependency.  At most, we might identify new places for epi 
hooks as we go. 

> On 13 Apr 2016, at 17:11, Paul Davis  wrote:
> 
> Hello everybody!
> 
> Wow, 56 repos! Hopefully we get an award somewhere for that. I've
> listed the repositories below in some crude groups to try and give an
> idea of what we're working with. I have to agree that this is getting
> a bit on the ridiculous side. Of all of the repos that the ASF
> actually develops I'm only seeing four or so Erlang apps (b64url,
> config, couch-collate, and khash) that are likely truly re-usable
> outside of CouchDB without mucking about.
> 
> I have previously (years ago) played around with trying to go back to
> a single repository. It generally works fine, the only issue that I
> found was that rebar's {vsn, git} tag for *.app.src files doesn't work
> in a single repo and it gets a bit complicated managing that by hand.
> However, I think it would be possible to add something like rebar's
> deps files and a small custom tool that either a) breaks the build if
> versions haven't changed or b) even better, automatically sets
> application versions based on a source file (and tweaks them like git
> describe when there's been a commit since the version). This lets us
> continue to "alias" commits with a human readable version and doesn't
> require a single version across all applications. (Alternatively, we
> could wipe the version info on every project and set a single version
> that is the same for all applications that matches the CouchDB
> version, but this might get weird for upstream dependencies).
> 
> That said, I'd agree with Bob that the new dependency format seems to
> be solving a problem we shouldn't have. I'd rather just pull
> everything into a single repo and use tooling to help maintain any
> sharp edges like the versioning issue I mentioned above.
> 
> Personally, what I'd like to see is to have all Erlang repos merged
> into the main couchdb.git repo and then have all upstream dependencies
> managed by git-subtree. I could go either way on having the node and
> spidermonkey view engines included or not. For the non Erlang parts of
> our release (fauxton and documentation) I'd keep them as separate
> repos so that their tooling doesn't need to be changed and/or adapted
> to work out of a repo subdirectory. The administration things also
> seem to make good sense to keep separate as they're not part of the
> product/release tarball/whatever.
> 
> If anyone has a strong objection to a monolithic Erlang repo I'd like
> to hear it. Otherwise I may work up a lengthier and more thorough
> proposal for dev@ to consider consolidating all of these repositories
> for sanity and profit.
> 
> Paul
> 
> Main repo:
> 
> couchdb.git
> 
> 
> Erlang repos developed by ASF:
> 
> couchdb-b64url.git
> couchdb-cassim.git
> couchdb-chttpd.git
> couchdb-config.git
> couchdb-couch-collate.git
> couchdb-couch-dbupdates.git
> couchdb-couch-epi.git
> couchdb-couch-event.git
> couchdb-couch-httpd.git
> couchdb-couch-index.git
> couchdb-couch-log-lager.git
> couchdb-couch-log.git
> couchdb-couch-mrview.git
> couchdb-couch-plugins.git
> couchdb-couch-replicator.git
> couchdb-couch-stats.git
> couchdb-couch.git
> couchdb-ddoc-cache.git
> couchdb-erlang-tests.git
> couchdb-ets-lru.git
> couchdb-fabric.git
> couchdb-global-changes.git
> couchdb-ioq.git
> couchdb-khash.git
> couchdb-mango.git
> couchdb-mem3.git
> couchdb-peruser.git
> couchdb-rexi.git
> couchdb-setup.git
> couchdb-snappy.git
> couchdb-twig.git
> 
> 
> Non-Erlang things we develop as part of a release:
> 
> couchdb-fauxton.git
> couchdb-documentation.git
> 
> 
> Mirrored repos of upstream Erlang deps:
> 
> couchdb-bear.git
> couchdb-folsom.git
> couchdb-goldrush.git
> couchdb-ibrowse.git
> couchdb-jiffy.git
> couchdb-lager.git
> couchdb-meck.git
> couchdb-mochiweb.git
> couchdb-oauth.git
> couchdb-rebar.git
> 
> 
> Query Servers:
> 
> couchdb-query-server-node.git
> couchdb-query-server-spidermonkey.git
> 
> 
> Unsure but has Erlang in it:
> 
> couchdb-examples.git
> 
> 
> Project Administrative Things Kinda:
> 
> couchdb-admin.git
> couchdb-ci.git
> couchdb-docker.git
> couchdb-www.git
> 
> 
> Client Library:
> 
> couchdb-nano.git
> 
> 
> JS CLI tool:
> 
> couchdb-nmo.git
> 
> 
> Empty:
> 
> couchdb-javascript-tests.git
> 
> Legacy:
> 
> couchdb-futon.git
> couchdb-jquery-couch.git
> 
> 
>> On Wed, Apr 13, 2016 at 3:41 AM, Garren Smith  wrote:
>> I like the idea of going back to a single repo for core db features. I
>> would like Fauxton to still be in its own repo.
>> As someone who wrote some very basic erlang code for CouchDB recently. I
>> found the multiple repos quite tricky to 

Re: On dependency management and CI issues associated with it

[Paul Davis]

Hello everybody!

Wow, 56 repos! Hopefully we get an award somewhere for that. I've
listed the repositories below in some crude groups to try and give an
idea of what we're working with. I have to agree that this is getting
a bit on the ridiculous side. Of all of the repos that the ASF
actually develops I'm only seeing four or so Erlang apps (b64url,
config, couch-collate, and khash) that are likely truly re-usable
outside of CouchDB without mucking about.

I have previously (years ago) played around with trying to go back to
a single repository. It generally works fine, the only issue that I
found was that rebar's {vsn, git} tag for *.app.src files doesn't work
in a single repo and it gets a bit complicated managing that by hand.
However, I think it would be possible to add something like rebar's
deps files and a small custom tool that either a) breaks the build if
versions haven't changed or b) even better, automatically sets
application versions based on a source file (and tweaks them like git
describe when there's been a commit since the version). This lets us
continue to "alias" commits with a human readable version and doesn't
require a single version across all applications. (Alternatively, we
could wipe the version info on every project and set a single version
that is the same for all applications that matches the CouchDB
version, but this might get weird for upstream dependencies).

That said, I'd agree with Bob that the new dependency format seems to
be solving a problem we shouldn't have. I'd rather just pull
everything into a single repo and use tooling to help maintain any
sharp edges like the versioning issue I mentioned above.

Personally, what I'd like to see is to have all Erlang repos merged
into the main couchdb.git repo and then have all upstream dependencies
managed by git-subtree. I could go either way on having the node and
spidermonkey view engines included or not. For the non Erlang parts of
our release (fauxton and documentation) I'd keep them as separate
repos so that their tooling doesn't need to be changed and/or adapted
to work out of a repo subdirectory. The administration things also
seem to make good sense to keep separate as they're not part of the
product/release tarball/whatever.

If anyone has a strong objection to a monolithic Erlang repo I'd like
to hear it. Otherwise I may work up a lengthier and more thorough
proposal for dev@ to consider consolidating all of these repositories
for sanity and profit.

Paul

Main repo:

couchdb.git


Erlang repos developed by ASF:

couchdb-b64url.git
couchdb-cassim.git
couchdb-chttpd.git
couchdb-config.git
couchdb-couch-collate.git
couchdb-couch-dbupdates.git
couchdb-couch-epi.git
couchdb-couch-event.git
couchdb-couch-httpd.git
couchdb-couch-index.git
couchdb-couch-log-lager.git
couchdb-couch-log.git
couchdb-couch-mrview.git
couchdb-couch-plugins.git
couchdb-couch-replicator.git
couchdb-couch-stats.git
couchdb-couch.git
couchdb-ddoc-cache.git
couchdb-erlang-tests.git
couchdb-ets-lru.git
couchdb-fabric.git
couchdb-global-changes.git
couchdb-ioq.git
couchdb-khash.git
couchdb-mango.git
couchdb-mem3.git
couchdb-peruser.git
couchdb-rexi.git
couchdb-setup.git
couchdb-snappy.git
couchdb-twig.git


Non-Erlang things we develop as part of a release:

couchdb-fauxton.git
couchdb-documentation.git


Mirrored repos of upstream Erlang deps:

couchdb-bear.git
couchdb-folsom.git
couchdb-goldrush.git
couchdb-ibrowse.git
couchdb-jiffy.git
couchdb-lager.git
couchdb-meck.git
couchdb-mochiweb.git
couchdb-oauth.git
couchdb-rebar.git


Query Servers:

couchdb-query-server-node.git
couchdb-query-server-spidermonkey.git


Unsure but has Erlang in it:

couchdb-examples.git


Project Administrative Things Kinda:

couchdb-admin.git
couchdb-ci.git
couchdb-docker.git
couchdb-www.git


Client Library:

couchdb-nano.git


JS CLI tool:

couchdb-nmo.git


Empty:

couchdb-javascript-tests.git

Legacy:

couchdb-futon.git
couchdb-jquery-couch.git


On Wed, Apr 13, 2016 at 3:41 AM, Garren Smith  wrote:
> I like the idea of going back to a single repo for core db features. I
> would like Fauxton to still be in its own repo.
> As someone who wrote some very basic erlang code for CouchDB recently. I
> found the multiple repos quite tricky to manage and I couldn't see how it
> made anything easier.
>
> On Wed, Apr 13, 2016 at 8:35 AM, Alexander Shorin  wrote:
>
>> Hi Robert,
>>
>> Point about flattening to a single repository is valid: in the end, we
>> have our apps repos in broken state all the time as they are not
>> declare their decencies. So noone can pick fabric@master and run it -
>> he'll spend quite a lot of time to figure the deps of the right
>> versions. But the idea to solve the problem by reducing set of
>> repositories we have to test is good.
>>
>>
>> Hi Iliya,
>>
>> I have alternative solution for you:
>>
>> - Turn-off Travis CI everywhere where we cannot be sure about testing
>> without depended PRs (all except 

Re: On dependency management and CI issues associated with it

[Alexander Shorin]

On Wed, Apr 13, 2016 at 6:46 PM, Ilya Khlopotov  wrote:
>
> > - If you rebase/update any of your subcomponent PRs you must update
> commit hash on apache/couchdb one;
>
> Above is an error prune step which actually makes the following false
> > - No new new steps/files/work introduced, so there is no need to care
> about learning curve;
>
> > Sounds simpler and better for me, how it does for you?
> Simpler - maybe. Better not so much.

Yes, I also found a case, when you have at least twice trigger CI
build: once for initial PR, and second after all related PRs get
merged, because merge operation produces merge commit which you should
write into rebar.config instead.

I think this could be automated as well (wonder if repo tool can be
extended for that kind of task), but there is still need to think
about and do some research.

> We can store deps in proposed format and make rebar understand json (rebar3 
> understands json BTW) but I am sure this approach would be at least frowned 
> upon if not rejected.

If rebar3 already supports proposed by you format, then good. More
points to upgrade that tortoise. But agree on with your latter part.

> Also I would really love to merge in my local deps I use in development. But 
> this is a different topic probably.

Good point about deps overlaying. It's nice feature to have.

P.S. Seems like the most simple way is to fold all the repos back to
one, but I won't think about the hell someone have to pass though to
set commit history in the right order without breaking anything...


--
,,,^..^,,,

Re: On dependency management and CI issues associated with it

[Ilya Khlopotov]

Hi Alex,

Thank you for your feedback.

> - If you rebase/update any of your subcomponent PRs you must update
commit hash on apache/couchdb one;

Above is an error prune step which actually makes the following false
> - No new new steps/files/work introduced, so there is no need to care
about learning curve;

> Sounds simpler and better for me, how it does for you?
Simpler - maybe. Better not so much. I would like to add some other
automatons on top latter on. Such as automatic generation of a Release
Notes between two tags for example. In order to do that I would need a list
of dependencies in simple format.
I always can call `bin/rebar -q -C rebar.config.script list-deps` from my
python scripts but I cannot do it every time I need to lookup something
since it slows things down significantly. So I have to read once and cache
it in memory and do other trickery. Relying on trickery makes the script
complicated which would raise additional questions on the PR with it.

We can store deps in proposed format and make rebar understand json (rebar3
understands json BTW) but I am sure this approach would be at least frowned
upon if not rejected.

Also I would really love to merge in my local deps I use in development.
But this is a different topic probably.

Best regards,
ILYA



From:   Alexander Shorin 
To: "dev@couchdb.apache.org" 
Date:   2016/04/12 11:35 PM
Subject:Re: On dependency management and CI issues associated with it



Hi Robert,

Point about flattening to a single repository is valid: in the end, we
have our apps repos in broken state all the time as they are not
declare their decencies. So noone can pick fabric@master and run it -
he'll spend quite a lot of time to figure the deps of the right
versions. But the idea to solve the problem by reducing set of
repositories we have to test is good.


Hi Iliya,

I have alternative solution for you:

- Turn-off Travis CI everywhere where we cannot be sure about testing
without depended PRs (all except third-party modules, fauxton, docs,
and few more independent projects like couch-epi);
- Require everyone to submit additional PR to apache/couchdb repo with
commit hashes update;
- On this apache/couchdb PR we'll run CI testing;
- If you rebase/update any of your subcomponent PRs you must update
commit hash on apache/couchdb one;

Pros:
- We won't forget to update rebar.config when new changes lands;
- We will always run complete integration testing with all the right
deps states;
- We won't have to invent any complicated integration solutions to
deal with sub-repos testing;
- No new new steps/files/work introduced, so there is no need to care
about learning curve;

Cons:
- Need to be a bit tricky on Travis builder to realize on which remote
(fork) new rebar.config hashes are to correctly checkout them, though
that is not a rocket science since we have access to git information
there.

The Jenkins CI role here is to ensure that we have master build right
and releases build right, on the various OSes.

Sounds simpler and better for me, how it does for you?

--
,,,^..^,,,


On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson
 wrote:
> I'd like us to instead consider flattening to a single repository. I've
found no value and only pain from the multiple repositories approach (43 in
total!).
>
> The contention is that multiple repositories enforces application
boundaries (low coupling / high cohesion) but I've not felt that in
reality. We don't, and couldn't meaningfully, release any of our components
separately, and, as Ilya makes clear, many enhancements require changes to
multiple repositories, and we break this into multiple commits, losing the
ability to look at an enhancement in toto.
>
> If what Ilya is proposing is the solution, I think it's the solution to a
problem we should not have.
>
> B.
>
>> On 12 Apr 2016, at 16:22, Ilya Khlopotov  wrote:
>>
>>
>>
>> Dear community,
>>
>>
>> There is a problem with contributors workflow which renders our CI
system
>> useless. As you might know couchdb project consists of multiple
>> repositories. Most of the time changes cross the repositories
boundaries.
>> When this happens the push to any of the repositories causes CI
failures.
>> CI fails since it uses the old version of dependencies from main
repository
>> of the project. Here is what we can do about it.
>>
>> # Proposal
>>
>> Let's use multiple files for dependency management.
>>
>> - deps.json - serves the same purpose as dependencies list from current
>> rebar.config.script
>> - proposed.deps.json - here we specify list of PRs we want to commit
>> atomically
>> - override.deps.json - local file outside of version control which we
>> consult in order to include development tools specific to contributor
(code
>> reloader, debugger, tracer, profiler, binpp, ...)
>>
>> Bellow is the example of a content of these files:
>>
>> ## deps.json
>> {
>>"src/b64url": 

Re: On dependency management and CI issues associated with it

[Garren Smith]

I like the idea of going back to a single repo for core db features. I
would like Fauxton to still be in its own repo.
As someone who wrote some very basic erlang code for CouchDB recently. I
found the multiple repos quite tricky to manage and I couldn't see how it
made anything easier.

On Wed, Apr 13, 2016 at 8:35 AM, Alexander Shorin  wrote:

> Hi Robert,
>
> Point about flattening to a single repository is valid: in the end, we
> have our apps repos in broken state all the time as they are not
> declare their decencies. So noone can pick fabric@master and run it -
> he'll spend quite a lot of time to figure the deps of the right
> versions. But the idea to solve the problem by reducing set of
> repositories we have to test is good.
>
>
> Hi Iliya,
>
> I have alternative solution for you:
>
> - Turn-off Travis CI everywhere where we cannot be sure about testing
> without depended PRs (all except third-party modules, fauxton, docs,
> and few more independent projects like couch-epi);
> - Require everyone to submit additional PR to apache/couchdb repo with
> commit hashes update;
> - On this apache/couchdb PR we'll run CI testing;
> - If you rebase/update any of your subcomponent PRs you must update
> commit hash on apache/couchdb one;
>
> Pros:
> - We won't forget to update rebar.config when new changes lands;
> - We will always run complete integration testing with all the right
> deps states;
> - We won't have to invent any complicated integration solutions to
> deal with sub-repos testing;
> - No new new steps/files/work introduced, so there is no need to care
> about learning curve;
>
> Cons:
> - Need to be a bit tricky on Travis builder to realize on which remote
> (fork) new rebar.config hashes are to correctly checkout them, though
> that is not a rocket science since we have access to git information
> there.
>
> The Jenkins CI role here is to ensure that we have master build right
> and releases build right, on the various OSes.
>
> Sounds simpler and better for me, how it does for you?
>
> --
> ,,,^..^,,,
>
>
> On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson
>  wrote:
> > I'd like us to instead consider flattening to a single repository. I've
> found no value and only pain from the multiple repositories approach (43 in
> total!).
> >
> > The contention is that multiple repositories enforces application
> boundaries (low coupling / high cohesion) but I've not felt that in
> reality. We don't, and couldn't meaningfully, release any of our components
> separately, and, as Ilya makes clear, many enhancements require changes to
> multiple repositories, and we break this into multiple commits, losing the
> ability to look at an enhancement in toto.
> >
> > If what Ilya is proposing is the solution, I think it's the solution to
> a problem we should not have.
> >
> > B.
> >
> >> On 12 Apr 2016, at 16:22, Ilya Khlopotov  wrote:
> >>
> >>
> >>
> >> Dear community,
> >>
> >>
> >> There is a problem with contributors workflow which renders our CI
> system
> >> useless. As you might know couchdb project consists of multiple
> >> repositories. Most of the time changes cross the repositories
> boundaries.
> >> When this happens the push to any of the repositories causes CI
> failures.
> >> CI fails since it uses the old version of dependencies from main
> repository
> >> of the project. Here is what we can do about it.
> >>
> >> # Proposal
> >>
> >> Let's use multiple files for dependency management.
> >>
> >> - deps.json - serves the same purpose as dependencies list from current
> >> rebar.config.script
> >> - proposed.deps.json - here we specify list of PRs we want to commit
> >> atomically
> >> - override.deps.json - local file outside of version control which we
> >> consult in order to include development tools specific to contributor
> (code
> >> reloader, debugger, tracer, profiler, binpp, ...)
> >>
> >> Bellow is the example of a content of these files:
> >>
> >> ## deps.json
> >> {
> >>"src/b64url": [
> >>"https://github.com/apache/couchdb-b64url;,
> >>"6895652d80f95cdf04efb14625abed868998f174"
> >>],
> >>"src/cassim": [
> >>"https://github.com/apache/couchdb-cassim;,
> >>"9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
> >>],
> >>"src/chttpd": [
> >>"https://github.com/apache/couchdb-chttpd;,
> >>"54e8f6147486d9afc5245e0143d15a4dd1185654"
> >>],
> >>"src/meck": [
> >>"https://github.com/apache/couchdb-meck;,
> >>"tree/0.8.2"
> >>],
> >> 
> >> }
> >>
> >> ## proposed.deps.json
> >> {
> >>"src/couch": "https://github.com/apache/couchdb-couch/pull/124;,
> >>"src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108;
> >>"src/couch_tests": [
> >>"https://github.com/apache/couchdb-erlang-tests;,
> >>"tree/branch"
> >>],
> >> }
> >>
> >> # Interface
> >>
> >> I propose to write a simple CLI tool to work with 

Re: On dependency management and CI issues associated with it

[Alexander Shorin]

Hi Robert,

Point about flattening to a single repository is valid: in the end, we
have our apps repos in broken state all the time as they are not
declare their decencies. So noone can pick fabric@master and run it -
he'll spend quite a lot of time to figure the deps of the right
versions. But the idea to solve the problem by reducing set of
repositories we have to test is good.


Hi Iliya,

I have alternative solution for you:

- Turn-off Travis CI everywhere where we cannot be sure about testing
without depended PRs (all except third-party modules, fauxton, docs,
and few more independent projects like couch-epi);
- Require everyone to submit additional PR to apache/couchdb repo with
commit hashes update;
- On this apache/couchdb PR we'll run CI testing;
- If you rebase/update any of your subcomponent PRs you must update
commit hash on apache/couchdb one;

Pros:
- We won't forget to update rebar.config when new changes lands;
- We will always run complete integration testing with all the right
deps states;
- We won't have to invent any complicated integration solutions to
deal with sub-repos testing;
- No new new steps/files/work introduced, so there is no need to care
about learning curve;

Cons:
- Need to be a bit tricky on Travis builder to realize on which remote
(fork) new rebar.config hashes are to correctly checkout them, though
that is not a rocket science since we have access to git information
there.

The Jenkins CI role here is to ensure that we have master build right
and releases build right, on the various OSes.

Sounds simpler and better for me, how it does for you?

--
,,,^..^,,,


On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson
 wrote:
> I'd like us to instead consider flattening to a single repository. I've found 
> no value and only pain from the multiple repositories approach (43 in total!).
>
> The contention is that multiple repositories enforces application boundaries 
> (low coupling / high cohesion) but I've not felt that in reality. We don't, 
> and couldn't meaningfully, release any of our components separately, and, as 
> Ilya makes clear, many enhancements require changes to multiple repositories, 
> and we break this into multiple commits, losing the ability to look at an 
> enhancement in toto.
>
> If what Ilya is proposing is the solution, I think it's the solution to a 
> problem we should not have.
>
> B.
>
>> On 12 Apr 2016, at 16:22, Ilya Khlopotov  wrote:
>>
>>
>>
>> Dear community,
>>
>>
>> There is a problem with contributors workflow which renders our CI system
>> useless. As you might know couchdb project consists of multiple
>> repositories. Most of the time changes cross the repositories boundaries.
>> When this happens the push to any of the repositories causes CI failures.
>> CI fails since it uses the old version of dependencies from main repository
>> of the project. Here is what we can do about it.
>>
>> # Proposal
>>
>> Let's use multiple files for dependency management.
>>
>> - deps.json - serves the same purpose as dependencies list from current
>> rebar.config.script
>> - proposed.deps.json - here we specify list of PRs we want to commit
>> atomically
>> - override.deps.json - local file outside of version control which we
>> consult in order to include development tools specific to contributor (code
>> reloader, debugger, tracer, profiler, binpp, ...)
>>
>> Bellow is the example of a content of these files:
>>
>> ## deps.json
>> {
>>"src/b64url": [
>>"https://github.com/apache/couchdb-b64url;,
>>"6895652d80f95cdf04efb14625abed868998f174"
>>],
>>"src/cassim": [
>>"https://github.com/apache/couchdb-cassim;,
>>"9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
>>],
>>"src/chttpd": [
>>"https://github.com/apache/couchdb-chttpd;,
>>"54e8f6147486d9afc5245e0143d15a4dd1185654"
>>],
>>"src/meck": [
>>"https://github.com/apache/couchdb-meck;,
>>"tree/0.8.2"
>>],
>> 
>> }
>>
>> ## proposed.deps.json
>> {
>>"src/couch": "https://github.com/apache/couchdb-couch/pull/124;,
>>"src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108;
>>"src/couch_tests": [
>>"https://github.com/apache/couchdb-erlang-tests;,
>>"tree/branch"
>>],
>> }
>>
>> # Interface
>>
>> I propose to write a simple CLI tool to work with this structure. Bellow is
>> a list of commands which we need to support (for minimal version)
>>
>> ## Adding new dependency
>>
>> git propose add https://github.com/apache/couchdb-foo
>> a2d5ad2eedc960248b806f61df0a1009462bdb46
>> git propose add https://github.com/apache/couchdb-bar tree/branch_name
>>
>> ## Adding new PR to the change set
>>
>> git propose add https://github.com/apache/couchdb-config/pull/4
>>
>> ## Checking out right dependencies
>>
>> git propose checkout
>>
>> ## Checking out release
>>
>> git propose checkout --release # this would ignore proposed.deps.json if it
>>