JamesMutton </JamesM>
On 1/17/14 11:09 AM, "Noah Slater" <nsla...@apache.org> wrote: >Good plan. What's your wiki username? I'll add you. > >On 17 January 2014 20:01, Mutton, James <jmut...@akamai.com> wrote: >> Might I suggest that this sounds like good info to document on the wiki >> for committers getting started. I'd add it but I'm not in the >>allow-list. >> :) >> >> </JamesM> >> >> >> >> >> On 1/17/14 4:00 AM, "Garren Smith" <gar...@apache.org> wrote: >> >>>I'm claiming 2nd person added! >>> >>>On 17 Jan 2014, at 1:28 PM, Noah Slater <nsla...@apache.org> wrote: >>> >>>> Psst. A little birdy tells me that if you ask nicely, the infra folks >>>> will add you to the Apache GitHub org too, so you can show off your >>>> Apache affiliation. I was the first person added. Because I may have >>>> been the first to ask. ;) >>>> >>>> On 17 January 2014 11:56, Noah Slater <nsla...@apache.org> wrote: >>>>> Awesome, thanks Paul. >>>>> >>>>> Note to all devs: if you want your contributions to CouchDB to show >>>>>up >>>>> on your GitHub profile, you have to star each of the repositories. >>>>> (That's just how GitHub mechanics work for repo mirrors.) >>>>> >>>>> You can find them all here: >>>>> >>>>> https://github.com/apache >>>>> >>>>> On 17 January 2014 00:00, Paul Davis <paul.joseph.da...@gmail.com> >>>>>wrote: >>>>>> New repos are up: https://git-wip-us.apache.org/repos/asf?s=couchdb >>>>>> >>>>>> I'm gonna go through and initialize them with history from master or >>>>>> one of the bigcouch and rcouch branches as appropriate. >>>>>> >>>>>> On Thu, Jan 16, 2014 at 2:12 PM, Paul Davis >>>>>><paul.joseph.da...@gmail.com> wrote: >>>>>>> Infrastructure ticket opened: >>>>>>>https://issues.apache.org/jira/browse/INFRA-7203 >>>>>>> >>>>>>> On Thu, Jan 16, 2014 at 1:42 PM, Jan Lehnardt <j...@apache.org> >>>>>>>wrote: >>>>>>>> >>>>>>>> On 16 Jan 2014, at 20:42 , Paul Davis >>>>>>>><paul.joseph.da...@gmail.com> >>>>>>>>wrote: >>>>>>>> >>>>>>>>> It doesn't appear that this is objectionable to anyone. Does >>>>>>>>>anyone >>>>>>>>> have an objection to us having infra/me create these repos to use >>>>>>>>>for >>>>>>>>> the bigcouch/rcouch merge work? This won't affect master or >>>>>>>>>releases >>>>>>>>> until those merges finish. >>>>>>>> >>>>>>>> no objections. >>>>>>>> >>>>>>>> Jan >>>>>>>> -- >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jan 14, 2014 at 11:02 PM, Paul J Davis >>>>>>>>> <paul.joseph.da...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Jan 14, 2014, at 8:37 PM, Benoit Chesneau >>>>>>>>>>><bchesn...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> On Wed, Jan 15, 2014 at 12:22 AM, Paul Davis >>>>>>>>>>><paul.joseph.da...@gmail.com>wrote: >>>>>>>>>>> >>>>>>>>>>>> I've recently been having discussions about how to handle the >>>>>>>>>>>> repository configuration for various bits of CouchDB >>>>>>>>>>>>post-merge. The >>>>>>>>>>>> work that Benoit has been doing on the rcouch merge branch >>>>>>>>>>>>have >>>>>>>>>>>>also >>>>>>>>>>>> touched on this topic as well. >>>>>>>>>>>> >>>>>>>>>>>> The background for those unfamiliar is that the standard >>>>>>>>>>>>operating >>>>>>>>>>>> procedure for Erlang is to have a single Erlang application >>>>>>>>>>>>per >>>>>>>>>>>> repository and then rely on rebar to fetch each dependency. >>>>>>>>>>>> Traditionally in CouchDB land we've always just included the >>>>>>>>>>>>source to >>>>>>>>>>>> all applications in a single monolithic repository and >>>>>>>>>>>>periodically >>>>>>>>>>>> reimport changes from upstream dependencies. >>>>>>>>>>>> >>>>>>>>>>>> Recently rcouch changed from the monolithic repository to use >>>>>>>>>>>>external >>>>>>>>>>>> repositories for some dependencies. Originally the BigCouch >>>>>>>>>>>>used an >>>>>>>>>>>> even more federated scheme that had each Erlang application in >>>>>>>>>>>>an >>>>>>>>>>>> external repository (and the core couch Erlang application was >>>>>>>>>>>>in the >>>>>>>>>>>> root repository). When Bob Newson and I did the initial >>>>>>>>>>>>hacking >>>>>>>>>>>>on the >>>>>>>>>>>> BigCouch merge we pulled those external dependencies into the >>>>>>>>>>>>root >>>>>>>>>>>> repository reverting back to the large monolithic approach. >>>>>>>>>>>> >>>>>>>>>>>> After trying to deal with the merge and contemplating how >>>>>>>>>>>>various >>>>>>>>>>>> Erlang release things might work it's become fairly apparent >>>>>>>>>>>>that the >>>>>>>>>>>> monolithic approach is a bit constrictive. For instance, part >>>>>>>>>>>>of >>>>>>>>>>>> rebar's versioning abilities lets you tag repositories to >>>>>>>>>>>>generate >>>>>>>>>>>> versions rather than manually updating versions in source >>>>>>>>>>>>files. >>>>>>>>>>>> Another thing I've found on other projects is that having each >>>>>>>>>>>> application in a separate repository requires developers to >>>>>>>>>>>>think a >>>>>>>>>>>> bit more detailed about the public internal interfaces used >>>>>>>>>>>>through >>>>>>>>>>>> out the system. We've done some work to this extent already >>>>>>>>>>>>with >>>>>>>>>>>> separating source directories but forcing commits to multiple >>>>>>>>>>>> repositories shoots up a big red flag that maybe there's a >>>>>>>>>>>>high >>>>>>>>>>>>level >>>>>>>>>>>> of coupling between two bits of code. >>>>>>>>>>>> >>>>>>>>>>>> Other benefits of having the multiple repository setup is that >>>>>>>>>>>>its >>>>>>>>>>>> possible that this lends itself to being integrated with the >>>>>>>>>>>>proposed >>>>>>>>>>>> plugin system. It'd be fairly trivial to have a script that >>>>>>>>>>>>went and >>>>>>>>>>>> fetched plugins that aren't developed at Apache (as a >>>>>>>>>>>>./configure time >>>>>>>>>>>> switch type of thing). Having a system like this would also >>>>>>>>>>>>allow us >>>>>>>>>>>> to have groups focused on particular bits of development not >>>>>>>>>>>>have to >>>>>>>>>>>> concern themselves with the unrelated parts of the system. >>>>>>>>>>>> >>>>>>>>>>>> Given all that, I'd like to propose that we move to having a >>>>>>>>>>>> repository for each application/dependency that we use to >>>>>>>>>>>>build >>>>>>>>>>>> CouchDB. Each repository would be hosted on ASF infra and >>>>>>>>>>>>mirrored to >>>>>>>>>>>> GitHub as expected. This means that we could have the root >>>>>>>>>>>>repository >>>>>>>>>>>> be a simple repo that contains packaging/release/build stuff >>>>>>>>>>>>that >>>>>>>>>>>> would enable lots of the ideas offered on configurable types >>>>>>>>>>>>of >>>>>>>>>>>> release generation. I've included an initial list of >>>>>>>>>>>>repositories at >>>>>>>>>>>> the end of this email. Its basically just the apps that have >>>>>>>>>>>>been >>>>>>>>>>>> split out in either rcouch or bigcouch plus a few other bits >>>>>>>>>>>>from >>>>>>>>>>>> CouchDB master. >>>>>>>>>>>> >>>>>>>>>>>> I would also point out that even though our main repo would >>>>>>>>>>>>need to >>>>>>>>>>>> fetch other dependencies from the internet to build the final >>>>>>>>>>>>output, >>>>>>>>>>>> we fully intend that our release tarballs would *not* have >>>>>>>>>>>>this >>>>>>>>>>>> requirement. Ie, when we go to cut a release part of the >>>>>>>>>>>>process the >>>>>>>>>>>> RM would run would be to pull all of those dependencies before >>>>>>>>>>>> creating a tarball that would be wholly self contained. Given >>>>>>>>>>>>an >>>>>>>>>>>> apache-couchdb-x.y.z.tar.gz release file, there won't be a >>>>>>>>>>>>requirement >>>>>>>>>>>> to have access to the ASF git repos. >>>>>>>>>>>> >>>>>>>>>>>> I'm not entirely sure how controversial this is for anyone. >>>>>>>>>>>>For >>>>>>>>>>>>the >>>>>>>>>>>> most part the reactions I remember hearing were more concerned >>>>>>>>>>>>on >>>>>>>>>>>> whether the infrastructure team would allow us to use this >>>>>>>>>>>>sort >>>>>>>>>>>>of >>>>>>>>>>>> configuration. I looked yesterday and asked and apparently its >>>>>>>>>>>> something we can request but as always we'll want to verify >>>>>>>>>>>>again if >>>>>>>>>>>> we have consensus to move in this direction. >>>>>>>>>>>> >>>>>>>>>>>> Anyone have comments or flames? Right now I'm just interested >>>>>>>>>>>>in >>>>>>>>>>>> feeling out what sort of (lack of?) consensus there is on >>>>>>>>>>>>such a >>>>>>>>>>>> change. If there's general consensus I'd think we'd do a vote >>>>>>>>>>>>in a >>>>>>>>>>>> couple weeks and if that passes then start on down this road >>>>>>>>>>>>for the >>>>>>>>>>>> two merge projects and then it would become part of master >>>>>>>>>>>>once >>>>>>>>>>>>those >>>>>>>>>>>> land (as opposed to doing this to master and then attempting >>>>>>>>>>>>to >>>>>>>>>>>>merge >>>>>>>>>>>> rcouch/bigcouch onto that somehow). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This is a quick pass at listing what extra repositories I'd >>>>>>>>>>>>have >>>>>>>>>>>> created. Some of these applications only exist in the bigcouch >>>>>>>>>>>>and/or >>>>>>>>>>>> rcouch branches so that's where the unfamiliar application >>>>>>>>>>>>names are >>>>>>>>>>>> from. I'd also point out that the documentation and fauxton >>>>>>>>>>>>things are >>>>>>>>>>>> just on a whim in that we could decouple that development from >>>>>>>>>>>>the >>>>>>>>>>>> erlang development. I can see arguments for an against those. >>>>>>>>>>>>I'm much >>>>>>>>>>>> less concerned on that aspect than the Erlang parts that are >>>>>>>>>>>>directly >>>>>>>>>>>> affected by rebar/Erlang conventions. >>>>>>>>>>>> >>>>>>>>>>>> chttpd >>>>>>>>>>>> config >>>>>>>>>>>> couch >>>>>>>>>>>> couch_collate >>>>>>>>>>>> couch_dbupdates >>>>>>>>>>>> couch_httpd >>>>>>>>>>>> couch_index >>>>>>>>>>>> couch_mrview >>>>>>>>>>>> couch_plugins >>>>>>>>>>>> couch_replicator >>>>>>>>>>>> documentation >>>>>>>>>>>> ddoc_cache >>>>>>>>>>>> ets_lru >>>>>>>>>>>> fabric >>>>>>>>>>>> fauxton >>>>>>>>>>>> ibrowse >>>>>>>>>>>> jiffy >>>>>>>>>>>> mem3 >>>>>>>>>>>> mochiweb >>>>>>>>>>>> oauth >>>>>>>>>>>> rebar >>>>>>>>>>>> rexi >>>>>>>>>>>> snappy >>>>>>>>>>>> twig >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I also contemplated this and and I am generally +1 on this. And >>>>>>>>>>>definitely >>>>>>>>>>> +1 to mirror them on the apache git if possible. I have a >>>>>>>>>>>couple of >>>>>>>>>>> comments though. >>>>>>>>>>> >>>>>>>>>>> Initially I also had everything separated in its own source >>>>>>>>>>>repository. 1 >>>>>>>>>>> year ago I merged back as one core repo the couchdb erlang >>>>>>>>>>>applications and >>>>>>>>>>> put all the dependencies in the refuge repository or in the >>>>>>>>>>>refuge CDN for >>>>>>>>>>> the spidermonkey and ICU sources. >>>>>>>>>>> >>>>>>>>>>> I merged back as one core repo the couchdb erlang applications >>>>>>>>>>>because they >>>>>>>>>>> were a little too much dependant. Especially couch_httpd, >>>>>>>>>>>couch_index and >>>>>>>>>>> couch_mrview. These applications are not yet enough by >>>>>>>>>>>themselves. >>>>>>>>>>> >>>>>>>>>>> Imo if we split everything in their own apps, then we should >>>>>>>>>>>make sure >>>>>>>>>>> that couch_httpd can be used without couch_index and >>>>>>>>>>>couch_mrview (which >>>>>>>>>>> means that "all_docs" is available in couch_httpd). Also we >>>>>>>>>>>should be able >>>>>>>>>>> to just launch couch without any of the above. And probably >>>>>>>>>>>without the >>>>>>>>>>> need of an ini. The couch_query_server module thing is an >>>>>>>>>>>interesting case. >>>>>>>>>>> bigcouch is also introducing `ddoc_cache` which I am not sure >>>>>>>>>>>why it is >>>>>>>>>>> provided as a standalone app. Does it means it can be replaced >>>>>>>>>>>by another >>>>>>>>>>> application eventually? Why not having it simply in the couch >>>>>>>>>>>application? >>>>>>>>>>> Does it needs to be updated separately? >>>>>>>>>>> >>>>>>>>>>> Also all our base applications should also be named spaced >>>>>>>>>>>correctly so >>>>>>>>>>> they will be strictly identified as erlang modules: "config" >>>>>>>>>>>is >>>>>>>>>>>too >>>>>>>>>>> generic, "ddoc_cache" too. Others are probably OK. >>>>>>>>>>> >>>>>>>>>>> There are probably other things that we could provide as apps: >>>>>>>>>>> >>>>>>>>>>> - couch_daemon, >>>>>>>>>>> - couch_js >>>>>>>>>>> - couch_external >>>>>>>>>>> - couch_stats >>>>>>>>>>> - couch_compaction_daemon >>>>>>>>>>> - couch_httpd_proxy >>>>>>>>>>> >>>>>>>>>>> Anyway again i'm +1 for this move, I really think it's a good >>>>>>>>>>>idea. >>>>>>>>>>> >>>>>>>>>>> - benoit >>>>>>>>>> >>>>>>>>>> I agree on most of this. Roughly I see three general points. >>>>>>>>>> >>>>>>>>>> First, deciding on whether some things are external deps is >>>>>>>>>>definitely up for discussion. Whether couch_mrview is a different >>>>>>>>>>app/repo is not necessarily clear cut. Personally I think I over >>>>>>>>>>engineered couch_index which blurs the lines a bit. If I could >>>>>>>>>>wave a wand I'd have just couch_mrview and it'd be separate. More >>>>>>>>>>importantly I think the separate repos makes these things more >>>>>>>>>>apparent. The fact were discussing this sort of architecture >>>>>>>>>>thing >>>>>>>>>>is suggestive that it's forcing us to think a bit harder. >>>>>>>>>> >>>>>>>>>> Second is the aspect of composability. For instance the mrview >>>>>>>>>>thing to me is obviously a different repo precisely so a user >>>>>>>>>>could import couch (_core?) directly without requiring the spider >>>>>>>>>>monkey dependency. The monolithic repo doesn't allow this without >>>>>>>>>>some very non-standard tooling. >>>>>>>>>> >>>>>>>>>> Thirdly, app naming is always a contention. The config name was >>>>>>>>>>actually a hot code upgrade concern. We couldn't reuse >>>>>>>>>>couch_config directly at the time. And Adam was also hopeful we >>>>>>>>>>could the it into a useful non-specific config app. >>>>>>>>>> >>>>>>>>>> Fourthly, and related to secondly, we'll also want to look at >>>>>>>>>>splitting other apps out as necessary. The ones you listed I >>>>>>>>>>think >>>>>>>>>>aren't controversial it's just that no one has done it yet. My >>>>>>>>>>list was purely what existed so far without attempting to carve >>>>>>>>>>things up more. I definitely agree we should carve more in just >>>>>>>>>>wanted to cover consensus that carving is the right direction. >>>>>>>>>> >>>>>>>>>> Fifthly, I'm done typing on my phone. I'll fill in more thoughts >>>>>>>>>>tomorrow. >>>>>>>>>> >>>>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Noah Slater >>>>> https://twitter.com/nslater >>>> >>>> >>>> >>>> -- >>>> Noah Slater >>>> https://twitter.com/nslater >>> >> > > > >-- >Noah Slater >https://twitter.com/nslater
