Re: why etc/default.d etc/local.d

[Paul J Davis]

I'd rather leave them or remove the functionality. Hiding the config chain 
seems wrong. 

On Nov 12, 2012, at 5:37 PM, Randall Leeds randall.le...@gmail.com wrote:

 +0 on removing them.
 
 
 On Mon, Nov 12, 2012 at 10:10 AM, Jan Lehnardt j...@apache.org wrote:
 
 
 On Nov 10, 2012, at 12:12 , Benoit Chesneau bchesn...@gmail.com wrote:
 
 I wonder why we handle this config files in the couchdb build. It's
 more the responsability of the Linux, BSD distributions to maintain
 them.
 
 We actually just create the directories, for users to put files into,
 we don’t use them ourselves.
 
 Cheers
 Jan
 --
 
 

Re: All The Numbers -- View Engine Performance Benchmarks

[Paul J. Davis]

I tried this a long time ago with the binary to JS code in C. It didn't make 
for a huge improvement in total speed.



On Jan 27, 2013, at 11:37 PM, Jason Smith j...@iriscouch.com wrote:

 Hey, Jan. This is a totally random and hypothetical idea:
 
 Do you think there would be any speedup to use term_to_binary() and
 binary_to_term() instead of encoding through JSON? The view server would of
 course need to support that codec. I have already implemented encoding in
 erlang.js: https://github.com/iriscouch/erlang.js
 
 My suspicion is that there would be minor or zero speedup. The Erlang side
 would get faster (term_to_binary is faster) but the JS side would get
 slower (custom decoding rather than JSON.parse()). The JS VM is slightly
 faster so the total change would reflect that.
 
 But I thought I'd share the idea.
 
 On Sun, Jan 27, 2013 at 12:50 PM, Jan Lehnardt j...@apache.org wrote:
 
 
 On Jan 27, 2013, at 13:22 , Alexander Shorin kxe...@gmail.com wrote:
 
 On Sun, Jan 27, 2013 at 3:55 PM, Jason Smith j...@iriscouch.com wrote:
 
 * Very little difference in different implementations (because stdio is
 the
 bottleneck)
 
 Why stdio is a bottleneck? I'm interesting underlay reasons.
 
 It is actually not the the stdio, but the serialisation form erlang-terms
 to JSON to JS Objects to JSON to erlang terms.
 
 Cheers
 Jan
 --
 
 
 
 As for my experience, the protocol design doesn't allows view and
 query servers works faster as they can. For example, we have 50 ddocs
 with validate functions. For each document save there would be
 executed from 100 commands (50 resets + 50 ddoc validate_doc_update
 calls) till 150 commands (+ddocs caches), while it's possible to
 process them in bulk mode.
 
 --
 ,,,^..^,,,
 
 
 
 
 -- 
 Iris Couch

Re: SpiderMonkey 1.8.5 upgrades

[Paul J. Davis]

Adam,

I'm with Bob on this one. 1.1 and 1.0 are forked so they stay the same. This 
should only concern trunk.

Also, just rewriting couchjs and letting configure choose for 1.2 might be 
doable. 

On Apr 2, 2011, at 6:09 AM, Robert Newson robert.new...@gmail.com wrote:

 +1 for supporting 1.8.5 exclusively from trunk (i.e, 1.2) upwards.
 Leave couchjs as is on 1.0.x and 1.1.x
 
 B.
 
 
 On 2 April 2011 02:59, Adam Kocoloski kocol...@apache.org wrote:
 On Apr 1, 2011, at 7:26 PM, Paul Davis wrote:
 
 Hey,
 
 Mozilla released a SM 1.8.5 source distribution this morning [1].
 We've been getting requests from various places to upgrade our couchjs
 to use this newer version for a couple weeks and now that its
 available, there's no better time to act.
 
 As can be expected, this new SpiderMonkey has a fairly significant API
 change from what we've been using in couchjs. Up until now we've been
 able to get away with supporting 1.7 and the 1.8.0rc1 tarballs without
 much hassle. The new API makes this much more difficult. Chris C
 Coulson from Ubuntu has been working on a patch that'll allow us to
 work with 1.8.5 and (IIRC) should work with the 1.8.0rc1 but it
 includes some extra gnarly ifdef magic to make things work.
 
 So my question is what versions of SM should we support? I would
 probably vote to drop everything in favor of 1.8.5 and no longer
 support the older APIs. There is a possibility of just having two
 versions of couchjs that we choose at compile time. But from what I've
 heard and seen, we're basically not going to be able to have a single
 compile time ifdef decision on versions without some super screw code.
 
 Thoughts?
 
 [1] http://ftp.mozilla.org/pub/mozilla.org/js/
 
 Well, we need to continue to support SM 1.7 on the 1.0.x branch, and if that 
 means 1.0.x doesn't work with SM 1.8.5 then so be it.  For 1.1.0 I'd want to 
 know what the availability of SM 1.8.5 in various package repositories looks 
 like before dropping support for SM 1.7. Ideally I'd like to ship at least 
 one version of CouchDB that works with both SM 1.7 and SM 1.8.5, but I've 
 seen Chris' work in COUCHDB-1078 and I don't relish the thought of making 
 that any more complicated than it already is.
 
 Trunk can drop support for SM 1.7.
 
 Adam
 
 

Re: Broken build

[Paul J. Davis]

Cool. I'll ignore it for now then. 



On Apr 9, 2011, at 7:13 AM, Jan Lehnardt j...@apache.org wrote:

 My nightly builds got tripped up on the make distcheck breakage, but seem to 
 work after that got fixed.
 
 
 On 9 Apr 2011, at 02:25, Paul Davis wrote:
 
 Apparently the initial ejson commit has been building on our build-bot
 instance for about four days. The commit info can be found at [1].
 Bottom line is that something hit an infinite loop spewing:
 
  sed: couldn't flush stdout: Broken pipe
 
 Don't bother trying to read the stdio log, its 44M of that message repeated.
 
 Has anyone noticed this locally? Is this just a transient thing that
 was caused by our mucking around in the build system?
 
 
 [1] http://ci.apache.org/builders/couchdb-coverage/builds/497
 

Re: Helping out with releases

[Paul J. Davis]

1.0.3 is on my list for today

On May 15, 2011, at 11:50 AM, till t...@php.net wrote:

 Just wanted to ask what the status of either release is?
 
 Are the two tests failing still failing, or can we move on to make dist? :-)
 
 Till
 
 On Thu, May 12, 2011 at 5:15 PM, Benoit Chesneau bchesn...@gmail.com wrote:
 On Thu, May 12, 2011 at 4:31 AM, Randall Leeds randall.le...@gmail.com 
 wrote:
 On Tue, May 10, 2011 at 06:54, Paul Davis paul.joseph.da...@gmail.com 
 wrote:
 On Tue, May 10, 2011 at 9:49 AM, Jan Lehnardt j...@apache.org wrote:
 Putting 140 on an infinite loop (while(true); do ./test/etap/run 
 test/etap/140-attachment-comp.t ; done) eventually gives me 
 http://www.friendpaste.com/2JCA8dvre9orA3UysYYTCe on both SSD Mac OS X 
 10.6 and Ubuntu 10.04 on disk in a VMWare.
 
 
 That's the error that I was getting.
 
 I also got one instance of 
 http://www.friendpaste.com/7EpB1eJGRoRFBvjXvmFG3V on the Ubuntu.
 
 
 Never seen this one which makes things more fun.
 
 The first one comes up reliably after half a minute or so and repeats 
 itself, it may well be a socket ulimit or something. The second one I 
 only ever saw once so far.
 
 Cheers
 Jan
 --
 
 Just distchecked 1.0.x and 1.1.x no problem.
 Give it a shot, Paul?
 
 the same, osx (with  without ssd)
 
 - benoît
 

Re: svn commit: r1125654 - in /couchdb/trunk: AUTHORS THANKS

[Paul J. Davis]

Woot!

On May 21, 2011, at 5:47 AM, rand...@apache.org wrote:

 Author: randall
 Date: Sat May 21 09:47:32 2011
 New Revision: 1125654
 
 URL: http://svn.apache.org/viewvc?rev=1125654view=rev
 Log:
 moving myself from THANKS to AUTHORS
 
 Modified:
couchdb/trunk/AUTHORS
couchdb/trunk/THANKS
 
 Modified: couchdb/trunk/AUTHORS
 URL: 
 http://svn.apache.org/viewvc/couchdb/trunk/AUTHORS?rev=1125654r1=1125653r2=1125654view=diff
 ==
 --- couchdb/trunk/AUTHORS (original)
 +++ couchdb/trunk/AUTHORS Sat May 21 09:47:32 2011
 @@ -16,5 +16,6 @@ documentation or developing software. So
  * Benoît Chesneau beno...@apache.org
  * Filipe Manana fdman...@apache.org
  * Robert Newson rnew...@apache.org
 + * Randall Leeds rand...@apache.org
 
 For a list of other credits see the `THANKS` file.
 
 Modified: couchdb/trunk/THANKS
 URL: 
 http://svn.apache.org/viewvc/couchdb/trunk/THANKS?rev=1125654r1=1125653r2=1125654view=diff
 ==
 --- couchdb/trunk/THANKS (original)
 +++ couchdb/trunk/THANKS Sat May 21 09:47:32 2011
 @@ -48,7 +48,6 @@ suggesting improvements or submitting ch
  * Joel Clark unsigned_c...@yahoo.com
  * Matt Lyon m...@flowerpowered.com
  * mikeal mikeal.rog...@gmail.com
 - * Randall Leeds randall.le...@gmail.com
  * Joscha Feth jos...@feth.com
  * Jarrod Roberson jar...@vertigrated.com
  * Jae Kwon jkwon.w...@gmail.com
 
 

Re: [VOTE] Apache CouchDB 1.1.0 release, round 3.

[Paul J. Davis]

+1 here. Motions pass on latest OS X. 

On May 30, 2011, at 8:09 PM, Randall Leeds randall.le...@gmail.com wrote:

 * Signature OK
 * MD5 OK
 * SHA1 OK
 * Make check OK
 * Futon OK
 
 +1
 
 On Mon, May 30, 2011 at 15:25, Robert Newson rnew...@apache.org wrote:
 Hello,
 
 I would like call a vote for the Apache CouchDB 1.1.0 release, round 3.
 
 Two further issues have been resolved since round 2;
 
 1) Compatibility with erlang R14B03.
 2) Release tarball now works on Windows (with Cygwin).
 
 We encourage the whole community to download and test these release 
 artifacts so
 that any critical issues can be resolved before the release is made. 
 Everyone is
 free to vote on this release, so get stuck in!
 
 We are voting on the following release artifacts:
 
  http://people.apache.org/~rnewson/dist/1.1.0/
 
 These artifacts have been built from the 1.1.0 tag in Subversion:
 
  http://svn.apache.org/repos/asf/couchdb/tags/1.1.0
 
 Please follow our test procedure:
 
 http://wiki.apache.org/couchdb/Test_procedure
 
 Happy voting,
 
 B.
 

Starting the Git Experiment

[Paul J. Davis]

Dear committers, 

We now have a green light from infrastructure to switch to using Git as our 
writable VCS. This is to be considered a live experiment. If something breaks 
its possible we'll have to revert back to SVN. But nothing will break and 
everyone will forgive me for any bugs that may crop up.

If there are no objections I would like to switch over soonish. Normally I 
would say Monday to give people a chance to respond to this email but we've had 
quite a few discussions on switching to Git already and no one has voiced 
opposition. Seeing as that's the case if I get a majority of +1's from the 
committers I'll start disabling SVN access as soon as I see the majority vote.

Paul Davis 

Re: Starting the Git Experiment

[Paul J. Davis]

Oops, forgot that status update.

I reran the scripts for the mirrors on git.apache.org using the same version of 
git-svn I used to generate the last writable git repo. The result was that it 
calculated the same hashes as what we had so I feel comfortable blaming this on 
the different git-svn versions.

Bottom line, any project that already has a mirror on git.apache.org will just 
use that as the basis for the new writable repo so that hashes will are 
preserved. 


On Friday, September 23, 2011 at 12:57 PM, Adam Kocoloski wrote:

 +1 assuming the writeable repo has the same hashes as the current read-only 
 mirror.
 
 Adam
 
 On Friday, September 23, 2011 at 1:52 PM, Paul J. Davis wrote:
 
  Dear committers, 
  
  We now have a green light from infrastructure to switch to using Git as our 
  writable VCS. This is to be considered a live experiment. If something 
  breaks its possible we'll have to revert back to SVN. But nothing will 
  break and everyone will forgive me for any bugs that may crop up.
  
  If there are no objections I would like to switch over soonish. Normally I 
  would say Monday to give people a chance to respond to this email but we've 
  had quite a few discussions on switching to Git already and no one has 
  voiced opposition. Seeing as that's the case if I get a majority of +1's 
  from the committers I'll start disabling SVN access as soon as I see the 
  majority vote.
  
  Paul Davis 

Re: Starting the Git Experiment

[Paul J. Davis]

SVN is now read only.

http://www.youtube.com/watch?v=T9uuPza41Uw 


On Friday, September 23, 2011 at 8:14 PM, Paul Davis wrote:

 
 That's enough of a consensus for me. I'll ask for SVN writes to be disabled 
 and bring up the new Git repo. I'll send emails when SVN goes down and Git 
 comes up. 
 
 
 On Sep 23, 2011, at 8:00 PM, Filipe David Manana fdman...@apache.org 
 (mailto:fdman...@apache.org) wrote:
 
  +1 as well Paul
  
  On Fri, Sep 23, 2011 at 4:18 PM, Robert Newson rnew...@apache.org 
  (mailto:rnew...@apache.org) wrote:
   +1
   
   On 23 September 2011 21:18, Randall Leeds randall.le...@gmail.com 
   (mailto:randall.le...@gmail.com) wrote:
+1

Thank you!

On Fri, Sep 23, 2011 at 13:52, Paul J. Davis 
paul.joseph.da...@gmail.com (mailto:paul.joseph.da...@gmail.com)wrote:

 Dear committers,
 
 We now have a green light from infrastructure to switch to using Git 
 as our
 writable VCS. This is to be considered a live experiment. If something
 breaks its possible we'll have to revert back to SVN. But nothing 
 will break
 and everyone will forgive me for any bugs that may crop up.
 
 If there are no objections I would like to switch over soonish. 
 Normally I
 would say Monday to give people a chance to respond to this email but 
 we've
 had quite a few discussions on switching to Git already and no one has
 voiced opposition. Seeing as that's the case if I get a majority of 
 +1's
 from the committers I'll start disabling SVN access as soon as I see 
 the
 majority vote.
 
 Paul Davis
  
  
  
  -- 
  Filipe David Manana,
  
  Reasonable men adapt themselves to the world.
   Unreasonable men adapt the world to themselves.
   That's why all progress depends on unreasonable men.

Re: Starting the Git Experiment

[Paul J. Davis]

The Git repo is up and running. Everyone should test it out and try and break 
it. Let me know if you find any issues. Thanks again for the patience from 
everyone.

Anonymous clones are up at:

http://git-wip-us.apache.org/repos/asf/couchdb.git

Committers need to clone from:

https://git-wip-us.apache.org/repos/asf/couchdb.git

General docs are up at:

http://git-wip-us.apache.org/

GitWeb is at:

http://git-wip-us.apache.org/repos/asf

And a commit:

http://git-wip-us.apache.org/repos/asf/couchdb/commit/f07c75fe



On Friday, September 23, 2011 at 10:12 PM, Paul J. Davis wrote:

 SVN is now read only.
 
 http://www.youtube.com/watch?v=T9uuPza41Uw 
 
 On Friday, September 23, 2011 at 8:14 PM, Paul Davis wrote:
 
  
  That's enough of a consensus for me. I'll ask for SVN writes to be disabled 
  and bring up the new Git repo. I'll send emails when SVN goes down and Git 
  comes up. 
  
  
  On Sep 23, 2011, at 8:00 PM, Filipe David Manana fdman...@apache.org 
  (mailto:fdman...@apache.org) wrote:
  
   +1 as well Paul
   
   On Fri, Sep 23, 2011 at 4:18 PM, Robert Newson rnew...@apache.org 
   (mailto:rnew...@apache.org) wrote:
+1

On 23 September 2011 21:18, Randall Leeds randall.le...@gmail.com 
(mailto:randall.le...@gmail.com) wrote:
 +1
 
 Thank you!
 
 On Fri, Sep 23, 2011 at 13:52, Paul J. Davis 
 paul.joseph.da...@gmail.com 
 (mailto:paul.joseph.da...@gmail.com)wrote:
 
  Dear committers,
  
  We now have a green light from infrastructure to switch to using 
  Git as our
  writable VCS. This is to be considered a live experiment. If 
  something
  breaks its possible we'll have to revert back to SVN. But nothing 
  will break
  and everyone will forgive me for any bugs that may crop up.
  
  If there are no objections I would like to switch over soonish. 
  Normally I
  would say Monday to give people a chance to respond to this email 
  but we've
  had quite a few discussions on switching to Git already and no one 
  has
  voiced opposition. Seeing as that's the case if I get a majority of 
  +1's
  from the committers I'll start disabling SVN access as soon as I 
  see the
  majority vote.
  
  Paul Davis
   
   
   
   -- 
   Filipe David Manana,
   
   Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
 

Re: Proposed couch hack in November or December

[Paul J Davis]

November 28th is Thanksgiving which is big family thing. As long as it's not 
that weekend I'm in. 

 On Sep 27, 2013, at 11:51 AM, Dave Cottlehuber d...@jsonified.com wrote:
 
 Hi everybody,
 
 Is there interest in having a hackathon again in late November or early 
 December?
 
 Ideally we'd focus this on something specific and big -- merging all the 
 forks -- and make the weekend a 4 day one.
 
 I can organise faciliities in Vienna -- which has the other advantage that I 
 can attend around family commitments. I think I used up my travel budget for 
 the year by now.
 
 A+
 Dave
 
 

Re: [DISCUSS] Multiple Repositories for Erlang Apps and Dependencies

[Paul J Davis]

 On Jan 14, 2014, at 8:47 PM, Benoit Chesneau bchesn...@gmail.com wrote:
 
 On Wed, Jan 15, 2014 at 12:22 AM, Paul Davis 
 paul.joseph.da...@gmail.comwrote:
 
 
 
 I would also point out that even though our main repo would need to
 fetch other dependencies from the internet to build the final output,
 we fully intend that our release tarballs would *not* have this
 requirement. Ie, when we go to cut a release part of the process the
 RM would run would be to pull all of those dependencies before
 creating a tarball that would be wholly self contained. Given an
 apache-couchdb-x.y.z.tar.gz release file, there won't be a requirement
 to have access to the ASF git repos.
 
 
 Just to be clear, the idea would be creating only  a release tarrball from
 all the git repositories or do you also think to create a full repository
 from all the dependencies? Something done by Cloudi for example which build
 the main repository [1] from the other repositories [2]. rcouch is only
 creating a tarball containing all erlang deps [3] .
 

I hadn't contemplated a synthetic repository until you mentioned it but my 
initial reaction is ick. Not sure I see a point but it's also not something I 
see as overly detrimental. 

Regardless of that I'd expect the release tarball would be able to be generated 
from just the federated repositories. 

 Also what about the npm modules that are fetched for futon and
 couchjs-node? Do you think we need to import them as well? At least maybe
 should we provide a mirror?
 
 - benoit
 

I don't know enough about the state of these modules to have an opinion. I've 
talked to a few people about the fauxton build in general but I don't have a 
clear picture on what's a build time vs configure time dependency as well as 
how fauxton configure/build steps map to the corresponding configure/build 
steps. 

For instance, if as part of a release we build a completed fauxton and include 
that as a static thing then id not pull the npm dependencies directly and 
instead consider them external dependencies. But it's also completely possible 
I have no idea what I'm talking about and that makes no sense. 

 
 [1] https://github.com/CloudI/CloudI
 [2] https://github.com/Cloudi and
 [3] https://github.com/refuge/rcouch/blob/master/Makefile#L120

Re: [DISCUSS] Multiple Repositories for Erlang Apps and Dependencies

[Paul J Davis]

 On Jan 14, 2014, at 8:37 PM, Benoit Chesneau bchesn...@gmail.com wrote:
 
 On Wed, Jan 15, 2014 at 12:22 AM, Paul Davis 
 paul.joseph.da...@gmail.comwrote:
 
 I've recently been having discussions about how to handle the
 repository configuration for various bits of CouchDB post-merge. The
 work that Benoit has been doing on the rcouch merge branch have also
 touched on this topic as well.
 
 The background for those unfamiliar is that the standard operating
 procedure for Erlang is to have a single Erlang application per
 repository and then rely on rebar to fetch each dependency.
 Traditionally in CouchDB land we've always just included the source to
 all applications in a single monolithic repository and periodically
 reimport changes from upstream dependencies.
 
 Recently rcouch changed from the monolithic repository to use external
 repositories for some dependencies. Originally the BigCouch used an
 even more federated scheme that had each Erlang application in an
 external repository (and the core couch Erlang application was in the
 root repository). When Bob Newson and I did the initial hacking on the
 BigCouch merge we pulled those external dependencies into the root
 repository reverting back to the large monolithic approach.
 
 After trying to deal with the merge and contemplating how various
 Erlang release things might work it's become fairly apparent that the
 monolithic approach is a bit constrictive. For instance, part of
 rebar's versioning abilities lets you tag repositories to generate
 versions rather than manually updating versions in source files.
 Another thing I've found on other projects is that having each
 application in a separate repository requires developers to think a
 bit more detailed about the public internal interfaces used through
 out the system. We've done some work to this extent already with
 separating source directories but forcing commits to multiple
 repositories shoots up a big red flag that maybe there's a high level
 of coupling between two bits of code.
 
 Other benefits of having the multiple repository setup is that its
 possible that this lends itself to being integrated with the proposed
 plugin system. It'd be fairly trivial to have a script that went and
 fetched plugins that aren't developed at Apache (as a ./configure time
 switch type of thing). Having a system like this would also allow us
 to have groups focused on particular bits of development not have to
 concern themselves with the unrelated parts of the system.
 
 Given all that, I'd like to propose that we move to having a
 repository for each application/dependency that we use to build
 CouchDB. Each repository would be hosted on ASF infra and mirrored to
 GitHub as expected. This means that we could have the root repository
 be a simple repo that contains packaging/release/build stuff that
 would enable lots of the ideas offered on configurable types of
 release generation. I've included an initial list of repositories at
 the end of this email. Its basically just the apps that have been
 split out in either rcouch or bigcouch plus a few other bits from
 CouchDB master.
 
 I would also point out that even though our main repo would need to
 fetch other dependencies from the internet to build the final output,
 we fully intend that our release tarballs would *not* have this
 requirement. Ie, when we go to cut a release part of the process the
 RM would run would be to pull all of those dependencies before
 creating a tarball that would be wholly self contained. Given an
 apache-couchdb-x.y.z.tar.gz release file, there won't be a requirement
 to have access to the ASF git repos.
 
 I'm not entirely sure how controversial this is for anyone. For the
 most part the reactions I remember hearing were more concerned on
 whether the infrastructure team would allow us to use this sort of
 configuration. I looked yesterday and asked and apparently its
 something we can request but as always we'll want to verify again if
 we have consensus to move in this direction.
 
 Anyone have comments or flames? Right now I'm just interested in
 feeling out what sort of (lack of?) consensus there is on such a
 change. If there's general consensus I'd think we'd do a vote in a
 couple weeks and if that passes then start on down this road for the
 two merge projects and then it would become part of master once those
 land (as opposed to doing this to master and then attempting to merge
 rcouch/bigcouch onto that somehow).
 
 
 This is a quick pass at listing what extra repositories I'd have
 created. Some of these applications only exist in the bigcouch and/or
 rcouch branches so that's where the unfamiliar application names are
 from. I'd also point out that the documentation and fauxton things are
 just on a whim in that we could decouple that development from the
 erlang development. I can see arguments for an against those. I'm much
 less concerned on that aspect than the Erlang parts that are directly
 

Re: silent view index file corruption

[Paul J Davis]

This corruption was quite odd in that there wasn't a conspicuous reason for it. 
 I didn't dive to deep into the whole thing so it's possible i missed something 
obvious. 

There are two things at play here.  How proactive should we be in provoking 
these errors and how much should we check for situations where our data file 
got trounced.

The extreme proactive position would be equivalent to a full table scan per 
write which is out of the question. So to some extent we won't be able to 
detect some errors until read time which is an unknowable interval. The other 
aspect is how rigorous should we check reads? This extreme would basically 
require a sha1 for every read or write no matter how small, not to mention the 
storage overhead. This part I'm not sure about. There's probably middle ground 
with crc sums and what not but i don't see a clear answer. 

Basically, the question is how much should we attempt to detect when hardware 
lies.  I reckon that there's probably a middle ground to report when an 
assumption is violated and full on table scans. Ideally such things would be 
fairly configurable but i sure don't see an obvious answer.


On Apr 6, 2010, at 10:06 PM, Randall Leeds randall.le...@gmail.com wrote:

 I immediately want to say 'ini file option' but I'm not sure whether to err
 on safety or speed.
 
 Maybe this is a good candidate for merkle trees or something else we can do
 throughout the view tree that might less overhead than md5 summing all the
 nodes? After all, most inner nodes shouldn't change most of the time. Some
 incremental, cheap checksum might be a worthwhile *option*.
 
 On Apr 6, 2010 6:04 PM, Adam Kocoloski kocol...@apache.org wrote:
 
 Hi all, we recently had an EC2 node go AWOL for about 12 hours.  When it
 came back, we noticed after a few days that a number of the view indexes
 stored on that node were not updating.  I did some digging into the error
 logs and with Paul's help pieced together what was going on.  I won't bother
 you with all the gory details unless you ask for them, but the gist of it is
 that those files are corrupted.
 
 The troubling thing for me is that we only discovered the corruption when it
 completely broke the index updates.  In one case, it did this by rearranging
 the bits so that couch_file thought that the btree node it was reading from
 disk had an associated MD5 checksum. It didn't (no btree nodes do), and so
 couch_file threw a file_corruption exception.  But if the corruption had
 shown up in another part of the file I might never have known.  In fact,
 some of the other indices on that node probably are silently corrupted.
 
 You might wonder how likely it is that a file becomes corrupted but still
 appears to be functioning.  I checked the last modified timestamps for three
 broken files.  One was last modified when the node went down, but the other
 two had timestamps in between the node's recovery and now.  To me, that
 means that the view indexer was able to update those files for quite a while
 (~2 days) before it bumped into a part of the btree that was corrupted.
 
 I wonder what we should do about this.  My first thought is to make it
 optional to write  btree nodes (possibly only for view index files?) using
 append_term_md5 instead of append_term.  It seems like a simple patch, but I
 don't know a priori what the performance hit would be.  Other thoughts?
 
 Best, Adam

Re: silent view index file corruption

[Paul J Davis]

On Apr 6, 2010, at 11:20 PM, Adam Kocoloski kocol...@apache.org wrote:

 On Apr 6, 2010, at 10:50 PM, Paul J Davis wrote:
 
 This corruption was quite odd in that there wasn't a conspicuous reason for 
 it.  I didn't dive to deep into the whole thing so it's possible i missed 
 something obvious. 
 
 The instance was unresponsive to ssh for 12 hours.  The report from AWS 
 Support was merely a problem with the underlying host followed by a 
 recommendation to launch a replacement at your earliest convenience.  I 
 don't know what the gremlins were doing behind the scenes, but I'm not 
 surprised the files are corrupted :)
 

Yeah I don't think that we should worry about high energy particles flipping 
bits too much here.

 There are two things at play here.  How proactive should we be in provoking 
 theseI errors and how much should we check for situations where our data 
 file got trounced.
 
 The extreme proactive position would be equivalent to a full table scan per 
 write which is out of the question. So to some extent we won't be able to 
 detect some errors until read time which is an unknowable interval.
 
 I'm totally comfortable with only detecting them at read-time.
 
 The other aspect is how rigorous should we check reads? This extreme would 
 basically require a sha1 for every read or write no matter how small, not to 
 mention the storage overhead. This part I'm not sure about. There's probably 
 middle ground with crc sums and what not but i don't see a clear answer.
 
 We currently store MD5 checksums with document bodies and validate them on 
 reads.  It hasn't proven to be an undue burden.
 

We do that for every doc body? Did not know that. Perhaps general 
append_term_md5 usage wouldn't be as big of a deal as i feared.

 Best, Adam
 
 Basically, the question is how much should we attempt to detect when 
 hardware lies.  I reckon that there's probably a middle ground to report 
 when an assumption is violated and full on table scans. Ideally such things 
 would be fairly configurable but i sure don't see an obvious answer.
 
 
 On Apr 6, 2010, at 10:06 PM, Randall Leeds randall.le...@gmail.com wrote:
 
 I immediately want to say 'ini file option' but I'm not sure whether to err
 on safety or speed.
 
 Maybe this is a good candidate for merkle trees or something else we can do
 throughout the view tree that might less overhead than md5 summing all the
 nodes? After all, most inner nodes shouldn't change most of the time. Some
 incremental, cheap checksum might be a worthwhile *option*.
 
 On Apr 6, 2010 6:04 PM, Adam Kocoloski kocol...@apache.org wrote:
 
 Hi all, we recently had an EC2 node go AWOL for about 12 hours.  When it
 came back, we noticed after a few days that a number of the view indexes
 stored on that node were not updating.  I did some digging into the error
 logs and with Paul's help pieced together what was going on.  I won't bother
 you with all the gory details unless you ask for them, but the gist of it is
 that those files are corrupted.
 
 The troubling thing for me is that we only discovered the corruption when it
 completely broke the index updates.  In one case, it did this by rearranging
 the bits so that couch_file thought that the btree node it was reading from
 disk had an associated MD5 checksum. It didn't (no btree nodes do), and so
 couch_file threw a file_corruption exception.  But if the corruption had
 shown up in another part of the file I might never have known.  In fact,
 some of the other indices on that node probably are silently corrupted.
 
 You might wonder how likely it is that a file becomes corrupted but still
 appears to be functioning.  I checked the last modified timestamps for three
 broken files.  One was last modified when the node went down, but the other
 two had timestamps in between the node's recovery and now.  To me, that
 means that the view indexer was able to update those files for quite a while
 (~2 days) before it bumped into a part of the btree that was corrupted.
 
 I wonder what we should do about this.  My first thought is to make it
 optional to write  btree nodes (possibly only for view index files?) using
 append_term_md5 instead of append_term.  It seems like a simple patch, but I
 don't know a priori what the performance hit would be.  Other thoughts?
 
 Best, Adam
 

Re: Preparing the 1.0.2 release

[Paul J. Davis]

I've already got the bits made, if you can double check what's online, that'd 
be great. Unfortunately I'm already back in Iowa so free time will be hit or 
miss.

Paul Davis

On Nov 24, 2010, at 4:35 PM, Noah Slater nsla...@apache.org wrote:

 I have an hour or two to help you through this process Paul.
 
 Are you around on IRC?
 
 On 24 Nov 2010, at 20:34, Paul Davis wrote:
 
 On Wed, Nov 24, 2010 at 12:00 PM, Paul Davis
 paul.joseph.da...@gmail.com wrote:
 This email is to request that the CouchDB developer community provide
 comments on moving forward with the 1.0.2 release. I am instructed to
 specifically ask that developers check the NEWS and CHANGES files in
 the 1.0.x branch to ensure that they list all of the applicable
 information.
 
 Thanks,
 Paul Davis
 
 
 This is *NOT* a vote on releasing 1.0.2.
 
 I've gone through the directions for preparing a release and I'd like
 to get some feedback on the artefacts themselves. This means things
 like verifying the tarball signature, the sha1 and md5 checksums and
 whether the tarball is missing anything.
 
 To check the signatures, you can use the directions at [1]. The
 artefacts themselves are located at [2].
 
 To check what's in the tarball vs what's not you can do this:
 
 $ mkdir tmp  cd tmp
 $ svn export http://svn.apache.org/repos/asf/couchdb/tags/1.0.2
 $ wget 
 http://people.apache.org/~davisp/dist/1.0.2/apache-couchdb-1.0.2.tar.gz
 $ tar -xvzf apache-couchdb-1.0.2.tar.gz
 $ diff -r apache-couchdb-1.0.2 1.0.2
 
 I repeat, this is *NOT* a vote on the release. We're merely working on
 improving our bus factor.
 
 Thanks,
 Paul Davis
 
 [1] http://people.apache.org/~davisp/dist/
 [2] http://people.apache.org/~davisp/dist/1.0.2/
 

Re: couchdb conflict resolution

[Paul J. Davis]

Here is some high level text on it. 

https://issues.apache.org/jira/browse/COUCHDB-988

On Feb 15, 2011, at 10:14 AM, Aaron Boxer boxe...@gmail.com wrote:

 Hello,
 
 I am very interested in understanding how conflict resolution works in 
 couchdb:
 
 Is there a technical overview, somewhere, of how a node decides which
 revision wins?
 After a conflict is resolved, are old revisions discarded?
 
 Any technical details, short of slogging through the code, would be
 very welcome.
 
 Thanks,
 Aaron

Re: [DISCUSSION] Move Fauxton to its own mailing list?

[Paul J Davis]

+1



 On Apr 14, 2015, at 3:50 PM, Andy Wenk andyw...@apache.org wrote:
 
 +1
 
 On 14 April 2015 at 21:02, Alexander Shorin kxe...@gmail.com wrote:
 
 On Tue, Apr 14, 2015 at 5:09 PM, Jan Lehnardt j...@apache.org wrote:
 We create new mailing list c...@couchdb.apache.org (or your favourite
 bike shed) that gets all JIRA and GitHub traffic.
 
 dev@ then is for discussion and decisions for all code projects
 (including Fauxton).
 
 If you want to keep track of tickets and pull requests, sign up for code@
 .
 
 There is a bit of a discrepancy between discussions in JIRA and dev@,
 but I’m willing to take that risk, as most people probably bulk-delete all
 JIRA mails anyway :)
 
 +1
 
 
 --
 ,,,^..^,,,
 
 
 
 -- 
 Andy Wenk
 Hamburg - Germany
 RockIt!
 
 GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588
 
 https://people.apache.org/keys/committer/andywenk.asc

Re: Windows build failing on couch_ejson_compare.c

[Paul J Davis]

The macro is totally fine. There are always plenty of those to account for 
windows/*nix differences so it's not dirty by any means. 



 On Jul 18, 2015, at 8:28 AM, Nick North nort...@gmail.com wrote:
 
 I see a couple of obvious solutions to this problem:
 
 
   1. Use the thread_local keyword rather than __thread. The semantics are
   not identical, so @davisp may have views on whether the behaviour is still
   OK. From the Windows build point of view, it would force us into building
   with Visual Studio 2015, which releases on 20 Jul, so we don't know whether
   it works with the rest of the build.
   2. Create a macro that expands to __declspec(thread) or __thread as
   appropriate. This feels less correct than the thread_local solution, but
   maybe better in the short term.
 
 Does anyone have any thoughts on which is the better way forward? Or have
 alternative suggestions?
 
 Nick
 
 On Fri, 17 Jul 2015 at 22:09 Nick North nort...@gmail.com wrote:
 
 Answering my own question: looks as if Microsoft uses __declspec(thread)
 rather than __thread.
 
 Nick
 
 On Fri, 17 Jul 2015 at 22:02 Nick North nort...@gmail.com wrote:
 
 I'm trying out @wohali's Windows build, which went well up to compilation
 of couch_ejson_compare.c, where it seems to be tripping up over the changes
 in the recent commit
 https://github.com/apache/couchdb-couch/commit/6b38dfacbb97c5cb7c89a27115d1227c7c52dbba
 to optimise performance. Specifically it gives this series of errors:
 
 Compiling
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c
 ERROR: compile failed while processing c:/couchdb/src/couch: rebar_abort
 Microsoft (R) C/C++ Optimizing Compiler Version 18.00.31101 for x86
 Copyright (C) Microsoft Corporation.  All rights reserved.
 
 cl : Command line warning D9002 : ignoring unknown option '-fno-common'
 couch_ejson_compare.c
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(44) :
 warnin
 g C4431: missing type specifier - int assumed. Note: C no longer supports
 defaul
 t-int
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(44) :
 error
 C2054: expected '(' to follow '__thread'
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(44) :
 error
 C2085: 'collator' : not in formal parameter list
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(44) :
 error
 C2143: syntax error : missing ';' before '='
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(58) :
 warnin
 g C4255: 'get_collator' : no function prototype given: converting '()' to
 '(void
 )'
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(62) :
 warnin
 g C4255: 'get_collator' : no function prototype given: converting '()' to
 '(void
 )'
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(66) :
 error
 C2065: 'collator' : undeclared identifier
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(66) :
 warnin
 g C4047: '!=' : 'int' differs in levels of indirection from 'void *'
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(67) :
 error
 C2065: 'collator' : undeclared identifier
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(67) :
 warnin
 g C4047: 'return' : 'UCollator *' differs in levels of indirection from
 'int'
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(70) :
 error
 C2065: 'collator' : undeclared identifier
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(70) :
 warnin
 g C4047: '=' : 'int' differs in levels of indirection from 'UCollator *'
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(73) :
 error
 C2065: 'collator' : undeclared identifier
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(73) :
 warnin
 g C4047: 'function' : 'UCollator *' differs in levels of indirection from
 'int'
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(73) :
 warnin
 g C4024: 'ucol_close_55' : different types for formal and actual
 parameter 1
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(78) :
 error
 C2065: 'collator' : undeclared identifier
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(78) :
 warnin
 g C4047: '=' : 'UCollator *' differs in levels of indirection from 'int'
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(84) :
 error
 C2065: 'collator' : undeclared identifier
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(84) :
 warnin
 g C4047: 'return' : 'UCollator *' differs in levels of indirection from
 'int'
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(256)
 : warni
 ng C4127: conditional expression is constant
 c:/couchdb/src/couch/priv/couch_ejson_compare/couch_ejson_compare.c(296)
 : warni
 ng C4127: conditional expression is constant
 
 I'll look into this further tomorrow, but someone more knowledgeable may
 immediately know what the 

Re: On dependency management and CI issues associated with it

[Paul J Davis]

> On Apr 15, 2016, at 3:13 AM, Jan Lehnardt  wrote:
> 
> 
>> On 14 Apr 2016, at 23:11, Joan Touzet  wrote:
>> 
>> Based on this information, are we in violation of ASF requirements? Can
>> anyone clarify for me what we actually need to be doing here?
> 
> There is no such policy. We are also not bundling SpiderMonkey or Erlang
> either. Neither do any of the Java projects bundle e.g. OpenJDK.
> 

My understanding is that anything included in an ASF release tarball must be in 
ASF source control which is why we mirror mochiweb but not Erlang.

There are also rules about downloading things to build ASF projects and the 
Java/Maven communities should have this info. Given that Erlang hasn't had a 
real package thing until semi recently its not something I've spent time 
figuring out. 

> The question of whether to have a “safe copy“ to be ensured against
> suddenly disappearing upstream is entirely* up to the project, but not
> ASF policy.
> 
> *upstream dependencies that have dual licensing that includes a GPL
> flavour or other incompatible license[1] can’t be mirrored on ASF
> source control and distribution servers (that’s why we don’t mirror
> SpiderMonkey or Erlang (although we could do Erlang now, that they
> switched to ASF 2, but I would not suggest we do this).
> 
> [1]: http://www.apache.org/legal/resolved.html#category-x
> 
> * * *
> 
> Personally, with npm’s new unpublish policy[2], I’m okay with having
> our dependencies there.
> 
> Because of the deep dependency tree, we should be very diligent about
> not accidentally including category-x licensed modules. I’m sure we
> can automate this into a npm postinstall script, so we know as soon
> as possible.
> 
> At the very least, we need an audit prior to any release.
> 
> [2]: http://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy
> 
> Best
> Jan
> --
> 
> 
>> 
>> -Joan
>> 
>> - Original Message -
>>> From: "Garren Smith" 
>>> To: dev@couchdb.apache.org, "Joan Touzet" 
>>> Sent: Thursday, April 14, 2016 2:43:10 AM
>>> Subject: Re: On dependency management and CI issues associated with it
>>> 
>>> Hi Joan,
>>> 
>>> Good point. Until about a week ago we use to keep all our
>>> dependencies in
>>> our repo. But we have just switched to webpack which allows us to
>>> manage
>>> our dependencies via npm (in case you are wondering, we don't depend
>>> on
>>> leftpad directly). So some of them are in our repo but the majority
>>> are
>>> downloaded and then bundled.
>>> 
>>> 
>>> Cheers
>>> Garren
>>> 
>>> On Wed, Apr 13, 2016 at 11:29 PM, Joan Touzet 
>>> wrote:
>>> 
 Garren, correct me if I'm wrong but Fauxton depends on a large
 number
 of JS dependencies that we don't keep copies of, correct? Or is it
 just
 for the build process?
 
 -Joan
 
 - Original Message -
> From: "Alexander Shorin" 
> To: dev@couchdb.apache.org
> Sent: Wednesday, April 13, 2016 2:08:20 PM
> Subject: Re: On dependency management and CI issues associated
> with it
> 
> On Wed, Apr 13, 2016 at 8:39 PM, Robert Newson
> 
> wrote:
>> It's a thread derail but this notion that we're being "fairly
>> rude"
>> needs resolving. It might be lost to history now but we got
>> here,
>> I think, with the best intentions of ensuring all the code that
>> appears in couchdb can be traced back to code hosted at asf. Is
>> it
>> a concrete requirement? I honestly forget but I thought so.
> 
> Yes, that's the answer why. If one day mochiweb owner will decide
> to
> drop his github repo, we shouldn't be leave with broken builds.
> See
> leftpad story as well. Initially, that requirement seems
> redundant,
> but recent npm drama showed that it has a huge point. Also there
> are
> some legal bits about.
> 
> --
> ,,,^..^,,,
> 
> -- 
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
> 

Re: [PROPOSAL] CouchDB 2.0 log to ./var/log/couchdb.log by default

[Paul J Davis]

Seems reasonable to me. I wonder if we should add a stdout log line that 
indicates where logs are going? Would be easy to add that as a module callback 
so it would work for stderr, file, and syslog. 

> On Aug 2, 2016, at 2:36 PM, Joan Touzet  wrote:
> 
> Presently, CouchDB 2.0 logs only to stderr. I have opened a PR
> to switch this behaviour to log to the ./var/log/couchdb.log
> release-local file by default:
> 
>  https://github.com/apache/couchdb/pull/435
> 
> This behaviour is easily overridden in the default.ini/local.ini
> files if desired.
> 
> I'm not sure if this is wanted by all stakeholders, so I haven't
> merged it into master. Please let me know either here or in the PR
> your thoughts. My intent is to merge this change by lazy consensus.

Re: Shards database name error in logs

[Paul J Davis]

Doh! Forgot about cassim. 

> On Sep 14, 2016, at 12:17 PM, Alexander Shorin  wrote:
> 
> Paul,
> 
> _metadata is our special database which is valid.
> https://github.com/apache/couchdb-cassim/blob/master/src/cassim_metadata_cache.erl#L63-L64
> --
> ,,,^..^,,,
> 
> 
>> On Wed, Sep 14, 2016 at 7:15 PM, Paul Davis  
>> wrote:
>> I'm pretty sure its saying that you're trying to create a database
>> named _metadata which isn't a valid database name for two reasons.
>> First, we require db names to start with a letter for file system
>> compatibility and to avoid surprises (ie, if someone tried to name a
>> database "../foo"). Second, we generally reserve the _ prefix in all
>> API places for CouchDB internal use.
>> 
>> On Tue, Sep 13, 2016 at 11:23 PM, Joey Samonte
>>  wrote:
>>> Good day,
>>> 
>>> 
>>> I am getting a lot of error like this one in the logs (Windows):
>>> 
>>> 
>>> {"error":"illegal_database_name","reason":"Name: 
>>> 'shards%2F-1fff%2F_metadata.1473815830'. Only lowercase 
>>> characters (a-z)
>>> 
>>> 
>>> Why is this so?
>>> 
>>> 
>>> Regards,
>>> 
>>> Joey
>>> 
>>> 
>>> 
>>>