Stabilizing our automated builds - help needed!

[Joan Touzet]

Hello,

I've been getting our CI workflows cleaned up a bit, including
re-enabling our OS matrix builds on Apache's Jenkins. Today, I finally
got my first all-green build on Jenkins and Travis at the same time. It
was a LOT harder than it should have been.

Why? Because our test suite (and build process) inconsistently fails.
I need your help to stabilize the build so we can rely on CI again.
I also need your help because I can't do all the work myself.

We have 1 build failure and 5 test failures that I'm aware of.


Currently there is one strict build failure, in fauxton. I believe this
 is a transient failure of trying to pull down an npm package:

--------------------------
grunt-couchapp@0.2.1 node_modules/grunt-couchapp
├── nano@3.3.0 (errs@0.2.4, request@2.9.203, follow@0.8.0)
├── couchapp@0.10.0 (watch@0.8.0, coffee-script@1.12.4, connect@3.6.0)
└── grunt@0.3.17 (dateformat@1.0.2-1.2.3, semver@1.0.14, async@0.1.22, 
colors@0.6.2, hooker@0.2.3, underscore@1.2.4, nopt@1.0.10, 
underscore.string@2.1.1, gzip-js@0.3.2, temporary@0.0.8, uglify-js@1.3.5, 
prompt@0.1.12, glob-whatev@0.1.8, jshint@0.9.1, connect@2.4.6, nodeunit@0.7.4)
npm ERR! Linux 3.19.0-65-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" "--production"
npm ERR! node v4.2.6
npm ERR! npm  v2.14.12

npm ERR! Callback called more than once.
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /usr/src/couchdb-checkout/src/fauxton/npm-debug.log

I've seen tell of some retry scripts on GH, is this something we could
consider for the build process?
--------------------------



The rest of the failures are in our test suite. 4 are in eunit and 1 is
in the JavaScript suite. Here's what I've seen:


--------------------------
module 'couch_log_writer_syslog_test'
  couch_log_writer_syslog_test: couch_log_writer_syslog_test_...*timed out*
  undefined

Not sure why this times out, can we increase the timeout maybe?


--------------------------
module 'couchdb_compaction_daemon_tests'
  Compaction daemon tests
    couchdb_compaction_daemon_tests:65: should_compact_by_default_rule...*timed 
out*
    undefined
  couchdb_compaction_daemon_tests:99: should_compact_by_dbname_rule...*timed 
out*
  undefined

Same as before, don't know why we time out, can we increase it?


--------------------------
module 'couchdb_compaction_daemon_tests'
  Compaction daemon tests
    couchdb_compaction_daemon_tests:65: 
should_compact_by_default_rule...*failed*
in function 
couchdb_compaction_daemon_tests:'-should_compact_by_default_rule/1-fun-2-'/1 
(test/couchdb_compaction_daemon_tests.erl, line 88)
in call from 
couchdb_compaction_daemon_tests:'-should_compact_by_default_rule/1-fun-7-'/1 
(test/couchdb_compaction_daemon_tests.erl, line 88)
**error:{assert,[{module,couchdb_compaction_daemon_tests},
         {line,88},
         {expression,"DbFrag2 < 70"},
         {expected,true},
         {value,false}]}
  output:<<"">>

Paul Davis (davisp) thinks he has fixed this in the pluggable storage
engine branch. Paul can you confirm?


--------------------------
module 'couch_log_config_listener_test'
  couch_log_config_listener_test: couch_log_config_test_...*failed*
in function couch_log_config_listener_test:'-check_restart_listener/0-fun-2-'/1 
(test/couch_log_config_listener_test.erl, line 38)
in call from couch_log_config_listener_test:check_restart_listener/0 
(test/couch_log_config_listener_test.erl, line 38)
**error:{assertEqual,[{module,couch_log_config_listener_test},
              {line,38},
              {expression,"get_handler ( )"},
              {expected,not_found},
              {value,{config_listener,{couch_log_sup,<0.3192.0>}}}]}
  output:<<"">>

No clue what's going on here.


--------------------------
test/javascript/tests/replication.js                           
    Error: expected '5', got '6'
Trace back (most recent call first):
    
  52: test/javascript/test_setup.js
      T(false,"expected '5', got '6'",(void 0))
 321: test/javascript/couch_test_runner.js
      TEquals(5,6)
 1620: test/javascript/tests/replication.js
      ()
  37: test/javascript/cli_runner.js
      runTest()
  48: test/javascript/cli_runner.js
      
fail


Need help analyzing this one.


Thanks in advance to anyone who's able to help out here.

-Joan