GitHub user kxepal opened a pull request:
https://github.com/apache/couchdb/pull/253
Replace etap tests with eunit
This PR replaces etap test with eunit.
Issues with etap:
1. It's etap (:
2. Almost for all suites each next test depends from the state of previous
one
3. Lack of integration with rebar out of box
4. Badly organization
Goals:
1. Refresh test suite: split and isolate each test case
2. Find and get rid heisenbugs
3. Make test suite ready for CouchDB 2.0 build system
Considerations:
1. Each target test function name reflects the test goal
2. Each target test function starts with `should_` prefix to 1) separate
test functions from helper ones 2) make names more clean
3. Otherwise test has description or reference to couchdb issue
4. Test suites which tests specific module named as this module is
5. Otherwise they starts with more common "couchdb_" prefix
6. Tests output should be human friendly
Fixed issues (at least):
- Timeouts are reduced to meanful values, all timer:sleep are removed
unless they are required for test logic
- 073-changes.t: heartbeats count is very depended from host's performance
- 090-task-status.t: get_task_prop silently resets Acc if searched task
isn't last in the list
- 140-attachments-comp.t: zlib:gzip only worked for small attachments
- 201-view-group-shutdown.t: race condition for {error, all_dbs_active}
assertion check. Previously was solved by timer:sleep.
Some stats:
- CouchDB:
- Etap:
- Files=52, Tests=1219
- time make check 107.92s user 11.17s system 35% cpu 5:38.20 total
- Eunit:
- Files=29, Tests=693
- time make check 74.41s user 11.68s system 51% cpu 2:48.48 total
- couch_mrview:
- Etap:
- Files=7, Tests=41
- time make check 3.02s user 0.79s system 73% cpu 5.191 total
- Eunit:
- Files=7, Tests=39
- time make check 1.48s user 0.32s system 52% cpu 3.405 total
- couch_replicator:
- Etap:
- Files=7, Tests=1832
- time make check 115.11s user 13.94s system 81% cpu 2:37.98 total
- Eunit:
- Files=7, Tests=127
- time make check 133.73s user 15.99s system 89% cpu 2:47.51 total
(Notice: as for etap each "test" actually means the assertion)
Expected output: https://www.friendpaste.com/2MCgzw3g18ATOWyLGBSQwZ
Notice: I'd ran tests both on fast and slow environment (single cored
virtualbox with 40% cpu cap), so execution timeouts fits both. However, I'm not
sure about all possible configurations, so if any test fails with *timeout* for
you please include in your report your system configuration - it
Notice: [error] messages in output is *FINE* - they happens because
`log=none` doesn't disables logging completely, allowing error messages to be
emit out.
P.S. All commits after "Goodbye etap!" will be squashed before merge.
COUCHDB-1963
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/couchdb 1963-eunit
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/couchdb/pull/253.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #253
----
commit c9c457607ad61b6c61bb51df8ebda36bcc809e3f
Author: Alexander Shorin <[email protected]>
Date: 2014-05-15T20:08:36Z
Add run script to execute eunit tests
Usage is the same as for test/etap/run:
./test/couchdb/run -v ${PATH}
-v runs in verbose mode, as etap does. Also, you can use make for that:
make check-eunit
which will run tests everywhere in project where Makefile contains
check-eunit subcommand definition.
The ${PATH} thing could be single file or directory. The latter should
contains *_tests.erl files which would be compiled and executed by
eunit.
The *_tests.erl
The reason of compiling on run instead of using autoconf for that is
to simplify tests developing and avoid situations, when you'd fixed
the test or add new one, but forgot to remove/compile beam file.
All test_*.beam files are been stored in test/couchdb/ebin directory,
the temporary test files will be places to test/couchdb/temp one.
Both directories will be removed by make clean/distclean command.
commit 47e61617ff0f444a0e74f9fce58562e3f4e6c3d5
Author: Alexander Shorin <[email protected]>
Date: 2014-05-15T23:46:16Z
Add common header for eunit test files
commit 33a95efbeb0613b121f4c36ef36092f0893ee00d
Author: Alexander Shorin <[email protected]>
Date: 2014-05-16T00:45:38Z
Port 001-load.t etap test suite to eunit
commit 01d6bca33c865bfce5798da1c9b492a1fa6bfe6d
Author: Alexander Shorin <[email protected]>
Date: 2014-05-16T01:19:05Z
Port 002-icu-driver.t etap test suite to eunit
See setup/0 comment for specific info about loading
couch_icu_driver with eunit.
commit 5bbb1140195f7a8e39bd36d4b422090fe3af4442
Author: Alexander Shorin <[email protected]>
Date: 2014-05-16T14:38:56Z
Port 010-file-basics.t and 011-file-headers.t etap test suites to eunit
Both merged into single suite since they tests single target and shares
common bits.
commit c3dc98de0ad62dfd6021777b27632e48ee30de6d
Author: Alexander Shorin <[email protected]>
Date: 2014-05-16T23:28:14Z
Port 020-btree-basics.t and 021-btree-reductions.t etap suites to eunit
Both merged into single suite since they tests single target and shares
common bits.
commit ed7b576a7267227d73a412c0066a04efefb5c955
Author: Alexander Shorin <[email protected]>
Date: 2014-05-16T23:42:01Z
Port 030-doc-from-json.t and 031-doc-to-json.t etap suites to eunit
Both merged into single suite since they tests single target and shares
common bits.
commit 33f5575e192f03cb450c332f6eb202e33f1b9565
Author: Alexander Shorin <[email protected]>
Date: 2014-05-17T00:16:38Z
Port 040-util.t etap test suite to eunit
commit 332596355db5a4e2d24574a14ec89c6fc0f34ed7
Author: Alexander Shorin <[email protected]>
Date: 2014-05-17T00:23:25Z
Port 041-uuid.t etap test suite to eunit
Config files are removed in favor of using couch_config API instead.
commit 6162c9541f46a23cd079f7901a4b0b260fc1b397
Author: Alexander Shorin <[email protected]>
Date: 2014-05-18T10:04:08Z
Port 042-work-queue.t etap test suite to eunit
Etap tests were made in flow style, testing the same things multiple
times without real need. For eunit they are split into small test cases
to focus on testing goals.
Timeout on receive is decreased from 3000 to 100.
commit de228dc0629aadd210e23cad055f9a68fbea5055
Author: Alexander Shorin <[email protected]>
Date: 2014-05-18T10:35:58Z
Port 043-find-in-binary.t etap test suite to eunit
It been merged into couch_util_tests suite.
commit 31c6634c32da99de14b3c00e9787a70ea4c0b78d
Author: Alexander Shorin <[email protected]>
Date: 2014-05-18T10:45:05Z
Port 050-stream.t etap test suite to eunit
commit d00e3a278769692d95db340c2e7fa5c3d1f07ff8
Author: Alexander Shorin <[email protected]>
Date: 2014-05-18T10:53:03Z
Port 06[0-5]-kt-*.t etap test suites to eunit
All merged into single suite since they're test the same module.
commit 3504d9882fbc66f06cd0309e137856ca9e26bfff
Author: Alexander Shorin <[email protected]>
Date: 2014-05-19T01:36:06Z
Port 070-couch-db.t etap test suite to eunit
Fix ERL_LIB environment variable setting. Add ?tempdb macros for
unique temporary database name generation.
commit d22c1e1c1ac6f066a9d03fadfbd6776a810e71fc
Author: Alexander Shorin <[email protected]>
Date: 2014-05-20T03:16:41Z
Port 072-cleanup.t etap test suite to eunit
requests functions from test_util were moved to test_request module
with nicer and simpler API.
commit 2c6c7249849c6f24cf822839647dc433c77a70eb
Author: Alexander Shorin <[email protected]>
Date: 2014-05-20T03:19:22Z
Port 073-changes.t etap test suite to eunit
For heartbeats test they don't being counted anymore since their
amount is heavy depends from the overall system performance and some
assertions may fail or not because of that. Instead of this, we just
ensure that their amount is going to increase over the time.
commit aff968819775a78a67b090dfa97f6e5df3f2e80b
Author: Alexander Shorin <[email protected]>
Date: 2014-05-21T14:58:12Z
Port 074-doc-update-conflicts.t etap test suite to eunit
Timeout decreased, added 10K clients case
commit 085e0ed289eafbae9a5e1c526cc25a2b39d9008a
Author: Alexander Shorin <[email protected]>
Date: 2014-05-22T09:07:24Z
Port 075-auth-cache.t etap test suite to eunit
Timeouts are removed.
commit e656e8d710e565fc0c7aa0420dee2b407dadd3f8
Author: Alexander Shorin <[email protected]>
Date: 2014-05-22T16:17:47Z
Port 076-file-compression.t etap test suite to eunit
The original test suite was decoupled into compaction and comparison
cases.
commit d01b88abbecbe7de2ddc51d0fc4dd88f3b32816d
Author: Alexander Shorin <[email protected]>
Date: 2014-05-22T16:31:09Z
Port 077-couch-db-fast-db-delete-create.t etap test suite to eunit
Merged into couch_db_tests suite.
commit c44c8d13c1ab1a88c08e3f3059138fae09430314
Author: Alexander Shorin <[email protected]>
Date: 2014-05-23T05:29:22Z
Port 080-config-get-set.t etap test suite to eunit
commit 09a0e6d369f45e97874698166cb0efbdaced48a5
Author: Alexander Shorin <[email protected]>
Date: 2014-05-25T18:02:19Z
Port 081-config-override.t etap test suite to eunit
Merged into couch_config_tests suite.
Setup fixtures.
commit 806f13bbd8d8985eae044b4650ca630ecb5fd13e
Author: Alexander Shorin <[email protected]>
Date: 2014-05-26T05:26:22Z
Port 082-config-register.t etap test suite to eunit
Merged into couch_config_tests suite.
commit ddbbd519b193bc0fb1d7ca124b233790979f4437
Author: Alexander Shorin <[email protected]>
Date: 2014-05-26T05:46:06Z
Port 083-config-no-files.t etap test suite to eunit
Merged into couch_config_tests suite.
commit 9227502e1f00920de4db3e177e76551563d99826
Author: Alexander Shorin <[email protected]>
Date: 2014-05-26T16:23:41Z
Port 090-task-status.t etap test suite to eunit
Split huge test case into multiple ones. Fix issue with get_task_prop
when Acc may be reset if searched task isn't last in the list.
commit fad32ab98af0abde6ea01694136ee1f79844f914
Author: Alexander Shorin <[email protected]>
Date: 2014-05-26T16:59:45Z
Port 100-ref-counter.t etap test suite to eunit
commit 8bc52a1171f2eb87a8523c8746399b1fd739a33c
Author: Alexander Shorin <[email protected]>
Date: 2014-05-26T19:57:38Z
Port 120-stats-collect.t etap test suite to eunit
commit 19c5402311dde98a79f870d6fc3814de5503498d
Author: Alexander Shorin <[email protected]>
Date: 2014-05-27T14:27:53Z
Port 121-stats-aggregates.t etap test suite to eunit
Merged into couch_stats_tests suite.
commit d0e50a0843cec3da88908b04585c750c878451a5
Author: Alexander Shorin <[email protected]>
Date: 2014-05-28T01:32:11Z
Port 130-attachments-md5.t etap test suite to eunit
Add random document id generator macros.
Have to use handmade http client instead of ibrowse since it makes too
complicated sending chunked requests.
commit 5726d0689d4b9e37df8476726e97ee6fa92c46bb
Author: Alexander Shorin <[email protected]>
Date: 2014-06-02T21:51:46Z
Port 140-attachments-comp.t etap test suite to eunit
- Merge into couchdb_attachments_tests suite;
- Add PUT requests to test_request util;
- Remove dependency from files outside fixtures directory;
- Group test cases to reduce amount of duplicate code;
- Fix hidden issue with gzip encoding: for encoding_length stub info
check using zlib:gzip on 2KiB+ files leads to mismatch by 2-4 bytes
and this difference grows with file size. Using gzip fun code from
couch_stream solves the issue.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
