[PATCH 01/13] test: Uniformly canonicalize actual and expected JSON
On Wed, Jul 25 2012, Austin Clements wrote: > Previously, we used a variety of ad-hoc canonicalizations for JSON > output in the test suite, but were ultimately very sensitive to JSON > irrelevancies such as whitespace. This introduces a new test > comparison function, test_expect_equal_json, that first pretty-prints > *both* the actual and expected JSON and the compares the result. > > The current implementation of this simply uses Python's json.tool to > perform pretty-printing (with a fallback to the identity function if > parsing fails). However, since the interface it introduces is > semantically high-level, we could swap in other mechanisms in the > future, such as another pretty-printer or something that does not > re-order object keys (if we decide that we care about that). The whole series looks good to me and was easy to read through. The use of Python json.tool is pretty reasonable; It is shipped in Python 2.6+ (For example perl does not ship json components by default, making it more complicated to set up for this purpose). If that is too "much", then we could also use tr -d '[:space:]' as an intermediate solution (but that eats whitespace in quoted content too...) I'll be afk for a few days then I test this... Tomi
Re: [PATCH 01/13] test: Uniformly canonicalize actual and expected JSON
On Wed, Jul 25 2012, Austin Clements amdra...@mit.edu wrote: Previously, we used a variety of ad-hoc canonicalizations for JSON output in the test suite, but were ultimately very sensitive to JSON irrelevancies such as whitespace. This introduces a new test comparison function, test_expect_equal_json, that first pretty-prints *both* the actual and expected JSON and the compares the result. The current implementation of this simply uses Python's json.tool to perform pretty-printing (with a fallback to the identity function if parsing fails). However, since the interface it introduces is semantically high-level, we could swap in other mechanisms in the future, such as another pretty-printer or something that does not re-order object keys (if we decide that we care about that). The whole series looks good to me and was easy to read through. The use of Python json.tool is pretty reasonable; It is shipped in Python 2.6+ (For example perl does not ship json components by default, making it more complicated to set up for this purpose). If that is too much, then we could also use tr -d '[:space:]' as an intermediate solution (but that eats whitespace in quoted content too...) I'll be afk for a few days then I test this... Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 01/13] test: Uniformly canonicalize actual and expected JSON
Previously, we used a variety of ad-hoc canonicalizations for JSON
output in the test suite, but were ultimately very sensitive to JSON
irrelevancies such as whitespace. This introduces a new test
comparison function, test_expect_equal_json, that first pretty-prints
*both* the actual and expected JSON and the compares the result.
The current implementation of this simply uses Python's json.tool to
perform pretty-printing (with a fallback to the identity function if
parsing fails). However, since the interface it introduces is
semantically high-level, we could swap in other mechanisms in the
future, such as another pretty-printer or something that does not
re-order object keys (if we decide that we care about that).
In general, this patch does not remove the existing ad-hoc
canonicalization because it does no harm. We do have to remove the
newline-after-comma rule from notmuch_json_show_sanitize and
filter_show_json because it results in invalid JSON that cannot be
pretty-printed.
Most of this patch simply replaces test_expect_equal and
test_expect_equal_file with test_expect_equal_json. It changes the
expected JSON in a few places where sanitizers had placed newlines
after commas inside strings.
---
test/crypto| 37 +++--
test/json | 14 +++---
test/maildir-sync | 11 ---
test/multipart | 34 +++---
test/search-output |2 +-
test/test-lib.sh | 17 +
6 files changed, 55 insertions(+), 60 deletions(-)
diff --git a/test/crypto b/test/crypto
index be752b1..5dd14c4 100755
--- a/test/crypto
+++ b/test/crypto
@@ -51,8 +51,7 @@ expected='[[[{"id": "X",
"headers": {"Subject": "test signed message 001",
"From": "Notmuch Test Suite ",
"To": "test_suite at notmuchmail.org",
- "Date": "Sat,
- 01 Jan 2000 12:00:00 +"},
+ "Date": "Sat, 01 Jan 2000 12:00:00 +"},
"body": [{"id": 1,
"sigstatus": [{"status": "good",
"fingerprint": "'$FINGERPRINT'",
@@ -64,7 +63,7 @@ expected='[[[{"id": "X",
{"id": 3,
"content-type": "application/pgp-signature"}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
"$output" \
"$expected"
@@ -85,8 +84,7 @@ expected='[[[{"id": "X",
"headers": {"Subject": "test signed message 001",
"From": "Notmuch Test Suite ",
"To": "test_suite at notmuchmail.org",
- "Date": "Sat,
- 01 Jan 2000 12:00:00 +"},
+ "Date": "Sat, 01 Jan 2000 12:00:00 +"},
"body": [{"id": 1,
"sigstatus": [{"status": "good",
"fingerprint": "'$FINGERPRINT'",
@@ -99,7 +97,7 @@ expected='[[[{"id": "X",
{"id": 3,
"content-type": "application/pgp-signature"}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
"$output" \
"$expected"
@@ -119,8 +117,7 @@ expected='[[[{"id": "X",
"headers": {"Subject": "test signed message 001",
"From": "Notmuch Test Suite ",
"To": "test_suite at notmuchmail.org",
- "Date": "Sat,
- 01 Jan 2000 12:00:00 +"},
+ "Date": "Sat, 01 Jan 2000 12:00:00 +"},
"body": [{"id": 1,
"sigstatus": [{"status": "error",
"keyid": "'$(echo $FINGERPRINT | cut -c 25-)'",
@@ -132,7 +129,7 @@ expected='[[[{"id": "X",
{"id": 3,
"content-type": "application/pgp-signature"}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
"$output" \
"$expected"
mv "${GNUPGHOME}"{.bak,}
@@ -193,8 +190,7 @@ expected='[[[{"id": "X",
"headers": {"Subject": "test encrypted message 001",
"From": "Notmuch Test Suite ",
"To": "test_suite at notmuchmail.org",
- "Date": "Sat,
- 01 Jan 2000 12:00:00 +"},
+ "Date": "Sat, 01 Jan 2000 12:00:00 +"},
"body": [{"id": 1,
"encstatus": [{"status": "good"}],
"sigstatus": [],
@@ -210,7 +206,7 @@ expected='[[[{"id": "X",
"content-type": "application/octet-stream",
"filename": "TESTATTACHMENT"}]}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
"$output" \
"$expected"
@@ -221,7 +217,7 @@ output=$(notmuch show --format=json --part=4 --decrypt
subject:"test encrypted m
expected='{"id": 4,
"content-type": "text/plain",
"content": "This is a test encrypted message.\n"}'
-test_expect_equal \
+test_expect_equal_json \
"$output" \
"$expected"
@@ -248,8 +244,7 @@ expected='[[[{"id": "X",
"headers": {"Subject": "test encrypted message 001",
"From": "Notmuch Test Suite ",
"To": "test_suite at notmuchmail.org",
- "Date": "Sat,
- 01 Jan 2000 12:00:00 +"},
+ "Date": "Sat, 01 Jan 2000 12:00:00 +"},
"body": [{"id": 1,
"encstatus": [{"status": "bad"}],
"content-type": "multipart/encrypted",
@@ -258,7 +253,7 @@ expected='[[[{"id": "X",
{"id": 3,
"content-type": "application/octet-stream"}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
"$output" \
"$expected"
mv "${GNUPGHOME}"{.bak,}
@@ -283,8 +278,7 @@ expected='[[[{"id": "X",
"headers": {"Subject": "test encrypted message 002",
"From": "Notmuch Test Suite ",
"To":
[PATCH 01/13] test: Uniformly canonicalize actual and expected JSON
Previously, we used a variety of ad-hoc canonicalizations for JSON
output in the test suite, but were ultimately very sensitive to JSON
irrelevancies such as whitespace. This introduces a new test
comparison function, test_expect_equal_json, that first pretty-prints
*both* the actual and expected JSON and the compares the result.
The current implementation of this simply uses Python's json.tool to
perform pretty-printing (with a fallback to the identity function if
parsing fails). However, since the interface it introduces is
semantically high-level, we could swap in other mechanisms in the
future, such as another pretty-printer or something that does not
re-order object keys (if we decide that we care about that).
In general, this patch does not remove the existing ad-hoc
canonicalization because it does no harm. We do have to remove the
newline-after-comma rule from notmuch_json_show_sanitize and
filter_show_json because it results in invalid JSON that cannot be
pretty-printed.
Most of this patch simply replaces test_expect_equal and
test_expect_equal_file with test_expect_equal_json. It changes the
expected JSON in a few places where sanitizers had placed newlines
after commas inside strings.
---
test/crypto| 37 +++--
test/json | 14 +++---
test/maildir-sync | 11 ---
test/multipart | 34 +++---
test/search-output |2 +-
test/test-lib.sh | 17 +
6 files changed, 55 insertions(+), 60 deletions(-)
diff --git a/test/crypto b/test/crypto
index be752b1..5dd14c4 100755
--- a/test/crypto
+++ b/test/crypto
@@ -51,8 +51,7 @@ expected='[[[{id: X,
headers: {Subject: test signed message 001,
From: Notmuch Test Suite test_su...@notmuchmail.org,
To: test_su...@notmuchmail.org,
- Date: Sat,
- 01 Jan 2000 12:00:00 +},
+ Date: Sat, 01 Jan 2000 12:00:00 +},
body: [{id: 1,
sigstatus: [{status: good,
fingerprint: '$FINGERPRINT',
@@ -64,7 +63,7 @@ expected='[[[{id: X,
{id: 3,
content-type: application/pgp-signature}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
$output \
$expected
@@ -85,8 +84,7 @@ expected='[[[{id: X,
headers: {Subject: test signed message 001,
From: Notmuch Test Suite test_su...@notmuchmail.org,
To: test_su...@notmuchmail.org,
- Date: Sat,
- 01 Jan 2000 12:00:00 +},
+ Date: Sat, 01 Jan 2000 12:00:00 +},
body: [{id: 1,
sigstatus: [{status: good,
fingerprint: '$FINGERPRINT',
@@ -99,7 +97,7 @@ expected='[[[{id: X,
{id: 3,
content-type: application/pgp-signature}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
$output \
$expected
@@ -119,8 +117,7 @@ expected='[[[{id: X,
headers: {Subject: test signed message 001,
From: Notmuch Test Suite test_su...@notmuchmail.org,
To: test_su...@notmuchmail.org,
- Date: Sat,
- 01 Jan 2000 12:00:00 +},
+ Date: Sat, 01 Jan 2000 12:00:00 +},
body: [{id: 1,
sigstatus: [{status: error,
keyid: '$(echo $FINGERPRINT | cut -c 25-)',
@@ -132,7 +129,7 @@ expected='[[[{id: X,
{id: 3,
content-type: application/pgp-signature}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
$output \
$expected
mv ${GNUPGHOME}{.bak,}
@@ -193,8 +190,7 @@ expected='[[[{id: X,
headers: {Subject: test encrypted message 001,
From: Notmuch Test Suite test_su...@notmuchmail.org,
To: test_su...@notmuchmail.org,
- Date: Sat,
- 01 Jan 2000 12:00:00 +},
+ Date: Sat, 01 Jan 2000 12:00:00 +},
body: [{id: 1,
encstatus: [{status: good}],
sigstatus: [],
@@ -210,7 +206,7 @@ expected='[[[{id: X,
content-type: application/octet-stream,
filename: TESTATTACHMENT}]}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
$output \
$expected
@@ -221,7 +217,7 @@ output=$(notmuch show --format=json --part=4 --decrypt
subject:test encrypted m
expected='{id: 4,
content-type: text/plain,
content: This is a test encrypted message.\n}'
-test_expect_equal \
+test_expect_equal_json \
$output \
$expected
@@ -248,8 +244,7 @@ expected='[[[{id: X,
headers: {Subject: test encrypted message 001,
From: Notmuch Test Suite test_su...@notmuchmail.org,
To: test_su...@notmuchmail.org,
- Date: Sat,
- 01 Jan 2000 12:00:00 +},
+ Date: Sat, 01 Jan 2000 12:00:00 +},
body: [{id: 1,
encstatus: [{status: bad}],
content-type: multipart/encrypted,
@@ -258,7 +253,7 @@ expected='[[[{id: X,
{id: 3,
content-type: application/octet-stream}]}]},
['
-test_expect_equal \
+test_expect_equal_json \
$output \
$expected
mv ${GNUPGHOME}{.bak,}
@@ -283,8 +278,7 @@ expected='[[[{id: X,
headers: {Subject: test encrypted message 002,
From: Notmuch Test Suite test_su...@notmuchmail.org,
To: test_su...@notmuchmail.org,
- Date: Sat,
- 01 Jan 2000 12:00:00 +},
+ Date: Sat, 01 Jan 2000 12:00:00 +},
body: [{id: 1,
encstatus: [{status:
