mmmmxa commented on PR #4407:
URL: https://github.com/apache/iggy/pull/4407#issuecomment-6086471136
Hi @hubcio!
I'll focus only on high-level thoughts as I haven't really comprehended 2.2k
lines of the test since I wanted to limit the time spent on this.
Observations:
- I have ran the test a number of times and went through code on a high
level.
- The two suggested parameter configurations work and fail as expected on my
Fedora 44.1.7 IF you ran the default config first, and then the bad one.
- However, running bad config first and then the default one after causes a
failure, which I did not thoroughly investigate or I don't have the expertise
in clustering yet to see what is the problem:
```
PANIC: panicked at core/integration/tests/cluster/chaos_monkey.rs:233:5:
2 violations, replay with IGGY_TEST_CHAOS_DURATION_SECS=120
IGGY_TEST_CHAOS_KILL_INTERVAL_SECS=15 IGGY_TEST_CHAOS_DOWNTIME_SECS=5
IGGY_TEST_CHAOS_SEED=7:
nodes did not reach the end state within 60s:
node 1: NodeView { partitions: Some([PartitionView { current_offset: 33969,
messages_count: 11320, segments_count: 1, stored_offsets: [Some(33969),
Some(33969)] }, PartitionView { current_offset: 32299, messages_count: 9650,
segments_count: 1, stored_offsets: [Some(32299)] }, PartitionView {
current_offset: 32419, messages_count: 9770, segments_count: 1, stored_offsets:
[Some(32419)] }, PartitionView { current_offset: 32179, messages_count: 9530,
segments_count: 1, stored_offsets: [Some(32179)] }]), history:
Some([PartitionView { current_offset: 17219, messages_count: 17220,
segments_count: 2, stored_offsets: [] }, PartitionView { current_offset: 16229,
messages_count: 16230, segments_count: 2, stored_offsets: [] }, PartitionView {
current_offset: 17369, messages_count: 17370, segments_count: 2,
stored_offsets: [] }, PartitionView { current_offset: 17269, messages_count:
17270, segments_count: 2, stored_offsets: [] }]), members: Some([]) }, node 0:
NodeView { partitions: Some([Par
titionView { current_offset: 33969, messages_count: 110, segments_count: 1,
stored_offsets: [Some(33969), Some(33969)] }, PartitionView { current_offset:
32299, messages_count: 9650, segments_count: 1, stored_offsets: [Some(32299)]
}, PartitionView { current_offset: 32419, messages_count: 9770, segments_count:
1, stored_offsets: [Some(32419)] }, PartitionView { current_offset: 32179,
messages_count: 9530, segments_count: 1, stored_offsets: [Some(32179)] }]),
history: Some([PartitionView { current_offset: 17219, messages_count: 17220,
segments_count: 2, stored_offsets: [] }, PartitionView { current_offset: 16229,
messages_count: 16230, segments_count: 2, stored_offsets: [] }, PartitionView {
current_offset: 17369, messages_count: 17370, segments_count: 2,
stored_offsets: [] }, PartitionView { current_offset: 17269, messages_count:
17270, segments_count: 2, stored_offsets: [] }]), members: Some([]) }
chaos-topic partition 0: node 1 holds 11320 messages, the first node read
holds 110; first difference at position 0: Some(LogEntry { offset: 22650, id:
92233720368547769320, payload: b"producer 4 seq 11240" }) vs Some(LogEntry {
offset: 33860, id: 18446744073709568616, payload: b"producer 0 seq 17000" })
FAIL [ 203.286s] (1/1) integration::mod
cluster::chaos_monkey::given_five_nodes_when_one_is_killed_every_interval_should_keep_acked_messages
Cancelling due to test failure:
────────────
Summary [ 203.295s] 1 test run: 0 passed, 1 failed, 1039 skipped
FAIL [ 203.286s] (1/1) integration::mod
cluster::chaos_monkey::given_five_nodes_when_one_is_killed_every_interval_should_keep_acked_messages
error: test run failed
```
Here are my thoughts. Please take them with a pinch of salt because I do not
have the information on the context of use and current intended scope of the
tool.
1. This test is already so large that it seems like it is going to evolve
into a small subsystem that could be deployed and ran somewhere. It could make
sense to put the whole comment section in the beginning into a separate README,
and rewrite it using simple AST English. (btw just mentioning "use simple ast
english" in the prompt turns out to be enough). I find that the writing style
currently is not good for understanding the new concepts, and is only okay when
you know the topic. Maybe known bugs section could also be useful here, not
sure about where to put all of the discoveries that can be potentially made
with this tool.
2. It could be useful for this test to have an artifact or a simple
structured report. At the moment, the test is bloating the terminal easily by
dumping all the logs in case of failure, causing the information to disappear
if the terminal buffer is set to default, which is always low for these
purposes. This file could record some details about the environment which would
enable to automate or facilitate decision making when someone uses it,
especially to submit a bug report.
3. The tests are randomly failing by the current design, and as I have
mentioned above, the sequence of running them also affects the outcomes. The
latter example shows a potential issue in discussions when collaborating on the
bugs found by this test. The signal is random and is heavily influenced by
whatever slop running on a random contributor's machine who's going to submit a
bug report generated by this system. I haven't thought of a solution yet, but
I'm thinking about the potential context of the system that we can capture so
that the decision making on the bug can be automated with agents or aligned
quicker between the discussing parties.
Please treat 1 and 2 as more concrete points and 3 as a discussion opener
into potential improvements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
