On 7/14/20 3:34 PM, Dumitru Ceara wrote: > On 6/30/20 3:24 AM, Ilya Maximets wrote: >> This patch set adds new stream provider and other functionality in >> order to record all the incoming data on all the steams (ssl, tcp, >> unixctl) of openvswitch library based applications and replay these >> streams later for debugging purposes or performance tests. >> >> For example, these changes allowed me to record the lifecycle of >> a standalone ovsdb-server in a fake-multinode cluster from ovn >> scale testing setup with 100 nodes. While having only replay files, >> user could reproduce transactions and connections in a big cluster on >> a local PC wile being able to tweak various log levels, run everything >> under debugger or tracer, measure performance difference with perf. >> >> Current implementation doesn't work with clustered databases since >> raft heavily depends on time events. This is a point of further >> improvement. >> >> Some more details in individual commit messages. Some documentation >> updates are missing, will be in next version of the patch-set. Code >> maybe not that elegant in a couple of places, I'm going to work on >> this too. >> >> ovn-northd might be a good candidate to integrate this functionality >> from the OVN side. >> >> Ilya Maximets (5): >> ovsdb-server: Allow using predictable UUIDs. >> stream: Add record/replay functionality. >> ovsdb-server: Integrate stream replay engine. >> ovsdb-server: Don't update manager status if replay engine is active. >> jsonrpc: Disable inactivity probes if replay engine is active. >> >> lib/automake.mk | 1 + >> lib/jsonrpc.c | 10 +- >> lib/stream-provider.h | 4 + >> lib/stream-replay.c | 561 ++++++++++++++++++++++++++++++++++++++++ >> lib/stream.c | 38 ++- >> lib/stream.h | 46 +++- >> lib/uuid.c | 34 +++ >> lib/uuid.h | 1 + >> ovsdb/ovsdb-client.c | 2 +- >> ovsdb/ovsdb-server.c | 30 ++- >> tests/test-jsonrpc.c | 2 +- >> utilities/ovs-vsctl.c | 2 +- >> vswitchd/ovs-vswitchd.c | 2 +- >> vtep/vtep-ctl.c | 2 +- >> 14 files changed, 717 insertions(+), 18 deletions(-) >> create mode 100644 lib/stream-replay.c >> > > Hi Ilya, > > This is really great and it will make performance testing easier indeed! > > This is not a complete review as I just played a bit with the > functionality and recorded and replayed some OVN DB executions but > something that I missed was a bit of documentation or maybe a short > example (apart from the samples in the commit logs).
Yeah. I know, this needs some chapter in documentation, otherwise it's not obvious how to use this functionality in a correct way. I'll work on this. > > Also, I'm wondering if there's any way possibility to expand this > feature and allow users to start/stop recording while the application is > already running. If that could be done in a reliable way it would make > it usable "online", potentially even in production. This is definitely a great feature to have. There are few points here that we should take care of in case we would like to have recording being enabled in runtime: 1. Randomness: we need to be able to re-initialize UUID generator in run-time, i.e. force it to start generation of non-random values. 2. Need to pause all the connections and record their initial states. 3. Need to dump current DB state to be able to start replay on it. 4. We'll probably need to resolve issues with time-based events first, or being able to disable them in runtime. Maybe it's not a full list, but this already will require some amount of work. However, this might be possible. One implementation idea is that we could have callbacks registered for each part of the code that interested in record/replay and these callbacks could be called to record initial states while enabling recording in runtime. E.g. one callback for UUID generator, one for each running stream, one for DB storage, etc. However, there are things that depends on a sequences of events that could be hard to restore without having exact same messages being processed in exact same order. For example, state of the ovsdb-idl state machine or something like that. We will probably end up storing big chunk of a process memory to keep things running as expected on replay. > > I have a few minor comments on the individual patches. > > Again, great work! Thanks for checking it out! > > Thanks, > Dumitru > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
