Hi Ben, As discussed earlier, during the OVN meeting, I've noticed a new performance issue with ovn-northd-ddlog when running it against a database from one of our more recent scale tests:
http://people.redhat.com/~dceara/ovn-northd-ddlog-tests/20210708/ovnnb_db.db ovn-northd-ddlog uses 100% CPU and never really reaches the point to perform the first transaction to the Southbound. Memory usage is also very high, I stopped it at 45GB RSS. To test I did: SANDBOXFLAGS="--nbdb-source=/tmp/ovnnb_db.db --ddlog" make sandbox (gdb) thread 4 [Switching to thread 4 (Thread 0x7fc079ab7700 (LWP 2440622))] #0 0x00007fc07a361158 in _int_malloc (av=av@entry=0x7fc070000020, bytes=bytes@entry=33) at malloc.c:3862 3862 fwd = fwd->fd; (gdb) bt #0 0x00007fc07a361158 in _int_malloc (av=av@entry=0x7fc070000020, bytes=bytes@entry=33) at malloc.c:3862 #1 0x00007fc07a36196e in _int_realloc (av=av@entry=0x7fc070000020, oldp=oldp@entry=0x7fb8c4f4ff30, oldsize=oldsize@entry=32, nb=48) at malloc.c:4600 #2 0x00007fc07a362d86 in __GI___libc_realloc (oldmem=0x7fb8c4f4ff40, bytes=32) at malloc.c:3235 #3 0x00000000014ca970 in alloc::raw_vec::finish_grow () #4 0x00000000014caa6d in alloc::raw_vec::RawVec<T,A>::reserve () #5 0x0000000000e511a3 in ddlog_rt::string_append_str () #6 0x000000000079162f in types::__Rule_Flow_80::{{closure}}::__f () #7 0x00000000010ec3e8 in differential_dataflow::operators::join::Deferred<K,V1,V2,T,R1,R2,R3,C1,C2,M,D>::work::{{closure}} () #8 0x000000000108b924 in differential_dataflow::operators::join::Deferred<K,V1,V2,T,R1,R2,R3,C1,C2,M,D>::work () #9 0x0000000001a07adf in <timely::dataflow::operators::generic::builder_raw::OperatorCore<T,L> as timely::scheduling::Schedule>::schedule () #10 0x00000000010258cd in <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule () #11 0x00000000011da3da in timely::worker::Wrapper::step () #12 0x000000000164f04f in timely::worker::Worker<A>::step_or_park () #13 0x00000000017b971f in differential_datalog::program::worker::DDlogWorker::flush () #14 0x00000000017b8682 in differential_datalog::program::worker::DDlogWorker::run () #15 0x000000000157b22a in std::sys_common::backtrace::__rust_begin_short_backtrace () #16 0x0000000000ff6a37 in core::ops::function::FnOnce::call_once{{vtable-shim}} () #17 0x0000000001265dfa in std::sys::unix::thread::Thread::new::thread_start () #18 0x00007fc07a767432 in start_thread (arg=<optimized out>) at pthread_create.c:477 #19 0x00007fc07a3d76d3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Regards, Dumitru _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
