On Thu, Mar 22, 2012 at 10:24 PM, Nikolay Samofatov <nikolay.samofa...@red-soft.biz> wrote: > Hello, All! > > We completed (conditional) implementation of lock manager buffer > 2 GB and > put it into production. > > The production system tends to be 100% CPU bound now (as there is a few days > back-log of work). > All 24 cores are busy, 450 GB RAM utilized for one database. > > Roughly ~50% of CPU time is spent in Linux kernel on average and lock manager > mutex wait hovers > around 40-50%. > > Live kernel profiler output of loaded system is as follows (generated with > OProfile): > > 7073900 3.6298 vmlinux vmlinux > copy_user_generic_string > 5554648 2.8502 vmlinux vmlinux __wake_up > 4695015 2.4091 vmlinux vmlinux > prepare_to_wait_exclusive > 4670651 2.3966 vmlinux vmlinux > futex_wait_setup > 3988807 2.0468 vmlinux vmlinux futex_wake > 2006843 1.0298 vmlinux vmlinux intel_idle > 1321253 0.6780 vmlinux vmlinux schedule > 1022927 0.5249 vmlinux vmlinux put_page > 1001815 0.5141 vmlinux vmlinux > finish_wait > 735902 0.3776 vmlinux vmlinux > try_to_wake_up > 716559 0.3677 vmlinux vmlinux > _atomic_dec_and_lock > 674752 0.3462 vmlinux vmlinux get_page > 646537 0.3318 vmlinux vmlinux > tick_nohz_stop_sched_tick > 628516 0.3225 vmlinux vmlinux > task_rq_lock > 594621 0.3051 vmlinux vmlinux > update_curr > 583779 0.2996 vmlinux vmlinux > native_write_msr_safe > 527673 0.2708 vmlinux vmlinux > select_nohz_load_balancer > 504132 0.2587 vmlinux vmlinux > get_futex_key > 460908 0.2365 vmlinux vmlinux > __audit_syscall_exit > 447328 0.2295 vmlinux vmlinux > enqueue_hrtimer > 443841 0.2277 vmlinux vmlinux > __wake_up_bit > 429380 0.2203 vmlinux vmlinux > thread_return > 401287 0.2059 vmlinux vmlinux > __switch_to > 397639 0.2040 vmlinux vmlinux > update_cfs_shares > 381845 0.1959 vmlinux vmlinux > rb_get_reader_page > 377837 0.1939 vmlinux vmlinux ktime_get > 371547 0.1907 vmlinux vmlinux > native_read_tsc > 369331 0.1895 vmlinux vmlinux > radix_tree_lookup_slot > > > This demonstrates that to scale effectively past 24-40 cores per node, > Firebird needs to implement > shared buffer cache. > Lock manager mutex operations and copying from file system cache to userspace > emerge as hard bottleneck. > > With lock manager change to 64-bit one 24 core node handles ~10 million user > transactions daily. > reliably, at times having >1500 concurrent INSERT/UPDATE requests. > > Shall we merge lock manager changes?
Another thing to try is to use a kernel > 3.2 if possible Some of the write locking and scalability issues were solved in kernel 3.2 http://news.ycombinator.com/item?id=3793973 http://www.reddit.com/r/Database/comments/rruh1/postgresql_did_i_say_32_cores_how_about_64/ ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
