Thanks! Here is the output after running /w pt-pmp: http://pastebin.com/uj1R0i4Q
Three things I have noticed: 1. using strace I see that 75% of the time the application is spent inside theses: futex, mmap, munmap 2. using pt-pmp the top hit is: ibis::array_t::reserve 3. using perf I see that we never utilize more than 3 cores (the box has 20+ cores, and we’re running 10 threads in this instance) After looking at ibis::array_t::reserve, it looks like ibis::fileManager::storage::storage is being called, which is using a mutex. Since we have threads executing fastbit in parallel I’m wondering if there is possibly some lock contention going on with the file manager? I am not currently setting fileManager.maxBytes, could this be part of the problem? Cheers, Sean On May 8, 2015, at 2:57 PM, Justin Swanhart <[email protected]<mailto:[email protected]>> wrote: Hi, You may be able use this tool to examine your application and see where it is waiting: http://www.percona.com/doc/percona-toolkit/2.2/pt-pmp.html http://www.percona.com/get/pt-pmp It uses gdb to gather stack traces and aggregates them together so that you can see where multiple threads are stuck at the same point of execution, such as a mutex or semaphore. It is kind of the opposite of 'perf', since it finds where things are waiting and doing nothing, rather than where things are consuming the most CPU. Full disclosure: I work for Percona. Regards, --Justin On Fri, May 8, 2015 at 10:28 AM, Sean McNamara <[email protected]<mailto:[email protected]>> wrote: Hey John- I had a fastbit question for you. We're running a process that uses N threads to query N partitions against fastbit in parallel. We're seeing some contention somewhere that appears to cause the threads to execute serially. Before I dig in further I though it would be worth asking you: is there some sort of mutex or synchronization that is happening inside fastbit that could cause it to only be able to execute a single query at a time within a given process? Thanks, Sean _______________________________________________ FastBit-users mailing list [email protected]<mailto:[email protected]> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected]<mailto:[email protected]> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
