Hi Brad, Patrick, All...
I think I've understood this second problem. In summary, it is memory
related.
This is how I found the source of the problem:
1./ I copied and adapted the user application to run in another
cluster of ours. The idea was for me to understand the application
and run it myself to collect logs and so on...
2./ Once I submit it to this other cluster, every thing went fine. I
was hammering cephfs from multiple nodes without problems. This
pointed to something different between the two clusters.
3./ I've started to look better to the segmentation fault message,
and assuming that the names of the methods and functions do mean
something, the log seems related to issues on the management of
objects in cache. This pointed to a memory related problem.
4./ On the cluster where the application run successfully, machines
have 48GB of RAM and 96GB of SWAP (don't know why we have such a
large SWAP size, it is a legacy setup).
# top
top - 00:34:01 up 23 days, 22:21, 1 user, load average: 12.06,
12.12, 10.40
Tasks: 683 total, 13 running, 670 sleeping, 0 stopped, 0 zombie
Cpu(s): 49.7%us, 0.6%sy, 0.0%ni, 49.7%id, 0.1%wa, 0.0%hi,
0.0%si, 0.0%st
Mem: 49409308k total, 29692548k used, 19716760k free, 433064k
buffers
Swap: 98301948k total, 0k used, 98301948k free, 26742484k
cached
5./ I have noticed that ceph-fuse (in 10.2.2) consumes about 1.5 GB
of virtual memory when there is no applications using the filesystem.
7152 root 20 0 1108m 12m 5496 S 0.0 0.0 0:00.04
ceph-fuse
When I only have one instance of the user application running,
ceph-fuse (in 10.2.2) slowly rises with time up to 10 GB of memory
usage.
if I submit a large number of user applications simultaneously,
ceph-fuse goes very fast to ~10GB.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18563 root 20 0 10.0g 328m 5724 S 4.0 0.7 1:38.00
ceph-fuse
4343 root 20 0 3131m 237m 12m S 0.0 0.5 28:24.56
dsm_om_connsvcd
5536 goncalo 20 0 1599m 99m 32m R 99.9 0.2 31:35.46
python
31427 goncalo 20 0 1597m 89m 20m R 99.9 0.2 31:35.88 python
20504 goncalo 20 0 1599m 89m 20m R 100.2 0.2 31:34.29
python
20508 goncalo 20 0 1599m 89m 20m R 99.9 0.2 31:34.20 python
4973 goncalo 20 0 1599m 89m 20m R 99.9 0.2 31:35.70
python
1331 goncalo 20 0 1597m 88m 20m R 99.9 0.2 31:35.72
python
20505 goncalo 20 0 1597m 88m 20m R 99.9 0.2 31:34.46 python
20507 goncalo 20 0 1599m 87m 20m R 99.9 0.2 31:34.37 python
28375 goncalo 20 0 1597m 86m 20m R 99.9 0.2 31:35.52 python
20503 goncalo 20 0 1597m 85m 20m R 100.2 0.2 31:34.09
python
20506 goncalo 20 0 1597m 84m 20m R 99.5 0.2 31:34.42 python
20502 goncalo 20 0 1597m 83m 20m R 99.9 0.2 31:34.32 python
6./ On the machines where the user had the segfault, we have 16 GB
of RAM and 1GB of SWAP
Mem: 16334244k total, 3590100k used, 12744144k free, 221364k
buffers
Swap: 1572860k total, 10512k used, 1562348k free, 2937276k
cached
7./ I think what is happening is that once the user submits his sets
of jobs, the memory usage goes to the very limit on this type
machine, and the raise is actually to fast that ceph-fuse segfaults
before OOM Killer can kill it.
8./ We have run the user application in the same type of machines
but with 64 GB of RAM and 1GB of SWAP, and everything goes fine also
here.
So, in conclusion, our second problem (besides the locks which was fixed
by Pat patch) is the memory usage profile of ceph-fuse in 10.2.2 which
seems to be very different than what it was in ceph-fuse 9.2.0.
Are there any ideas how can we limit the virtual memory usage of
ceph-fuse in 10.2.2?
Cheers
Goncalo
On 07/08/2016 09:54 AM, Brad Hubbard wrote:
Hi Goncalo,
If possible it would be great if you could capture a core file for this with
full debugging symbols (preferably glibc debuginfo as well). How you do
that will depend on the ceph version and your OS but we can offfer help
if required I'm sure.
Once you have the core do the following.
$ gdb /path/to/ceph-fuse core.XXXX
(gdb) set pag off
(gdb) set log on
(gdb) thread apply all bt
(gdb) thread apply all bt full
Then quit gdb and you should find a file called gdb.txt in your
working directory.
If you could attach that file to http://tracker.ceph.com/issues/16610
Cheers,
Brad
On Fri, Jul 8, 2016 at 12:06 AM, Patrick Donnelly <[email protected]> wrote:
On Thu, Jul 7, 2016 at 2:01 AM, Goncalo Borges
<[email protected]> wrote:
Unfortunately, the other user application breaks ceph-fuse again (It is a
completely different application then in my previous test).
We have tested it in 4 machines with 4 cores. The user is submitting 16
single core jobs which are all writing different output files (one per job)
to a common dir in cephfs. The first 4 jobs run happily and never break
ceph-fuse. But the remaining 12 jobs, running in the remaining 3 machines,
trigger a segmentation fault, which is completely different from the other
case.
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (()+0x297fe2) [0x7f54402b7fe2]
2: (()+0xf7e0) [0x7f543ecf77e0]
3: (ObjectCacher::bh_write_scattered(std::list<ObjectCacher::BufferHead*,
std::allocator<ObjectCacher::BufferHead*> >&)+0x36) [0x7f5440268086]
4: (ObjectCacher::bh_write_adjacencies(ObjectCacher::BufferHead*,
std::chrono::time_point<ceph::time_detail::real_clock,
std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, long*,
int*)+0x22c) [0x7f5440268a3c]
5: (ObjectCacher::flush(long)+0x1ef) [0x7f5440268cef]
6: (ObjectCacher::flusher_entry()+0xac4) [0x7f5440269a34]
7: (ObjectCacher::FlusherThread::entry()+0xd) [0x7f5440275c6d]
8: (()+0x7aa1) [0x7f543ecefaa1]
9: (clone()+0x6d) [0x7f543df6893d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
This one looks like a very different problem. I've created an issue
here: http://tracker.ceph.com/issues/16610
Thanks for the report and debug log!
--
Patrick Donnelly
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW 2006
T: +61 2 93511937
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
