Adar Dembo has posted comments on this change. Change subject: KUDU-1931: fix SIGSEGV in file cache RandomAccessFile::memory_footprint() ......................................................................
Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/6359/1//COMMIT_MSG Commit Message: PS1, Line 22: For now, I've : added some coverage but we really should be running at least one end-to-end : test on all block managers. > can you create a jira to make some bm tests parameterized so that we run th The block manager tests (block_manager-test and block_manager-stress-test) _are_ parameterized on both; the issue is that neither test suites ever invoke ReadableBlock::memory_footprint(), because it's really something that is useful to test at this low level (short of just calling it to make sure it doesn't crash). Tests that do invoke memory_footprint() are even higher level (i.e. cfile-test, tablet-test, any cluster test) and never run against the FBM. I filed KUDU-1932. http://gerrit.cloudera.org:8080/#/c/6359/1/src/kudu/fs/block_manager-test.cc File src/kudu/fs/block_manager-test.cc: PS1, Line 438: LOG(INFO) > is this useful without further context? consider moving to VLOG It is useful: this is the only call to ReadableBlock::memory_footprint() in all of our block manager parameterized tests. That's why I added it, and that's why it shouldn't be VLOG(), where it won't run by default. I'll add a comment explaining this so no one will remove it by accident. http://gerrit.cloudera.org:8080/#/c/6359/1/src/kudu/util/file_cache-test.cc File src/kudu/util/file_cache-test.cc: PS1, Line 291: class RandomAccessFileCacheTest : public FileCacheTest<RandomAccessFile> { : }; > why not make this a typed test like the others? still covers what you want That won't work; memory_footprint() is only a method on RandomAccessFile, not RWFile. http://gerrit.cloudera.org:8080/#/c/6359/1/src/kudu/util/file_cache.cc File src/kudu/util/file_cache.cc: PS1, Line 350: // Some anecdotal memory measurements taken inside gdb: : // - glibc 2.23 malloc_usable_size() on make_shared<FileType>: 88 bytes. : // - tcmalloc malloc_usable_size() on make_shared<FileType>: 96 bytes. : // - sizeof(std::shared_ptr::_Sp_counted_base): 16 bytes. : // - sizeof(*this): 72 bytes. > would making the descriptor inherit from enable_shared_from this help? alte _Sp_counted_base isn't portable: there's no class with the same name in libc++. I'm not even sure if libc++'s shared_ptr control block is the same size; I took a quick glance at the code and its implementation is quite different. I don't think it's worth the effort to conditionally do something for libc++ and something else for libstdc++. I first experimented with enable_shared_from_this() and saw some interesting results. Within memory_footprint(), the address was still offset by the size of the shared_ptr control block. But instead of crashing, malloc_usable_size() returned a garbage value (-16). The code I was using looked like this: size_t memory_footprint() const override { - return kudu_malloc_usable_size(this) + + return kudu_malloc_usable_size(shared_from_this().get()) + once_.memory_footprint_excluding_this() + base_.filename().capacity(); } It makes sense that enable_shared_from_this doesn't help. Yes, it lets you construct shared_ptrs using 'this', but those shared_ptrs are stack objects, and calling .get() yields the same object pointer as if enable_shared_from_this wasn't in use, one where the base of the allocation is actually 16 bytes prior. -- To view, visit http://gerrit.cloudera.org:8080/6359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I9e57c06dd35252d03729c10ff64fe43d79288513 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Adar Dembo <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Dan Burkert <[email protected]> Gerrit-Reviewer: David Ribeiro Alves <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-HasComments: Yes
