Tim Armstrong has uploaded a new patch set (#7). Change subject: IMPALA-3202: implement spill-to-disk in new buffer pool ......................................................................
IMPALA-3202: implement spill-to-disk in new buffer pool See https://goo.gl/0zuy97 for a high-level summary of the design. Unpinned pages can now be written to disk to free up memory. After unpinning, pages enter a "dirty" state. Each client initiates asynchronous writes for dirty page to free up memory to allocate more buffers. After the write completes, pages are "clean" and can be evicted from memory by any client that needs the buffer. This is implemented by moving pages between lists in ClientImpl (a new internal class that stores the client's state) and BufferPool. I/O: ---- The mechanics of I/O to scratch files is handled by the TmpFileMgr mechanisms introduced in earlier IMPALA-3202 patches. The decision to start a write I/O is based on two factors. First, each client maintains the invariant that bytes of the dirty pages should not exceed the unused reservation. This is to ensure that any client's reservation can always be fulfilled by evicting a clean page. Second, additional writes are initiated to proactively write data to disk, so that clean pages are available when needed. The buffer pool tries to keep enough writes in flight to keep all disks busy. The buffer pool avoids read I/O whenever possible: if an unpinned page is pinned again and its buffer is still in memory, no I/O is required to read back the data. Locking: -------- Concurrency is managed using client, page, and clean page list locks. The design uses intrusive doubly-linked lists to track pages. This patch adds a LockType parameter to InternalQueue and a FakeLock type to support a non-concurrent InternalList type that is more convenient for the buffer pool's locking scheme. Testing: -------- Added some basic unit tests to BufferPoolTest that exercise the new code paths. More test coverage will be added later with system tests and porting of tests from BufferedBlockMgrTest - see the draft test plan for IMPALA-3200. Change-Id: I8c6119a4557da853fefa02e17c49d8be0e9fbe01 --- M be/src/common/compiler-util.h A be/src/runtime/bufferpool/buffer-pool-internal.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/runtime/bufferpool/suballocator-test.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/aligned-new.h A be/src/util/fake-lock.h M be/src/util/internal-queue.h 10 files changed, 1,009 insertions(+), 297 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/5584/7 -- To view, visit http://gerrit.cloudera.org:8080/5584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c6119a4557da853fefa02e17c49d8be0e9fbe01 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong <[email protected]>
