[PATCH 0/8] Speed up cache loading time

2014-05-13 Thread Nguyễn Thái Ngọc Duy 
On Fri, May 9, 2014 at 5:27 PM, Duy Nguyen pclo...@gmail.com wrote:
 The below patch implements such a daemon to cache the index. It takes
 91ms and 377ms to load a 25MB index with and without the daemon. I use
 share memory instead of pipe, but the format is still on disk not
 in memory for simplicity. I think we're good even without in memory
 format.

Here is a better version (on top of split-index). I duplicated webkit
index 8 times to get its size to 199MB (version 2), close to what
Facebook tried last time [1]. read_cache() on index v2, v4, with the
daemon caching v2 and v4 respectively is 2994.861ms (199MB index
file), 2245.113ms (118MB) and 663.399ms and 880.935ms. The best number
is 4.5 times better the worst.

That is clocked at 800 MHz. A repository at this size deserves a
better CPU. At 2.5 GHz we spend 183.228ms on loading the index. A
reasonable number to me. If we scale other parts of git-status as well
as this, we should be able to make git status within 1 or 2 seconds.

The tested index does not have fully populated cache-tree so real
world numbers could be a bit higher.

[1] http://thread.gmane.org/gmane.comp.version-control.git/189776/focus=190156

Nguyễn Thái Ngọc Duy (8):
  read-cache: allow to keep mmap'd memory after reading
  unix-socket: stub impl. for platforms with no unix socket support
  daemonize: set a flag before exiting the main process
  Add read-cache--daemon for caching index and related stuff
  read-cache: try index data from shared memory
  read-cache--daemon: do not read index from shared memory
  read-cache: skip verifying trailing SHA-1 on cached index
  read-cache: inform the daemon that the index has been updated

 .gitignore |   1 +
 Documentation/config.txt   |   4 +
 Documentation/git-read-cache--daemon.txt (new) |  27 
 Makefile   |   8 +
 builtin/gc.c   |   2 +-
 cache.h|   7 +-
 config.c   |  12 ++
 config.mak.uname   |   1 +
 daemon.c   |   2 +-
 environment.c  |   1 +
 read-cache--daemon.c (new) | 208 +
 read-cache.c   | 116 +-
 setup.c|   4 +-
 submodule.c|   1 +
 unix-socket.h  |  18 +++
 wrapper.c  |  14 ++
 16 files changed, 414 insertions(+), 12 deletions(-)
 create mode 100644 Documentation/git-read-cache--daemon.txt
 create mode 100644 read-cache--daemon.c

-- 
1.9.1.346.ga2b5940

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/8] Speed up cache loading time

2014-05-13 Thread Stefan Beller 
 That is clocked at 800 MHz. A repository at this size deserves a
 better CPU. At 2.5 GHz we spend 183.228ms on loading the index. A
 reasonable number to me. If we scale other parts of git-status as well
 as this, we should be able to make git status within 1 or 2 seconds.


Which harddrive do you use? Traditional or SSDs?
Does have harddrive loading time significant impact here? (just a
guess/question)
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/8] Speed up cache loading time

2014-05-13 Thread Duy Nguyen 
On Tue, May 13, 2014 at 9:24 PM, Stefan Beller stefanbel...@gmail.com wrote:
 That is clocked at 800 MHz. A repository at this size deserves a
 better CPU. At 2.5 GHz we spend 183.228ms on loading the index. A
 reasonable number to me. If we scale other parts of git-status as well
 as this, we should be able to make git status within 1 or 2 seconds.


 Which harddrive do you use? Traditional or SSDs?

Traditional

 Does have harddrive loading time significant impact here? (just a
 guess/question)

In the hot cache case, I assume the index stays in OS cache anyway so
hard drive should not impact much (the other parts of git-status like
index refresh or untracked file listing is a different story and some
may fall out of cache). My laptop has 4G ram, with my repeated tests,
I guess the index (even 200mb) stayed in the cache (but did not really
verify it).
-- 
Duy
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html