Re: [PATCH 1/1] fsmonitor: don't fill bitmap with entries to be removed

2019-10-07 Thread William Baker 
On 10/3/19 4:36 PM, Junio C Hamano wrote:

>> +if (pos >= istate->cache_nr)
>> +BUG("fsmonitor_dirty has more entries than the index 
>> (%"PRIuMAX" >= %"PRIuMAX")",
>> +(uintmax_t)pos, (uintmax_t)istate->cache_nr);
> 
> This is how we show size_t values without using "%z" that we avoid,
> but are "pos" and 'cache_nr" size_t or ssize_t?  I thought they are
> plain boring unsigned, so shouldn't we use the plain boring "%u"
> without casting?
> 
> The same comment applies to other uses of uintmax_t cast in this
> patch.
> 

Thanks for catching this.  I will update these BUGs in the next
patch to avoid casting.

>> +# Use test files that start with 'z' so that the entries being added
>> +# and removed appear at the end of the index.
> 
> In other words, future developers are warned against adding entries
> to and leaving them in the index that sort later than z100 in new
> tests they add before this point.  Is the above wording clear enough
> to tell them that, I wonder?
> 

You're understanding is correct, and I agree this comment could be
clearer.  I will fix this up in v2.

Thanks for the feedback!
William

Re: [PATCH 1/1] fsmonitor: don't fill bitmap with entries to be removed

2019-10-03 Thread Junio C Hamano 
"William Baker via GitGitGadget"  writes:

>  create mode 100755 t/t7519/fsmonitor-env
> ...
> + if (pos >= istate->cache_nr)
> + BUG("fsmonitor_dirty has more entries than the index 
> (%"PRIuMAX" >= %"PRIuMAX")",
> + (uintmax_t)pos, (uintmax_t)istate->cache_nr);

This is how we show size_t values without using "%z" that we avoid,
but are "pos" and 'cache_nr" size_t or ssize_t?  I thought they are
plain boring unsigned, so shouldn't we use the plain boring "%u"
without casting?

The same comment applies to other uses of uintmax_t cast in this
patch.

>  void fill_fsmonitor_bitmap(struct index_state *istate)
>  {
> - unsigned int i;
> + unsigned int i, skipped = 0;
>   istate->fsmonitor_dirty = ewah_new();
> - for (i = 0; i < istate->cache_nr; i++)
> - if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
> - ewah_set(istate->fsmonitor_dirty, i);
> + for (i = 0; i < istate->cache_nr; i++) {
> + if (istate->cache[i]->ce_flags & CE_REMOVE)
> + skipped++;
> + else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
> + ewah_set(istate->fsmonitor_dirty, i - skipped);
> + }
>  }

Matches the explanation in the proposed log message pretty well.
Good job.

> @@ -354,4 +354,16 @@ test_expect_success 'discard_index() also discards 
> fsmonitor info' '
>   test_cmp expect actual
>  '
>  
> +# Use test files that start with 'z' so that the entries being added
> +# and removed appear at the end of the index.

In other words, future developers are warned against adding entries
to and leaving them in the index that sort later than z100 in new
tests they add before this point.  Is the above wording clear enough
to tell them that, I wonder?

> +test_expect_success 'status succeeds after staging/unstaging ' '
> + test_commit initial &&
> + removed=$(test_seq 1 100 | sed "s/^/z/") &&

Thanks.

[PATCH 1/1] fsmonitor: don't fill bitmap with entries to be removed

2019-10-03 Thread William Baker via GitGitGadget 
From: William Baker 

While doing some testing with fsmonitor enabled I found
that git commands would segfault after staging and
unstaging an untracked file.  Looking at the crash it
appeared that fsmonitor_ewah_callback was attempting to
adjust bits beyond the bounds of the index cache.

Digging into how this could happen it became clear that
the fsmonitor extension must have been written with
more bits than there were entries in the index.  The
root cause ended up being that fill_fsmonitor_bitmap was
populating fsmonitor_dirty with bits for all entries in
the index, even those that had been marked for removal.

To solve this problem fill_fsmonitor_bitmap has been
updated to skip entries with the the CE_REMOVE flag set.
With this change the bits written for the fsmonitor
extension will be consistent with the index entries
written by do_write_index.  Additionally, BUG checks
have been added to detect if the number of bits in
fsmonitor_dirty should ever exceed the number of
entries in the index again.

Another option that was considered was moving the call
to fill_fsmonitor_bitmap closer to where the index is
written (and where the fsmonitor extension itself is
written).  However, that did not work as the
fsmonitor_dirty bitmap must be filled before the index
is split during writing.

Signed-off-by: William Baker 
---
 fsmonitor.c | 29 -
 t/t7519-status-fsmonitor.sh | 12 
 t/t7519/fsmonitor-env   | 24 
 3 files changed, 60 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env

diff --git a/fsmonitor.c b/fsmonitor.c
index 231e83a94d..fa0b96d84e 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -14,8 +14,13 @@ struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR);
 static void fsmonitor_ewah_callback(size_t pos, void *is)
 {
struct index_state *istate = (struct index_state *)is;
-   struct cache_entry *ce = istate->cache[pos];
+   struct cache_entry *ce;
+   
+   if (pos >= istate->cache_nr)
+   BUG("fsmonitor_dirty has more entries than the index 
(%"PRIuMAX" >= %"PRIuMAX")",
+   (uintmax_t)pos, (uintmax_t)istate->cache_nr);
 
+   ce = istate->cache[pos];
ce->ce_flags &= ~CE_FSMONITOR_VALID;
 }
 
@@ -50,17 +55,24 @@ int read_fsmonitor_extension(struct index_state *istate, 
const void *data,
}
istate->fsmonitor_dirty = fsmonitor_dirty;
 
+   if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+   BUG("fsmonitor_dirty has more entries than the index 
(%"PRIuMAX" > %"PRIuMAX")",
+   (uintmax_t)istate->fsmonitor_dirty->bit_size, 
(uintmax_t)istate->cache_nr);
+
trace_printf_key(&trace_fsmonitor, "read fsmonitor extension 
successful");
return 0;
 }
 
 void fill_fsmonitor_bitmap(struct index_state *istate)
 {
-   unsigned int i;
+   unsigned int i, skipped = 0;
istate->fsmonitor_dirty = ewah_new();
-   for (i = 0; i < istate->cache_nr; i++)
-   if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
-   ewah_set(istate->fsmonitor_dirty, i);
+   for (i = 0; i < istate->cache_nr; i++) {
+   if (istate->cache[i]->ce_flags & CE_REMOVE)
+   skipped++;
+   else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
+   ewah_set(istate->fsmonitor_dirty, i - skipped);
+   }
 }
 
 void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
@@ -71,6 +83,10 @@ void write_fsmonitor_extension(struct strbuf *sb, struct 
index_state *istate)
uint32_t ewah_size = 0;
int fixup = 0;
 
+   if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+   BUG("fsmonitor_dirty has more entries than the index 
(%"PRIuMAX" > %"PRIuMAX")",
+   (uintmax_t)istate->fsmonitor_dirty->bit_size, 
(uintmax_t)istate->cache_nr);
+
put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
strbuf_add(sb, &hdr_version, sizeof(uint32_t));
 
@@ -236,6 +252,9 @@ void tweak_fsmonitor(struct index_state *istate)
}
 
/* Mark all previously saved entries as dirty */
+   if (istate->fsmonitor_dirty->bit_size > 
istate->cache_nr)
+   BUG("fsmonitor_dirty has more entries than the 
index (%"PRIuMAX" > %"PRIuMAX")",
+   
(uintmax_t)istate->fsmonitor_dirty->bit_size, (uintmax_t)istate->cache_nr);
ewah_each_bit(istate->fsmonitor_dirty, 
fsmonitor_ewah_callback, istate);
 
/* Now mark the untracked cache for fsmonitor usage */
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index 81a375fa0f..85df54f07c 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -354,4 +354,16 @@ test_expect_success 'discard_index() also disca