Re: [PATCH 03/15] add, reset: ensure submodules can be added or reset

[Brandon Williams]

On 07/26, Junio C Hamano wrote:
> Stefan Beller  writes:
> 
> > On Tue, Jul 25, 2017 at 2:39 PM, Brandon Williams  wrote:
> >> Commit aee9c7d65 (Submodules: Add the new "ignore" config option for
> >> diff and status) ...
> >
> > introduced in 2010, so quite widely spread.
> >
> >> ...  introduced the ignore configuration option for
> >> submodules so that configured submodules could be omitted from the
> >> status and diff commands.  Because this flag is respected in the diff
> >> machinery it has the unintended consequence of potentially prohibiting
> >> users from adding or resetting a submodule, even when a path to the
> >> submodule is explicitly given.
> >>
> >> Ensure that submodules can be added or set, even if they are configured
> >> to be ignored, by setting the `DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG` diff
> >> flag.
> >>
> >> Signed-off-by: Brandon Williams 
> >> ---
> >>  builtin/add.c   | 1 +
> >>  builtin/reset.c | 1 +
> >>  2 files changed, 2 insertions(+)
> >>
> >> diff --git a/builtin/add.c b/builtin/add.c
> >> index e888fb8c5..6f271512f 100644
> >> --- a/builtin/add.c
> >> +++ b/builtin/add.c
> >> @@ -116,6 +116,7 @@ int add_files_to_cache(const char *prefix,
> >> rev.diffopt.output_format = DIFF_FORMAT_CALLBACK;
> >> rev.diffopt.format_callback = update_callback;
> >> rev.diffopt.format_callback_data = 
> >> +   rev.diffopt.flags |= DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
> >
> >
> > This flag occurs once in the code base, with the comment:
> > /*
> >  * Unless the user did explicitly request a submodule
> >  * ignore mode by passing a command line option we do
> >  * not ignore any changed submodule SHA-1s when
> >  * comparing index and parent, no matter what is
> >  * configured. Otherwise we won't commit any
> >  * submodules which were manually staged, which would
> >  * be really confusing.
> >  */
> > int diff_flags = DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
> >
> > in prepare_commit, so commit ignores the .gitmodules file.
> >
> > This allows git-add to add ignored submodules, currently ignored submodules
> > would have to be added using the plumbing
> > git update-index --add --cacheinfo 16,$SHA1,
> 
> Let me play devil's advocate (as I have this suspicion that .ignore
> thing specific for submodule is probably misdesigned and certainly
> its implementation is backwards).  Is the primary use case for this
> .ignore thing to be able to do
> 
>   git add .
> 
> without having to worry about adding the submodule marked as such?  
> And if so, wouldn't it surprise these users who do use .ignore if
> "git add" suddenly started adding them?
> 
> I think the right tool to use these days for excluding some paths
> when adding all others is the negative pathspec; perhaps back when
> the .ignore thing was added, it didn't exist or not widely known?  
> 
> I suspect that it may result in a better system overall if we can
> deprecate and remove the submodule-specific .ignore thing.  At
> least, I think the DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG is backwards
> in that .ignore causes a submodule to be excluded from the diff by
> default and forces paths that care about differences to opt into the
> "override" thing, which is wrong---the specific UI thing that wants
> not to show them should instead opt into ignoring, while keeping the
> default not to special case such a flag that can only be set to a
> submodule path.

It looks like .ignore was added with aee9c7d65 (Submodules: Add the new
"ignore" config option for diff and status, 2010-08-06) in order to
ignore particular submodules with 'status' and 'diff' commands.  I don't
think it was intended to ignore submodules with commands like add and
reset.  Either way I agree that some of the things with most of the
submodules config seem a bit backwards and we may want to migrate away
from them completely as we begin to add more support for submodules into
the builtin commands.

> 
> > This makes sense, though a test demonstrating the change in behavior
> > would be nice, but git-add doesn't seem to change as it doesn't even load
> > the git modules config?
> >
> >> rev.max_count = 0; /* do not compare unmerged paths with stage #2 
> >> */
> >> run_diff_files(, DIFF_RACY_IS_MODIFIED);
> >> return !!data.add_errors;
> >> diff --git a/builtin/reset.c b/builtin/reset.c
> >> index 046403ed6..772d078b8 100644
> >> --- a/builtin/reset.c
> >> +++ b/builtin/reset.c
> >> @@ -156,6 +156,7 @@ static int read_from_tree(const struct pathspec 
> >> *pathspec,
> >> opt.output_format = DIFF_FORMAT_CALLBACK;
> >> opt.format_callback = update_index_from_diff;
> >> opt.format_callback_data = _to_add;
> >> +   opt.flags |= DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
> >
> > same here? Also as this is not failing any test, it may be worth adding one
> > to document the behavior of the 

Re: [PATCH 03/15] add, reset: ensure submodules can be added or reset

[Junio C Hamano]

Stefan Beller  writes:

> On Tue, Jul 25, 2017 at 2:39 PM, Brandon Williams  wrote:
>> Commit aee9c7d65 (Submodules: Add the new "ignore" config option for
>> diff and status) ...
>
> introduced in 2010, so quite widely spread.
>
>> ...  introduced the ignore configuration option for
>> submodules so that configured submodules could be omitted from the
>> status and diff commands.  Because this flag is respected in the diff
>> machinery it has the unintended consequence of potentially prohibiting
>> users from adding or resetting a submodule, even when a path to the
>> submodule is explicitly given.
>>
>> Ensure that submodules can be added or set, even if they are configured
>> to be ignored, by setting the `DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG` diff
>> flag.
>>
>> Signed-off-by: Brandon Williams 
>> ---
>>  builtin/add.c   | 1 +
>>  builtin/reset.c | 1 +
>>  2 files changed, 2 insertions(+)
>>
>> diff --git a/builtin/add.c b/builtin/add.c
>> index e888fb8c5..6f271512f 100644
>> --- a/builtin/add.c
>> +++ b/builtin/add.c
>> @@ -116,6 +116,7 @@ int add_files_to_cache(const char *prefix,
>> rev.diffopt.output_format = DIFF_FORMAT_CALLBACK;
>> rev.diffopt.format_callback = update_callback;
>> rev.diffopt.format_callback_data = 
>> +   rev.diffopt.flags |= DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
>
>
> This flag occurs once in the code base, with the comment:
> /*
>  * Unless the user did explicitly request a submodule
>  * ignore mode by passing a command line option we do
>  * not ignore any changed submodule SHA-1s when
>  * comparing index and parent, no matter what is
>  * configured. Otherwise we won't commit any
>  * submodules which were manually staged, which would
>  * be really confusing.
>  */
> int diff_flags = DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
>
> in prepare_commit, so commit ignores the .gitmodules file.
>
> This allows git-add to add ignored submodules, currently ignored submodules
> would have to be added using the plumbing
> git update-index --add --cacheinfo 16,$SHA1,

Let me play devil's advocate (as I have this suspicion that .ignore
thing specific for submodule is probably misdesigned and certainly
its implementation is backwards).  Is the primary use case for this
.ignore thing to be able to do

git add .

without having to worry about adding the submodule marked as such?  
And if so, wouldn't it surprise these users who do use .ignore if
"git add" suddenly started adding them?

I think the right tool to use these days for excluding some paths
when adding all others is the negative pathspec; perhaps back when
the .ignore thing was added, it didn't exist or not widely known?  

I suspect that it may result in a better system overall if we can
deprecate and remove the submodule-specific .ignore thing.  At
least, I think the DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG is backwards
in that .ignore causes a submodule to be excluded from the diff by
default and forces paths that care about differences to opt into the
"override" thing, which is wrong---the specific UI thing that wants
not to show them should instead opt into ignoring, while keeping the
default not to special case such a flag that can only be set to a
submodule path.

> This makes sense, though a test demonstrating the change in behavior
> would be nice, but git-add doesn't seem to change as it doesn't even load
> the git modules config?
>
>> rev.max_count = 0; /* do not compare unmerged paths with stage #2 */
>> run_diff_files(, DIFF_RACY_IS_MODIFIED);
>> return !!data.add_errors;
>> diff --git a/builtin/reset.c b/builtin/reset.c
>> index 046403ed6..772d078b8 100644
>> --- a/builtin/reset.c
>> +++ b/builtin/reset.c
>> @@ -156,6 +156,7 @@ static int read_from_tree(const struct pathspec 
>> *pathspec,
>> opt.output_format = DIFF_FORMAT_CALLBACK;
>> opt.format_callback = update_index_from_diff;
>> opt.format_callback_data = _to_add;
>> +   opt.flags |= DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
>
> same here? Also as this is not failing any test, it may be worth adding one
> to document the behavior of the "submodule..ignore" flag in tests?
>
>>
>> if (do_diff_cache(tree_oid, ))
>> return 1;
>> --
>> 2.14.0.rc0.400.g1c36432dff-goog
>>

Re: [PATCH 03/15] add, reset: ensure submodules can be added or reset

[Brandon Williams]

On 07/25, Stefan Beller wrote:
> On Tue, Jul 25, 2017 at 2:39 PM, Brandon Williams  wrote:
> > Commit aee9c7d65 (Submodules: Add the new "ignore" config option for
> > diff and status) ...
> 
> introduced in 2010, so quite widely spread.
> 
> > ...  introduced the ignore configuration option for
> > submodules so that configured submodules could be omitted from the
> > status and diff commands.  Because this flag is respected in the diff
> > machinery it has the unintended consequence of potentially prohibiting
> > users from adding or resetting a submodule, even when a path to the
> > submodule is explicitly given.
> >
> > Ensure that submodules can be added or set, even if they are configured
> > to be ignored, by setting the `DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG` diff
> > flag.
> >
> > Signed-off-by: Brandon Williams 
> > ---
> >  builtin/add.c   | 1 +
> >  builtin/reset.c | 1 +
> >  2 files changed, 2 insertions(+)
> >
> > diff --git a/builtin/add.c b/builtin/add.c
> > index e888fb8c5..6f271512f 100644
> > --- a/builtin/add.c
> > +++ b/builtin/add.c
> > @@ -116,6 +116,7 @@ int add_files_to_cache(const char *prefix,
> > rev.diffopt.output_format = DIFF_FORMAT_CALLBACK;
> > rev.diffopt.format_callback = update_callback;
> > rev.diffopt.format_callback_data = 
> > +   rev.diffopt.flags |= DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
> 
> 
> This flag occurs once in the code base, with the comment:
> /*
>  * Unless the user did explicitly request a submodule
>  * ignore mode by passing a command line option we do
>  * not ignore any changed submodule SHA-1s when
>  * comparing index and parent, no matter what is
>  * configured. Otherwise we won't commit any
>  * submodules which were manually staged, which would
>  * be really confusing.
>  */
> int diff_flags = DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
> 
> in prepare_commit, so commit ignores the .gitmodules file.
> 
> This allows git-add to add ignored submodules, currently ignored submodules
> would have to be added using the plumbing
> git update-index --add --cacheinfo 16,$SHA1,
> 
> This makes sense, though a test demonstrating the change in behavior
> would be nice, but git-add doesn't seem to change as it doesn't even load
> the git modules config?

I can add a comment to the code but its already being tested in the
submodule test suite.  The only reason this doesn't cause any changes
now is that the gitmodules config is never loaded, but that may
change if we decide to allow lazy-loading of the gitmodules file (like
the last couple patches in this series do).

-- 
Brandon Williams

Re: [PATCH 03/15] add, reset: ensure submodules can be added or reset

[Stefan Beller]

On Tue, Jul 25, 2017 at 2:39 PM, Brandon Williams  wrote:
> Commit aee9c7d65 (Submodules: Add the new "ignore" config option for
> diff and status) ...

introduced in 2010, so quite widely spread.

> ...  introduced the ignore configuration option for
> submodules so that configured submodules could be omitted from the
> status and diff commands.  Because this flag is respected in the diff
> machinery it has the unintended consequence of potentially prohibiting
> users from adding or resetting a submodule, even when a path to the
> submodule is explicitly given.
>
> Ensure that submodules can be added or set, even if they are configured
> to be ignored, by setting the `DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG` diff
> flag.
>
> Signed-off-by: Brandon Williams 
> ---
>  builtin/add.c   | 1 +
>  builtin/reset.c | 1 +
>  2 files changed, 2 insertions(+)
>
> diff --git a/builtin/add.c b/builtin/add.c
> index e888fb8c5..6f271512f 100644
> --- a/builtin/add.c
> +++ b/builtin/add.c
> @@ -116,6 +116,7 @@ int add_files_to_cache(const char *prefix,
> rev.diffopt.output_format = DIFF_FORMAT_CALLBACK;
> rev.diffopt.format_callback = update_callback;
> rev.diffopt.format_callback_data = 
> +   rev.diffopt.flags |= DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;


This flag occurs once in the code base, with the comment:
/*
 * Unless the user did explicitly request a submodule
 * ignore mode by passing a command line option we do
 * not ignore any changed submodule SHA-1s when
 * comparing index and parent, no matter what is
 * configured. Otherwise we won't commit any
 * submodules which were manually staged, which would
 * be really confusing.
 */
int diff_flags = DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;

in prepare_commit, so commit ignores the .gitmodules file.

This allows git-add to add ignored submodules, currently ignored submodules
would have to be added using the plumbing
git update-index --add --cacheinfo 16,$SHA1,

This makes sense, though a test demonstrating the change in behavior
would be nice, but git-add doesn't seem to change as it doesn't even load
the git modules config?

> rev.max_count = 0; /* do not compare unmerged paths with stage #2 */
> run_diff_files(, DIFF_RACY_IS_MODIFIED);
> return !!data.add_errors;
> diff --git a/builtin/reset.c b/builtin/reset.c
> index 046403ed6..772d078b8 100644
> --- a/builtin/reset.c
> +++ b/builtin/reset.c
> @@ -156,6 +156,7 @@ static int read_from_tree(const struct pathspec *pathspec,
> opt.output_format = DIFF_FORMAT_CALLBACK;
> opt.format_callback = update_index_from_diff;
> opt.format_callback_data = _to_add;
> +   opt.flags |= DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;

same here? Also as this is not failing any test, it may be worth adding one
to document the behavior of the "submodule..ignore" flag in tests?

>
> if (do_diff_cache(tree_oid, ))
> return 1;
> --
> 2.14.0.rc0.400.g1c36432dff-goog
>

[PATCH 03/15] add, reset: ensure submodules can be added or reset

[Brandon Williams]

Commit aee9c7d65 (Submodules: Add the new "ignore" config option for
diff and status) introduced the ignore configuration option for
submodules so that configured submodules could be omitted from the
status and diff commands.  Because this flag is respected in the diff
machinery it has the unintended consequence of potentially prohibiting
users from adding or resetting a submodule, even when a path to the
submodule is explicitly given.

Ensure that submodules can be added or set, even if they are configured
to be ignored, by setting the `DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG` diff
flag.

Signed-off-by: Brandon Williams 
---
 builtin/add.c   | 1 +
 builtin/reset.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/builtin/add.c b/builtin/add.c
index e888fb8c5..6f271512f 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -116,6 +116,7 @@ int add_files_to_cache(const char *prefix,
rev.diffopt.output_format = DIFF_FORMAT_CALLBACK;
rev.diffopt.format_callback = update_callback;
rev.diffopt.format_callback_data = 
+   rev.diffopt.flags |= DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
rev.max_count = 0; /* do not compare unmerged paths with stage #2 */
run_diff_files(, DIFF_RACY_IS_MODIFIED);
return !!data.add_errors;
diff --git a/builtin/reset.c b/builtin/reset.c
index 046403ed6..772d078b8 100644
--- a/builtin/reset.c
+++ b/builtin/reset.c
@@ -156,6 +156,7 @@ static int read_from_tree(const struct pathspec *pathspec,
opt.output_format = DIFF_FORMAT_CALLBACK;
opt.format_callback = update_index_from_diff;
opt.format_callback_data = _to_add;
+   opt.flags |= DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG;
 
if (do_diff_cache(tree_oid, ))
return 1;
-- 
2.14.0.rc0.400.g1c36432dff-goog