Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-12 01:46, Ross Berteig wrote:

In any case, you don't generally want to do addremove and commit in a
single operation because that doesn't give you a chance to review (and
test) what it decided to add and remove before it is committed to
immutable history.


That's certainly true for a public open source project but I wouldn't do 
it for an internal one.



___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 23:41, Scott Robison wrote:

Okay, so you *do* want (or at least expected) the use of --ignore (in
the context of addremove) to "rm" files already being managed. Which
is not an unreasonable desire, certainly could make some work flows
easier. The addremove command was structured around the idea of "this
unmanaged file that exists in the working directory should be added at
the next commit, and this managed file that no longer exists in the
woking directory should be removed at the next commit".

In any case, the "unmanaged files" text is a good addition to both, I
think, since add can be used to add the contents of a directory.


Sounds great. Thanks.

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Ross Berteig]

On 4/11/2017 3:24 PM, Thomas wrote:

This is actually one of the sources I based my exclusion list on.
I added other files too. I replaced all # characters at the beginning 
of each line with semicolons, extracted the files like [Tt]umbs.db to 
Thumbs.db and thumbs.db, saved it, and let my batch file create a 
comma-separated list for the --ignore command line switch.


Which raises a documentation point. I just realized that the GLOB syntax 
that fossil understands is more powerful than fossil's documentation 
explicitly shows. The documentation is vague and only shows examples 
using the star wildcard. But it can do more


GLOB matching is implemented[1] (like many low level constructs) by 
calling into SQLite. In this case, the SQLite GLOB operator is used to 
do the match. The SQLite documentation[2] says "The GLOB operator is 
similar to LIKE but uses the Unix file globbing syntax for its 
wildcards. Also, GLOB is case sensitive, unlike LIKE.  The infix 
GLOB operator is implemented by calling the function glob(Y,X) and can 
be modified by overriding that function."


It also says[3] "The glob(X,Y) function is equivalent to the expression 
"Y GLOB X". Note that the X and Y arguments are reversed in the glob() 
function relative to the infix GLOB operator. If the 
sqlite3_create_function() interface is used to override the glob(X,Y) 
function with an alternative implementation then the GLOB operator will 
invoke the alternative implementation."


It doesn't actually define what it meant by "Unix file globbing syntax", 
but one could easily assume that meant the obvious use of '*' and '?'. 
But all Unix shells also support character classes, and some support 
even more features that could be lumped in with globbing (such as '~' 
for $HOME). Various on-line tutorials (not official SQLite 
documentation) say that [abc] character classes work too. I had to 
resort to the SQLite source to find a definitive definition[4]:


Globbing rules:
  '*'   Matches any sequence of zero or more characters.
  '?'   Matches exactly one character.
 [...]  Matches one character from the enclosed list of
characters.
 [^...] Matches one character not in the enclosed list.
With the [...] and [^...] matching, a ']' character can be included
in the list by making it the first character after '[' or '^'.  A
range of characters can be specified using '-'.  Example:
"[a-z]" matches any single lower-case letter.  To match a '-', make
it the last character in the list.

So the examples from the gitignore project can be used as is if comments 
are stripped out. Adding support for comments to the glob parser in 
fossil might not be that difficult, I may look into that if I get time 
as it is something I've wanted in the past.You mentioned changing the 
git comments into semicolons. I'm not sure that does anything at all in 
fossil. If it does, that should be documented too.


Something I definitely will do (unless another developer beats me to the 
punch) is figure out a good place to drop the definition of what a glob 
can be into the documentation. There's probably more than one place that 
makes sense, but I don't want to repeat it in every command that 
mentions supporting one of the glob settings or has an option that uses 
a glob, so I'm looking for the sensible best place to put it.


[1]: https://www.fossil-scm.org/index.html/artifact/2209933ddc94df7e
[2]: https://sqlite.org/lang_expr.html
[3]: https://sqlite.org/lang_corefunc.html#glob
[4]: http://www.sqlite.org/src/artifact/9d52522cc8ae7f5c?ln=616-631

--
Ross Berteig   r...@cheshireeng.com
Cheshire Engineering Corp.   http://www.CheshireEng.com/
+1 626 303 1602

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Ross Berteig]

On 4/11/2017 3:24 PM, Thomas wrote:

On 2017-04-11 23:09, Ross Berteig wrote:

On 4/10/2017 11:48 AM, Thomas wrote:


Actually, I got a batch file that reads the file filter settings from
another file and creates the binary-glob and the ignore-glob files on
the fly before an addremove and a commit (crlf-glob is not created and
only contains an asterisk now).


Why do this on every commit?


That's a legit question. The reason is that at first I only learned 
about the command line swiches, that take the list comma separated.
My files however contained a mask on each line, basically the same way 
the .fossil-settings files are outlined. I only learned about the 
.fossil-settings folder later here in the mailing list, hence I just 
quickly changed the batch file to create the required files.


The reason why I had my files in this format is because I assembled my 
lists from sources on the web with common file name extensions, file 
and folder names.




The whole point of the versioned settings is that they can be set once
and stored, and are carried with the repository into every working
folder on fossil open.


The only differences to my file is that it's got a different filename 
extension for which Notepad++ allows me to insert syntax-highlighted 
coments, and that it sorts the list. That's all.




You figure out what files never need saving, and put globs in
ignore-glob that match them. Note that the globs are matched on the
whole pathname in the working folder, so you can exclude entire folders
too if that is handy. Which is nice for things like the Release and
Debug folders that VS tends to create in its projects. You will want to
work out carefully what files your IDE needs to be treated like source
code, which of those are binary (a stupid mistake made by too many IDEs
is to use binary files for project configuration which makes using
version control harder than necessary), and which files are build 
products.


Something like this is my usual starting point for ignore-glob for a lot
of projects. I usually create ignore-glob right after creating the
repository and before checking things in. I always include globs for
whatever backup files get created by my text editors, all the build
products I can identify, and if practical, any folders created by 
building.


~*,*.bak
*.o,*.d
*.obj,*.exe,*.dll
*.bin,*.hex
*Build*/*
*build*/*
ship/*
*.zip,*.tar*

The fossil extras command is useful for finding things that belong in
ignore-glob. It lists the files that fossil addremove would add or
fossil clean would delete.

IDEs come with an extra burden. They always seem to be written by people
that believe there is no world outside of the IDE, and especially no
version control. Worse, they also seem to be hostile to all the other
IDE developers, and highly resistant to any sort of standardized naming
of things. On the positive side, there is this project at Github where
people are tracking what to tell Git to ignore. The syntax isn't
identical for fossil, but the data is still valuable.

  https://github.com/github/gitignore


This is actually one of the sources I based my exclusion list on.
I added other files too. I replaced all # characters at the beginning 
of each line with semicolons, extracted the files like [Tt]umbs.db to 
Thumbs.db and thumbs.db, saved it, and let my batch file create a 
comma-separated list for the --ignore command line switch.


My key point here is that this is a one-time setup for a repository. 
Possibly a second time when you discover something you missed, or add a 
new tool to the project. Not something you should need to edit constantly.


I've been using fossil for years with projects in C and Lua, for many 
different target architectures, with and without several IDEs, and even 
for documentation-only projects with text in Markdown or LaTeX where 
builds produce PDFs.


I essentially never use fossil addremove. It is not part of my workflow 
at all.


I do use fossil extras and fossil changed as part of my habitual review 
before I commit changes. The extras command tells me if I forgot about a 
new file. The changes command tells me what files I actually changed. In 
both cases, I want to know that I'm not committing large swaths of 
unexpected files.


To my mind, addremove is simply too powerful since it assumes that 
missing and new files are intentionally missing and new. That means that 
fossil addremove will also mis-handle a renamed file, which would 
otherwise be correctly handled by using fossil mv to tell fossil the 
file changed names.


I actually almost never use fossil clean for much the same reason: it is 
too powerful, I don't want it breaking my IDE just because after a tool 
upgrade it suddenly is writing some new intermediate file I hadn't yet 
heard of.


I tune the ignore-glob based on what fossil extras is saying, generally 
aiming for extras to be an empty list.


I always use .fossil-settings for complex settings like the various 
globs. It is too 

Re: [fossil-users] Issue with ignore-glob

[Kees Nuyt]

[Default] On Tue, 11 Apr 2017 21:25:31 +0100, Thomas
 wrote:

>On 2017-04-11 21:04, Ross Berteig wrote:
>> The fossil addremove command is a convenience command that scans the
>> tree, obeying some of the glob settings, and applies fossil add and
>> fossil forget command as needed to make the list of files now in the
>> repository consistent with the settings and the directory tree in the
>> working checkout.
>
> That's right. This means it defeats the idea of using rm beforehand, as 
> addremove invokes another add for the very same file later on, adding it 
> again. That's why this doesn't work. I've tried that too.

In your use case, I wouldn't blindly addremove and commit,
addremove has a --dry-run option for a reason.
Use it, setup the required ignore filters, versioned or not.
Then fossil addremove --dry-run until satisfied.
After the final fossil addremove, check the "fossil ls" list of
managed files; "fossil forget" any files you don't want before
finally committing.

I would say you were in a hurry and only made a beginners
mistake, like we all did, and there is no need to change
anything. I admit the insignificance of --ignore and the
ignore-glob setting on managed files could be stressed a bit
more in the docs.

-- 
Kind Regards,
Kees Nuyt
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Tue, Apr 11, 2017 at 4:11 PM, Thomas  wrote:
> On 2017-04-11 22:51, Scott Robison wrote:
>>
>> On Tue, Apr 11, 2017 at 3:39 PM, Scott Robison 
>> wrote:
>>>
>>> On Tue, Apr 11, 2017 at 3:21 PM, Thomas  wrote:

 On 2017-04-11 22:11, Thomas wrote:

 add
--ignoreIgnore unmanaged files matching
 patterns from the comma separated list of
 glob patterns.
--exclude   Exclude files matching patterns from
 the comma separated list of glob patterns.

 The same for addremove, I guess.
>>>
>>>
>>> I've updated the documentation for --ignore for add and addremove.
>>> Adding exclude is more than I have time for at this moment. Baby
>>> steps.
>>
>>
>> I still think there is a fundamental misunderstanding about addremove.
>
>
> I think not a misunderstanding but I made a minor error. There's nothing
> wrong with the add command as it currently is. This --exclude would only
> apply to addremove because the add command has it already (= rm or delete).

{snipped}

Okay, so you *do* want (or at least expected) the use of --ignore (in
the context of addremove) to "rm" files already being managed. Which
is not an unreasonable desire, certainly could make some work flows
easier. The addremove command was structured around the idea of "this
unmanaged file that exists in the working directory should be added at
the next commit, and this managed file that no longer exists in the
woking directory should be removed at the next commit".

In any case, the "unmanaged files" text is a good addition to both, I
think, since add can be used to add the contents of a directory.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 23:09, Ross Berteig wrote:

On 4/10/2017 11:48 AM, Thomas wrote:


Actually, I got a batch file that reads the file filter settings from
another file and creates the binary-glob and the ignore-glob files on
the fly before an addremove and a commit (crlf-glob is not created and
only contains an asterisk now).


Why do this on every commit?


That's a legit question. The reason is that at first I only learned 
about the command line swiches, that take the list comma separated.
My files however contained a mask on each line, basically the same way 
the .fossil-settings files are outlined. I only learned about the 
.fossil-settings folder later here in the mailing list, hence I just 
quickly changed the batch file to create the required files.


The reason why I had my files in this format is because I assembled my 
lists from sources on the web with common file name extensions, file and 
folder names.




The whole point of the versioned settings is that they can be set once
and stored, and are carried with the repository into every working
folder on fossil open.


The only differences to my file is that it's got a different filename 
extension for which Notepad++ allows me to insert syntax-highlighted 
coments, and that it sorts the list. That's all.




You figure out what files never need saving, and put globs in
ignore-glob that match them. Note that the globs are matched on the
whole pathname in the working folder, so you can exclude entire folders
too if that is handy. Which is nice for things like the Release and
Debug folders that VS tends to create in its projects. You will want to
work out carefully what files your IDE needs to be treated like source
code, which of those are binary (a stupid mistake made by too many IDEs
is to use binary files for project configuration which makes using
version control harder than necessary), and which files are build products.

Something like this is my usual starting point for ignore-glob for a lot
of projects. I usually create ignore-glob right after creating the
repository and before checking things in. I always include globs for
whatever backup files get created by my text editors, all the build
products I can identify, and if practical, any folders created by building.

~*,*.bak
*.o,*.d
*.obj,*.exe,*.dll
*.bin,*.hex
*Build*/*
*build*/*
ship/*
*.zip,*.tar*

The fossil extras command is useful for finding things that belong in
ignore-glob. It lists the files that fossil addremove would add or
fossil clean would delete.

IDEs come with an extra burden. They always seem to be written by people
that believe there is no world outside of the IDE, and especially no
version control. Worse, they also seem to be hostile to all the other
IDE developers, and highly resistant to any sort of standardized naming
of things. On the positive side, there is this project at Github where
people are tracking what to tell Git to ignore. The syntax isn't
identical for fossil, but the data is still valuable.

  https://github.com/github/gitignore


This is actually one of the sources I based my exclusion list on.
I added other files too. I replaced all # characters at the beginning of 
each line with semicolons, extracted the files like [Tt]umbs.db to 
Thumbs.db and thumbs.db, saved it, and let my batch file create a 
comma-separated list for the --ignore command line switch.



___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 22:51, Scott Robison wrote:

On Tue, Apr 11, 2017 at 3:39 PM, Scott Robison  wrote:

On Tue, Apr 11, 2017 at 3:21 PM, Thomas  wrote:

On 2017-04-11 22:11, Thomas wrote:

add
   --ignoreIgnore unmanaged files matching
patterns from the comma separated list of
glob patterns.
   --exclude   Exclude files matching patterns from
the comma separated list of glob patterns.

The same for addremove, I guess.


I've updated the documentation for --ignore for add and addremove.
Adding exclude is more than I have time for at this moment. Baby
steps.


I still think there is a fundamental misunderstanding about addremove.


I think not a misunderstanding but I made a minor error. There's nothing 
wrong with the add command as it currently is. This --exclude would only 
apply to addremove because the add command has it already (= rm or delete).




You create a new repo. You run fossil addremove and get everything in
there, including things you don't want because you've never setup
ignore globs. You commit.

Now you know you need ignore, so you type fossil addremove --ignore
'*.whatever'. Files matching *.whatever that are already in the repo
will not be removed from the repo. Files matching *.whatever that are
not in the repo will not be added to the repo. fossil addremove does
not commit files, it just prepares to add or remove them in the next
commit.


Yes. That's how I understood it.
Since it doesn't make much sense to just run addremove alone, I always 
run addremove and then commit immediately. I got a batch file that does 
this.




Has your point been that running "addremove --ignore '*.whatever'"
treat files matching those patterns as missing so they will be removed
at the next commit? I don't think so because you said you didn't
expect it to remove the files. But you are surprised that --ignore
isn't ignoring files in addremove, and according to each and every
test I've run, it does ignore those unmanaged files so they are not
added.


It (--ignore) only ignores unmanaged files. But basically, what I 
expected it to do is remove these files from the next commit, as if I 
had removed them with rm or delete manually. Of course they're still in 
the repository in the previous revision.


So, the issue only arises with addremove. Since after the first 
addremove and commit all files are in the repository in the first 
revision, --ignore has no function for consecutive invocations of 
addremove and commit, as all files are now managed. The files happily 
appear in the commit comment but you won't get any clue about why.




Maybe you could try a new repository from scratch and try it the way I
described in my earlier posts. Or maybe you could give me the exact
command line you are running, and the fossil version output, and even
the OS / shell you are using so that I can see if there is something
else surprising about this.


It's exactly the way you described it.

- New repository.
- open
- Copy project files in the checked out folder structure.
- addremove + commit
- ui to see that lots of files are in the repository that shouldn't be there
- Use --ignore with the next addremove + commit cycle and see that it 
doesn't work as you'd expect



___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Ross Berteig]

On 4/10/2017 11:48 AM, Thomas wrote:


Actually, I got a batch file that reads the file filter settings from 
another file and creates the binary-glob and the ignore-glob files on 
the fly before an addremove and a commit (crlf-glob is not created and 
only contains an asterisk now).


Why do this on every commit?

The whole point of the versioned settings is that they can be set once 
and stored, and are carried with the repository into every working 
folder on fossil open.


You figure out what files never need saving, and put globs in 
ignore-glob that match them. Note that the globs are matched on the 
whole pathname in the working folder, so you can exclude entire folders 
too if that is handy. Which is nice for things like the Release and 
Debug folders that VS tends to create in its projects. You will want to 
work out carefully what files your IDE needs to be treated like source 
code, which of those are binary (a stupid mistake made by too many IDEs 
is to use binary files for project configuration which makes using 
version control harder than necessary), and which files are build products.


Something like this is my usual starting point for ignore-glob for a lot 
of projects. I usually create ignore-glob right after creating the 
repository and before checking things in. I always include globs for 
whatever backup files get created by my text editors, all the build 
products I can identify, and if practical, any folders created by building.


~*,*.bak
*.o,*.d
*.obj,*.exe,*.dll
*.bin,*.hex
*Build*/*
*build*/*
ship/*
*.zip,*.tar*

The fossil extras command is useful for finding things that belong in 
ignore-glob. It lists the files that fossil addremove would add or 
fossil clean would delete.


IDEs come with an extra burden. They always seem to be written by people 
that believe there is no world outside of the IDE, and especially no 
version control. Worse, they also seem to be hostile to all the other 
IDE developers, and highly resistant to any sort of standardized naming 
of things. On the positive side, there is this project at Github where 
people are tracking what to tell Git to ignore. The syntax isn't 
identical for fossil, but the data is still valuable.


  https://github.com/github/gitignore

Now I wonder how hard it would be to add support for comments to 
fossil's versioned settings files, as well as the [Aa] to the globs. 
Comments are probably easy, but I suspect the glob syntax is tied to 
SQLite. I may look into that.

--

Ross Berteig   r...@cheshireeng.com
Cheshire Engineering Corp.   http://www.CheshireEng.com/
+1 626 303 1602

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 22:21, Thomas wrote:

On 2017-04-11 22:11, Thomas wrote:

On 2017-04-11 22:01, Scott Robison wrote:

I was thinking about that earlier (well, a warning, not an error,
which presumes you can't continue). Then the questions I put above
came into my mind so I didn't bring it up. What would you suggest
calling the command?


Change the text for --ignore and .fossil-settings/ignore-glob to:
unmanaged matching files to ignore

Add another switch --exclude and .fossil-settings/exclude-glob and
explain that this applies to all, managed and unmanaged files
unconditionally.

But right after that you'd have to start drinking beer as my offer still
stands ;-)



add
   --ignoreIgnore unmanaged files matching
patterns from the comma separated list of
glob patterns.
   --exclude Exclude files matching patterns from
the comma separated list of glob patterns.

The same for addremove, I guess.


Oups, sorry. Not for add. Only for addremove.
-- exclude for add already exists in the form of rm or delete, right?



___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[David Mason]

I've been using fossil for several years now, so when I set up a new fossil
my first operation is to copy over an existing .fossil-settings and commit,
so I haven't been exposed to this problem for a while.  I certainly
remember when I first used it that it did some unexpected things. Perhaps
if there is no ignore-glob or --ignore fossil should give an error pointing
out that it will put *ALL* files into the repo. (When I say error here, and
in the earlier comment) I meant a prompt that you can override (like a
cr/lf file).)

I just did the following:

: /tmp ; fossil new foo.fossil
project-id: 4718320ad8895a1e5b2878f7d88ebb6b349c3777
server-id:  bf202e7dcd2d960fc5a4ead57fcd3b0ce6edb3d1
admin-user: dmason (initial password is "8bd847")
: /tmp ; cd foo
: /tmp/foo ; fossil open ../foo.fossil
project-name: 
repository:   /private/tmp/foo/../foo.fossil
local-root:   /private/tmp/foo/
config-db:/Users/dmason/.fossil
project-code: 4718320ad8895a1e5b2878f7d88ebb6b349c3777
checkout: 7ea66bcf3c7871acfb653662f743aa8b027d0dcc 2017-04-11 21:30:40
UTC
tags: trunk
comment:  initial empty check-in (user: dmason)
check-ins:1
: /tmp/foo ; echo 'int main(){}' >foo.c
: /tmp/foo ; make foo.o
cc-c -o foo.o foo.c
: /tmp/foo ; make foo
cc   foo.o   -o foo
: /tmp/foo ; fs addremove
ADDED  foo
ADDED  foo.c
ADDED  foo.o
added 3 files, deleted 0 files
: /tmp/foo ; fossil ci -m new
./foo contains binary data. Use --no-warnings or the "binary-glob" setting
to disable this warning.
Commit anyhow (a=all/y/N)?

I would have it similarly tell you that you haven't set up "ignore-glob"
and that it will add everything and let you commit anyway, maybe with a
list of all the files it is committing. To make it go away, all you'd have
to do is ``touch .fossil-settings/ignore-blob``.

../Dave
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Tue, Apr 11, 2017 at 3:39 PM, Scott Robison  wrote:
> On Tue, Apr 11, 2017 at 3:21 PM, Thomas  wrote:
>> On 2017-04-11 22:11, Thomas wrote:
>>
>> add
>>--ignoreIgnore unmanaged files matching
>> patterns from the comma separated list of
>> glob patterns.
>>--exclude   Exclude files matching patterns from
>> the comma separated list of glob patterns.
>>
>> The same for addremove, I guess.
>
> I've updated the documentation for --ignore for add and addremove.
> Adding exclude is more than I have time for at this moment. Baby
> steps.

I still think there is a fundamental misunderstanding about addremove.

You create a new repo. You run fossil addremove and get everything in
there, including things you don't want because you've never setup
ignore globs. You commit.

Now you know you need ignore, so you type fossil addremove --ignore
'*.whatever'. Files matching *.whatever that are already in the repo
will not be removed from the repo. Files matching *.whatever that are
not in the repo will not be added to the repo. fossil addremove does
not commit files, it just prepares to add or remove them in the next
commit.

Has your point been that running "addremove --ignore '*.whatever'"
treat files matching those patterns as missing so they will be removed
at the next commit? I don't think so because you said you didn't
expect it to remove the files. But you are surprised that --ignore
isn't ignoring files in addremove, and according to each and every
test I've run, it does ignore those unmanaged files so they are not
added.

Maybe you could try a new repository from scratch and try it the way I
described in my earlier posts. Or maybe you could give me the exact
command line you are running, and the fossil version output, and even
the OS / shell you are using so that I can see if there is something
else surprising about this.
-- 
Scott Robison
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Tue, Apr 11, 2017 at 3:21 PM, Thomas  wrote:
> On 2017-04-11 22:11, Thomas wrote:
>>
>> On 2017-04-11 22:01, Scott Robison wrote:
>>>
>>> I was thinking about that earlier (well, a warning, not an error,
>>> which presumes you can't continue). Then the questions I put above
>>> came into my mind so I didn't bring it up. What would you suggest
>>> calling the command?
>>
>>
>> Change the text for --ignore and .fossil-settings/ignore-glob to:
>> unmanaged matching files to ignore
>>
>> Add another switch --exclude and .fossil-settings/exclude-glob and
>> explain that this applies to all, managed and unmanaged files
>> unconditionally.
>>
>> But right after that you'd have to start drinking beer as my offer still
>> stands ;-)
>
>
>
> add
>--ignoreIgnore unmanaged files matching
> patterns from the comma separated list of
> glob patterns.
>--exclude   Exclude files matching patterns from
> the comma separated list of glob patterns.
>
> The same for addremove, I guess.

I've updated the documentation for --ignore for add and addremove.
Adding exclude is more than I have time for at this moment. Baby
steps.
-- 
Scott Robison
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Tue, Apr 11, 2017 at 3:11 PM, Thomas  wrote:
> On 2017-04-11 22:01, Scott Robison wrote:
>>
>> On Tue, Apr 11, 2017 at 2:31 PM, David Mason  wrote:
>>>
>>> I think --ignore should give an error if the --ignore matches a file
>>> already
>>> in the repository.  The current behaviour is clearly somewhat ambiguous.
>>
> Change the text for --ignore and .fossil-settings/ignore-glob to:
> unmanaged matching files to ignore
>
> Add another switch --exclude and .fossil-settings/exclude-glob and explain
> that this applies to all, managed and unmanaged files unconditionally.
>
> But right after that you'd have to start drinking beer as my offer still
> stands ;-)

I'll be happy to enjoy a root beer with you. That's as far as I'll go. :)
-- 
Scott Robison
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Tue, Apr 11, 2017 at 3:04 PM, Thomas  wrote:
> On 2017-04-11 19:34, Scott Robison wrote:
>>
>> No, I try to explain why what you see isn't a design flaw, and
>> apparently fail. But I'll keep trying!
>
>
> Since I've never heard of any software that would not ignore files it is
> told to ignore you're going to have a hard time to convince me ;-)

I'll concede that the thoughts of the fossil devs heavily influence
how options work and what they think they should do. I would be
surprised if a file I told the system to track was not because of some
other set of files I wanted to ignore.

>>> Source code management != backup
>>
>>
>> Never said it was. Keeping an obj file in a repo because there is no
>> corresponding source from which to build it is valuable so that other
>> people can get access to all the build artifacts required.
>
>
> Keeping hundreds of megabytes of SQLite Visual Studio Intellisense databases
> is not something required to build the software. It is also not something
> desirable to delete on the originating machine just to satisfy the odd
> behaviour of the SCM. To be fair, Fossil does a very good job in only
> storing the compressed differences. Although the files are quite big, the
> repository only grew by about 10 or 12 MiB with each commit.

I never suggested anyone would want to track megabytes of such files.

> I believe that Fossil users are software developers and know what happens to
> object files they exclude. I believe they're smart enough to either rename
> object files they need in the repository if they set up exclusion filters,
> or that they would set up appropriate filters. They'd find out as soon as
> they test it, and they can find out with "fossil ui". In my opinion this is
> a very weak excuse. ;)

Or maybe they're smart enough to use the software as designed. #zing
#sorry #lowblow

I kid, really. I understand most of your points. There is definitely
room for confusion.

>>> I would like to emphasise that --ignore (or .fossil-settings\ignore-glob)
>>> is
>>> an _explicit_ command, clearly stating the user's desire for exlusion of
>>> these files, following the documentation.
>>> Silently ignoring this wish can't be the correct process.
>>
>>
>> No, it is an explicit command clearly stating the user's desire for
>> exclusion of these files *that are not already under source control*.
>
>
> That's not what the documentation says ;-)

To be fair, the documentation doesn't really say either way. One would
need a standards body to write documentation that has no issues. And
even then there are still issues because someone comes along later and
disagrees with what a set of words should be defined to mean.

Unless of course you're talking about the documentation in the source
code fragment you quoted earlier, in which case it did say that it was
only building a list of unmanaged files.

>> The fact that the user does not remember or did not realize they
>> issues conflicting commands does not mean that fossil should suddenly
>> stop tracking the file, or so it seems to me.
>>
>> If a file was previously added to a repository (indicating a desire to
>> keep track of modifications to the file), that is more important than
>> ignoring the file.
>
>
> Isn't it a natural thing that the first step everyone does when trying out a
> revision management system is commit everything they got as they haven't set
> up any exclusions at that time?

Not for me. I run changes and extras commands almost every time I run
addremove just to be sure there aren't any surprises, and if I find
any, I will update glob patterns as needed.

> I can't imagine that this is really its intended function. If it really is,
> then this should be marked in triple-bold and red with green and pink
> stripes in the docs.

One of my earliest replies was a concession that documentation may
need to be updated.

>>> File add.c on line 672 says:
>>>   /* step 1:
>>>   ** Populate the temp table "sfile" with the names of all unmanaged
>>>   ** files currently in the check-out, except for files that match the
>>>   ** --ignore or ignore-glob patterns and dot-files.  Then add all of
>>>   ** the files in the sfile temp table to the set of managed files.
>>>   */
>>> According to this, it seems it's a design flaw.
>>
>> The key words are at the end of line 673 and the beginning of line
>> 674: "unmanaged files". By definition, a managed file (one that has
>> been previously added to the repository) is not an unmanaged file.
>> Thus it is working as described.
>
> That's from the source code of add.c. I don't think many people skim through
> the source code when there's other accompanying documentation provided. ;-)

But you did. :) I agree it is not the place most users will go for
documentation. But it is awfully good evidence that it is working as
designed and intended. CHECKMATE! :)
-- 
Scott Robison
___
fossil-users mailing list

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 22:11, Thomas wrote:

On 2017-04-11 22:01, Scott Robison wrote:

I was thinking about that earlier (well, a warning, not an error,
which presumes you can't continue). Then the questions I put above
came into my mind so I didn't bring it up. What would you suggest
calling the command?


Change the text for --ignore and .fossil-settings/ignore-glob to:
unmanaged matching files to ignore

Add another switch --exclude and .fossil-settings/exclude-glob and
explain that this applies to all, managed and unmanaged files
unconditionally.

But right after that you'd have to start drinking beer as my offer still
stands ;-)



add
   --ignoreIgnore unmanaged files matching   
patterns from the comma separated list of   
glob patterns.
   --exclude Exclude files matching patterns from
the comma separated list of glob patterns.

The same for addremove, I guess.

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 19:34, Scott Robison wrote:

No, I try to explain why what you see isn't a design flaw, and
apparently fail. But I'll keep trying!


Since I've never heard of any software that would not ignore files it is 
told to ignore you're going to have a hard time to convince me ;-)




Source code management != backup


Never said it was. Keeping an obj file in a repo because there is no
corresponding source from which to build it is valuable so that other
people can get access to all the build artifacts required.


Keeping hundreds of megabytes of SQLite Visual Studio Intellisense 
databases is not something required to build the software. It is also 
not something desirable to delete on the originating machine just to 
satisfy the odd behaviour of the SCM. To be fair, Fossil does a very 
good job in only storing the compressed differences. Although the files 
are quite big, the repository only grew by about 10 or 12 MiB with each 
commit.


I believe that Fossil users are software developers and know what 
happens to object files they exclude. I believe they're smart enough to 
either rename object files they need in the repository if they set up 
exclusion filters, or that they would set up appropriate filters. They'd 
find out as soon as they test it, and they can find out with "fossil 
ui". In my opinion this is a very weak excuse. ;)




I would like to emphasise that --ignore (or .fossil-settings\ignore-glob) is
an _explicit_ command, clearly stating the user's desire for exlusion of
these files, following the documentation.
Silently ignoring this wish can't be the correct process.


No, it is an explicit command clearly stating the user's desire for
exclusion of these files *that are not already under source control*.


That's not what the documentation says ;-)



The fact that the user does not remember or did not realize they
issues conflicting commands does not mean that fossil should suddenly
stop tracking the file, or so it seems to me.

If a file was previously added to a repository (indicating a desire to
keep track of modifications to the file), that is more important than
ignoring the file.


Isn't it a natural thing that the first step everyone does when trying 
out a revision management system is commit everything they got as they 
haven't set up any exclusions at that time?


I actually expected that some defaults would have been applied already 
but that's not the case, which is of course good. All files and folders 
got committed, apart from empty directories. It turned out Fossil 
doesn't know about folders (according to the documentation), hence I 
just created an empty file in each empty folder.




A switch that doesn't work is either a huge design flaw or a bug. A --ignore
switch that doesn't ignore is a huge security bug (and a trap) too.


Ignoring does work as desired. It only applies to files that match the
pattern that are not already in source control.


Yepp, that's what we figured out now. ;)
Since the software is a single executable it comes without any ignore 
settings, hence it applies to precisely how many files after the first 
commit? Correct: Precisely _zero_! ;)


So, --ignore or .fossil-settings\ignore-glob have _no_ function at all 
at first. That's not a lot for a command-line switch that's supposed to 
ignore files. ;-)




Going back to my examples from yesterday: I had an ignore-glob of *.a.
I ran addremove and it ignored the a.a file, but not the b.b file. If
I ran add a.a, it warned me that it was in the ignore pattern, but
allowed me to add it anyway.


I agree that I would have seen a warning with add. However, with 
thousands of files in a folder, that command is not very efficient 
compared to addremove.




--ignore is not a "remove existing files from the repository rule"
switch. It is an "ignore unmanaged files that match a pattern" switch.


I wasn't and I am not expecting it to remove existing files from the 
repository. I'd expect it to simply ignore these files for the current 
commit.


Again, that switch is completely useless after a first commit with all 
files. It does absolutely nothing. I find it a bit strange that I'm 
supposed to be the first one to notice this. Fossil is not just a couple 
of weeks old.




Just because you can't see it doesn't mean it isn't there. It is an
intentional design to allow the ignoring of unmanaged files.


And since all files are managed after the first addremove and commit, 
and every consecutive invocation(s), there's no unmanaged files left, 
rendering the switch's function to no function at all.


I can't imagine that this is really its intended function. If it really 
is, then this should be marked in triple-bold and red with green and 
pink stripes in the docs.




File add.c on line 672 says:
  /* step 1:
  ** Populate the temp table "sfile" with the names of all unmanaged
  ** files currently in the check-out, except for files that match the
  ** --ignore or ignore-glob patterns and 

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Tue, Apr 11, 2017 at 2:31 PM, David Mason  wrote:
>
> On 11 April 2017 at 14:34, Scott Robison  wrote:
>>
>> No, it is an explicit command clearly stating the user's desire for
>> exclusion of these files *that are not already under source control*.
>> The fact that the user does not remember or did not realize they
>> issues conflicting commands does not mean that fossil should suddenly
>> stop tracking the file, or so it seems to me.
>>
>> If a file was previously added to a repository (indicating a desire to
>> keep track of modifications to the file), that is more important than
>> ignoring the file.
>
>
> I think --ignore should give an error if the --ignore matches a file already
> in the repository.  The current behaviour is clearly somewhat ambiguous.

It would be relatively easy to warn if a modified file matches the
glob. Would you want it to search an entire checkout each and every
time for all files that are monitored to see if they match? Should it
always scan all files at every commit (even if they aren't in the
current checkout) to find conflicts?

I'm not trying to be flippant. I'm really not sure just how far to go
down the rabbit hole, especially for a bit of functionality that has
worked as expected for most people (presumably) for years (though
maybe it isn't working the way anyone expects except for the few
people who've weighed in defending it).

> Alternately (or additionally!) a command that would check and report if
> there is any file(s) that would match the --ignore or the ignore-glob flag
> would be helpful.

I was thinking about that earlier (well, a warning, not an error,
which presumes you can't continue). Then the questions I put above
came into my mind so I didn't bring it up. What would you suggest
calling the command?
-- 
Scott Robison
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[David Mason]

On 11 April 2017 at 14:34, Scott Robison  wrote:

> No, it is an explicit command clearly stating the user's desire for
> exclusion of these files *that are not already under source control*.
> The fact that the user does not remember or did not realize they
> issues conflicting commands does not mean that fossil should suddenly
> stop tracking the file, or so it seems to me.
>
> If a file was previously added to a repository (indicating a desire to
> keep track of modifications to the file), that is more important than
> ignoring the file.
>

I think --ignore should give an error if the --ignore matches a file
already in the repository.  The current behaviour is clearly somewhat
ambiguous.

Alternately (or additionally!) a command that would check and report if
there is any file(s) that would match the --ignore or the ignore-glob flag
would be helpful.

../Dave
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 21:04, Ross Berteig wrote:

The fossil addremove command is a convenience command that scans the
tree, obeying some of the glob settings, and applies fossil add and
fossil forget command as needed to make the list of files now in the
repository consistent with the settings and the directory tree in the
working checkout.


That's right. This means it defeats the idea of using rm beforehand, as 
addremove invokes another add for the very same file later on, adding it 
again. That's why this doesn't work. I've tried that too.



___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Ross Berteig]

On 4/11/2017 9:36 AM, Thomas wrote:
I would like to emphasise that --ignore (or 
.fossil-settings\ignore-glob) is an _explicit_ command, clearly 
stating the user's desire for exlusion of these files, following the 
documentation.

Silently ignoring this wish can't be the correct process.


But that isn't what the --ignore switch (and globs) means. It means 
ignore these files in this command's invocation. That generally has 
nothing to do with what fossil will do with files that are already in 
the repository.


To stop tracking revisions of a file that is checked in, you must ask 
fossil to do that explicitly. There are two ways to achieve that:


1. fossil rm somefile
2. rm somefile && fossil addremove

The fossil rm command marks the file as deleted as of a specific 
checkin, which is also on a specific branch. That is, the removal is 
itself versioned. This is the proper way to remove a source file that is 
no longer needed. It disappears from this branch and all descendants, 
but not from any other branch, or from the past.


(Fossil rm can also be used to reverse an unintended fossil add *before* 
it is checked in. In this case, it operates on the working checkout 
only. But the intent is the same.)


Generally fossil rm only acts on the repository, but there has been 
discussion of whether not also deleting the file from disk is or is not 
surprising to a user. So there is a setting that changes the default, 
and a command line option that forces it to also delete from disk. There 
is also a near-synonym "forget" that will never delete from disk. See 
fossil help rm for all the gory details.


The fossil addremove command is a convenience command that scans the 
tree, obeying some of the glob settings, and applies fossil add and 
fossil forget command as needed to make the list of files now in the 
repository consistent with the settings and the directory tree in the 
working checkout.


Knowing a subfolder is currently a mess, you can freely say fossil 
addremove --ignore mymessypart without meaning anything more or less 
than do the addremove operation except in that folder right now.




A switch that doesn't work is either a huge design flaw or a bug. A 
--ignore switch that doesn't ignore is a huge security bug (and a 
trap) too.


What would I need a --ignore switch for when I got to delete the files 
first manually anyway? That's completely contrary to its purpose, 
isn't it? The reason why I'm using --ignore is because I _don't want_ 
to delete these files.


You should use fossil forget or fossil rm to remove the files from 
further tracking, and set the ignore-glob (and check it in) to prevent 
further accidental tracking.


The --ignore switch is documented to "Ignore files matching patterns 
from the comma separated list of glob patterns." And that is exactly 
what it does. The fossil addremove command does not touch those files 
either on disk or in the repository. It ignored them.


--

Ross Berteig   r...@cheshireeng.com
Cheshire Engineering Corp.   http://www.CheshireEng.com/
+1 626 303 1602

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Tue, Apr 11, 2017 at 10:36 AM, Thomas  wrote:
> On 2017-04-11 05:22, Scott Robison wrote:
>>
>> Perhaps it should be documented, but I don't think it is a bug. It is
>> the software doing the job it was originally told to do (track versions
>> of a file) instead of doing the job it was subsequently told to do
>> (ignore untracked files with a given glob).
>>
>> For example, I have licensed libraries in the past that are shipped as
>> obj files. They need to be tracked, even though most obj files don't.
>> This model allows you to add certain obj files which will be tracked
>> while ignoring all others.
>
>
> You're a good salesman. ;-)
>
> Do you work for Microsoft by any chance? You're trying to sell a design flaw
> or bug as a feature. ;-)

No, I try to explain why what you see isn't a design flaw, and
apparently fail. But I'll keep trying!

> Source code management != backup

Never said it was. Keeping an obj file in a repo because there is no
corresponding source from which to build it is valuable so that other
people can get access to all the build artifacts required.

> I would like to emphasise that --ignore (or .fossil-settings\ignore-glob) is
> an _explicit_ command, clearly stating the user's desire for exlusion of
> these files, following the documentation.
> Silently ignoring this wish can't be the correct process.

No, it is an explicit command clearly stating the user's desire for
exclusion of these files *that are not already under source control*.
The fact that the user does not remember or did not realize they
issues conflicting commands does not mean that fossil should suddenly
stop tracking the file, or so it seems to me.

If a file was previously added to a repository (indicating a desire to
keep track of modifications to the file), that is more important than
ignoring the file.

> A switch that doesn't work is either a huge design flaw or a bug. A --ignore
> switch that doesn't ignore is a huge security bug (and a trap) too.

Ignoring does work as desired. It only applies to files that match the
pattern that are not already in source control.

> What would I need a --ignore switch for when I got to delete the files first
> manually anyway? That's completely contrary to its purpose, isn't it? The
> reason why I'm using --ignore is because I _don't want_ to delete these
> files.

Going back to my examples from yesterday: I had an ignore-glob of *.a.
I ran addremove and it ignored the a.a file, but not the b.b file. If
I ran add a.a, it warned me that it was in the ignore pattern, but
allowed me to add it anyway.

Today I went back to that example repository and created a file c.a
and ran addremove. It ignored c.a because I had not added it
previously.

If I delete the ignore-glob file and use the --ignore switch with
'*.a' it behaves identically.

> Every virus scanner, backup software, mirroring software, every other source
> code revision management, webserver, and god knows what else, follows its
> exclusion rules.

--ignore is not a "remove existing files from the repository rule"
switch. It is an "ignore unmanaged files that match a pattern" switch.

> Sorry, I can't see any logic behind this, apart from that this bug or design
> issue may have slipped in accidentally and now no one's willing to fix it,
> but trying to defend and advertise it as a feature... ;)

Just because you can't see it doesn't mean it isn't there. It is an
intentional design to allow the ignoring of unmanaged files.

> File add.c on line 672 says:
>   /* step 1:
>   ** Populate the temp table "sfile" with the names of all unmanaged
>   ** files currently in the check-out, except for files that match the
>   ** --ignore or ignore-glob patterns and dot-files.  Then add all of
>   ** the files in the sfile temp table to the set of managed files.
>   */
> According to this, it seems it's a design flaw.

The key words are at the end of line 673 and the beginning of line
674: "unmanaged files". By definition, a managed file (one that has
been previously added to the repository) is not an unmanaged file.
Thus it is working as described.

The only thing I can think at this point is that you believe that if
*any* file matching a glob pattern is in the repository, all files
matching that same glob pattern will be added to the repository
despite the glob pattern. This is not how my current version of fossil
(downloaded from the fossil-scm site a few days ago) behaves. Only
files already in the repository (managed files) with changes will be
committed.
-- 
Scott Robison
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 10:02, Mark Janssen wrote:

That's not a security hole at all. Once a file was added, ignoring it
will not remove past version from the repository. History in fossil is
immutable. If you inadvertently added a file which shouldn't be there
you should shun it instead.


The way I understand shunning works is that it won't add that particular 
version of the file anymore.


https://www.fossil-scm.org/xfer/doc/trunk/www/shunning.wiki

"Every Fossil repository maintains a list of the hash names of "shunned" 
artifacts. Fossil will refuse to push or pull any shunned artifact. 
Furthermore, all shunned artifacts (but not the shunning list itself) 
are removed from the repository whenever the repository is reconstructed 
using the "rebuild" command."


A hash changes as soon as the file's content changes, or am I wrong?

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 05:22, Scott Robison wrote:

Perhaps it should be documented, but I don't think it is a bug. It is
the software doing the job it was originally told to do (track versions
of a file) instead of doing the job it was subsequently told to do
(ignore untracked files with a given glob).

For example, I have licensed libraries in the past that are shipped as
obj files. They need to be tracked, even though most obj files don't.
This model allows you to add certain obj files which will be tracked
while ignoring all others.


You're a good salesman. ;-)

Do you work for Microsoft by any chance? You're trying to sell a design 
flaw or bug as a feature. ;-)


Source code management != backup

I would like to emphasise that --ignore (or 
.fossil-settings\ignore-glob) is an _explicit_ command, clearly stating 
the user's desire for exlusion of these files, following the documentation.

Silently ignoring this wish can't be the correct process.

A switch that doesn't work is either a huge design flaw or a bug. A 
--ignore switch that doesn't ignore is a huge security bug (and a trap) too.


What would I need a --ignore switch for when I got to delete the files 
first manually anyway? That's completely contrary to its purpose, isn't 
it? The reason why I'm using --ignore is because I _don't want_ to 
delete these files.


Every virus scanner, backup software, mirroring software, every other 
source code revision management, webserver, and god knows what else, 
follows its exclusion rules.


Sorry, I can't see any logic behind this, apart from that this bug or 
design issue may have slipped in accidentally and now no one's willing 
to fix it, but trying to defend and advertise it as a feature... ;)


File add.c on line 672 says:
  /* step 1:
  ** Populate the temp table "sfile" with the names of all unmanaged
  ** files currently in the check-out, except for files that match the
  ** --ignore or ignore-glob patterns and dot-files.  Then add all of
  ** the files in the sfile temp table to the set of managed files.
  */
According to this, it seems it's a design flaw.

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 10:02, Mark Janssen wrote:

That's not a security hole at all. Once a file was added, ignoring it
will not remove past version from the repository. History in fossil is
immutable. If you inadvertently added a file which shouldn't be there
you should shun it instead.


It is very well a security issue if I place a password in a file at a 
later point in time and use --ignore to not include it anymore but that 
command is silently dropped.


My intuition tells me that the old version without password is in the 
repository while the new version containing a password is not.


That's got nothing to do with immutable. It's just what I told it to do. 
Ignore the file from now on.


https://www.fossil-scm.org/xfer/help/addremove

https://www.fossil-scm.org/xfer/help/add
"If files are attempted to be added explicitly on the command line which
match "ignore-glob", a confirmation is asked first. This can be prevented
using the -f|--force option."

That's not the case for addremove.

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Mark Janssen]

That's not a security hole at all. Once a file was added, ignoring it
will not remove past version from the repository. History in fossil is
immutable. If you inadvertently added a file which shouldn't be there
you should shun it instead.

On Tue, Apr 11, 2017 at 1:27 AM, Thomas  wrote:
> On 2017-04-11 00:01, Thomas wrote:
>>
>> The --ignore argument as well as the .fossil-settings\ignore-glob file
>> don't work for files or file masks that have been committed at some
>> point after the repository has been created. Your work-around worked.
>> After deleting some of these files, committing, changing, and committing
>> again, they were ignored/not checked in afterwards.
>>
>> I'd say this is either a big design flaw or a bug.
>> It's not mentioned anywhere in the documentation and is anything but
>> logical and reasonable.
>
>
> That's also a big security hole.
>
> Someone checks in a file
> password.this_is_so_confidential_you_should_never_disclose_it_to_anyone.txt.
>
> Bang.
>
>
> ___
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Apr 10, 2017 5:02 PM, "Thomas"  wrote:

On 2017-04-10 22:28, Scott Robison wrote:

> On Mon, Apr 10, 2017 at 2:57 PM, Thomas  wrote:
>
>> I reckon I owe you a beer! ;-)
>>
>
> Not at all. I don't drink, anyway. Well, not beer. :)
>

You're probably missing one of the best parts in life here ;-)



Anyway, your suggestion sounds very reasonable (ok, it doesn't sound
>> reasonable at all to be honest, but I think that's what happened).
>>
>
> I think if you add a file to a repo then add an ignore, it is safer to
> keep maintaining it rather than suddenly ignoring it. The two events
> are conflicting, so fail safe. Removing the file allows it to be
> ignored going forward.
>

I just tried it. Yepp, that was it. ;-)

The --ignore argument as well as the .fossil-settings\ignore-glob file
don't work for files or file masks that have been committed at some point
after the repository has been created. Your work-around worked. After
deleting some of these files, committing, changing, and committing again,
they were ignored/not checked in afterwards.

I'd say this is either a big design flaw or a bug.
It's not mentioned anywhere in the documentation and is anything but
logical and reasonable.


Perhaps it should be documented, but I don't think it is a bug. It is the
software doing the job it was originally told to do (track versions of a
file) instead of doing the job it was subsequently told to do (ignore
untracked files with a given glob).

For example, I have licensed libraries in the past that are shipped as obj
files. They need to be tracked, even though most obj files don't. This
model allows you to add certain obj files which will be tracked while
ignoring all others.


I reckon everyone starts using software with the default settings.

I'd even go a step further and say when someone starts using a source code
version management software the first thing they do is give it a go and
check out and in again pretty much everything they got scattered around
their harddrive and every memory stick in reach. ;)

The least I'd expect is that these files are not just silently uploaded but
that I'd receive a message, explaining why they are uploaded. I thought I
got to chuck the box out the window after I'd been asked again so many time
to enter a commit note including all the files I didn't want it to include.
Bear in mind, I explicitely told the software not to upload them, but it
secretly ignores/drops my command. That is far from being user-friendly. ;-)

Of course I'm glad to know now how to circumvent this issue.
Thanks again.


___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-11 00:01, Thomas wrote:

The --ignore argument as well as the .fossil-settings\ignore-glob file
don't work for files or file masks that have been committed at some
point after the repository has been created. Your work-around worked.
After deleting some of these files, committing, changing, and committing
again, they were ignored/not checked in afterwards.

I'd say this is either a big design flaw or a bug.
It's not mentioned anywhere in the documentation and is anything but
logical and reasonable.


That's also a big security hole.

Someone checks in a file 
password.this_is_so_confidential_you_should_never_disclose_it_to_anyone.txt.


Bang.

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-10 22:28, Scott Robison wrote:

On Mon, Apr 10, 2017 at 2:57 PM, Thomas  wrote:

I reckon I owe you a beer! ;-)


Not at all. I don't drink, anyway. Well, not beer. :)


You're probably missing one of the best parts in life here ;-)



Anyway, your suggestion sounds very reasonable (ok, it doesn't sound
reasonable at all to be honest, but I think that's what happened).


I think if you add a file to a repo then add an ignore, it is safer to
keep maintaining it rather than suddenly ignoring it. The two events
are conflicting, so fail safe. Removing the file allows it to be
ignored going forward.


I just tried it. Yepp, that was it. ;-)

The --ignore argument as well as the .fossil-settings\ignore-glob file 
don't work for files or file masks that have been committed at some 
point after the repository has been created. Your work-around worked. 
After deleting some of these files, committing, changing, and committing 
again, they were ignored/not checked in afterwards.


I'd say this is either a big design flaw or a bug.
It's not mentioned anywhere in the documentation and is anything but 
logical and reasonable.


I reckon everyone starts using software with the default settings.

I'd even go a step further and say when someone starts using a source 
code version management software the first thing they do is give it a go 
and check out and in again pretty much everything they got scattered 
around their harddrive and every memory stick in reach. ;)


The least I'd expect is that these files are not just silently uploaded 
but that I'd receive a message, explaining why they are uploaded. I 
thought I got to chuck the box out the window after I'd been asked again 
so many time to enter a commit note including all the files I didn't 
want it to include.
Bear in mind, I explicitely told the software not to upload them, but it 
secretly ignores/drops my command. That is far from being user-friendly. ;-)


Of course I'm glad to know now how to circumvent this issue.
Thanks again.

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Mon, Apr 10, 2017 at 2:57 PM, Thomas  wrote:
> I reckon I owe you a beer! ;-)

Not at all. I don't drink, anyway. Well, not beer. :)

I was surprised my first guess was off base, but actually sitting down
and trying some stuff outside the large repo you're using was useful
to see how things worked.

> Anyway, your suggestion sounds very reasonable (ok, it doesn't sound
> reasonable at all to be honest, but I think that's what happened).

I think if you add a file to a repo then add an ignore, it is safer to
keep maintaining it rather than suddenly ignoring it. The two events
are conflicting, so fail safe. Removing the file allows it to be
ignored going forward.

> After reading your mail I remembered that all files that have been ignored
> all along via ignore-glob or --ignore are files I deleted manually at one
> point or another.
>
> The files that always go into the repro I've never touched but I'd added
> them to the ignore list later.

Glad I was able to point you in a useful direction.

FYI: In looking a little at the code, I can see that there was some
work related to versioned settings a couple years ago now that seems
to have been put in place to check either the file in the repo or the
file on the disk depending on what is available. I don't have time to
understand it completely, but that seems to be the source of my
confusion as to when / where versioned settings are accessed / read.

-- 
Scott Robison
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-10 21:36, Scott Robison wrote:

Next I added a.a explicitly (it warned me and I said okay) and
committed. Then I made a change to a.a and it was identified as a
change for the next commit.

So my best guess at the moment is: During one of your earlier attempts
at adding things, you added a bunch of files and committed them to the
repo. Now that those files are in the repo, fossil will not ignore
them because they are part of the repo. If you were to remove the
files from the work directory so that fossil was no longer tracking
them, commit those changes, then try addremove again, it might work
more to your liking.

At least, that worked for me here with a simple little repo with only
a few files in it.

My version: "This is fossil version 2.1 [83e3445f67] 2017-03-10 17:07:08 UTC"


I reckon I owe you a beer! ;-)

I haven't tried it yet because soe of the files I should remove are 
quite big. I'll try to move them out of the open path later. I'm sure 
Visual Studio is going to rebuild its Intellisense SQLite databases 
again but this might take a long time for various projects.


Anyway, your suggestion sounds very reasonable (ok, it doesn't sound 
reasonable at all to be honest, but I think that's what happened).


After reading your mail I remembered that all files that have been 
ignored all along via ignore-glob or --ignore are files I deleted 
manually at one point or another.


The files that always go into the repro I've never touched but I'd added 
them to the ignore list later.


___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Mon, Apr 10, 2017 at 1:58 PM, Thomas  wrote:
> On 2017-04-10 20:34, Scott Robison wrote:
>>
>> On Mon, Apr 10, 2017 at 1:05 PM, Thomas  wrote:
>>>
>>> On 2017-04-10 20:00, Scott Robison wrote:
>>
>> Let's say you have a repo named bob. You have not yet committed any
>> .fossil-settings files. You create the .fossil-settings files then run
>> addremove and commit. The addremove command will not recognize any of
>> the files you are ignoring at this point because you've never
>> committed the .fossil-settings files themselves.
>
>
> Thanks for your extensive explanation. You're right, I would have never
> expected it to work like this. In fact, I believe if it really behaves as
> you described it, that's something that should be mentioned in the
> documentation.
>
> I can't imagine I'd be the only or first one who'd stumble over this. It
> means that whatever you do or want to do the first commit automatically
> stores all files unconditionally in the repository.
>
> I'd even go as far as to say this defeats the whole idea of having (only)
> these files in the first place. Or, in other words, what good is an
> exclusion or filter list that doesn't exclude or filter right from the
> start?
>
>
>> If you *are* doing this, maybe you would be satisfied with a flow like
>> this:
>>
>> 1. update .fossil-settings
>> 2. add (if necessary) and commit those changes
>> 3. now run addremove for the entire working copy
>> 4. now commit those changes.
>
>
> I am using these files at the moment, yes.
>
> But even before the first addremove or commit I had the .VC.db or .tlog
> files exluded with the --ignore parameter on the command line for addremove.
> So, no this behavior can't be the reason why Fossil ignores some files but
> not others.
>
> fossil addremove
> and
> fossil commit
> must have been run hundreds of times now, and the files are still updated in
> the repository every time addremove is performed.

I'm running on very little sleep, so I hope I'm helping more than just rambling.

Anyway, I just created an empty test repo:

> fossil init test.fossil

then opened it

> fossil open test.fossil

then created three files

> echo '*.a' > .fossil-settings\ignore-glob
> echo a > a.a
> echo b > b.b

then ran fossil addremove and it only added b.b. My guess is that
since I last cared about that bit of functionality, someone changed it
to work more intuitively.

Given that .fossil-settings\ignore-glob is in a hidden dot folder,
addremove does not automatically add the ignore-glob file itself. I
did that separately.

Next I added a.a explicitly (it warned me and I said okay) and
committed. Then I made a change to a.a and it was identified as a
change for the next commit.

So my best guess at the moment is: During one of your earlier attempts
at adding things, you added a bunch of files and committed them to the
repo. Now that those files are in the repo, fossil will not ignore
them because they are part of the repo. If you were to remove the
files from the work directory so that fossil was no longer tracking
them, commit those changes, then try addremove again, it might work
more to your liking.

At least, that worked for me here with a simple little repo with only
a few files in it.

My version: "This is fossil version 2.1 [83e3445f67] 2017-03-10 17:07:08 UTC"

-- 
Scott Robison
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-10 20:34, Scott Robison wrote:

On Mon, Apr 10, 2017 at 1:05 PM, Thomas  wrote:

On 2017-04-10 20:00, Scott Robison wrote:

Let's say you have a repo named bob. You have not yet committed any
.fossil-settings files. You create the .fossil-settings files then run
addremove and commit. The addremove command will not recognize any of
the files you are ignoring at this point because you've never
committed the .fossil-settings files themselves.


Thanks for your extensive explanation. You're right, I would have never 
expected it to work like this. In fact, I believe if it really behaves 
as you described it, that's something that should be mentioned in the 
documentation.


I can't imagine I'd be the only or first one who'd stumble over this. It 
means that whatever you do or want to do the first commit automatically 
stores all files unconditionally in the repository.


I'd even go as far as to say this defeats the whole idea of having 
(only) these files in the first place. Or, in other words, what good is 
an exclusion or filter list that doesn't exclude or filter right from 
the start?




If you *are* doing this, maybe you would be satisfied with a flow like this:

1. update .fossil-settings
2. add (if necessary) and commit those changes
3. now run addremove for the entire working copy
4. now commit those changes.


I am using these files at the moment, yes.

But even before the first addremove or commit I had the .VC.db or .tlog 
files exluded with the --ignore parameter on the command line for 
addremove. So, no this behavior can't be the reason why Fossil ignores 
some files but not others.


fossil addremove
and
fossil commit
must have been run hundreds of times now, and the files are still 
updated in the repository every time addremove is performed.




___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Mon, Apr 10, 2017 at 1:05 PM, Thomas  wrote:
> On 2017-04-10 20:00, Scott Robison wrote:
>>
>> On Apr 10, 2017 12:48 PM, "Thomas" > > wrote:
>> Example of .fossil-settings\ignore-glob:
>> *.obj
>> *.tlog
>> *.VC.db
>>
>> The real file of course contains a much bigger list. I only picked
>> these three masks as an example.
>>
>> Fossil happily ignores the .obj files (and many others), but no
>> matter how hard I try, it keeps adding all .tlog and all .VC.db
>> files (and it ignores many others too). I can't figure out a pattern
>> that would tell me why some files/filters are accepted and work as
>> advertised in the ignore-glob, others aren't.
>>
>> Is there anything I overlooked?
>>
>> Any help or idea is highly appreciated.
>>
>>
>> I think it reads those from the repo, so you won't see anything until
>> you've committed the files once or after changes are made.
>
>
> The files it ignores (.obj, .iobj, etc) are neither in the local repository
> nor in the remote one.
>
> The files it doesn't ignore (.VC.db, .log, .tlog, tec) are in both, the
> local and the remote repository.
>
> I got autosnyc on. Sorry, I probably should have mentioned this.

No problem, I was more brief that I maybe should have been because I
was just about to head into a meeting. So here is what I was trying to
say:

Let's say you have a repo named bob. You have not yet committed any
.fossil-settings files. You create the .fossil-settings files then run
addremove and commit. The addremove command will not recognize any of
the files you are ignoring at this point because you've never
committed the .fossil-settings files themselves.

Let's say you have set some glob patterns in the past, either through
the web interface, the command line, or via a .fossil-settings file.
Then you update the .fossil-settings files, run addremove and commit.
Any changes you made to the .fossil-settings files will not be honored
because you've not committed them.

It very well may be that this is not what you are doing, but I can see
how it would not be intuitive that fossil will not use the
.fossil-settings file you have updated in the opened working copy but
not yet committed.

If you *are* doing this, maybe you would be satisfied with a flow like this:

1. update .fossil-settings
2. add (if necessary) and commit those changes
3. now run addremove for the entire working copy
4. now commit those changes.

-- 
Scott Robison
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Thomas]

On 2017-04-10 20:00, Scott Robison wrote:

On Apr 10, 2017 12:48 PM, "Thomas" > wrote:
Example of .fossil-settings\ignore-glob:
*.obj
*.tlog
*.VC.db

The real file of course contains a much bigger list. I only picked
these three masks as an example.

Fossil happily ignores the .obj files (and many others), but no
matter how hard I try, it keeps adding all .tlog and all .VC.db
files (and it ignores many others too). I can't figure out a pattern
that would tell me why some files/filters are accepted and work as
advertised in the ignore-glob, others aren't.

Is there anything I overlooked?

Any help or idea is highly appreciated.


I think it reads those from the repo, so you won't see anything until
you've committed the files once or after changes are made.


The files it ignores (.obj, .iobj, etc) are neither in the local 
repository nor in the remote one.


The files it doesn't ignore (.VC.db, .log, .tlog, tec) are in both, the 
local and the remote repository.


I got autosnyc on. Sorry, I probably should have mentioned this.


___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Issue with ignore-glob

[Scott Robison]

On Apr 10, 2017 12:48 PM, "Thomas"  wrote:

Hello,

As stated in one of my earlier mails, I also got an issue with files to
ignore.

I have now created a folder .fossil-settings and placed the glob files in
it.

Actually, I got a batch file that reads the file filter settings from
another file and creates the binary-glob and the ignore-glob files on the
fly before an addremove and a commit (crlf-glob is not created and only
contains an asterisk now).

Before, the batch file read the external files with the
crlf/binary/exclusion masks and created command line arguments, separated
by commata, now it just writes its output to the mentioned files in
.fossil-settings, each file mask on a different line.

The outcome is exactly the same. Some files are added to the repository
every single time although their - in my opinion - correct name masks are
in correctly added to ignore-glob.

Example of .fossil-settings\ignore-glob:
*.obj
*.tlog
*.VC.db

The real file of course contains a much bigger list. I only picked these
three masks as an example.

Fossil happily ignores the .obj files (and many others), but no matter how
hard I try, it keeps adding all .tlog and all .VC.db files (and it ignores
many others too). I can't figure out a pattern that would tell me why some
files/filters are accepted and work as advertised in the ignore-glob,
others aren't.

Is there anything I overlooked?

Any help or idea is highly appreciated.


I think it reads those from the repo, so you won't see anything until
you've committed the files once or after changes are made.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users