Re: [fossil-users] Repairing trunk/timeline after a shun

[Andy Goth]

On 5/17/2016 9:01 PM, John P. Rouillard wrote:
> Would it be worth mentioning that shunning a commit does not shun the
> artifacts checked in during that commit?

The shun page should give more information about its effective use.

> Is there any way to identify and remove these unconnected artifacts?

They show as "unknown" in the list of artifacts (/bloblist).  I'd have
to review the code for the exact reasons for this annotation.  I see
lots of unknowns in Fossil's own list of artifacts, by the way, so it
could also be there are deficiencies in the artifact classification logic.

Here's one case: http://fossil-scm.org/index.html/info/e791da28f3fbc43f

And another: http://fossil-scm.org/index.html/info/f39e8905c78826c5

For the most part, it looks like "unknown" means a control artifact (not
a manifest).  We should fix that, probably also change "unknown" to
"orphaned" or "unreferenced" or some such.  And then we should add a
page listing such unreferenced artifacts, or update /bloblist to have
filter and/or sort options.

-- 
Andy Goth | 



signature.asc
Description: OpenPGP digital signature
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[Richard Hipp]

On 5/17/16, John P. Rouillard  wrote:
>
> Is there an expand(content) or something that I
> could use?
>

Use the "content()" SQL function to decode the blobs.  The argument is
any symbolic name (like a prefix of the SHA1 hash).  Example:

 SELECT content('trunk');

Returns the complete text of the check-in artifact for the trunk check-in.

To find the files in a check-in use the files_of_checkin virtual table:

SELECT * FROM files_of_checkin WHERE
checkinid=symbolic_name_to_rid('trunk');


-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[John P. Rouillard]

Hi Andy:

In message <0dc19ff1-f9f1-67af-7119-62bf62462...@gmail.com>,
Andy Goth writes:
>On 5/14/2016 4:59 PM, John P. Rouillard wrote:
>> fossil init shun.fossil
>> mkdir shun
>> cd shun
>> fossil open ../shun.fossil
>> echo 2 > 2
>> fossil addremove
>> fossil ci -m "initial commit"
>> echo 2a > 2
>> fossil ci -m "second commit"
>> echo 2b > 2
>> fossil ci -m "third commit"
>> echo 2shun > 2
>> fossil ci -m "fourth shunable commit"
>> echo 2c > 2
>> fossil ci -m "fifth commit"
>> echo 2d > 2
>> fossil ci -m "sixth commit"
>> 
>> then shun the forth commit. You can rebuild the repo if you like it
>> makes no difference.
>
>Go look in /bloblist and you'll find that 2shun is still there, even
>after a rebuild.  You've not actually succeeded in removing the
>dangerous data from your repository.  It's still there despite not being
>connected to anything.

Eep. If I shun a commit artifact, should all the artifacts for that
commit also be shunned? I think the answer is no since shunning should
be an advanced operation.  Mistakenly shunning a commit (rather than a
file) could delete other files that shouldn't be shunned.

Would it be worth mentioning that shunning a commit does not shun the
artifacts checked in during that commit? I kind of expected them to be
removed since they were added as a single transaction. (I understand
why this expectation is wrong but...)

Is there any way to identify and remove these unconnected artifacts?
Would rebuild --vacuum clean them out? I suspect deconstruct would not
dump those disconnected artifacts. So would

   fossil deconstruct
   fossil reconstruct 

make a new repo without the disconnected data.

Is there any way using "fossil sqlite3" to dump the data so I can grep
for the banned data to make sure it's been erased?

As I figured:

   select uuid,content from blob;

doesn't do it.  Is there an expand(content) or something that I
could use?

In any case, thank you for showing the error of my ways. I have to
shun the files with the privileged info and not the commit (unless the
privileged info is in the commit message). Well back to trying to
sanitize things.

Have a great week.

--
-- rouilj
John Rouillard
===
My employers don't acknowledge my existence much less my opinions.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[Andy Goth]

On 5/16/2016 8:00 PM, Richard Hipp wrote:
> On 5/16/16, Andy Goth  wrote:
>> I've wanted reparenting capability to support importing old history into
>> a Fossil repository.
> 
> I interpret your sentence above as "I volunteer to test out the new
> code!".  Thanks!

I should have known there'd be a price to pay.

> I added a "fossil reparent" command on the "reparent" branch
> (https://www.fossil-scm.org/fossil/info/5c5a364d5e106d9c).  It is only
> superficially tested so far.  Let me know what problems you find.

I had to make a single-character fix, no big deal.

Here's my first test:

$ alias f=~/fossil/fossil
$ f new shun.fossil
$ mkdir shun
$ cd shun
$ f open ../shun.fossil
$ echo good_data > file
$ f addremove
$ f commit -m good_data
New_Version: 8c70055d3a86ce658b8c32e12fc02d350a5a2efc
$ echo bad_data > file
$ f commit -m bad_data
New_Version: 350e4c5b3f9c85f8b7c31cd0d161d2f7dd5d0b47
$ echo corrected_data > file
$ echo more_bad_data > file2
$ f addremove
$ f commit -m more_bad_data
New_Version: aafdc5705546ef5bd0da111f83b6eedce7269989
$ f rm -hard file2
$ f commit -m 'remove more_bad_data'
New_Version: 14e90133a47d0c119cf0539e4bf60e19d84f9235
# [used web interface to shun the two commits containing bad_data]
# [did not actually shun bad_data and more_bad_data, by the way...]
$ f rebuild
$ f reparent -test 14e9 8c70
$ f rebuild
$ f reparent 14e9 8c70
$ f rebuild

My second test was to stitch together two repositories.

$ f new old_repo.fossil
$ f new new_repo.fossil
$ mkdir checkout
$ cd checkout
$ f open ../old_repo.fossil
checkout: 9495df1464768e21b280947402f1936ae0dcdb28 2016-05-17 04:46:21 UTC
$ echo version_1 > file
$ f addremove
$ f commit -m version_1
New_Version: 45c8061a7462f00e10f8511d1dfc3486c0c67f86
$ echo version_2 > file
$ f commit -m version_2
New_Version: e308d948a2cd23781417fdcbb2caf62991f50d4f
$ f close
$ f open ../new_repo.fossil
checkout: 6542758abd71985d53757bd86b46c2a4162a6eac 2016-05-17 04:46:24 UTC
$ echo version_3 > file
$ f addremove
$ f commit -m version_3
New_Version: e86d82a264de0f2734a8710d0e91a118e43169ac
$ echo version_4 > file
$ f commit -m version_4
New_Version: 1eb483f157614b35448449a57d02290719559798
$ f close
$ cd ..
$ mkdir repo
$ f deconstruct -R old_repo.fossil repo
$ f deconstruct -R new_repo.fossil repo
$ f reconstruct merged_repo.fossil repo
$ f reparent -R merged_repo.fossil e86d e308

I then had to shun 6542758abd71985d53757bd86b46c2a4162a6eac, which is
the initial empty check-in for new_repo.fossil.

The web interface correctly showed the version_3 check-in modifying file
from "version_2" to "version_3", rather than file being created new.

However!

$ f leaves -R merged_repo.fossil
   (1) 2016-05-17 04:47:20 [1eb483f157] version_4 (tags: trunk)
   (2) 2016-05-17 04:46:50 [e308d948a2] version_2 (tags: trunk)

Does a rebuild clear this up?  Not exactly.

$ f rebuild -R merged_repo.fossil
$ f leaves -R merged_repo.fossil

That's right.  Now there are ZERO leaves.

When I open the repository, it picks version_2, not version_4.  Then
when I update, it goes to version_4.

I'm able to make further commits to the repository, but trunk continues
to have no leaf.  Trunk finally acquires a leaf when I make a branch.

-- 
Andy Goth | 



signature.asc
Description: OpenPGP digital signature
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[Andy Goth]

On 5/14/2016 4:59 PM, John P. Rouillard wrote:
> fossil init shun.fossil
> mkdir shun
> cd shun
> fossil open ../shun.fossil
> echo 2 > 2
> fossil addremove
> fossil ci -m "initial commit"
> echo 2a > 2
> fossil ci -m "second commit"
> echo 2b > 2
> fossil ci -m "third commit"
> echo 2shun > 2
> fossil ci -m "fourth shunable commit"
> echo 2c > 2
> fossil ci -m "fifth commit"
> echo 2d > 2
> fossil ci -m "sixth commit"
> 
> then shun the forth commit. You can rebuild the repo if you like it
> makes no difference.

Go look in /bloblist and you'll find that 2shun is still there, even
after a rebuild.  You've not actually succeeded in removing the
dangerous data from your repository.  It's still there despite not being
connected to anything.

-- 
Andy Goth | 



signature.asc
Description: OpenPGP digital signature
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[Richard Hipp]

On 5/16/16, Andy Goth  wrote:
>
> I've wanted reparenting capability to support importing old history into
> a Fossil repository.
>

I interpret your sentence above as "I volunteer to test out the new
code!".  Thanks!

I added a "fossil reparent" command on the "reparent" branch
(https://www.fossil-scm.org/fossil/info/5c5a364d5e106d9c).  It is only
superficially tested so far.  Let me know what problems you find.

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[Andy Goth]

On 5/16/2016 11:41 AM, John P. Rouillard wrote:
> Richard Hipp writes:
>> On 5/15/16, Andy Goth  wrote:
>>>
>>> To fix, you're going to have to go back to the version prior to the shun
>>> and redo every check-in since that point.
>>
>> Another option is to "reparent" the first check-in after the shun such
>> that it's parent becomes the last check-in before the shun.  There is
>> a "parent" tag for this purpose.
> 
> That is what I was hoping for. Is there any documentation on the
> parent tag? A full text search of "parent NEAR tag" returns too many hits
> and none of the documentaion hits seem to apply.

I've wanted reparenting capability to support importing old history into
a Fossil repository.

For example, I'm given a snapshot of version 2 of a software system and
am told to use it as my baseline.  I make it be the initial import of a
new Fossil repository, then I do my thing.  But then I have questions
about history, and someone gives me a snapshot of version 1.  I want to
import that so I can do diffs and other things through the Fossil
interface, but I can't without recreating the repository using pretty
much the same technique I outlined in my first reply to this thread.

Another scenario would be me being given CVS, SVN, RCS, Git, or other
history data some time after initially creating my repository from one
or a small number of snapshot versions.  I can import these
incrementally, but I can't retroactively integrate what's imported with
the timeline, only keep it in a branch off to the side.

If I can check in control artifacts to facilitate stitching these
otherwise disparate timelines together, then I'll be able to sanely
manage situations such as the above, as well as fix the hole caused by
shunning a non-leaf check-in.

-- 
Andy Goth | 



signature.asc
Description: OpenPGP digital signature
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[John P. Rouillard]

Hello Richard:

In message

Re: [fossil-users] Repairing trunk/timeline after a shun

[Richard Hipp]

On 5/15/16, Andy Goth  wrote:
>
> To fix, you're going to have to go back to the version prior to the shun
> and redo every check-in since that point.

Another option is to "reparent" the first check-in after the shun such
that it's parent becomes the last check-in before the shun.  There is
a "parent" tag for this purpose.

Unfortunately, I'm not sure I ever added the code to have SQLite ever
do anything with the "parent" tag.  Maybe this is our excuse to go in
and fix that

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[Stephan Beal]

On Mon, May 16, 2016 at 12:37 AM, John P. Rouillard <
rouilj+fos...@cs.umb.edu> wrote:

> Fortunately there are no artifact references in
> tickets/wiki/comments. But if there was, would some SQL magic
> be needed to find/change these references?
>

The SQL-usable parts of the repo are just a cache for the underlying
metadata. If you deleted/modified any of it, it would reappear after the
next rebuild.


> >In retrospect, you would have been better off not shunning the entire
> >check-in but rather only the file containing the security credentials.
>
> That makes sense. What is the easiest way to get the artifact
> ID for the file?
>
>fossil timeline -v
>
> gives you the file name, but not the artifact number.  I would
> have expected the artifact number as part of the listing:
>

IIRC, ls -v will show that:

http://www.fossil-scm.org/index.html/help?cmd=ls



> It looks like I need to "fossil finfo 2" to get all artifacts for
> all versions of the file named 2. Then I need to search the finfo
> for the checkin artifact number to figure out which artifact I
> need to wipe.
>
> Is there an easier way?
>

Try ls with the -r CHECKIN_VERSION flag.


> Might it make sense to add a verification step if any of the
> shunned artifacts are a commit record since this misstep is a
> pain to recover from?
>

Shunning is generally considered to be "the nuclear option" and recovery
isn't generally a consideration :/. In your case it's simply cosmetic (if
i'm understanding right) - there's a "hole" in the history, but there's no
way to recover from that except, as Andy suggested, recreating the history
starting at the oldest edge of that hole.

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[Tony Papadimitriou]

In case it applies to you, if you haven't rebuilt the repo since the mistake 
shunning, I believe you can un-shun what was shunned, and it should all come 
back to the way it was.



On 5/14/2016 4:59 PM, John P. Rouillard wrote:
I had to recently shun some artifacts from a fossil repo (due to embedded 
security credentials). Now when I look at the fossil timeline that spans 
that shunning I no longer have a

single connected trunk.

So is there some way to reintegrate the trunk across the shun?



___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Repairing trunk/timeline after a shun

[John P. Rouillard]

Hi Andy:

In message <5538e894-bdfa-ad31-be79-26383e8ea...@gmail.com>,
Andy Goth writes:
>On 5/14/2016 4:59 PM, John P. Rouillard wrote:
>> I had to recently shun some artifacts from a fossil repo (due
>> to embedded security credentials). Now when I look at the
>> fossil timeline that spans that shunning I no longer have a
>> single connected trunk.
>> 
>> So is there some way to reintegrate the trunk across the shun?
>> Is this split timeline just a side-effect of shunning?
>
>Quite simply, you have a hole in your repository.

As I suspected, I removed a link in the middle of the timeline chain.

>To fix, you're going to have to go back to the version prior to
>the shun and redo every check-in since that point.  Use the
>-date-override and -user-override commit options to make your
>commits resemble the originals.  As far as I know, no one has
>scripted this kind of reconstruction yet, so you're on your own.

Fair enough. If I come up with a script I'll post it.

>A consequence is every manifest artifact ID since the shun will
>change.  If you have any references to the old artifact IDs,
>e.g. in tickets or subsequent check-in comments, you may want to
>update them as well.

Fortunately there are no artifact references in
tickets/wiki/comments. But if there was, would some SQL magic
be needed to find/change these references?

>In retrospect, you would have been better off not shunning the entire
>check-in but rather only the file containing the security credentials.

That makes sense. What is the easiest way to get the artifact
ID for the file?

   fossil timeline -v

gives you the file name, but not the artifact number.  I would
have expected the artifact number as part of the listing:

  [305556e6e5] fourth shunable commit (user: rouilj, tags: trunk) 
 EDITED 2 [artifact_ID]

Then I could "fossil what artifact_ID" to get the full sha1 sum.

It looks like I need to "fossil finfo 2" to get all artifacts for
all versions of the file named 2. Then I need to search the finfo
for the checkin artifact number to figure out which artifact I
need to wipe.

Is there an easier way?

[[ As a side note "fossil json timeline checkin -v" does have the
uuid's for the files in it. I am not sure parsing json is easier
even with tools like jq, but... ]]

The timeline view in the web interface also doesn't supply
artifact id's for the files created during a checkin. I am
thinking of something like:

  [305556e6e5] fourth shunable commit (user: rouilj, tags: trunk) 
 EDITED 2 [artifact ID here] [diff]

If I had seen the artifact ID's, I would most likely not have
shunned he checkin and shunned the files instead 8-(. I would
still have to click on the file names to get the whole sha1
artifact number for shunning but it would have made me think twice.

Might it make sense to add a verification step if any of the
shunned artifacts are a commit record since this misstep is a
pain to recover from?

>Then do a check-in that corrects or removes the file.

There was a following checkin that fixed the issue before I
shunned the earlier versions with the credentials.

>It is legal, though certainly not ideal, to have one or more
>manifests refer to artifacts not present in the repository.
>Fossil's handling of this situation isn't quite as graceful as
>it could be, but I don't think it's going to outright die.

Doesn't seem to have died.

In my test case repo, accessing the artifact by UUID returns an
unknown page notice in the web interface. Accessing the artifact
with "fossil artifact" again just reports not found.

Looking at the commit the diff (and mlink) table report that the
shunned artifact is deleted. "timeline -v" shows:

  21:15:19 [305556e6e5] fourth shunable commit (user: rouilj tags: trunk)
   DELETED 2

and "fossil finfo 2"  doesn't show a fourth commit at all.

Looking at the timeline (with file listing) in the web ui, I do
see a (deleted) notation on the shunned file and a subsequent
(new) notation on the following revision. So a little misleading
but amending the checkin comment can explain it away, so not a
real issue.

Looks like the missing artifact is being handled reasonably.

Also it looks like you must support manifests with missing
artifacts otherwise shunning can't work at all. As it always
leaves references to missing artifacts (the commit record) if
nowhere else.

>> Would doing a fossil git-export | fossil git-import "fix" things?
>No, nor would a rebuild, nor would a deconstruct followed by a
>reconstruct.

Right because I deleted the commit artifact needed to stitch
the commits before and after the shunned commit together.

Thanks for your feedback.
--
-- rouilj
John Rouillard
===
My employers don't acknowledge my existence much less my opinions.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org

Re: [fossil-users] Repairing trunk/timeline after a shun

[Andy Goth]

On 5/14/2016 4:59 PM, John P. Rouillard wrote:
> I had to recently shun some artifacts from a fossil repo (due to
> embedded security credentials). Now when I look at the fossil timeline
> that spans that shunning I no longer have a single connected trunk.
> 
> So is there some way to reintegrate the trunk across the shun? Is this
> split timeline just a side-effect of shunning?

Quite simply, you have a hole in your repository.

To fix, you're going to have to go back to the version prior to the shun
and redo every check-in since that point.  Use the -date-override and
-user-override commit options to make your commits resemble the
originals.  As far as I know, no one has scripted this kind of
reconstruction yet, so you're on your own.

A consequence is every manifest artifact ID since the shun will change.
If you have any references to the old artifact IDs, e.g. in tickets or
subsequent check-in comments, you may want to update them as well.

In retrospect, you would have been better off not shunning the entire
check-in but rather only the file containing the security credentials.
Then do a check-in that corrects or removes the file.  It is legal,
though certainly not ideal, to have one or more manifests refer to
artifacts not present in the repository.  Fossil's handling of this
situation isn't quite as graceful as it could be, but I don't think it's
going to outright die.

> Would doing a fossil git-export | fossil git-import "fix" things?

No, nor would a rebuild, nor would a deconstruct followed by a
reconstruct.

-- 
Andy Goth | 



signature.asc
Description: OpenPGP digital signature
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

[fossil-users] Repairing trunk/timeline after a shun

[John P. Rouillard]

Hi Everybody:

I had to recently shun some artifacts from a fossil repo (due to
embedded security credentials). Now when I look at the fossil timeline
that spans that shunning I no longer have a single connected trunk.

To reproduce use:

fossil init shun.fossil
mkdir shun
cd shun
fossil open ../shun.fossil
echo 2 > 2
fossil addremove
fossil ci -m "initial commit"
echo 2a > 2
fossil ci -m "second commit"
echo 2b > 2
fossil ci -m "third commit"
echo 2shun > 2
fossil ci -m "fourth shunable commit"
echo 2c > 2
fossil ci -m "fifth commit"
echo 2d > 2
fossil ci -m "sixth commit"

then shun the forth commit. You can rebuild the repo if you like it
makes no difference.

fossil ui's timeline shows the initial checkin on the leftmost rail
with nodes for the initial, first, second, third commits. The third
commit is marked as a leaf. Then next to the left rail is a second
rail with no nodes until the fifth and sixth commits. The sixth commit
was marked as leaf before the shunning but not after the shunning.

So the fourth commit (shunned) is missing, but the timeline is
disjoint at the point of the shunning. Also the end of the trunk
branch is located at the fourth commit (which is marked as a leaf).

If I try to: fossil up trunk
in the shun subdirectory (with the 6th commit) I get:

  fossil: ../fossil/src/bag.c:146: bag_find: Assertion `e>0' failed.
  Aborted

which seems wrong.

I tried to reconnect the timelines by adding a new propagating branch
tag called trunk to commit 5. This did make commit 6 into a leaf, but
the timeline is still split. However I do get the following when
running 'fossil update trunk'

checkout: 7d1873d50037c56a9d610ef557590f33cc1ccbe1 2016-05-14 21:43:24 UTC
tags: trunk
comment:  sixth commit (user: rouilj)
changes:  None. Already up-to-date
WARNING: multiple open leaf check-ins on trunk:
  (1) 2016-05-14 21:43:24 [7d1873d500] (current)
  (2) 2016-05-14 21:43:19 [43059e2375]

So is there some way to reintegrate the trunk across the shun? Is this
split timeline just a side-effect of shunning? Would doing a
fossil git-export | fossil git-import
"fix" things?

(As a side note, is there a shun cli command? fossil help/help -all
doesn't seem to show one.)

Thanks for any insight.

--
-- rouilj
John Rouillard
===
My employers don't acknowledge my existence much less my opinions.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users