Re: [fossil-users] fossil-scm as an SQLite db with schema/data revision control

[Stephan Beal]

On Wed, Jul 27, 2016 at 8:55 PM, Adam Jensen  wrote:

> Does anyone know of a [libfossil](http://fossil.wanderinghorse.net)
> mailing list? Or is it cool to talk about such things here?
>

No and yes.

i'm the originator of libfossil, but a severe inflamed elbow nerve, caused
by too much typing, limits me, more often than not, to single-handed
typing, which has meant essentially no programming for me since December,
2014 :/. So... the brakes are on libfossil's development until either my
elbow recovers or someone steps up to get it going again. (Volunteers are
of course welcomed :D.)

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] fossil-scm as an SQLite db with schema/data revision control

[John McMurloc]

?

-Original Message-
From: fossil-users-boun...@lists.fossil-scm.org
[mailto:fossil-users-boun...@lists.fossil-scm.org] On Behalf Of Adam Jensen
Sent: 27. juli 2016 20:56
To: fossil-users@lists.fossil-scm.org
Subject: Re: [fossil-users] fossil-scm as an SQLite db with schema/data
revision control

On 07/27/2016 10:37 AM, Ron W wrote:
> On Sun, Jul 24, 2016 at 10:58 PM, Adam Jensen  > wrote:
> 
> 
> [The Session Extension](http://www.sqlite.org/sessionintro.html)
pointed
> out by Eduardo seems to have a lot of potential.
> 
> "The session extension provides a mechanism for recording changes to
> some or all of the rowid tables in an SQLite database, and packaging
> those changes into a "changeset" or "patchset" file that can later be
> used to apply the same set of changes to another database with the
same
> schema and compatible starting data. A "changeset" may also be
inverted
> and used to "undo" a session."
> 
---

> Eduardo's description of the SQLite extension only talks about table
> data. This would still leave a need for managing schema updates.
> 

Interesting, I didn't interpret it that way.

> If this SQLite extension also handles schema updates, then probably no
> reason to use tickets.
> 

I suspect it does but I haven't tested that suspicion yet. I've compiled
SQLite with support for the Session Extension but section 2.0 of the
documentation - "Using The Session Extension" - is empty.

> In either case, the actual changesets should be suitable to manage as
> files in Fossil.
>  

I suppose when Fossil manages a file, from one version (check-in) to the
next it is a diff of some kind that is actually recorded and generating
the diff is an internal mechanism that is part of the check-in process.

Imagining a custom system for distributed version control of an SQLite
database, composed of components harvested from Fossil (but not a
standard-off-the-shelf Fossil system), the file being managed might be
something like a .dump of the database and the current Fossil diff
mechanism could (abstract concept) be replaced with the Session
Extension method of generating diffs (changesets).

Does anyone know of a [libfossil](http://fossil.wanderinghorse.net)
mailing list? Or is it cool to talk about such things here?

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] fossil-scm as an SQLite db with schema/data revision control

[Adam Jensen]

On 07/27/2016 10:37 AM, Ron W wrote:
> On Sun, Jul 24, 2016 at 10:58 PM, Adam Jensen  > wrote:
> 
> 
> [The Session Extension](http://www.sqlite.org/sessionintro.html) pointed
> out by Eduardo seems to have a lot of potential.
> 
> "The session extension provides a mechanism for recording changes to
> some or all of the rowid tables in an SQLite database, and packaging
> those changes into a "changeset" or "patchset" file that can later be
> used to apply the same set of changes to another database with the same
> schema and compatible starting data. A "changeset" may also be inverted
> and used to "undo" a session."
> 
---

> Eduardo's description of the SQLite extension only talks about table
> data. This would still leave a need for managing schema updates.
> 

Interesting, I didn't interpret it that way.

> If this SQLite extension also handles schema updates, then probably no
> reason to use tickets.
> 

I suspect it does but I haven't tested that suspicion yet. I've compiled
SQLite with support for the Session Extension but section 2.0 of the
documentation - "Using The Session Extension" - is empty.

> In either case, the actual changesets should be suitable to manage as
> files in Fossil.
>  

I suppose when Fossil manages a file, from one version (check-in) to the
next it is a diff of some kind that is actually recorded and generating
the diff is an internal mechanism that is part of the check-in process.

Imagining a custom system for distributed version control of an SQLite
database, composed of components harvested from Fossil (but not a
standard-off-the-shelf Fossil system), the file being managed might be
something like a .dump of the database and the current Fossil diff
mechanism could (abstract concept) be replaced with the Session
Extension method of generating diffs (changesets).

Does anyone know of a [libfossil](http://fossil.wanderinghorse.net)
mailing list? Or is it cool to talk about such things here?

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Purging data from Fossil

[Andy Bradford]

Thus said David Mason on Tue, 26 Jul 2016 16:41:29 -0400:

> I want to  share these fossils with some people  who cannot be allowed
> to see the grades files. So I need to remove the data.

Do they actually need the fossil  (e.g. all the history of the changes),
or would  an export  (e.g. fossil  zip, followed  by removing  files) be
sufficient?

Thanks,

Andy
-- 
TAI64 timestamp: 4000579902c4


___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Purging data from Fossil

[Warren Young]

On Jul 27, 2016, at 6:55 AM, Steve Stefanovich  wrote:
> 
>> (A)  Suggest a better name than "fossil trim"
> 
> (I realise standalone is supposed to mean something else in current bundle 
> world, but it currently doesn't work anyway). In this context it will create 
> a standalone repo that can be opened and worked on its own. 

Yes, it would indeed be nice if “standalone” bundles truly were standalone, 
this whole thread notwithstanding.  It’s a misleading option name as it stands.

That by itself wouldn’t suffice to provide my wished-for repo sharding feature, 
since bundles currently don’t have a way to include artifacts based on a file 
path pattern, but that should be a straightforward extension, since Fossil 
stores file artifacts based on their full path.

> So we'd need a file glob in args here

Both glob and regex patterns are helpful, each in different cases.

Classic Unix tools that support both usually accept glob patterns by default, 
then accept a -r, -R, or --regex flag to make it accept REs instead.  Obviously 
we can’t use -R for this with Fossil, but double-dash flags are commonly used 
in Fossil for such non-default features.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Purging data from Fossil

[Warren Young]

On Jul 27, 2016, at 12:41 AM, Stephan Beal  wrote:
> 
> On Tue, Jul 26, 2016 at 11:12 PM, Eric Rubin-Smith  wrote:
> 
> this sounds more like it might belong in a "filtered export” 

Perhaps that would also allow repository sharding.

That is, given a repository that contains multiple independent projects — as 
may happen when converting from another VCS, or just due to naive use of Fossil 
— you may wish to export all but one of those projects to independent repos, 
then trim them out of the main repo.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Purging data from Fossil

[Richard Hipp]

On 7/27/16, David Mason  wrote:
> On 26 July 2016 at 17:13, Richard Hipp  wrote:
>
>> SELECT DISTINCT uuid
>>   FROM blob, mlink, filename
>>
>
> Works a charm (18 UUIDs show up).  Now if there were a way to shun from the
> command line, I'd be golden.

The entry box on the /shun webpage allows you to past all 18 SHA1
hashes in a single go.

BTW:  Be sure to "fossil rebuild" after shunning.  That extra step is
necessary to fully purge the unwanted artifacts.  Also look into using
the "fossil scrub" command on any repo that you want to send to
unprivileged users.

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Purging data from Fossil

[David Mason]

Another nice use of this capability would be to make nested fossils - one
of the really nice unique features of fossil.

Eg. fossil X with directories A, B and C (and lots of files and commits).
If you could (via export or strip) C into C.fossil and everything *but* C
into RestX.fossil then you could now open C.fossil nested under a RestX
opened directory and the fossils can be shared and maintained separately...

So ARGS should allow you to include or exclude a directory and everything
under it.

On 26 July 2016 at 17:04, Richard Hipp  wrote:

> Further thoughts:
>
> With shunning, Fossil remembers the SHA1 hashes of the shunned
> artifacts so that they can never again be received by subsequent
> pulls.  It seems like this is more than you need.  Suppose we created
> a new command
>
> "fossil trim ARGS..."
>
> This new command would remove selected artifacts from the local copy
> of the repository, but the removal would be temporary.  The artifacts
> would reappear the next time the local repository is synced with some
> other repo that holds the artifacts.  (The removal would, of course,
> be permanent if there were no other copies of the repo to sync with.)
> It seems like the "trim" command might serve several purposes:
>
> (1) Allow the creation of a repo that omits selected historical
> information that one does not want to share.
>
> (2) Allows putting Fossil into a weird state in order to stress the
> sync logic for testing purposes.
>
> I could have made use of this feature for purpose (1) last week, had
> it been available.  So I am now officially interested in adding it.
>
> The ARGS part would embody several options for doing things like
> removing all instances of a specific file, or all check-ins within a
> range of dates, all tickets matching some criteria, all wiki pages
> whose names match a GLOB pattern, etc.  The problem I had last week
> was I needed a cut-down instance of a repository that only contained
> data for two specific check-ins out of around 3000.
>
> Community members, I want your help as follows:
>
> (A)  Suggest a better name than "fossil trim"
>
> (B)  Define the syntax of ARGS.
>
> (C)  Define a safety mechanism that allows content to be restored if
> it is accidentally trimmed when there are no other repos available
> with which to sink.  Perhaps the trimmed content gets written into a
> separate "trash-can" database?
>
> On 7/26/16, David Mason  wrote:
> > I have a problem.
> >
> > I use fossil for my courses - one I've used for 4 years.  In each year
> > there are grades files (with student names, student numbers and grades),
> > and they get changed and committed over the semester, so there are many
> > versions in the fossil (hundreds of checkins).
> >
> > I want to share these fossils with some people who cannot be allowed to
> see
> > the grades files. So I need to remove the data.  I could shun the
> artifacts
> > and rebuild (and, in fact that's what I started to do), but finding all
> the
> > versions of the files seems rather daunting, and it appears I can only
> shun
> > from the UI, so I can't even write a script to do it.
> >
> > Any ideas on how to clean this up? fossil purge looked like it might
> help,
> > but then not so much...
> >
> > BTW, on the shun page, when it lists pending shuns at the bottom of the
> > page, it would be very convenient if it listed the files that correspond
> to
> > that hash (probably only 1, modulo renaming, but useful regardless).
> >
> > Thanks for any help!
> >
> > ../Dave
> >
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
>
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Purging data from Fossil

[David Mason]

On 26 July 2016 at 17:13, Richard Hipp  wrote:

> SELECT DISTINCT uuid
>   FROM blob, mlink, filename
>

Works a charm (18 UUIDs show up).  Now if there were a way to shun from the
command line, I'd be golden.

BTW, swapping "uuid" and "filename.name" in the SQL lists the files that
match a uuid.

Thanks  ../Dave
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] fossil-scm as an SQLite db with schema/data revision control

[Ron W]

On Sun, Jul 24, 2016 at 10:58 PM, Adam Jensen  wrote:

>
> [The Session Extension](http://www.sqlite.org/sessionintro.html) pointed
> out by Eduardo seems to have a lot of potential.
>
> "The session extension provides a mechanism for recording changes to
> some or all of the rowid tables in an SQLite database, and packaging
> those changes into a "changeset" or "patchset" file that can later be
> used to apply the same set of changes to another database with the same
> schema and compatible starting data. A "changeset" may also be inverted
> and used to "undo" a session."
>
> Wow, whoever designed that extension had Vision.
>
> Given this capability, would you still use Fossil tickets to record,
> manage, and communicate the changesets or do you suppose the changesets
> (or some variation) might somehow be stored in the blob table?


I was only suggesting tickets for the schema updates.

Eduardo's description of the SQLite extension only talks about table data.
This would still leave a need for managing schema updates.

If this SQLite extension also handles schema updates, then probably no
reason to use tickets.

In either case, the actual changesets should be suitable to manage as files
in Fossil.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Purging data from Fossil

[Steve Stefanovich]

> (A)  Suggest a better name than "fossil trim"

How about existing "fossil bundle export"? See (C) discussion below. Having 
bundles, trim and shun is more confusing for newcomers than single bundle 
export (or equivalent new command).

(B)  Define the syntax of ARGS.

This needs some use case discussion first, but I think existing bundle export 
args will be a good start. What needs to be added is file or directory options.

(C)  Define a safety mechanism that allows content to be restored if
it is accidentally trimmed when there are no other repos available
with which to sink.  Perhaps the trimmed content gets written into a
separate "trash-can" database?

This will be solved by extending bundles functionality. You effectively create 
a trimmed repo with 'fossil bundle export --standalone' (I realise standalone 
is supposed to mean something else in current bundle world, but it currently 
doesn't work anyway). In this context it will create a standalone repo that can 
be opened and worked on its own. 

Regarding use cases,  I think there are 2 orthogonal ‎categories of use which 
may intersect or combine:

1. Strip out files and directories, i.e. get rid of all artifacts that don't 
relate to this files (or export only those that touch these files).  Obviously 
for sensitive or proprietary information, but this can be useful  for repo 
'sharding' where only a subdirectory tree can be exported for someone to 
develop on, or to manage third party libraries. 

So we'd need a file glob in args here, maybe even as a file checked in 
.fossil_options like empty-dirs for lots of trimming, and so it can be set for 
synching once we get to that. It would be nice if glob worked from both include 
or exclude perspective (export only these files, or export all files except 
these). 

2. Strip out history of work, i.e. --from --to tags, --branch. For performance 
reasons on large repos, as it is unlikely that anything older than xxx years or 
months is required for active development. Or perhaps to submit drive-by 
patches or features, which bundle currently covers. Or rudimentary permissions 
management, where some devs are not to see somebody else's sensitive work in 
another branch, but others should. 

Args here would need to be recognised as hashes, tags or dates. 

I would also think that this kind of export should ensure that if a subset of 
artifacts or commits is exported, all the files touched must check out. Perhaps 
an initial checkin that adds the missing file, is tagged in a special way and 
never syncs anywhere. A special case of 'fossil rebase' :) The alternative is 
that the file simply doesn't check out with a warning, which is the current 
bundle export functionality and is not that useful. 

I am much more interested in 1. Don't care that much about 2. - but I can use 
branch export where all the files touched are guaranteed to check out, for 
subcontracting type work. 

I think if this functionality or some subset of it does get implemented, it 
would make Fossil much more malleable for users coming from Git world and make 
big differences between the two (sharding vs replication, rewriting history) 
matter less. 

Cheers,
Steve
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Purging data from Fossil

[Stephan Beal]

On Tue, Jul 26, 2016 at 11:12 PM, Eric Rubin-Smith 
wrote:

>
>> (A)  Suggest a better name than "fossil trim"
>>
>> (B)  Define the syntax of ARGS.
>>
>> (C)  Define a safety mechanism that allows content to be restored if
>> it is accidentally trimmed when there are no other repos available
>> with which to sink.  Perhaps the trimmed content gets written into a
>> separate "trash-can" database?
>>
>
> You might consider this from the converse perspective, allowing 'fossil
> export' to a new fossil repo, but specifying some sort of complex boolean
> selector on export.  Then you at least get the safety for free since the
> source repo is only read from.
>

fwiw, that was also my gut reaction: this sounds more like it might belong
in a "filtered export"

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users