Re: [fossil-users] tags manifest

[Ross Berteig]

On 8/6/2016 12:02 PM, Jan Danielsson wrote:

On 05/08/16 02:51, Ross Berteig wrote:
   Many thanks; this saved me a lot of time, as a non-native tcl
speaker.  And as such, I also would like someone to quickly look over
the additions I've made.  It seems to work, but I have a feeling I have
have reinvented the wheel.


Your additions look fine to me, but I too am a non-native Tcl speaker.



   I added more tests in the same file, but given the "set-" prefix I
assume it would have been better to split the manifest.tags content
tests to a separate test.  On the other hand, there aren't many tests
overall so it doesn't hurt having them all in a single file for now.



This file seems sensible to me. My logic is that the feature here is the 
effect of the "manifest" setting. It originally just controlled whether the



 If anything, I should add a few tests related to the specific content 
of the other two kinds of manifest files that are all controlled by the 
single manifest setting.


I think I'll lean on your new file_contains function to do that for 
manifest. The manifest.uuid is easy to compare to the output of fossil info.


--
Ross Berteig   r...@cheshireeng.com
Cheshire Engineering Corp.   http://www.CheshireEng.com/
+1 626 303 1602
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] tags manifest

[Richard Hipp]

On 8/3/16, Jan Danielsson  wrote:

>We've been using a version of the branch for a pretty long time now,
> and I'm starting to feel that it's ready for trunkification.
>
>Thoughts?

I am not completely happy with the format of the manifest.tags file.
The current format is:

  branch=jan-manifest-tags
  jan-manifest-tags

I think it should be:

  branch jan-manifest-tags
  tag jan-manifest-tags

Furthermore, it appears the "branch" line is only written for the
manifest.tags of a local checkout and is omitted for tarballs and zip
archives.  Why is that?

Should we move this discussion to fossil-...@lists.fossil-scm.org?

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] tags manifest

[Jan Danielsson]

On 05/08/16 02:51, Ross Berteig wrote:
[---]
> I just created test cases for fossil set manifest that verify that the
> files manifest, manifest.uuid, and manifest.tags appear and disappear as
> the setting is changed. I did not verify that any of the files have the
> right content.

   Many thanks; this saved me a lot of time, as a non-native tcl
speaker.  And as such, I also would like someone to quickly look over
the additions I've made.  It seems to work, but I have a feeling I have
have reinvented the wheel.

>> I'm not sure offhand how our test coverage looks for all settings.
>> Writing tests that cover every setting is probably a worthy goal for a
>> long term and low priority project for someone who wants to visit all
>> the nooks and crannies in the fossil source kit.
> 
> On the assumption that there will be more tests related to settings, I
> named the new file test/set-manifest.test so there is an obvious naming
> pattern to follow for tests of other features primarily exposed through
> settings.

   I added more tests in the same file, but given the "set-" prefix I
assume it would have been better to split the manifest.tags content
tests to a separate test.  On the other hand, there aren't many tests
overall so it doesn't hurt having them all in a single file for now.

-- 
Kind Regards,
Jan
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] tags manifest

[Jan Danielsson]

On 04/08/16 20:44, Warren Young wrote:
>> That's one of the things I realized after a few
>> failures -- whatever scheme you come up with, it has its limitations.
> 
> That’s part of what I mean, but I’m not just talking about simple 
> limitations.  The second and third applications of the code in new contexts 
> often stretch the original design in ways you had no reason to even consider 
> when initially designing the feature.
> 
> This is why too much up-front design is a problem.  It isn’t until you begin 
> trying to apply the design to the real world that you can see all of your 
> initial design’s weaknesses.

   Sure, and it's appreciated.  We've been using it for well over half a
year, and have formed some build systems around how the feature works.
And it would be naive to say I don't suffer from some tunnel-vision.

   That said, I explicitly didn't want to invent a complete new big
system.  I wanted to simply expose data which is already there in a
simple-to-access manner and - importantly - allow the information to be
accessed in tarballs and zipfiles.  There's a lot more one could do if
one were willing to do more plumbing, so to speak.

[---]
> Given the behavior you show, I wonder if your feature should create 
> manifest.branches, not manifest.tags.

   The way I view it is that it's used more like a label than anything,
and in that sense "manifest.tags" is more fitting.

   On a more technical note:  I use a query from "tags list" source to
generate manifest.tags.

>>> Does your build system use this project name to detect which binary 
>>> package(s) to build?
>>
>>   Sorry, I don't understand the question; interpreting from context
>> I'm going to take a stab at answering.  I apologize if I completely
>> misunderstood (it's well past bed-time here..).
>>
>>   The build server gets a request to build a specific version.
> 
> I was operating under the assumption that you had some kind of Continuous 
> Integration system going, so that every checkin could trigger a build, so the 
> CI system would need a way to figure out which binary packages to build: 
> release client, release server, experimental development builds of both, etc.

   No; some components have nightly builds, but there's still too much
manual intervention required for some of the near-leaf-node builds.  I
have worked in places which use CruiseControl and Jenkins; and there's a
long-term goal to use a CI-system here as well, but .. so many other
things on the todo-list..


-- 
Kind Regards,
Jan
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] tags manifest

[Ross Berteig]

On 8/4/2016 2:33 PM, Ross Berteig wrote:

I just built it and ran the existing test suite over it on Windows. It
passes all existing tests. I didn't spot any test cases for the new
feature


I just created test cases for fossil set manifest that verify that the 
files manifest, manifest.uuid, and manifest.tags appear and disappear as 
the setting is changed. I did not verify that any of the files have the 
right content.



I'm not sure offhand how our test coverage looks for all settings.
Writing tests that cover every setting is probably a worthy goal for a
long term and low priority project for someone who wants to visit all
the nooks and crannies in the fossil source kit.


On the assumption that there will be more tests related to settings, I 
named the new file test/set-manifest.test so there is an obvious naming 
pattern to follow for tests of other features primarily exposed through 
settings.


--
Ross Berteig   r...@cheshireeng.com
Cheshire Engineering Corp.   http://www.CheshireEng.com/
+1 626 303 1602
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] tags manifest

[Ross Berteig]

On 8/3/2016 9:24 AM, Jan Danielsson wrote:

   For those who are afraid of new features messing with your current
setup:  The jan-manifest-tags branch is explicitly designed to not
interfere with your current setup.  You need to take action for a change
to occur.  If you use "set manifest on" or "set manifest off" it will
work exactly like it always has.  I.e. if you don't care about this
feature, you - as an end user - will not notice it's there (apart an
updates settings field and help text).


I just built it and ran the existing test suite over it on Windows. It 
passes all existing tests. I didn't spot any test cases for the new 
feature. While I don't see that as blocking a merge to trunk (it didn't 
break any existing tested feature) I do think that we should write some 
test cases for the new treatment of the manifest setting, as well as the 
new tag manifest before the next "release" of fossil after this feature 
is merged.


I'm not sure offhand how our test coverage looks for all settings. 
Writing tests that cover every setting is probably a worthy goal for a 
long term and low priority project for someone who wants to visit all 
the nooks and crannies in the fossil source kit.


--
Ross Berteig   r...@cheshireeng.com
Cheshire Engineering Corp.   http://www.CheshireEng.com/
+1 626 303 1602
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] tags manifest

[Warren Young]

On Aug 3, 2016, at 7:48 PM, Jan Danielsson  wrote:
> 
> And it gets really fun when people try
> to be clever and have "funny" schemes like "the version converges
> towards pi.”

It’s funny that you should mention that.  On reading the announcement of the 
upcoming “pi” release of SQLite, I began having this amusing fantasy that 
someone would step up and begin offering a “no new features” version of SQLite, 
being 3.14 plus backported bug fixes only, asymptotically approaching the 
perfect SQLite. :)

(Not that I’m against features, but I’m certain that there’s a nontrivial 
segment of the user base that would adopt such a fork.)

> I can't answer your question without dedicating some time to it, and
> I can't say that I would come up with an good manifest.tags-solution for
> your specific case.

Understand, I’m not asking you to do that so that you do my work for me.  I 
don’t have much incentive to replace my existing working system this late in 
the game.  I offered it as a useful exercise which might help you to knock some 
sharp corners off of your current design.

I’ve observed that the first release of a feature is usually good for only one 
use case.  It isn’t until you’ve tried to reuse that code in three completely 
unrelated contexts that the feature becomes sufficiently generalized that you 
can expect it to drop into more new situations without needing further changes.

> That's one of the things I realized after a few
> failures -- whatever scheme you come up with, it has its limitations.

That’s part of what I mean, but I’m not just talking about simple limitations.  
The second and third applications of the code in new contexts often stretch the 
original design in ways you had no reason to even consider when initially 
designing the feature.

This is why too much up-front design is a problem.  It isn’t until you begin 
trying to apply the design to the real world that you can see all of your 
initial design’s weaknesses.

> What we
> found though was that everyone gets it 100% correct the first few times,
> but after a while people tend to get into the "This part is easy, I know
> this by heart"-mindset, and they check the "I did stage x, y, z" boxes
> without actually doing all the verifications.
> 
>   So there are different ways to tackle such problems.  Management say
> it's done by adjusting processes and procedures.  I'm tend to take a
> "develop better tools" approach.

I agree.  If it is at all possible to make the tooling smart enough to 
understand the rules, do it that way.  But equally, if it’s just as difficult 
to give the tools enough info that they can work out the correct answer as to 
do the manual process, you might as well continue to do it manually.

This is why it is valuable to have version numbers in as few places as possible 
and to create tools (or use existing ones) that can transfer that version into 
all the other places it needs to appear.  It creates fewer things for the human 
to worry about, and the process for scattering that version around is the same, 
every time.

>   People may think I'm being oddly picky about version numbers, but
> there's context to it.  Some of these systems are developed under
> contracts which state that "if there's a flaw in the traceability chain,
> we can demand money back”.

Never mind drastic scenarios like that.  Sloppy version numbering can cost an 
hour or so per incident in downstream costs.  If your process takes only a few 
hours to set up and hardly any maintenance beyond that, as ours does, it’s 
probably paying for itself just in decreased debugging time.

>>> The build system iterates through each line of manifest.tags
>> 
>> I don’t see what your build system is getting from manifest.tags that it 
>> couldn’t already get from the T card in the current manifest file.
> 
>   Hmm..
> 
>   $ fossil status
>   repository:   /Volumes/ramdisk/mytest/../mytest.fossil
>   local-root:   /Volumes/ramdisk/mytest/
>   config-db:/Users/jan/.fossil
>   checkout: 504d0bfcda9be2e7b365be2331468b1363a3f719 2016-08-03
> 14:21:07 UTC
>   parent:   569e2a7bb99c919dad0bbb9d6158a2b31aaf4850 2016-08-03
> 14:17:57 UTC
>   tags: trunk, mytest-0.0.1
>   comment:  Added test.c (user: jan)
> 
>   $ grep ^T manifest
>   $ grep mytest manifest
>   $ cat manifest.tags
>   branch=trunk
>   trunk
>   mytest-0.0.1
> 
>   Is there supposed to be a T-entry in the manifest file?  *confused*

T is for tags, not branches.  Or, to be more precise here, since it matters, T 
is for non-propagating tags, rather than propagating tags.

Thus if you say “f up mytest-0.0.1”, a T card should appear in the regular 
manifest file, while it won’t if you’re on the mytest-0.0.1 branch but a few 
checkins down from the tagged checkin.

I think Fossil’s existing behavior is more useful, since the checkin 
immediately following the one tagged mytest-0.0.1 begins a new version, so you 
are no longer on version 0.0.1.

Re: [fossil-users] tags manifest

[Jan Danielsson]

On 04/08/16 00:08, Warren Young wrote:
>> This means that the same
>> information is available in two different places
> 
> Only two?  Aren’t you the lucky one. :)

   Not that lucky, unfortunately.  :(  That was mostly referring to
cases such as sqlite and even fossil itself.  In our case it refers to
how some of the more recent repositories have been set up.  There are
old monolithic repositories which acted as "catch-all" repositories back
in the day and which we have been unable to break up for various
reasons.  They have many different components and versions in the same
repositories.  Then again, some of them are stuck in CVS and Subversion,
but my hope is to someday split them up as much as can possible be done
and convert them to fossil repositories..

> In my current main projects, the release version number appears in:
> 
> - the top-level build system file (configure.ac, CMakeLists.txt, etc.)
> - the most recent section at the top of the ChangeLog
> - the source file feeding “./myprogram -v” or the About screen
> - the binary package description file(s) (e.g. *.spec for RPM)
> - the Fossil release tag
> - the comment on the Fossil checkin creating the release tag
> 
> We drive everything from the first item on that list, except that the 
> ChangeLog remains manually-updated.  In that sense, we have “only” two places 
> to change the version number, too.
> 
> The formal release version number in our system has two parts: the 
> human-facing version number (e.g. 1.2.3) and the binary package release 
> number (e.g. -1) which combine to give the formal release number, 1.2.3-1.  
> These two parts may be in the same top-level file (e.g. CMakeLists.txt for a 
> CMake based project) or in separate files (e.g. configure.ac and *.spec.in 
> for an Autoconf + RPM based project).
> 
> We use the binary package release number to distinguish two different Fossil 
> checkins both tagged someproject-1.2.3.  The checkin comment for the first 
> tag will say “Release someproject v1.2.3-1”, and the second “…-2”.
> 
> The Fossil tag checkin and its checkin comment are generated by a mkrel 
> script we wrote which parses the version number out of one of the generated 
> files, such as *.spec file in an Autoconf + RPM based project, which as I 
> have said is generated from *.spec.in and configure.ac.
> 
> Other files come off of this, such as a version.h file for a C++ project, 
> generated from version.h.in by Autoconf, which gets it from configure.ac.
> 
> Because we want the binary package release number in the Fossil release tag’s 
> checkin comment, generated by the mkrel script, Fossil doesn’t get involved 
> until nearly the end of this process.  My initial thought from that 
> observation is that this means we can’t really replace our current scheme 
> with your scheme. If you had to redesign our scheme to use your new feature, 
> how would you go about it?

   Yeah, so you just perfectly summarized why versioning is a massive
pain in the you-know-what, and that tools even today are lacking in
helping manage version numbers.  And it gets really fun when people try
to be clever and have "funny" schemes like "the version converges
towards pi.".  (Which I myself thought was funny and clever until I
started writing tools for managing version numbers..).

   I can't answer your question without dedicating some time to it, and
I can't say that I would come up with an good manifest.tags-solution for
your specific case.  That's one of the things I realized after a few
failures -- whatever scheme you come up with, it has its limitations.

   There's much I can relate to in the procedures you described, so I
definitely understand where you're coming from.

> I don’t mean to dismiss your efforts, Jan.  It solves a real problem, and I 
> wish it was available before we started solving these problems on our own, 
> many years ago.  I’m willing to offer a weak sort of endorsement for 
> trunkifying it, since I haven’t played with it, but I can see that it’s 
> useful already.

   No worries, I didn't take it as a dismissal; but you're highlighting
something I should have been clearer about:  I was primarily aiming to
solve one particular problem for one, as I see it, fairly typical case.
Looking at the open source projects I follow closely, such a scheme
could be used for the majority of them.  That said, there are plenty of
cases where its use would be very limited.

>> Someone(tm) needs to make sure they match.
> 
> Or, as we have both discovered, some *thing*.  This is a task for automata, 
> not humans.

   Exactly.

>> Each new released version is tagged using a -
>> tag.
> 
> Tagged how?  Manually?

   Yes, but we have some tools to assist in that as well.  I have a
script which automatically tags "next version" (for fossil projects):

   $ revup major
   mytest-1.0.1 -> mytest-2.0.0
   $ revup minor
   mytest-1.0.1 -> mytest-1.1.0
   $ revup
   mytest-1.0.1 -> mytest-1.0.2

   ... but it has been kind of flaky, 

Re: [fossil-users] tags manifest

[Warren Young]

On Aug 3, 2016, at 10:24 AM, Jan Danielsson  wrote:
> 
> This means that the same
> information is available in two different places

Only two?  Aren’t you the lucky one. :)

In my current main projects, the release version number appears in:

- the top-level build system file (configure.ac, CMakeLists.txt, etc.)
- the most recent section at the top of the ChangeLog
- the source file feeding “./myprogram -v” or the About screen
- the binary package description file(s) (e.g. *.spec for RPM)
- the Fossil release tag
- the comment on the Fossil checkin creating the release tag

We drive everything from the first item on that list, except that the ChangeLog 
remains manually-updated.  In that sense, we have “only” two places to change 
the version number, too.

The formal release version number in our system has two parts: the human-facing 
version number (e.g. 1.2.3) and the binary package release number (e.g. -1) 
which combine to give the formal release number, 1.2.3-1.  These two parts may 
be in the same top-level file (e.g. CMakeLists.txt for a CMake based project) 
or in separate files (e.g. configure.ac and *.spec.in for an Autoconf + RPM 
based project).

We use the binary package release number to distinguish two different Fossil 
checkins both tagged someproject-1.2.3.  The checkin comment for the first tag 
will say “Release someproject v1.2.3-1”, and the second “…-2”.

The Fossil tag checkin and its checkin comment are generated by a mkrel script 
we wrote which parses the version number out of one of the generated files, 
such as *.spec file in an Autoconf + RPM based project, which as I have said is 
generated from *.spec.in and configure.ac.

Other files come off of this, such as a version.h file for a C++ project, 
generated from version.h.in by Autoconf, which gets it from configure.ac.

Because we want the binary package release number in the Fossil release tag’s 
checkin comment, generated by the mkrel script, Fossil doesn’t get involved 
until nearly the end of this process.  My initial thought from that observation 
is that this means we can’t really replace our current scheme with your scheme. 
If you had to redesign our scheme to use your new feature, how would you go 
about it?

I don’t mean to dismiss your efforts, Jan.  It solves a real problem, and I 
wish it was available before we started solving these problems on our own, many 
years ago.  I’m willing to offer a weak sort of endorsement for trunkifying it, 
since I haven’t played with it, but I can see that it’s useful already.

> Someone(tm) needs to make sure they match.

Or, as we have both discovered, some *thing*.  This is a task for automata, not 
humans.

> Each new released version is tagged using a -
> tag.

Tagged how?  Manually?

> The build system iterates through each line of manifest.tags

I don’t see what your build system is getting from manifest.tags that it 
couldn’t already get from the T card in the current manifest file.

>   Thoughts?

Your vague use of  makes me wonder if you’re using this in any 
Fossil repositories that host multiple related projects that nevertheless build 
to separate outputs, such as the client and server for a distributed system, 
giving two separate binary packages, each possibly with different version 
numbering.

Does your build system use this project name to detect which binary package(s) 
to build?
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] tags manifest

[Robertson, Joseph M.]

I do the opposite.

Coders check in code, but the build system manages both the version id files 
and the labeling, so they are always in sync.
(we are not using fossil as our vcs at work)

My steps are:
Check out code
Increment versionids in .h or .java files
Build
(verify using human eyeballs and quick tests)
Checkin work products and .h/.java id files
Label sources and work products and .h/.java id files.

This process should work for any vcs, whereas yours only works for fossil.


Joe


-Original Message-
From: fossil-users [mailto:fossil-users-boun...@lists.fossil-scm.org] On Behalf 
Of Jan Danielsson
Sent: Wednesday, August 03, 2016 11:24 AM
To: Fossil SCM user's discussion
Subject: [fossil-users] tags manifest

Hello,

   Since I've finally got a few days off, I've done some updates to the 
jan-manifest-tags; merge with trunk, made some bugfixes, and thought I'd 
reintroduce the branch (since it's been a while).

   The basic idea is the following:  It's common practice for projects to tag 
new release versions of their software.  Separately from that, there's a 
version number somewhere in the source tree (like #define VER_MAJ 1, #define 
VER_MIN 2, etc).  This means that the same information is available in two 
different places and Someone(tm) needs to make sure they match.  The idea for a 
tags manifest-like feature came about when I was working in a high-assurance 
project a long time ago and it happened once or twice that due to human errors, 
the two versions didn't match.  (not-so-high-assurance..).

   Tags manifests allows build systems to be able to extract the tagged version 
information from manifests generated by fossil.  This has the positive 
side-effect that the build system will work in tarballs and zipfiles generated 
by fossil, as they will contain the manifest files (provided the server has the 
appropriate manifest settings..).

   One addition (as per feedback) was to include the current branch at the top 
of the manifest.tags.  It uses the special "branch=" prefix.
Note that this is only done when operating in an open working tree.
I.e. the branch=... will not be available in generated tarballs and zipfiles, 
so don't make your build system rely on the "branch="-entry if you want you 
want the build system to work in exported tarballs or zipfiles.

   How we use it:

   Each new released version is tagged using a - tag.  
Our build server checks out a clean copy of a tagged version.  The build system 
iterates through each line of manifest.tags and looks for -X.Y.Z 
(where X, Y, and Z are numbers).  Once found, it either uses the version 
numbers to generate a header file, or a -DVERSION=... argument to the compiler. 
 For non-release builds, such an entry will not be found in manifest.tags, and 
it will use a special version number to indicate it's a development build.

   Other uses include using special (propagating) tag names (like
"experimental") to make the build system take special actions.

   We've been using a version of the branch for a pretty long time now, and I'm 
starting to feel that it's ready for trunkification.

   For those who are afraid of new features messing with your current
setup:  The jan-manifest-tags branch is explicitly designed to not interfere 
with your current setup.  You need to take action for a change to occur.  If 
you use "set manifest on" or "set manifest off" it will work exactly like it 
always has.  I.e. if you don't care about this feature, you - as an end user - 
will not notice it's there (apart an updates settings field and help text).

   Thoughts?

--
Kind Regards,
Jan
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

-- 
CONFIDENTIALITY
This e-mail message and any attachments thereto, is intended only for use 
by the addressee(s) named herein and may contain legally privileged and/or 
confidential information. If you are not the intended recipient of this 
e-mail message, you are hereby notified that any dissemination, 
distribution or copying of this e-mail message, and any attachments 
thereto, is strictly prohibited. If you have received this e-mail message 
in error, please immediately notify the sender and permanently delete the 
original and any copies of this email and any prints thereof.
ABSENT AN EXPRESS STATEMENT TO THE CONTRARY HEREINABOVE, THIS E-MAIL IS NOT 
INTENDED AS A SUBSTITUTE FOR A WRITING. Notwithstanding the Uniform 
Electronic Transactions Act or the applicability of any other law of 
similar substance and effect, absent an express statement to the contrary 
hereinabove, this e-mail message its contents, and any attachments hereto 
are not intended to represent an offer or acceptance to enter into a 
contract and are not otherwise intended to bind the sender, Sanmina 
Corporation (or any of its subsidiaries), or any other person or entity.

-- 
CONFIDENTIALITY
This