Re: [boinc_dev] [boinc_projects] keywords

[David Anderson]

On 7/20/2017 2:11 AM, Oliver Bock wrote:

just meant to improve things - to make BOINC better. For instance, I for
one just can't take over a responsible (!) role as release manager for,
say, Android unless there's a codebase and a workflow I can trust. If

As you know, client releases (including Android) are build from
a stable branch like client_release/7/7.8.
Not from master.
The workflow for master isn't relevant here.
Let's not confuse things.

Adopting a particular SCM model isn't a magic wand that will somehow fix 
everything.
The issues you mention are, I think, not much related to SCM; e.g.:
- The reasons small research groups aren't creating BOINC projects is that the
  risk/reward economics are unfavorable and the documentation is sub-par.
- Some projects made changes to the server code and didn't do the 10% extra work
  of merging them into master, so it becomes progressively harder for them to 
upgrade.


___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[David Anderson]

This discussion comes down to two contrasting models for software development:

1) The "waterfall model":
https://en.wikipedia.org/wiki/Waterfall_model
New features are done as a unit, with sequential phases
(requirements, design, implementation, verification, maintenance).
E.g., requirements and design don't change during implementation.
Typically, development is done in a separate branch,
which is merged with master at the end.

2) The "agile model":
https://en.wikipedia.org/wiki/Agile_software_development
New features are divided into smaller self-contained pieces,
which are completed and merged with master frequently.
Requirements and design can change on the fly.
Development need not be done in a separate branch.

The role of master differs somewhat.
It changes more frequently in the agile model.
Master is unstable in both models; releases are done from stable branches.
In both models there may be periods when master doesn't work.
These periods are more frequent in the agile model.

---

Each model has pros and cons.
The waterfall model is generally regarded as old and ineffective.
The corporate software world has found that the agile model generally works better 
for them,

and has moved to it en masse.
For distributed open-source projects like BOINC,
the agile model has some aspects - such as a daily face-to-face "scrum" meeting 
-
that would be difficult to do, so it must be adapted somewhat.

Oliver prefers the waterfall model,
so strongly that he won't work with people who use agile.
I'll leave it to him to explain this preference.

I prefer the agile model for several reasons:

1) In the way I develop, requirements/design/implementation proceed iteratively and 
in parallel.

I find many situations where the implementation has become difficult or awkward,
and I can fix this by changing the design or even the requirements.

2) I often work on several things at once.
If I do each one in a separate branch,
I inevitably commit code to the wrong branch,
and it's hard to untangle things later.

3) I want my code reviewed and tested by others ASAP.
I have only Win7 and Linux computers.
If my code doesn't work on other platforms, I want to know this ASAP.
I want to commit to master in small-size, frequent units.

4) It's worked extremely well so far.
BOINC has evolved rapidly and continuously over 15 years.
It's still very malleable and well-structured, making it easy to change and add.
Very few big software systems can say this.


Which model should we use in BOINC?
I don't think we need to pick one or the other.
The owner of each software area (client, manager, etc.) could choose one.
Or we could leave it up to the developer.

And, as I've said repeatedly, the major problems facing BOINC -
such as the inadequacy of documentation
and the lack of volunteers for software release management -
don't involve software development,
the choice of model doesn't have much bearing on them.
Let's not get hung up on it.

Also, let's not confuse the choice of model with how we do stable branches.
Of course we need stable branches for server software and API.
The agile model doesn't preclude this.

-- David

___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[Richard Haselgrove]

I think this sort of discussion is exactly what I was thinking of when I asked 
in my initial working group paper (Christian, please copy to Bernd and Oliver, 
and explain the background - if you haven't already), when I asked something 
like:

What is the current status of BOINC? Is it a computer science experiment, or is 
it mature scientific infrastructure?

To me, as a relative outsider and not a technical expert at your levels, it 
feels as if "throw everything into master and see if it floats" is the 
experimentalist's answer. I hope I'm not distorting your words too much if I 
suggest that the 'mature infrastructure' approach - if it is to survive the 
next 15 years - needs pre-release development branches for each of client, 
server, and API, all with their own distinct version control; and a separate 
branch for major direction changes like the NSF-funded TBD (which I suspect - 
though it wasn't made explicit at the time of the commit - is what really 
sparked this thread off).

On Thursday, 20 July 2017, 16:12, Oliver Bock  
wrote:
 

 On 20/07/17 11:11 , Oliver Bock wrote:
> If you don't adapt and progress it
> can only get worse, in particular from the point of view of newly
> interested scientists and contributors.

Just in case this came across in an offending way: the "you" wasn't
meant personally, it was meant as in "one" or referring to BOINC as a
project. Sorry if I phrased this ambiguously in my non-native tongue.

Oliver
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

   
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[Oliver Bock]

On 20/07/17 11:11 , Oliver Bock wrote:
> If you don't adapt and progress it
> can only get worse, in particular from the point of view of newly
> interested scientists and contributors.

Just in case this came across in an offending way: the "you" wasn't
meant personally, it was meant as in "one" or referring to BOINC as a
project. Sorry if I phrased this ambiguously in my non-native tongue.

Oliver



smime.p7s
Description: S/MIME Cryptographic Signature
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[Christian Beer]

Just so we are all on the same page about what issue is discussed here
and how an "integration branch" can solve this.

A project wants to use a version of the BOINC code that is known to be
stable (because usually you choose a released version of a software to use)
The BOINC server part has no versioning thus projects can only checkout
the branch named "master" at the moment.
The code in this branch is not guaranteed to work for the project, it is
not even guaranteed that the code in this branch compiles successfully.
So currently you could characterize "master" as a development branch.
All development is happening on this branch and it is supposed to be
buggy and broken from time to time. That is not what a project wants to
use for a public facing project.
Because using a development branch means the project needs to do tests
before deploying the code on the public project or risk a lot of cleanup
work if problems are encountered afterwards. This is usually not what
project administrators are paid for, it might even be that the project
admin does not have the skills to troubleshoot problems in the BOINC code.

Remember projects want to have the possibility to get source code that
has already been tested (to some degree) and at least is guaranteed to
compile. Having an "integration branch" would solve this issue.
Because the workflow changes a bit when new features and bugfixes are
developed and a branch is named the "integration branch".
New features are developed in so called "feature branches" that are
forked from the "integration branch".
When a feature is mature enough and tested to do what it should do it is
then merged into the "integration branch".
This way it is guaranteed that the "integration branch" has some
properties that are wanted by projects (see above).
In this scheme the "feature branches" are characterized as "development
branches" and the branch called "master" is the "integration branch".

I found a nice graphical overview how this "Integration branch" system
works too: https://backlogtool.com/git-guide/en/stepup/stepup1_5.html

In this approach the burden to test features and bugfixes is moved from
the project to the developer (where it rightfully belongs). Each
developer can also make sure that he/she is not meddling with
development going on at the same time by other developers (because they
are using different feature branches). This approach also helps
developers to make sure that they don't introduce unnecessary bugs or
typos into a branch that is used by projects because they would be
spotted in the feature branch.

This "integration branch" approach can later be augmented by automated
tests that regularly run on the "integration branch" to make sure it is
as stable as we want to have it. You can simply increase stability of
the "integration branch" by simply adding useful unit tests that test
individual features as well as an overall integration test that tests
all features (to some extend).

I hope this clears up how the "integration branch" approach can help to
fix the issue Oliver and I presented.

Regards
Christian

On 20.07.2017 12:06, David Anderson wrote:
> As far as I can tell, none of the issues you mention have anything to
> do with whether
> we call master a "development branch" or an "integration branch".
>
> You seem to think that people who don't buy into your world view are
> resistant to change.
> Not the case.
>
> If we had more energy going to understanding code, fixing bugs,
> writing documents, etc.,
> from the people who are able to do so, rather than dogmatic posturing,
> we'd all be better off.
>
> On 7/20/2017 2:11 AM, Oliver Bock wrote:
>> On 20/07/17 10:09 , David Anderson wrote:
 Sorry, but this isn't a sustainable model.
>>> Well, it's sustained us this far.
>> Well, in that case we have a different understanding of "sustainable".
>> Yes, BOINC survived, barely, but did it strive? Is it easy (let alone
>> fun) to use for downstream projects? Did it attract a healthy number of
>> volunteer developers after the alleged transition into a community
>> driven open source software (OOS) project? Sure, source code management
>> (SCM) isn't all there's to it but it's a significant part of it. Maybe
>> you underestimate its effect? I'm also nearing a ten-year involvement in
>> BOINC and way too often it's been a pain to use it as a downstream
>> project because of the way its SCM is done. It's codebase is brittle and
>> just can't be trusted, unnecessarily. 15 years ago the tools and best
>> practices were different. But these things are moving. Why not
>> participate in the progress that's been made since then, making
>> everyone's professional life easier and BOINC more attractive? Yes, that
>> means change, but it doesn't hurt. If you don't adapt and progress it
>> can only get worse, in particular from the point of view of newly
>> interested scientists and contributors.
>>
>> What I'm trying to say is: responding to constructive criticism with
>> 

Re: [boinc_dev] [boinc_projects] keywords

[Oliver Bock]

On 20/07/17 12:06 , David Anderson wrote:
> As far as I can tell, none of the issues you mention have anything to do
> with whether
> we call master a "development branch" or an "integration branch".

I know. That's why I keep trying to address that.

> You seem to think that people who don't buy into your world view are
> resistant to change.
> Not the case.

If you really think that this is "my" (exclusive) world view than that's
part of the problem. Sure, our opinions differ and that's ok. Please
consider that your world view is also subjective and should be reflected
upon not to get skewed. Do you honestly think that either BOINC's
software engineering significantly improved over the last 10 years or
that it's just fine as it is? Why do projects still face upgrade issues
and broken builds (see GPUGrid)? Why are client releases like 7.8 still
broken, despite having a comparatively elaborate release workflow? SCM
methodology plays a large role in this.

> If we had more energy going to understanding code, fixing bugs, writing
> documents, etc.,
> from the people who are able to do so,

You're right. But in order to do that I think two things need to happen:

1) Improve the processes (SE/SCM workflow, automation, etc) such that
energy can be used for the things you listed. Don't waste energy on bugs
and self-inflicted issues that could be avoided by improved processes.
That said, I'm proposing ways to increase efficiency which is
particularly important considering the scarce resources at hand. That
should be in BOINC's best interest.

2) Attract more energy by being an attractive OSS project. Granted, we'd
have to agree on what spurs attractiveness first.

> rather than dogmatic posturing,

Oh, that's what I do? "Dogmatic posturing"? To the contrary, I'm making
proposals that are fully open to discussion. I like to start discussions
to improve things that I (and a number of others) consider to be in need
of improving. Sadly, I'm often enough the only one (or one of very few)
who actually (still) publicly sticks his neck out and voices
constructive criticism. Do you think that calling my well-intended
attempts "dogmatic posturing" can be considered "open", helpful to a
previously factual discussion or is by any means inviting further
contributions? There's no need to make this personal.

Oliver



smime.p7s
Description: S/MIME Cryptographic Signature
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[Jord van der Elst]

On Thu, Jul 20, 2017 at 1:12 PM, Bernd Machenschalk <
bernd.machensch...@aei.mpg.de> wrote:

> Rosetta's is way older (2009?)
>

Correction: Rosetta updated their hardware and BOINC version to 'the
latest' on 23 June 2017 (http://boinc.bakerlab.org/rosetta/) (
http://boinc.bakerlab.org/rosetta/forum_thread.php?id=6893)

-- Jord van der Elst.
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[Bernd Machenschalk]

On 2017-07-20 10:09, David Anderson wrote:

On 7/20/2017 12:37 AM, Oliver Bock wrote:


Sorry, but this isn't a sustainable model.

Well, it's sustained us this far.


I'm not so sure about that, at least if "us" should cover more than 
SETI. Most major projects that I know of use BOINC server code that is 
as outdated as their requirements allow, and do their own development 
based on that old version. Ours (E@H) dates back to early 2012, 
Rosetta's is way older (2009?), WCG uses the first git migration (late 
2012).


Projects are worried first about stability and reliability, then about 
features they may need. You certainly don't want to build your project 
code - that should reliably manage some 100k machines of some 10k users 
that you don't want to alienate - from a branch that every intermediate 
half-baked commit of a handful of features that is worked on in parallel 
gets committed to. That's what's driving GPUgrid away, and that shows 
that - in my understanding of 'sustain' - it DOESN'T sustain.


We could have a server stable branch, as you've suggested, if we could 
figure out how to test the server software.


A system as complex as BOINC can only be trusted as a whole by building 
up trust from the ground up, i;e; by building trust to its components. 
This means that testing the "server software" means and requires testing 
its components. Testing, of course, can only mean testing the behavior 
compared to a specification (even if the specification isn't written 
down explicitly; then the test defines the specification). This, in 
turn, means you would better have a specification, at least for each 
module and interface, which first should be verified on its own for 
correctness and completeness. Then you can write a test 
case/program/whatever for this module. Then you can test this module to 
conform to your specification.


Basically there is no point in testing a module or a system as long its 
(desired) behavior isn't specified.


Best,
Bernd

___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[David Anderson]

As far as I can tell, none of the issues you mention have anything to do with 
whether
we call master a "development branch" or an "integration branch".

You seem to think that people who don't buy into your world view are resistant to 
change.

Not the case.

If we had more energy going to understanding code, fixing bugs, writing 
documents, etc.,
from the people who are able to do so, rather than dogmatic posturing, we'd all be 
better off.


On 7/20/2017 2:11 AM, Oliver Bock wrote:

On 20/07/17 10:09 , David Anderson wrote:

Sorry, but this isn't a sustainable model.

Well, it's sustained us this far.

Well, in that case we have a different understanding of "sustainable".
Yes, BOINC survived, barely, but did it strive? Is it easy (let alone
fun) to use for downstream projects? Did it attract a healthy number of
volunteer developers after the alleged transition into a community
driven open source software (OOS) project? Sure, source code management
(SCM) isn't all there's to it but it's a significant part of it. Maybe
you underestimate its effect? I'm also nearing a ten-year involvement in
BOINC and way too often it's been a pain to use it as a downstream
project because of the way its SCM is done. It's codebase is brittle and
just can't be trusted, unnecessarily. 15 years ago the tools and best
practices were different. But these things are moving. Why not
participate in the progress that's been made since then, making
everyone's professional life easier and BOINC more attractive? Yes, that
means change, but it doesn't hurt. If you don't adapt and progress it
can only get worse, in particular from the point of view of newly
interested scientists and contributors.

What I'm trying to say is: responding to constructive criticism with
"that's how we've done it for 10+ years, no need to change" isn't very
helpful if you want to see things improve. My proposal is (once again)
just meant to improve things - to make BOINC better. For instance, I for
one just can't take over a responsible (!) role as release manager for,
say, Android unless there's a codebase and a workflow I can trust. If
ideas, proposals or contributions aren't discussed at all or rejected
without meaningful reasoning people will stop trying. I'm sure that's
not in your interest.

Best,
Oliver



___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[David Anderson]

On 7/20/2017 12:37 AM, Oliver Bock wrote:


Sorry, but this isn't a sustainable model.

Well, it's sustained us this far.
We could have a server stable branch, as you've suggested,
if we could figure out how to test the server software.
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[Oliver Bock]

Hi David,

On 20/07/17 9:11 , David Anderson wrote:
> Master is for new development.

Sorry, but this isn't a sustainable model. Since only the client uses
release branches *all* other components (e.g. the server!) depend on
master being "usable". Therefore master should always be as stable as
possible. That said, development has to happen in feature/fix branches
that are merged into master *after* they're tested. In that sense the
master branch ought to be considered as an integration branch, not a
development branch.

Please revisit what I wrote here for more details:
https://github.com/BOINC/boinc/issues/1874

Best,
Oliver




smime.p7s
Description: S/MIME Cryptographic Signature
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [boinc_projects] keywords

[David Anderson]

On 7/19/2017 2:57 AM, Christian Beer wrote:

understand that some exploratory implementation should be done in the
meantime and I'm fine with that. But this should be happening in a
separate branch NOT in the master branch which is currently used to
build a 7.8 release.

The 7.8 client release uses the "client_release/7/7.8" branch, not master.
Master is for new development.
This is explained here:
https://boinc.berkeley.edu/trac/wiki/AdminReleaseManagement

-- David
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.