Re: [boinc_dev] GoogleTest

[Kevin Reed]

> From: Vitalii Koshura 
> To: BOINC Developers Mailing List 
> Date: 01/17/2018 02:36 PM
> Subject: [boinc_dev] GoogleTest
> Sent by: "boinc_dev" 
>
> Hello @all,
>
> As I wrote before I want to add unit test support to test existing XML
> parser and move to another 3rd-party XML parser further.
>
> So I have a question: can we use Google Test library in our project ()? I
am curious about the license. I
> do not have understanding with all these licenses so I ask you to help me
> with this.
>
> Also does anyone have any objections to use this library?
>

I don't have insight into the license usage, but WCG is about to start (in
the next two weeks) working on submitting a series of changes to the server
code as part of our efforts to update our server to the latest.  We would
like to be able to leverage this framework during this work so that we can
implement test cases while we add our changes.  As a result I would say
that we look forward to using the framework.

Thanks,
Kevin
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] server stable release issues

[Kevin Reed]

"boinc_dev"  wrote on 01/12/2018
02:54:44 AM:

> From: Laurence 
> To: 
> Date: 01/12/2018 02:56 AM
> Subject: Re: [boinc_dev] server stable release issues
> On 11/01/18 22:29, Marius Millea wrote:
> >
> >> 3) Assuming the cycles are independent, how should we number server
> >> releases?
> >> We could use the same numbering for both, but this has problems:
> >> - A sequence of releases in one part would cause gaps in the release
> >> numbers
> >>of the other part.
> >> - It would create the false impression that there's a connection or
> >>dependency between client and server releases with the same number.
> >> So my recommendation is that we start with 1.1.1 for the server
version
> >> number.
> >>
> > In the limited number of "releases of
> > boinc-server-docker that I've made to-date, I struggled with using
semantic
> > versioning (like 1.1.1) because it was always tough to know whether
> > something was a backwards compatible change (if we're being honest, I
think
> > *alot* of server commits are technically backwards incompatible), and
> > whether something was a bug-fix or true feature. Ultimately, it wasn't
> > clear it really mattered or will matter for us, the real goal is just
> > having a stable point. For that reason, I would recommend just doing
> > something easy and clear like "calendar versioning"
> > with e.g. YY.MM.MINOR. Anyway, I don't feel extremely strongly about
this,
> > but thought I'd suggest it.
> >
> The server must always be backwards compatible with the client. The
> general approach is roll forward with versioning. Hence, the most
> important aspect is that the version numbers are incremental. Try to
> avoid adding too many semantics to them.  The basic major.minor.patch
> semantics will lead to divergence of version number between the client
> and the server. This would be an argument for having them in separate
> repositories. But practically it does not matter.
>

Just my 2 cents.

1) The server should strive to be compatible with versions of the client
released with the past few years (maybe 4-5 years).
2) I vote in favor of semantic numbering.  Since we might test version 1.2
of the server and then release it in say Feb 2018 as version 1.2.5.  Master
might move on to version 1.3 (assuming same dev/release odd/even numbering
as client).  Then in June we might find a critical bug and fix the 1.2
release and release it as 1.2.6 but also start a 1.4 release cycle from the
1.3.  It would be unclear with a date oriented numbering what we would call
1.2.6 and 1.4.1.  This assumes that we will create stable release branches
for the server like we do for the client (which is something I am in favor
of).

thanks,
Kevin
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

[boinc_dev] BOINC Server Test Integration POC

[Kevin Reed]

Greetings,

I've finished putting together a proof of concept for a integration test
framework for testing the BOINC server.  I'd like it if a few people could
take a look and let me know if they think it is useful and could be
extended by the community as needed.  The goal would be to make this part
of the tests that are started by github when a new pull request is created
in order to validate that the system works as expected.  More information
and instructions to run the tests are available here:
https://github.com/TheAspens/boinc-server-test  I've tested it on Cent OS 7
and Ubuntu 16.04.

The goal is not to build a set of tests up front but rather add tests as
needed to help ensure that the server code is stable and working as
expected as we make changes to it.  This would complement the work proposed
by Vitalii back in Nov to use the google test project (
https://lists.ssl.berkeley.edu/pipermail/boinc_dev/2017-November/023055.html
) to do unit level testing.

Please give this a try and let me know what people think.  This would be
contributed to BOINC and be set up as a repository under
https://github.com/BOINC if we move forward with this.

Thanks,

Kevin

...

 knr...@us.ibm.com
 Donate your computer's unused power to humanitarian research:
www.worldcommunitygrid.org

WCG
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

[boinc_dev] Prosed changes to BOINC Governance Document from the Working Committee

[Kevin Reed]

The following proposed changes were developed by the working committee and
each item was approved by the committee as a recommendation to the PMC for
consideration. The major changes consist of the following:

  Add the supporters role to recognize the special role and
  responsabilities that non-developers can play in the ongoing support
  and maintenance of the project
  Create the position of PMC Secretary who is responsible for various
  tasks related to documenting and publish the activities of the PMC
  Set the PMC Chair and PMC Secretary to have one year terms
  Establish guidelines for conducting elections for PMC Chair and PMC
  Secretary
  Clarify that people thought leaders from other communities can become
  part of the PMC if they have a interest in supporting the BOINC
  project and volunteer computing
  Establish a process for removing inactive PMC members

For more details about these changes and to discuss these changes, please
see the BOINC admin mailing discussion here:
https://groups.google.com/forum/#!topic/boinc_admin/Fj-bgcM4qoM

Thanks,

Kevin
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] Build Windows Client

[Kevin Reed]

Juha Sointusalo <juha.sointus...@gmail.com> wrote on 11/16/2017 03:23:50
PM:

>> On 16 November 2017 at 23:04, Kevin Reed <knr...@us.ibm.com> wrote:
>> That makes sense.  Thanks for letting me know.  Has there been a
decision
>> about when BOINC will stop supporting Windows XP?
>
> David said recently, somewhere can't remember where, that he would
> like that XP is still supported. I think Seti still has/had a couple
> of thousand XP host.

Ok.  WCG currently has 2,042 Windows XP hosts with expavg_credit > 0.5.
That is 0.7% of our active hosts and 0.5% of our computing power.

I haven't been tracking that as a trend over time, but if 7.8 were to be
the last client supporting Windows XP we would be ok with that.  Most
Windows XP users and projects would be able to continue to accept
contributions from them using the last version of the client that support
Windows XP for several years.  We still have a surprising number of users
using the BOINC 6.10 client.

However, I don't know if 2013 (or newer) offers any advantages over 2010 so
I'm not making a recommendation or proposal - just citing some information.
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] Build Windows Client

[Kevin Reed]

Juha Sointusalo <juha.sointus...@gmail.com> wrote on 11/16/2017 02:29:51
PM:

>
>> On 15 November 2017 at 23:40, Kevin Reed <knr...@us.ibm.com> wrote:
>> The dependencies included in the 7.8.2 client version appear to be newer
>> than the ones in the boinc_depends_win_vs2013 repository so I was
wondering
>> if there was a different source.
>
> I believe there was some problem with XP compatibility with VS2013
> builds so Rom and David went back to building the official releases
> with VS2010. I suppose at that point keeping the VS2013 dependencies
> up-to-date was forgotten.
>

That makes sense.  Thanks for letting me know.  Has there been a decision
about when BOINC will stop supporting Windows XP?

For the record, building the client using vs2013 and the 2013 dependencies
was pretty straight forward and the binary ran correctly on Windows 7 (in
case anybody was wondering if the build even worked).
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

[boinc_dev] Build Windows Client

[Kevin Reed]

Greetings,

I'm building the windows client as per the directions found on
https://boinc.berkeley.edu/trac/wiki/CompileClient

Are the instructions for getting the dependencies still correct based on:
https://boinc.berkeley.edu/trac/wiki/SourceCodeGit#Windowsbuilddependencies
?

The dependencies included in the 7.8.2 client version appear to be newer
than the ones in the boinc_depends_win_vs2013 repository so I was wondering
if there was a different source.

Thanks,

Kevin

...

 knr...@us.ibm.com
 Donate your computer's unused power to humanitarian research:
www.worldcommunitygrid.org

WCG
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] Proposed update to code review guidance

[Kevin Reed]

Kevin Reed/Chicago/Contr/IBM wrote on 11/14/2017 05:18:05 PM:
>
> Based on a discussion the working committee had today, I've created
> a pull request to update the guidance for code reviews to reflect
> the balance between rigorous code reviews and rapid code reviews.
>
> The new version of the document is viewable here:  https://
> github.com/BOINC/boinc-policy/blob/knr-code-reviews/
> Expectations_of_Code_Review.md
>
> The pull request is here: https://github.com/BOINC/boinc-policy/pull/8
>
> I've requested a vote of the PMC to approve these changes in this thread:

> (removed link because it was the wrong one)
>
> If anyone has any feedback about the proposed changes, please
> contribute them on that discussion thread.

Actually the link to the request of the vote of the PMC is here:
https://groups.google.com/forum/#!topic/boinc_admin/bfRl6-heSYs
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

[boinc_dev] Proposed update to code review guidance

[Kevin Reed]

Based on a discussion the working committee had today, I've created a pull
request to update the guidance for code reviews to reflect the balance
between rigorous code reviews and rapid code reviews.


The new version of the document is viewable here:
https://github.com/BOINC/boinc-policy/blob/knr-code-reviews/Expectations_of_Code_Review.md


The pull request is here: https://github.com/BOINC/boinc-policy/pull/8


I've requested a vote of the PMC to approve these changes in this thread:
https://groups.google.com/forum/#!msg/boinc_admin/9mFdTzp4rq4/2CoBV2SuCAAJ

If anyone has any feedback about the proposed changes, please contribute
them on that discussion thread.

Thanks,

Kevin
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

[boinc_dev] Proposal for Use of GitHub Project Boards

[Kevin Reed]

Greetings,

I have seen previous discussions by at least Christian Beer and Vitalii
Koshura about using GitHub project boards.  I'd like to propose the
following guidelines for using them:
https://github.com/BOINC/boinc-policy/blob/project-boards/Issue_and_Pull_Request_Management.md

I've opened the following pull request to have this merged into the
BOINC/boinc-policy repository: https://github.com/BOINC/boinc-policy/pull/7
It would be great if we could have the discussion about the proposed
guidelines in the "conversation" section of the pull request.

The rationale for this change is that there are currently 339 issues that
can be seen here:  https://github.com/BOINC/boinc/issuesThis view of
issues makes it hard to see which ones belong to which component of BOINC,
which ones are higher priority than others and which ones have been
reviewed and are ready to implement vs ones that have been quickly opened
and don't have sufficient detail for a developer to implement.

If we use project boards, its makes it much easier for contributors to find
issues that pertain to a part of the system that they can contribute to and
that they know is important and ready to implement vs ones that are less
critical or are not ready to implement.

The goal for this proposal is for the guidelines to be simple and useful.
Please review and suggest ways to make the guidelines simpler and more
useful.

Since this is a development process document, it requires a consensus vote
of the committers as outlined in the governance document (
https://github.com/BOINC/boinc-policy/blob/project-boards/Governance.md#511-consensus-voting
).  What this means practically, is that we can discuss this for a week
updating the document based on comments and then if no-one objects, a
committer other than me can merge my pull request into the master branch of
boinc-policy.  In the future, if someone wants to change it, they simply
submit a pull request that changes the document which then starts another
discussion period.

Thanks,

Kevin

...

 knr...@us.ibm.com
 Donate your computer's unused power to humanitarian research:
www.worldcommunitygrid.org

WCG
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] Unit tests to validate XML requests and responses

[Kevin Reed]

On 09/11/17 2:14 Laurence Feld wrote:
> On 05/11/17 14:19, Vitalii Koshura wrote:
> > What do you think, guys?
> I think this is a good idea. I would also like to see the client-server
> RPC documented and tested.

I'd also like to vote in support of this idea.  WCG is about to start
working on updating our server software (in 2-3 weeks) and as I integrate
our changes in I wanted to have a TDD framework available for unit testing
and this looks like it will work well.

Thanks,

Kevin

...

 knr...@us.ibm.com
 Donate your computer's unused power to humanitarian research:
www.worldcommunitygrid.org

WCG
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

[boinc_dev] Tool for Building Website

[Kevin Reed]

Greetings,

At the workshop last week, Marius Millea demonstrated how he builds and
runs Cosmology@Home using docker containers and how these docker containers
are available for use by everyone.  You can see his project here:
https://github.com/marius311/boinc-server-docker

I've written some scripts using ansible that leverages Marius project that
will let you easily build the BOINC server code from an arbitrary branch.
This makes it easy to build a project from scratch for testing purposes.
For example, I used this code to recreate the bug fixed in pull request
https://github.com/BOINC/boinc/pull/2110 by building from master and then
rebuild the project from 'progger's branch to test that it was fixed.

This might be useful to other people doing development or reviewing other
people's work on the BOINC repository so I've made it available here:
https://github.com/TheAspens/boinc-server-test

Later this will be enhanced so that it can run a set of basic integration
tests against the newly built containers.
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

[boinc_dev] Project management changes proposed

[Kevin Reed]

The BOINC PMC chartered a committee in July to review how the BOINC
community was operating and identify areas for improvement.  The committee
is meeting periodically to talk through and produce concrete proposals for
review and discussion by the community. The work of this committee will be
continuing with additional proposals coming later.

The first products of this effort are documents around the process for
developing code.  There are two documents that we would like to open up to
the community for discussion and review.  After a period of discussion with
community, the final version of these documents will be voted on for
adoption by the BOINC PMC.

The initial documents are: A revised BOINC Governance Document (
http://boinc.berkeley.edu/trac/wiki/BOINC_Governance_Document_v2) and a
Development Workflow (
http://boinc.berkeley.edu/trac/wiki/Development_Workflow).

We would like to receive feedback on these proposals from the BOINC
community via either the discussion on the BOINC message boards (
https://boinc.berkeley.edu/dev/forum_thread.php?id=11812) or in the
boinc_dev mailing list.
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
https://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [eah_android] Re: android: transfers in Projects tab

[Kevin Reed]

I agree.  Succinct and informative.

thanks,

Kevin Reed
Senior IT Architect



...

 knr...@us.ibm.com
 312.529.2802
 ibminteractive.com Follow IBM Interactive on Facebook and Twitter:
facebook twitter


Donate your computer's unused power to humanitarian research:
www.worldcommunitygrid.org



From:   Oliver Bock oliver.b...@aei.mpg.de
To: Joachim Fritzsch joachim.fritz...@gmail.com
Cc: Kevin Reed/Chicago/IBM@IBMUS, Keith J
Uplinger/Austin/IBM@IBMUS, eah_andr...@aei.mpg.de
eah_andr...@aei.mpg.de, Rom Walton r...@romwnet.org,
boinc_dev@ssl.berkeley.edu boinc_dev@ssl.berkeley.edu
Date:   10/07/2013 04:04 AM
Subject:Re: [eah_android] Re: android: transfers in Projects tab



On 10/1/13 19:55 , Joachim Fritzsch wrote:
 I have uploaded screen shots to my public dropbox folder:
 https://www.dropbox.com/sh/i1ebp0xffoujx3m/H_jKaIQwNa

 It is quite hard to capture, since most transfers are over within
 seconds on a decent wifi connection.

 On one of the screen shots you can also see the implementation of the
 Notices tab:
 - The notices are filtered and only show entries of the RSS feed,
 scheduler notices are shown in the Projects tab (additional storage
 space needed) and client notices (update available) is not needed on
 Android.
 - A touch on one of the entries takes the user to the web link, if
available
 - The time stamp is the item creation time.

Looks good to me and seems reasonable.

Best,
Oliver

inline: 19352916.jpginline: 19462290.jpginline: 19463613.gifinline: 19231529.jpginline: 19144776.gifinline: 19440247.gifinline: graycol.gif___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [eah_android] BOINC o.A. - client power management bug

[Kevin Reed]

The purpose of the setting temperature  Y causing a suspended state is
to prevent it reaching an overheated state.  I would suggest that we not
use the text OVERHEATED as that would imply that we had failed to protect
the users device.

Perhaps something such as THERMAL_PROTECTION would be more appropriate.

suspend reason THERMAL_PROTECTION

Indicates that the software is proactive and protective.


 From: boinc_dev [mailto:boinc_dev-boun...@ssl.berkeley.edu] On
 Behalf Of Bruce Allen
 Sent: Monday, April 08, 2013 3:41 AM
 To: Joachim Fritzsch
 Cc: boinc_dev@ssl.berkeley.edu; eah_andr...@aei.mpg.de
 Subject: Re: [boinc_dev] [eah_android] BOINC o.A. - client power
 management bug

 Hi All,

 In order to help keep communication with the user as clear as
 possible, I suggest that you introduce one additional suspend reason:

  - run_on_batteries = 0
  -- charger connected, battery level  X, temperature  Y: not suspended
  -- charger connected, battery level  X, temperature  Y: suspend
 reason CHARGING
  -- charger connected, battery level  X, temperature  Y: suspend
 reason OVERHEATED
  -- charger connected, battery level  X, temperature  Y: suspend
 reason OVERHEATED

 suggested suspend reason for this case: OVERHEATED_AND_CHARGING

  -- on batteries: suspend reason BATTERIES
 
  - run_on_batteries = 1
  -- charger connected, battery level  X, temperature  Y: not suspended
  -- charger connected, battery level  X, temperature  Y: suspend
 reason CHARGING
  -- charger connected, battery level  X, temperature  Y: suspend
 reason OVERHEATED
  -- charger connected, battery level  X, temperature  Y: suspend
 reason OVERHEATED

 same here: OVERHEATED_AND_CHARGING

  -- on batteries, battery level  X, temperature  Y: not suspended
  -- on batteries, battery level  X, temperature  Y: suspend reason
CHARGING
  -- on batteries, battery level  X, temperature  Y: suspend
 reason OVERHEATED

 same here: OVERHEATED_AND_CHARGING

  -- on batteries, battery level  X, temperature  Y: suspend
 reason OVERHEATED

 Cheers,
Bruce


 ___
 boinc_dev mailing list
 boinc_dev@ssl.berkeley.edu
 http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
 To unsubscribe, visit the above URL and
 (near bottom of page) enter your email address.


 --

 Subject: Digest Footer

 ___
 boinc_dev mailing list
 boinc_dev@ssl.berkeley.edu
 http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev


 --

 End of boinc_dev Digest, Vol 107, Issue 4
 *

___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] Changes to client scheduler from 6.12.33 to 7.0.25?

[Kevin Reed]

If min_rpc_delay  last_rpc_duration*5, then use a backoff for the project
up to something like max(100*last_rpc_duration, 100*min_rpc_delay).  In the
event that last_rpc_duration*5  min_rpc_delay, clear the backoff.

This will force projects that use a min_rpc_delay setting is appropriate
for their infrastructure and will automatically protect other projects.  It
will also help projects that have a temporary overload.


 From: David Anderson da...@ssl.berkeley.edu
 
 Can anyone think of a solution for this?
 -- David
 
 On 11-Aug-2012 11:40 AM, john.mcl...@sybase.com wrote:
  OK, how about this scenario for a problem.
 
  low latency project has a current value of 1 minute for
min_rpc_delay.? It
  takes 10 seconds for an RPC.
  SETI@Home is overloaded and takes 90 seconds to respond to an RPD
delay.
  Neither is currently handing out work.
 
  Result:? Any host connected to both where both are the top two for
work
  fetch is going to have some difficulty fetching work from anywhere
else.
 
  jm7
 
 
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] file upload problem

[Kevin Reed]

Bernd,

I think that the result file record includes a md5sum that you could check
during validation and assimilation to confirm that it is the same file.
This would prevent improper assimilation.

How often have you seen this?

thanks,
Kevin


 Message: 2
 Date: Tue, 24 Jul 2012 09:58:56 +0200
 From: Bernd Machenschalk bernd.machensch...@aei.mpg.de
 Subject: [boinc_dev] file upload problem
 To: BOINC Developers Mailing List boinc_dev@ssl.berkeley.edu
 Message-ID: 500e55c0.40...@aei.mpg.de
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed

 Hi!

 We recently had a case where a result file was uploaded multiple
 times over and over again, even after the result was reported.

 One of the uploads happened between validation and assimilation,
 resulting in assimilation of a 'canonical' result that has never
 been validated (and
 was actually broken).

 Certainly this one incident is mainly the result of a misbehaving
 Client (7.0.3 IIRC), but in any case I think the server code should be
robust
 against such clients.

 I can think of a couple of options to prevent this from happening
 again in the future:

 - Move the result files from the upload directory when the result is
 reported; validate and assimilate it from this new location
 (scheduler would need
 to touch (move) result files)

 - Make the file upload handler check the result status in the DB
 (fuh needs DB access, additional DB load)

 - My current favorite: While receiving a file, the file upload
 handler appends .part to the filename. Only when the upload was
 successfully
 completed, the file is renamed to the result file name, but only if
 there is not already a file with that desired name yet. (the
 antique_file_deleter
 will eventually clean up partial uploads, too). Anything that still
 could go wrong with that behavior?

 Best,
 Bernd

___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] --allow_multiple_clients ignored

[Kevin Reed]

WCG won't run without the protection against creating multiple hosts.  We
have too many users who have the client installed on machines that are
re-imaged nightly or weekly which cause a lot of abandoned results if the
hosts are not linked with previous entries.

We need to update our server so we will work on that (option 2)

In the meantime, I can throw a hack onto our server if needed for a
specific user.  I haven't been following on this thread but if the user
sends an email to supp...@worldcommunitygrid.org, then I can help them.

thanks,

Kevin Reed
 .   .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .

i b m   i n t e r a c t i v e
71 S. Wacker Dr




From:   David Anderson da...@ssl.berkeley.edu
To: Rod Walker rodney.wal...@physik.uni-muenchen.de, BOINC
Developers Mailing List boinc_dev@ssl.berkeley.edu
Cc: Kevin Reed/Chicago/i...@ibmus, Mark Silberstein
ma...@cs.technion.ac.il
Date:   06/29/2010 11:56 AM
Subject:Re: [boinc_dev] --allow_multiple_clients ignored



I think I figured this out.
It turns out that the --allow_multiple_clients option
assumes a corresponding server option that tells the scheduler
to allow multiple concurrent requests from a given host.
I implemented this scheme for a particular project, superl...@technion.
No other BOINC projects (including WCG) use this server option,
which is why Rod gets the Another instance errors.

I have fixed this as follows: the client now tells the server
that it was run with --allow_multiple_clients,
and the server handles it accordingly, with no special server flags.

Now, there are 2 ways to make things work with WCG:

1) WCG uses its current server software and sets the
multiple_clients_per_host flag in its config file.
No client changes needed.
The drawback: WCG loses its protection against buggy clients
creating duplicate host records.

2) WCG updates its server software with changes in [21839],
and Rod uses clients built from the SVN trunk.

Kevin, any thoughts?

-- David

PS to Mark Silberstein: this change means that when up upgrade your
server software to [21839] or later,
you'll need to start using a new client.

On 28-Jun-2010 1:10 PM, Rod Walker wrote:
 There's no problem with multiple clients per host, as long as:
 1) you use the --allow_multiple_clients option
 2) each client has its own data directory,
 and you don't copy client_state.xml into this directory
 (client_state.xml contains host IDs, and each client must
 have a separate host ID).
 Are you doing both of these things?
 Yes. I do not provide client_state.xml and this file is not in the job
 run directory after boinc has run. Choosing 2 clients which ran on the
 same host at the same time...
 13-Jun-2010 16:55:22 [---] Data directory: /tmp/ri32buz/boinc_TN5146
 13-Jun-2010 16:56:08 [World Community Grid] Started download of
 wcg_hcc1_img_6.0
 8_i686-pc-linux-gnu

 and then ran to completion. The other

 13-Jun-2010 16:55:27 [---] Data directory: /tmp/ri32buz/boinc_CL5241
 13-Jun-2010 16:56:12 [World Community Grid] Message from project server:
 Not sen
 ding work - last request too recent: 5 sec

 Both are started with args
 /boinc --attach_project www.worldcommunitygrid.
 org 43ec69412c60d394929d1d31f2495411 --fetch_minimal_work
 --exit_when_idle --no_
 gui_rpc --no_priority_change --allow_multiple_clients

 but in the log I only see Config lines
 13-Jun-2010 16:55:27 [---] Config: run apps at regular priority
 13-Jun-2010 16:55:27 [---] Config: report completed tasks immediately
 13-Jun-2010 16:55:27 [---] Config: fetch minimal work

 Should a line for --allow_multiple_clients show up here? I do not see
 that in the code. Recall that I built this from the svn head in order to
 get the --fetch_minimal_work functionality.
 The other error I see, in a different log, is
 13-Jun-2010 16:56:14 [World Community Grid] Message from project server:
 Another
 scheduler instance is running for this host

 I do NOT see this error though
 log_message_error(Another instance of BOINC is running.);
 so I guess config.allow_multiple_clients is set. The problem seems to be
 on the project side, like the project uses the hostname rather than a
 hostID.

 I don't know what WLCG and WN stand for.
 WLCG is an acronym factory. World LHC Computing Grid, i.e. the computing
 part of the atom smasher in Geneva. WN=Worker node (host in a cluster).

 This security info should be well received. For sure, IBM do not want to
 get sued when boincers go bad.

 Cheers,
 Rod.


 BOINC projects are not peer-reviewed in general. However:

 1) IBM World Community Grid has only applications that are
 reviewed by IBM security experts, and that do some type of
 good-of-humanity science (biomedicine, environment).
 If you attach only to WCG, you have very strong security.

 2) The projects listed on http://boinc.berkeley.edu/projects.php
 are known to me, and can be trusted.

 -- David

inline

Re: [boinc_dev] host punishment mechanism revisited

[Kevin Reed]

One thing that Bruce Allen had suggested at one point and I think is a good
idea as well is to break validation into two stages.

The first stage is a validation of everything that can be done looking at a
single file.  There are a lot of checks that we do during the init_result
method that checks if the files were transmitted properly, values are
within reasonable ranges, etc.  The second stage is where we actually
compare information between two different results.

Bruce would need to validate this but I remember that he asserted that a
high number of results that were eventually marked as invalid could have
been flagged as invalid much earlier with this first stage check.  For WCG
we could probably catch about half of invalid results in a similar fashion.

The advantage of this is that the first stage of validation could be
performed shortly after result was returned and therefore provide some
useful and timely information into the 'host throttle' mechanism.  The
second stage would still suffer from the delayed effect and uncertain
usefulness.

In the SETI case, can analysis of a single result determine if the result
is garbage?

thanks,

Kevin Reed




From:   john.mcl...@sybase.com
To: Raistmer raist...@mail.ru
Cc: BOINC Developers Mailing List boinc_dev@ssl.berkeley.edu,
boinc_dev-boun...@ssl.berkeley.edu, Kevin
Reed/Chicago/i...@ibmus
Date:   05/27/2010 07:10 AM
Subject:Re: [boinc_dev] host punishment mechanism revisited



Most tasks are validated within days, not weeks.

Do you have a suggestion that will actually work better?  For all projects?
In particular the project that has the problem the worst has no mechanism
for telling that there is a problem other than validation.

jm7



 Raistmer
 raist...@mail.ru
   To
 Sent by:  Kevin Reed knr...@us.ibm.com,
 boinc_dev-bounce john.mcl...@sybase.com
 s...@ssl.berkeley.ed  cc
 uBOINC Developers Mailing List
   boinc_dev@ssl.berkeley.edu,
   boinc_dev-boun...@ssl.berkeley.edu
 05/27/2010 06:10  Subject
 AMRe: [boinc_dev] host punishment
   mechanism revisited










Punishment for invalid validation will not help at all IMO with broken
CUDA GPUs.
Tasks reported in seconds scale, validated in weeks scale and time of
existance such broken GPU ~day (host reboot will fix it).
That is, in most cases this delayed validation accounting will punish for

long ago forgotten problem. Not very effective way to deal with this
problem.

- Original Message -
From: john.mcl...@sybase.com
To: Kevin Reed knr...@us.ibm.com
Cc: BOINC Developers Mailing List boinc_dev@ssl.berkeley.edu;
boinc_dev-boun...@ssl.berkeley.edu
Sent: Thursday, May 27, 2010 9:18 AM
Subject: Re: [boinc_dev] host punishment mechanism revisited


+ and - 1 are too slow to be of much use unless the range is very short
 (i.e. 1 to 10 with 10 being completely trusted, and 1 being completely
 untrusted).  If we split the allocation between report and validation, we
 could have +1 for success, +1 for validation (for a total of +2 for a
 successful valid result), and -1 for error and -2 for invalid (for -1 for
 each of the cases of unsuccessful and successful but invalid).  The
 invalid
 punishment has to be -(success reward) + (whatever the punishment for an
 invalid result should be - which is a negative number).  To make it
 completely balanced, it would be 0.5 for reward for each of success and
 valid, and -1 for unsuccessful and -1.5 for successful but invalid.

 We also need to catch validation errors has has been proven by the
ongoing
 CUDA problem where CUDA cards start returning junk after a while and it
 takes a computer reboot to correct the problem.

 Which is the more stringent requirement depends on run times.  If a task
 takes minutes to error out, the maximum # of tasks in progress is the
more
 stringent.  If it takes over a day / task, then the max # downloads is
the
 more stringent requirement.  Since the project should protect itself
 against slow problems as well as fast problems, I am not completely
 certain
 that is the right division.

 Perhaps if a host is in state #2, we allow a per individual resource
 allocation / day.  The allocation in state #2 should reduce as we become
 more suspicious of the resource type.  If the host is in state #3, we
 allow
 an allocation per resource type / day (preferably an allocation of one
for
 each of CPU and each GPU type).  If we know it is a problem, we are
 waiting
 for the host to be fixed.

 I would suggest that the punishment and quota be per resource type
instead
 of per host

Re: [boinc_dev] [Fwd: [BOINC] #924: Partial Transfer Fails]

[Kevin Reed]

Rattledagger rattledag...@online.no wrote on 06/25/2009 01:08:30 PM:

 Kevin Reed

 On Wed, 24 Jun 2009 16:51:09 -0300, you wrote:
 
 But it also has this code in file_xfer.cpp (error checking removed to
make
 email shorter)
 
 // for downloads: if we requested a partial transfer,
 // and the HTTP response is 200,
 // and the file is larger than it should be,
 // the server or proxy must have sent us the entire file
 // (i.e. it doesn't understand Range: requests).
 // In this case, trim off the initial part of the file
 

 Wouldn't this mean the client already handles this correctly, as long as
WCG
 re-configures their server to ignore the range-request and sends the
 whole file?


You are correct.  I hadn't tested that setting yet and had instead looked
at the code in the wrong file to see how it would handle the condition if
we were to send the whole file.  I have now tested this and BOINC does
handle a full response correctly.

We have resolved this issue by adding the following lines to our
httpd.conf file for the download directory.

RequestHeader unset Range
Header unset Accept-Ranges

I scanned our log files for 206 and 416 response codes.  It looks like
about 0.2% of our download requests are retries.  This will have an effect
on some users, but the vast majority of our files are quite small (less
than 200k after compression) so full re-downloads will not be an
overwhelming.,
___
boinc_dev mailing list
boinc_dev@ssl.berkeley.edu
http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Re: [boinc_dev] [Fwd: [BOINC] #924: Partial Transfer Fails]

[Kevin Reed]

David,

I think that the following might be the way to go:

When the BOINC client is downloading any file and the transfer is
interrupted, then it should check for the existence of the Accept-Ranges:
bytes header for that file transfer.  If that header is found, then the
client behavior remains unchanged.

If that header is not found, then the client should not save any partially
downloaded file.  It should delete it and during the next request it should
request the entire file.

The example code associated with this article:
http://www.linuxdevcenter.com/pub/a/linux/2005/05/05/libcurl.html contains
info on how to get the request headers.

This will resolve our issues.

However, the boinc client has this code in http_curl.cpp:

if ((response/100)*100 == HTTP_STATUS_OK) {
http_op_retval = 0;
} else if ((response/100)*100 == HTTP_STATUS_CONTINUE) {
return;
} else {

This is essentially converting all of the 2XX response codes to be the same
as HTTP_STATUS_OK.  The 200 vs 206 response code is therefore being treated
the same by the BOINC code.  I think that the BOINC client should handle
the 200 and 206 return codes differently and if 200 is returned then the
file downloaded should replace any existing file.  If 206 is returned, then
the client should append to any existing file.

These behaviors would allow the client to operate as expected with regards
to HTTP headers and response codes.  This also does give control of the
client behavior on the server side as the presence or absence of the
'Accept-Ranges' header would control what the client did.

In the interim, I will modify our server so that it it ignores the Ranges
request header and does not contain the Accept-Ranges response.  This will
cause the entire file to be returned.  Since the BOINC client will append
this to the existing partial file, it will fail the md5sum check.  However,
that will result in an immediate failure instead of it taking waiting for
the timeout like it is doing now.  Once users start using the 6.10 client,
then the behavior will work as desired.

thanks,
Kevin




|
| From:  |
|
  
--|
  |David Anderson da...@ssl.berkeley.edu  
 |
  
--|
|
| To:|
|
  
--|
  |Kevin Reed/Chicago/i...@ibmus
  |
  
--|
|
| Cc:|
|
  
--|
  |Carl Christensen carl...@yahoo.com, BOINC Developers Mailing List 
boinc_dev@ssl.berkeley.edu  |
  
--|
|
| Date:  |
|
  
--|
  |06/22/2009 11:08 PM  
 |
  
--|
|
| Subject:   |
|
  
--|
  |Re: [Fwd: [BOINC] #924: Partial Transfer Fails]  
 |
  
--|





It seems like any solution is going to involve a client change.
So what about my previous suggestion,
namely a no_partial_transfer flag in the file_info.
That would provide control at the server side, which is desirable.
-- DPA

Carl Christensen wrote:
 I don't think libcurl distinguishes between a 206  200 (from this
message I
 found from the main libcurl guy, although it's a two

Re: [boinc_dev] [Fwd: [BOINC] #924: Partial Transfer Fails]

[Kevin Reed]

David,

It is worth mentioning that this is only an issue since we started to
'pre-compress' the files.  These files no-longer use mod_deflate.  I store
them with the suffix .gzb and add the following header to httpd.conf:

AddEncoding gzip gzb

libCurl then detects the gzip format and decompresses it during transfer.

If you are using the techniques outlined at
http://boinc.berkeley.edu/trac/wiki/FileCompression

then a partial transfer will work correctly since mod_deflate will handle
the range correctly using the uncompressed file size.

I am wondering if I might have been incorrect to open that bug.  A fourth
option might be to ignore the request for a range transfer and instead
return the whole file with a http 200 code for these file types.


6.4.2 Range Retrieval Requests


HTTP retrieval requests using conditional or unconditional GET methods MAY
request one or more sub-ranges of the entity, instead of the entire entity,
using the Range request header, which applies to the entity returned as the
result of the request:


   Range = Range : ranges-specifier



A server MAY ignore the Range header. However, HTTP/1.1 origin servers and
intermediate caches ought to support byte ranges when possible, since Range
supports efficient recovery from partially failed transfers, and supports
efficient partial retrieval of large entities.


If the server supports the Range header and the specified range or ranges
are appropriate for the entity:
  The presence of a Range header in an unconditional GET modifies what
  is returned if the GET is otherwise successful. In other words, the
  response carries a status code of 206 (Partial Content) instead of
  200 (OK). 


I.e. I should add

RequestHeader unset Range

so that apache will not process the range request.  However, you can not
set conditions on the RequestHeader like you can with Header so this would
take effect for all download requests (we don't do this for all of our
projects).   Any ideas out there?

thanks,

Kevin,


|
| From:  |
|
  
--|
  |David Anderson da...@ssl.berkeley.edu  
 |
  
--|
|
| To:|
|
  
--|
  |BOINC Developers Mailing List boinc_dev@ssl.berkeley.edu   
 |
  
--|
|
| Cc:|
|
  
--|
  |Carl Christensen carl...@yahoo.com, Kevin Reed/Chicago/i...@ibmus  
  |
  
--|
|
| Date:  |
|
  
--|
  |06/22/2009 03:14 PM  
 |
  
--|
|
| Subject:   |
|
  
--|
  |[Fwd: [BOINC] #924: Partial Transfer Fails]  
 |
  
--|





It's not clear to me how to solve this problem.
Synopsis:
- projects can configure their web server to send files in
   a compressed form to clients that can handle it; see
   http://boinc.berkeley.edu/trac/wiki/FileCompression
   In modern (= 5.4) BOINC clients,
   the client tells libCurl to handle such compression.
- The client keeps track of the bytes transferred N,
   and if a transfer fails and is retried,
   it sends a Range: header telling the web server to skip the first N