At least SETI requests a 5 minute delay between RPC attempts, and under this scenario, the host should respect both this and its own increasing backoffs when no work is allocated (there's no min_rpc_delay). That should allow enough elbow-room between the low-latency requests for #3, #4, #5... to be contacted.
>________________________________ >From: David Anderson <[email protected]> >To: [email protected] >Cc: "[email protected]" <[email protected]>; Richard >Haselgrove <[email protected]> >Sent: Sunday, 12 August 2012, 22:36 >Subject: Re: [boinc_dev] Changes to client scheduler from 6.12.33 to 7.0.25? > >Can anyone think of a solution for this? >-- David > >On 11-Aug-2012 11:40 AM, [email protected] wrote: >> OK, how about this scenario for a problem. >> >> low latency project has a current value of 1 minute for min_rpc_delay. It >> takes 10 seconds for an RPC. >> SETI@Home is overloaded and takes 90 seconds to respond to an RPD delay. >> Neither is currently handing out work. >> >> Result: Any host connected to both where both are the top two for work >> fetch is going to have some difficulty fetching work from anywhere else. >> >> jm7 >> >> >> >> From: David Anderson <[email protected]> >> >> To: Richard Haselgrove <[email protected]> >> >> Cc: "[email protected]" <[email protected]> >> >> Date: 08/10/2012 03:47 PM >> >> Subject: Re: [boinc_dev] Changes to client scheduler from 6.12.33 to >>7.0.25? >> >> Sent by: <[email protected]> >> >> >> >> >> >> >> Projects that use next_rpc_delay need to monitor their server load >> and adjust the value accordingly. >> -- David >> >> On 10-Aug-2012 12:02 PM, Richard Haselgrove wrote: >>> Provided, that is, that the project with the low next_rpc_delay value >> provides >>> sufficient server resources to manage the workflow expeditiously. >>> I've been noticing recently that the SETI servers can commonly take up to >> 90 >>> seconds to respond to a routine RPC. While waiting for that reply, the >> client >>> appears to be inhibited from initiating an RPC to another project. >>> Nobody would suggest that SETI should ever consider going low-latency, >> but I'm >>> using the example to question whether the combination of low-latency and >> a >>> sluggish server could lead to the sort of problems for other projects >> that John >>> described. >>> >>> *From:* David Anderson <[email protected]> >>> *To:* [email protected] >>> *Sent:* Friday, 10 August 2012, 19:43 >>> *Subject:* Re: [boinc_dev] Changes to client scheduler from 6.12.33 >> to 7.0.25? >>> >>> There's nothing wrong with a small next_rpc_delay value, e.g. 1 >> minute. >>> This will have no effect on other projects. >>> >>> The semantics of this are: >>> - the client issues a scheduler RPC to the project every 1 minute >>> - If the project has the highest scheduling priority, >>> this RPC will include work requests for resource types >>> that are not above the max buffer limit >>> (regardless of per-resource backoff). >>> >>> It a project (like Patricio's) has sporadic job availability, >>> then it will typically have highest scheduling priority. >>> However, as it processes jobs the priority will decrease, >>> so it won't block out other project. >>> >>> -- David >>> >>> On 10-Aug-2012 7:36 AM, Richard Haselgrove wrote: >>> > I would hope, and expect, that if anyone set an extreme >> low-latency value >>> > like a next_rpc_delay of 1 minute or less on a public-facing >> volunteer >>> > project, peer pressure from other projects would rapidly force >> them to change >>> > their mind. >>> > >>> > But as the front page of the website makes clear, BOINC is also >> designed to >>> > work as a virtual campus supercomputer centre, or as a company >> desktop grid. >>> > Under those circumstances, the risk of blocking other projects is >> lessened, >>> > and I think BOINC should work as described in the Wiki. >>> > >>> > >>> >> ________________________________ From: "[email protected] >>> <mailto:[email protected]>" >>> >> <[email protected] <mailto:[email protected]>> To: >> Patricio >>> Vidal <[email protected] <mailto:[email protected] >>>> >>> >> Cc: [email protected] <mailto:[email protected] >>> ; >>> [email protected] >>> <mailto:[email protected]> Sent: >>> >> Friday, 10 August 2012, 15:27 Subject: Re: [boinc_dev] Changes to >> client >>> >> scheduler from 6.12.33 to 7.0.25? >>> >> >>> >> There is still no point in the connection if the client is not >> going to >>> >> ask for work from the project anyway. >>> >> >>> >> The observation is that it would only take a very small number of >>> >> low-latency projects to starve all other projects of connections >> if the >>> >> connection period to the server is short enough. >>> >> >>> >> For example, if a connection takes 15 seconds and there are 4 >> projects >>> >> with a server specified connection period of a minute, then no >> other >>> >> projects will ever be contacted for anything as there will always >> be a >>> >> pending connection to a low-latency project. If these >> connections only >>> >> happen when the client would ask for work from the project if it >> were >>> >> contactable, then the situation improves greatly. It can still >> be the case >>> >> where all 4 rise to the top of the list of projects from which >> the client >>> >> wants work - in that case, the client will get no work from >> anywhere else >>> >> until one of these projects has supplied work. >>> >> >>> >> jm7 >>> >> >>> >> >>> >> |------------> | From: | |------------> >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >>> >>> |Patricio Vidal <[email protected] >>> <mailto:[email protected]>> >>> | >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >>> >>> |------------> >>> >> | To: | |------------> >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >>> >>> |<[email protected] <mailto:[email protected]>> >>> >> | >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >>> >>> |------------> >>> >> | Date: | |------------> >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >>> >>> |08/10/2012 10:04 AM >>> | >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >>> >>> |------------> >>> >> | Subject: | |------------> >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >>> >>> |Re: [boinc_dev] Changes to client scheduler from 6.12.33 to 7.0.25? >>> | >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >>> >>> |------------> >>> >> | Sent by: | |------------> >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >>> >>> |<[email protected] >>> <mailto:[email protected]>> >>> >> | >>> >>> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >>> >>> In my previous email I meant to say ".. we can't have clients waiting >>> >> hours..." >>> >> >>> >> I just found in the wiki that the next_rpc_delay is exactly >> designed for >>> >> my purpose: http://boinc.berkeley.edu/trac/wiki/LowLatency# >>> >> >>> >> Based on the wiki it seems to me the clients should contact the >> server >>> >> using the rpc_delay regardless they have work from the project. >>> >> >>> >> Patricio. >>> >> >>> >> >>> >> >>> >> >>> >> yoyo <yoyo@mailueberfall .de> To Patricio Vidal 08/10/2012 01:21 >> AM >>> >> <[email protected] <mailto:[email protected]>> >> cc >>> [email protected] <mailto:[email protected]> >>> >> >>> >> Subject Re: [boinc_dev] Changes to client scheduler from 6.12.33 >> to >>> >> 7.0.25? >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> The client doesn't wait for hours if the server has work. Even >> not if the >>> >> workunits are very short. Before the client runs out of work it >> fetches >>> >> new ones. Additional you can configure on the server, that the >> client >>> >> should periodic, e.g. every hour, connect to the server. I use it >> and let >>> >> the client connect at least every 5h. Was this next_epc_delay >> removed from >>> >> trunc? I use the client connect to check if a workunit which is >> in the >>> >> queue of the client can be deleted. >>> >> >>> >> yoyo >>> >> >>> >> Patricio Vidal schrieb: Yes, we need that functionality. Our jobs >> last from >>> >> minutes to few hours so we can have the clients waiting hours to >> contact >>> >> the server. >>> >> >>> >> Is there a reason why the client blocks the other projects if it >> has a >>> >> very short request cycle and it does not have work? I would think >> that if >>> >> it doesn't has work it should jump to the next project in the >> list. >>> >> >>> >> This functionality is critical for any project that has relative >> short >>> >> (few hours) jobs and it needs the results back as soon as >> possible. I >>> >> guess this usage is more typical of Boinc projects deployments in >> private >>> >> networks. >>> >> >>> >> Would it make sense to add another option to the configuration >> file for >>> >> this behavior? Something like <next_work_request_rpc_delay> ? Or >> just >>> >> restore part of the behavior of <next_rpc_delay> ? The old >> behavior has >>> >> been that way for years, right? Is it a issue with other projects >> so that >>> >> the behavior was changed? >>> >> >>> >> Patricio. >>> >> >>> >> >>> >> >>> >> >>> >> John.McLeod@sybase. com To 08/09/2012 04:50 PM yoyo >>> >> <[email protected] <mailto:[email protected]>> cc >>> [email protected] <mailto:[email protected]>, >>> >> >>> >> [email protected] >>> <mailto:[email protected]>, Patricio Vidal >>> >> <[email protected] <mailto:[email protected]>> >>> >> >>> >> Subject Re: [boinc_dev] Changes to client scheduler from 6.12.33 >> to >>> >> 7.0.25? >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> They want the clients not to "go to sleep" for a long time if >> there is no >>> >> work for a bit. Trickle messages only work if there is work from >> that >>> >> project actually on the client. >>> >> >>> >> There are some problems with contacting projects constantly under >> certain >>> >> circumstances. If there is no work on the client from the client >> and the >>> >> client is not interested in fetching work from that project, then >> there is >>> >> no point in a contact at all. There would be nothing that the >> server >>> >> could legitimately do. In this case, it is just taking up >> bandwidth that >>> >> some users pay for by the byte or have caps on usage. >>> >> >>> >> The code was modified so that no contact would be made if there >> was no >>> >> work on the client from the project. It could be modified so >> that the >>> >> client would talk to the project on the period specified if it >> either had >>> >> work from that project, or was interested in work from that >> project. i.e. >>> >> The queue is not full and that project was currently the top of >> the list >>> >> for work fetch. The problem with this is if this project has a >> very short >>> >> request cycle and it does not have work, it will block the client >> from >>> >> asking any other projects for work and the client would go idle >> as a >>> >> result. >>> >> >>> >> jm7 >>> >> >>> >> >>> >> |------------> | From: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |yoyo <[email protected] <mailto:[email protected]>> >>> >> | >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |------------> >>> >> | To: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |Patricio Vidal <[email protected] < >> mailto:[email protected]>> >>> >> | >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |------------> >>> >> | Cc: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |<[email protected] <mailto:[email protected]>>, >>> <[email protected] <mailto:[email protected]>>, >>> >> <[email protected] >>> <mailto:[email protected]>> | >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |------------> >>> >> | Date: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |08/09/2012 03:37 PM >>> >> | >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |------------> >>> >> | Subject: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |Re: [boinc_dev] Changes to client scheduler from 6.12.33 to >>> >> 7.0.25? | >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> For what reason should the client contact the server so often? >>> >> You just can use trickle messages to be sent by the client. >>> >> >>> >> yoyo >>> >> >>> >> Patricio Vidal schrieb: Yes, for our project we need the clients >> to contact >>> >> the server every few minutes. We send jobs that last from 20 >> minutes to few >>> >> hours. If the clients don't contact the server for several hour >> then we are >>> >> in trouble. >>> >> >>> >> Regards, Patricio. >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> John.McLeod@sybase. >>> >> >>> >> com >>> >> >>> >> To 08/09/2012 11:21 AM Patricio Vidal >>> >> >>> >> >>> >> <[email protected] <mailto:[email protected]>> >>> >> >>> >> cc >>> >> >>> >> [email protected] <mailto:[email protected]>, >>> >> >>> >> >>> >> [email protected] >>> <mailto:[email protected]> Subject Re: [boinc_dev] >> Changes to >>> >> client scheduler from 6.12.33 to 7.0.25? >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> It was removed in order to avoid hammering projects from clients >> that were >>> >> not going to ask for work and had no work they were working on. >>> >> >>> >> Perhaps, the code could be modified slightly again. Only ignore >> the >>> >> setting for contact server every X if the client has no work from >> the >>> >> server and the client would not ask for work from the project. >>> >> >>> >> BTW, a really short contact period will play havoc with multi >> project >>> >> clients as they will not be able to get work from elsewhere while >> you have >>> >> no work. >>> >> >>> >> jm7 >>> >> >>> >> >>> >> |------------> | From: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |Patricio Vidal <[email protected] < >> mailto:[email protected]>> >>> >> | >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |------------> >>> >> | To: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |<[email protected] <mailto:[email protected]>> >>> >> | >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |------------> >>> >> | Date: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |08/09/2012 11:00 AM >>> >> | >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |------------> >>> >> | Subject: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |[boinc_dev] Changes to client scheduler from 6.12.33 to >>> >> 7.0.25? | >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |------------> >>> >> | Sent by: | |------------> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> |<[email protected] >>> <mailto:[email protected]>> >>> >> | >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------------------------------------------------------------------------| >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> Hello, >>> >> >>> >> I noticed a change in the client scheduler when moving from >> 6.12.33 to >>> >> 7.0.25: I have the <next_rpc_delay> set 180 in the config.xml. >>> >> >>> >> For 6.12.33 the clients contact the server even if there is no >> work, which >>> >> is our desired behavior: >>> >> >>> >> 8/8/2012 11:39:09 PM | AlgoGrid | Requesting new tasks for CPU >> 8/8/2012 >>> >> 11:42:12 PM | AlgoGrid | Requesting new tasks for CPU 8/8/2012 >> 11:45:15 PM >>> >> | AlgoGrid | Requesting new tasks for CPU 8/8/2012 11:48:18 PM | >> AlgoGrid | >>> >> Requesting new tasks for CPU >>> >> >>> >> >>> >> For 7.0.25, when there are available task the scheduler contacts >> the >>> >> server as expected: >>> >> >>> >> 8/8/2012 9:26:33 PM | AlgoGrid | Requesting new tasks for CPU >> 8/8/2012 >>> >> 9:27:10 PM | AlgoGrid | Requesting new tasks for CPU >>> >> >>> >> ... but when there is no tasks it starts delaying the request >>> >> exponentially: >>> >> >>> >> 8/8/2012 9:49:45 PM | AlgoGrid | Requesting new tasks for CPU >> 8/8/2012 >>> >> 10:08:23 PM | AlgoGrid | Requesting new tasks for CPU 8/8/2012 >> 10:32:36 PM >>> >> | AlgoGrid | Requesting new tasks for CPU 8/8/2012 11:37:35 PM | >> AlgoGrid | >>> >> Requesting new tasks for CPU 8/9/2012 2:59:13 AM | AlgoGrid | >> Requesting >>> >> new tasks for CPU 8/9/2012 10:41:06 AM | AlgoGrid | Requesting >> new tasks >>> >> for CPU >>> >> >>> >> >>> >> Is there a new option to force the rpc call even if there is no >> tasks >>> >> available? We need this behavior because our boinc project needs >> a quick >>> >> response from the clients when we schedule a job (we have >> workunits of >>> >> about 1 min processing time and we schedule thousands at a time). >>> >> >>> >> Thank you, Patricio. >>> >> >>> >> >>> >> The server made the following annotations >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------- >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> This message contains information that may be privileged or >>> >> confidential and is the property of Beckman Coulter, Inc. It is >> intended >>> >> only for the person to whom it is addressed. If you are not the >> intended >>> >> recipient, you are not authorized to read, print, retain, copy, >>> >> disseminate, distribute or use this message or any part thereof. >> If you >>> >> receive this message in error, please notify the sender >> immediately and >>> >> delete all copies of this message. >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------- >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ >>> >> boinc_dev mailing list [email protected] >>> <mailto:[email protected]> >>> >> http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To >> unsubscribe, >>> >> visit the above URL and (near bottom of page) enter your email >> address. >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> The server made the following annotations >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------- >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> This message contains information that may be privileged or >>> >> confidential and is the property of Beckman Coulter, Inc. It is >> intended >>> >> only for the person to whom it is addressed. If you are not the >> intended >>> >> recipient, you are not authorized to read, print, retain, copy, >>> >> disseminate, distribute or use this message or any part thereof. >> If you >>> >> receive this message in error, please notify the sender >> immediately and >>> >> delete all copies of this message. >>> >> >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------- >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ >>> >> boinc_dev mailing list [email protected] >>> <mailto:[email protected]> >>> >> http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To >> unsubscribe, >>> >> visit the above URL and (near bottom of page) enter your email >> address. >>> >> >>> >> >>> >> >>> >> >>> >> -- Rate Me, MySkype (yoyo_rkn)Skype Me?! , myICQ 139003243 , >> myIRC >>> >> Rechenkraft.net e.V. - Verein zur Förderung von Bildung, >> Forschung und >>> >> Wissenschaft durch Einsatz vernetzter Computer weitere >> interessante >>> >> Projekte und Hilfe auf unserer Webseite www.Rechenkraft.net und >> im Chat >>> >> Rechenkraft.net e.V. - Non-profit association for the promotion >> of >>> >> education, research and science through the use of networked >> computers >>> >> other interesting projects and help on our website >> www.Rechenkraft.net and >>> >> on IRC >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> The server made the following annotations >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------- >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> This message contains information that may be privileged or >>> >> confidential and is the property of Beckman Coulter, Inc. It is >> intended >>> >> only for the person to whom it is addressed. If you are not the >> intended >>> >> recipient, you are not authorized to read, print, retain, copy, >>> >> disseminate, distribute or use this message or any part thereof. >> If you >>> >> receive this message in error, please notify the sender >> immediately and >>> >> delete all copies of this message. >>> >> >>> >> >>> >> >>> >> --------------------------------------------------------------------------------- >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ >>> >> boinc_dev mailing list [email protected] >>> <mailto:[email protected]> >>> >> http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To >> unsubscribe, >>> >> visit the above URL and (near bottom of page) enter your email >> address. >>> >> >>> >> >>> >> >>> >> >>> >> -- Rate Me, MySkype (yoyo_rkn)Skype Me?! , myICQ 139003243 , >> myIRC >>> >> Rechenkraft.net e.V. - Verein zur Förderung von Bildung, >> Forschung und >>> >> Wissenschaft durch Einsatz vernetzter Computer weitere >> interessante >>> >> Projekte und Hilfe auf unserer Webseite www.Rechenkraft.net und >> im Chat >>> >> Rechenkraft.net e.V. - Non-profit association for the promotion >> of >>> >> education, research and science through the use of networked >> computers >>> >> other interesting projects and help on our website >> www.Rechenkraft.net and >>> >> on IRC >>> >> >>> >> >>> >> >>> >> The server made the following annotations >>> >> >>> >> >>> >> --------------------------------------------------------------------------------- >> >>> >> >>> >> >>> >> >>> >> >>> This message contains information that may be privileged or >> confidential >>> >> and is the property of Beckman Coulter, Inc. It is intended only >> for the >>> >> person to whom it is addressed. If you are not the intended >> recipient, >>> >> you are not authorized to read, print, retain, copy, disseminate, >>> >> distribute or use this message or any part thereof. If you >> receive this >>> >> message in error, please notify the sender immediately and delete >> all >>> >> copies of this message. >>> >> >>> >> >>> >> --------------------------------------------------------------------------------- >> >>> >> >>> >> >>> >> >>> _______________________________________________ >>> >> boinc_dev mailing list [email protected] >>> <mailto:[email protected]> >>> >> http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To >> unsubscribe, >>> >> visit the above URL and (near bottom of page) enter your email >> address. >>> >> >>> >> >>> >> >>> >> _______________________________________________ boinc_dev mailing >> list >>> >> [email protected] <mailto:[email protected]> >>> >> http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To >> unsubscribe, >>> >> visit the above URL and (near bottom of page) enter your email >> address. >>> >> >>> >> >>> >> >>> > _______________________________________________ boinc_dev mailing >> list >>> > [email protected] <mailto:[email protected]> >>> > http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To >> unsubscribe, >>> > visit the above URL and (near bottom of page) enter your email >> address. >>> > >>> _______________________________________________ >>> boinc_dev mailing list >>> [email protected] <mailto:[email protected]> >>> http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev >>> To unsubscribe, visit the above URL and >>> (near bottom of page) enter your email address. >>> >>> >> _______________________________________________ >> boinc_dev mailing list >> [email protected] >> http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev >> To unsubscribe, visit the above URL and >> (near bottom of page) enter your email address. >> >> >> >_______________________________________________ >boinc_dev mailing list >[email protected] >http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev >To unsubscribe, visit the above URL and >(near bottom of page) enter your email address. > > > _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
