Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Martin
Paul D. Buck wrote: [...] As a participant I ask for little other than a fair hearing if I see problems, and a visible and interested project staff. I can only think of a couple projects where I feel that is true today. A sad, sad commentary on the state of the BOINC Community ...

Re: [boinc_dev] 6.6.20 and work scheduling

2009-04-30 Thread John . McLeod
OK, this is pretty much what is already done. However, there are more things to be done than just CPU and work fetch scheduling. The sleep has to be 1 second because there ARE things in the loop that have to run that frequently. For example the messages from the science applications have to be

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread John . McLeod
jm7 boinc_dev-boun...@ssl.berkeley.edu wrote on 04/29/2009 04:36:26 PM: Martin m_boinc...@ml1.co.uk Sent by: boinc_dev-boun...@ssl.berkeley.edu 04/29/2009 04:36 PM To BOINC Developers List boinc_dev@ssl.berkeley.edu cc Subject Re: [boinc_dev] Work Scheduling (pt 2, Cooperative

Re: [boinc_dev] FastCGI (was: Work Scheduling pt 2)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 12:16:00 john.mcl...@sybase.com escribió: I have the feeling that nobody ever used the FastCGI scheduler correctly, that server admins compile it with FastCGI support as if that was all that needs to be done. I don't know enough about Fast CGI to have any hope of

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 12:19:05 john.mcl...@sybase.com escribió: And then there is CPDN trickle-based kill which is completely different. The server sends a trickle-down to that host, knowing it has that task. The application handles the trickle and quits, saying it finished processing.

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread John . McLeod
jm7 boinc_dev-boun...@ssl.berkeley.edu wrote on 04/30/2009 11:35:16 AM: Martin m_boinc...@ml1.co.uk Sent by: boinc_dev-boun...@ssl.berkeley.edu 04/30/2009 11:35 AM To BOINC Developers List boinc_dev@ssl.berkeley.edu cc Subject Re: [boinc_dev] Work Scheduling (pt 2, Cooperative

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 13:06:36 Martin escribió: Even if you use multiple separate download servers, you just simply have message sockets to your WU pool allocator process(es) to report what is successfully transferred and when. Similarly so for setting what URL is to point to what WU on the web

Re: [boinc_dev] 6.6.20 and work scheduling

2009-04-30 Thread Paul D. Buck
On Apr 30, 2009, at 5:51 AM, john.mcl...@sybase.com wrote: OK, this is pretty much what is already done. However, there are more things to be done than just CPU and work fetch scheduling. The sleep has to be 1 second because there ARE things in the loop that have to run that frequently.

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Martin
john.mcl...@sybase.com wrote: [...] I may have gotten confused. I thought that someone was claiming that it ***could*** be done without DB access, and be triggered by the file download. 'Tis I! Some notes: It has to be backwards compatible - client versions that do not do this exist,

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Martin
Nicolás Alvarez wrote: BURP uses download servers provided by volunteer users. A few projects use download servers distributed across the world. File downloads *must* work on plain old HTTP servers without having to 'call home'. Then that is an Achilles heel for gaining feedback on what

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Paul D. Buck
On Apr 30, 2009, at 8:07 AM, Nicolás Alvarez wrote: El Jue 30 Abr 2009 10:21:31 john.mcl...@sybase.com escribió: Rom did an analysis at one time that indicated that the basic connection was about 10 times as expensive as the work for a single report or request.

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 13:39:18 Paul D. Buck escribió: Take the simplest example, The first stage of a task is the creation of the Result, which generates one or more Tasks. Neither of these are of interest or should be visible to the participant. So, why are they in the running tasks and

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Paul D. Buck
On Apr 30, 2009, at 8:16 AM, john.mcl...@sybase.com wrote: Maintaining an index is also expensive. Every insert into the table changes the index. In particular, the link between a WU and a User would have to have an index that is updated on every assignment (add) and every transfer

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 13:45:38 Martin escribió: Then that is an Achilles heel for gaining feedback on what was accepted by the client... Apache can report what URL downloaded whatever, so that is one means of feedback for servers under direct project control. Another means could be that that

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread John . McLeod
jm7 boinc_dev-boun...@ssl.berkeley.edu wrote on 04/30/2009 12:45:38 PM: Martin m_boinc...@ml1.co.uk Sent by: boinc_dev-boun...@ssl.berkeley.edu 04/30/2009 12:45 PM To boinc_dev@ssl.berkeley.edu cc Subject Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?) Nicolás

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 13:55:46 john.mcl...@sybase.com escribió: At present, how do the download servers know to delete downloaded/expired WUs if they can't 'phone home'? The Validator deletes the files. The validator compares files and chooses a canonical result. The assimilator does

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 13:58:27 john.mcl...@sybase.com escribió: Once a day if there was no CPU request, or with the CPU request if there is one. This is a substantially lower count than if every project is asked for a work proposal for every work request. You're adding more every words than we

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread John . McLeod
The assimilator has to have DB access to do its job. It determines which files to be deleted and passes the information on to the file deleter ( and I do not know the mechanism - it is quite possibly through the DB as the two are completely asynchronous). jm7 boinc_dev-boun...@ssl.berkeley.edu

Re: [boinc_dev] 6.6.20 and work scheduling

2009-04-30 Thread John . McLeod
Not until the next state is acted on - which happens after the state is fully determined. Is there a bug someplace, yes, but merely setting the flag does NOT actually change the state until the flag is acted upon. jm7 boinc_dev-boun...@ssl.berkeley.edu wrote on 04/30/2009 01:05:01 PM: Paul D.

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Paul D. Buck
On Apr 30, 2009, at 9:58 AM, john.mcl...@sybase.com wrote: jm7 Paul D. Buck p.d.b...@comcast.net wrote on 04/30/2009 12:51:30 PM: Paul D. Buck p.d.b...@comcast.net 04/30/2009 12:51 PM To Nicolás Alvarez nicolas.alva...@gmail.com cc BOINC Developers Mailing List

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Mikus Grinbergs
Some projects have real world deadlines that have to be met. I have no problem with this being a goal for a project. But in my opinion, to expect it to happen consistently is wishful thinking. Therefore, it should be the __duty__ of project administrators to plan for less than 100%

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 13:32:01 Mikus Grinbergs escribió: [ And what if the result that's returned is useless because of a bug in the project's application? ] And what if their output files are binary, a bug in the project's application causes invalid results, and sample_bitwise_validator marks

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Josef W. Segur
On 30 Apr 2009 at 9:21, john.mcl...@sybase.com wrote: Talk about spamming the servers. Ask every one for an offer (requires DB access). Accept some subset of those (requires DB access). Rom did an analysis at one time that indicated that the basic connection was about 10 times as

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Paul D. Buck
On Apr 30, 2009, at 8:07 AM, Nicolás Alvarez wrote: El Jue 30 Abr 2009 10:21:31 john.mcl...@sybase.com escribió: Rom did an analysis at one time that indicated that the basic connection was about 10 times as expensive as the work for a single report or request.

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread John . McLeod
Sorry, I was reporting from memory without the actual data in front of me. Still, a 3X reduction is important. jm7 Josef W. Segur jse...@westelcom

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 15:26:40 john.mcl...@sybase.com escribió: The two are at different layers in the protocol stack. No news to me. Using HTTP, we're *forced* to use individual requests, each sending complete information, and the server has to do eg. user/host lookup on every request. Using a

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread John . McLeod
OK, you want to switch to something other than http at the application level. (possibly hand rolled) at least for the updates. That might be some interesting design anyway. The obvious questions: 1) What about version compatibility? A) Old client to new server B) New client to

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Nicolás Alvarez
El Jue 30 Abr 2009 15:35:10 Nicolás Alvarez escribió: El Jue 30 Abr 2009 15:26:40 john.mcl...@sybase.com escribió: The two are at different layers in the protocol stack. No news to me. Using HTTP, we're *forced* to use individual requests, each sending complete information, and the server

Re: [boinc_dev] Work Scheduling (pt 2, Cooperative Scheduling?)

2009-04-30 Thread Rom Walton
The problem with persistent connections is the overall memory consumed in just maintaining the connection. Even if you striped down the scheduler to a self contained daemon you still will have to deal with the basics of maintaining connection state for which ever clients are connected and