Process Monitor can be used to "watch the things a process does" (you have to set up correct filters, etc.)... but I'm not sure if that includes sleeps. But if the process is waiting on a file or something, though, it should be able to tell you. Worth looking into.
https://technet.microsoft.com/en-us/library/bb896645.aspx Regards, Jacob Date: Mon, 18 May 2015 10:41:16 -0700 From: da...@ssl.berkeley.edu To: r.haselgr...@btopenworld.com; onec...@hotmail.com; jacob_w_kl...@msn.com CC: boinc_dev@ssl.berkeley.edu Subject: Re: [boinc_dev] [boinc_alpha] BOINC re-using slot directories without ensuring they're empty I looked at this and couldn't figure out the source of the 12-sec delay. In general, delays could happen because 1) the client does something that takes a long time (like copying a 5 GB file) 2) the client sleeps (i.e. calls boinc_sleep()). It does this in a few situations, like backing off and retrying a file system operation. But there's no indication that either of these is happening here. Does Windows have a way of logging the system calls that a process makes (like strace on Unix)? If so that might reveal what the client is doing during those 12 seconds. -- David On 16-May-2015 8:01 AM, Richard Haselgrove wrote: Here is the message log file for a GPUGrid task finish. The 12-second delay appears again between 14:26:35 and 14:26:47 - that's after the slot directory has been cleared, and the exiting task has changed state from 'running' to 'uploading'. Two new tasks have been assigned to the GPU, but their (small) startup files have not yet been linked to their respective slot directories. I also attach directory listings for the slot and GPUGrid project folders at various stages of the cleanup: the slot held 34 files totalling 44,186,727 bytes, which doesn't sound excessive: the largest file deletion (94,783,960 bytes) occurred several minutes later, when that file finished uploading. I'll enable similar logging and watch what happens when the next GPUGrid task starts up, but from memory, the disruption to BOINC is less severe at startup. On Tuesday, 12 May 2015, 23:29, David Anderson <da...@ssl.berkeley.edu> wrote: BTW: the client isn't completely single-threaded; it uses a separate thread to do CPU throttling. It would be feasible to also use separate threads for serving GUI RPC connections, which would allow client to remain responsive even while e.g. copying thousands of files to a slot dir. -- David On 12-May-2015 2:40 AM, Seke Rob wrote: > Reminds me of the Clean Energy Project, Phase 2 and why we have app_config and > <max_concurrent> and a default control of allowing 1 'In Progress' on a host. This > project sets up in slot copying near 6700 files [symlinking proposed long ago as > is done on several other WCG projects for the static files]. If more than one CEP2 > is started the machine feels at times like a snail, responsiveness of the BOINC > manager is poor, many a time the less powerful systems incurring error zero status > exits or total fail. On an 8 core observed it could take over an hour before > actual computing commenced [CPU time logged]. Boot cycle requires manually > starting of tasks one by one. Kevin Reed few years ago raised a ticket for > staggered starting, where the models can reach several GB and bigger in the > coming. At any rate, as much as these 6700 files are copied, they also then are > needing of deletion at completion [physical or symlink references]. The effect of > starting 1 CEP2 and finishing / packaging / zipping and transmitting can easily > lead to several minutes of there not being any computing, just whirring, for > minutes, just elapsed being logged. The more run the more the issue compounds, > with the effect of what many incur, the exit zero status series, resetting to > start or last checkpoint with often hours of computing time lost. > > Maybe you'd like to get in touch with your confederates at WCG [Keith Uplinger], > to discuss the issue further as this is now nearing a 5 year continues frustration > [June 2010 launch, and a huge limitation on the speed of progress on this project]. > > --SekeRob. > > On 12-5-2015 1:55, David Anderson wrote: >> That delay looks like it's caused by deleting files or by process cleanup. >> Does GPUGrid make lots of (non-output) files in the slot dir? >> >> Please try to repro it with slot_debug, task_debug, and heartbeat_debug set >> (gui_rpc_debug not needed). >> >> -- David >> >> On 11-May-2015 10:54 AM, Richard Haselgrove wrote: >>> Here's another example of a case where BOINC finds that it can't walk and chew >>> gum at the same time. The event of interest is >>> >>> 11/05/2015 18:35:34 | GPUGRID | Computation for task >>> e10s9_e7s6f4-GERARD_FXCXCL12_LIG_6282622-0-1-RND7898_0 finished >>> >>> Following that, there's a 12-second interval where neither heartbeats nor GUI >>> RPC traffic was logged: during that time, the Task tab of the Manager was >>> unchanging, not showing the regular update of elapsed time for running tasks. >>> >>> async_file_debug was active at the time, but found no events to log. >>> >>> These particular GPUGrid tasks generate around 90 MB of upload files, but I >>> think they are generated directly in the project folder and don't need to be >>> copied anywhere. >>> >>> Main log as attached file only. >>> >>> I'll catch a CMS-dev log later this evening, but after that, I'll be away for a >>> few days and I'll have to leave the bug-chase until the weekend. >>> >>> >>> >>> >>> On Monday, 11 May 2015, 9:42, Jacob Klein <jacob_w_kl...@msn.com> wrote: >>> >>> >>> >>> I have seen this problem before, where the UI becomes unresponsive. If I >>> recall, it happens when a T4T task is being set up (ie: after everything was >>> downloaded). For me, I don't recall the problem ever "screwing over other >>> tasks", though. >>> >>> Try this to reproduce it: Attach to T4T, and get a task. It may take a while >>> to do that download, so you can "step away" for a bit. Then, once that task >>> is going, abort it. Downloading the 2nd task should be instantaneous >>> (nothing really to download), but instantiation of that 2nd task should >>> cause the UI to hang (showing the "Please wait" messagebox in the manager). >>> >>> Does that help? >>> > Date: Sun, 10 May 2015 23:19:24 -0700 >>> > From: da...@ssl.berkeley.edu <mailto:da...@ssl.berkeley.edu> >>> > To: r.haselgr...@btopenworld.com <mailto:r.haselgr...@btopenworld.com>; >>> onec...@hotmail.com <mailto:onec...@hotmail.com> >>> > CC: boinc_al...@ssl.berkeley.edu <mailto:boinc_al...@ssl.berkeley.edu> >>> > Subject: Re: [boinc_alpha] BOINC re-using slot directories without >>> ensuring they're empty >>> > >>> > I did some initial testing and couldn't repro this; >>> > the client remains responsive while copying a 5 GB file to a slot dir. >>> > Does anyone else see this behavior? >>> > >>> > While testing this, please set "async_file_debug" log flag. >>> > This says when asynchronous file operations start and end. >>> > >>> > -- David >>> > >>> > On 10-May-2015 12:31 PM, Richard Haselgrove wrote: >>> > > One thing that may need attention if very large files become the norm is >>> the >>> > > single-threaded nature of some parts of the core client. My 1-hour CMS >>> test has >>> > > just finished, and a new 24-hour test started. >>> > > >>> > > >>> > > I watched this happening, and part of the process is copying a 1.33 GB >>> initial >>> > > .vmi image file (downloaded previously by BOINC from CERN) from the project >>> > > directory to the slot directory. This took about 90 seconds: during that >>> time, all >>> > > Manager updating stopped. I'm sure it's the copying process which inhibited >>> > > updates: I was watching the slot directory, and the .vmi image file had >>> appeared, >>> > > but other essential startup files hadn't. >>> > > >>> > > >>> > > When BOINC regained its ability to communicate, three running tasks had >>> exited >>> > > with the dreaded (and false) 'you may need to reset the project' advice. >>> inline >>> > > log follows: because my last log got mangled by my ISP's new mail >>> interface, I'll >>> > > attach it as a text file as well. >>> > > >>> > > >>> > > 10/05/2015 20:12:56 | LHC@home <mailto:LHC@home> 1.0 | Computation for task >>> > > >>> sd_FCChh_bs25_beta30_xing120_int1.0_emit2.0_tunex117.216_tuney118.226_6D_V4__1__s__118.31_117.32__4.1_4.2__6__20_1_sixvf_boinc701_1 >>> >>> > > finished >>> > > 10/05/2015 20:12:56 | CMS-dev | Starting task CMS_31107_1427806626.783437_0 >>> > > 10/05/2015 20:12:56 | CMS-dev | [cpu_sched] Starting task >>> > > CMS_31107_1427806626.783437_0 using CMS version 4615 (vbox64) in slot 7 >>> > > 10/05/2015 20:14:25 | climateprediction.net | Task >>> > > hadam3p_anz_e3g7_2013_1_009760406_0 exited with zero status but no >>> 'finished' file >>> > > 10/05/2015 20:14:25 | climateprediction.net | If this happens repeatedly >>> you may >>> > > need to reset the project. >>> > > 10/05/2015 20:14:25 | NumberFields@home <mailto:NumberFields@home> | Task >>> > > wu_sf3_DS-10x271_Grp503196of682667_0 exited with zero status but no >>> 'finished' file >>> > > 10/05/2015 20:14:25 | NumberFields@home <mailto:NumberFields@home> | If >>> this happens repeatedly you may need >>> > > to reset the project. >>> > > 10/05/2015 20:14:25 | SETI@home <mailto:SETI@home> | Task >>> 05jl12ab.3911.10292.438086664199.12.207_1 >>> > > exited with zero status but no 'finished' file >>> > > 10/05/2015 20:14:25 | SETI@home <mailto:SETI@home> | If this happens >>> repeatedly you may need to reset >>> > > the project. >>> > > 10/05/2015 20:14:25 | climateprediction.net | [cpu_sched] Restarting task >>> > > hadam3p_anz_e3g7_2013_1_009760406_0 using hadam3p_anz version 610 in slot 5 >>> > > 10/05/2015 20:14:25 | NumberFields@home <mailto:NumberFields@home> | >>> [cpu_sched] Restarting task >>> > > wu_sf3_DS-10x271_Grp503196of682667_0 using GetDecics version 200 in slot 0 >>> > > 10/05/2015 20:14:25 | SETI@home <mailto:SETI@home> | [cpu_sched] >>> Restarting task >>> > > 05jl12ab.3911.10292.438086664199.12.207_1 using setiathome_v7 version >>> 700 (cuda42) >>> > > in slot 2 >>> > > 10/05/2015 20:14:27 | LHC@home <mailto:LHC@home> 1.0 | Started upload of >>> > > >>> sd_FCChh_bs25_beta30_xing120_int1.0_emit2.0_tunex117.216_tuney118.226_6D_V4__1__s__118.31_117.32__4.1_4.2__6__20_1_sixvf_boinc701_1_0 >>> > > 10/05/2015 20:14:30 | LHC@home <mailto:LHC@home> 1.0 | Finished upload of >>> > > >>> sd_FCChh_bs25_beta30_xing120_int1.0_emit2.0_tunex117.216_tuney118.226_6D_V4__1__s__118.31_117.32__4.1_4.2__6__20_1_sixvf_boinc701_1_0 >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > On Sunday, 10 May 2015, 19:59, Seke Rob <onec...@hotmail.com >>> <mailto:onec...@hotmail.com>> wrote: >>> > > >>> > > >>> > > >>> > > Excellent this is all fixed and tested. Interest is/was that WCG's Clean >>> > > Energy at some point in time was to run very large models, talk of >>> 4-8GB IIRC. >>> > > >>> > > --SekeRob >>> > > >>> > > On May 10, 2015 20:27, Richard Haselgrove >>> <r.haselgr...@btopenworld.com <mailto:r.haselgr...@btopenworld.com> >>> > > <mailto:r.haselgr...@btopenworld.com >>> <mailto:r.haselgr...@btopenworld.com>>> wrote: >>> > > CMS only has stock applications configured for delivery to 64-bit >>> platforms. >>> > > I've made an anonymous platform configuration using the 32-bit VBox >>> Windows >>> > > wrapper: it has downloaded and is running its first 1-hour task. If that >>> > > completes successfully (it seems to have reached the >>> fully-operational stage), >>> > > I'll try a full 24-hour task, which under current operational >>> circumstances >>> > > should generate a >4 GB file locally. >>> > > >>> > > >>> > > On Sunday, 10 May 2015, 18:28, David Anderson >>> <da...@ssl.berkeley.edu <mailto:da...@ssl.berkeley.edu> >>> > > <mailto:da...@ssl.berkeley.edu <mailto:da...@ssl.berkeley.edu>>> wrote: >>> > > >>> > > >>> > > >>> > > NTFS handles > 4GB files, even if the hardware and/or OS is only 32-bit. >>> > > 32-bit versions of Windows have APIs (like _stat64()) for handling > >>> 4GB files. >>> > > BOINC needs to use these; we fixed one place where it wasn't. >>> > > >>> > > On Unix (Linux and Mac), BOINC uses the regular APIs (like lseek()) >>> but is >>> > > built with a >>> > > -D_FILE_OFFSET_BITS=64 flag that causes these functions to 64-bit size. >>> > > However, it's possible that BOINC has bugs involving > 4GB files on >>> Unix too. >>> > > If anyone has a 32-bit Linux system, please test with the CMS project. >>> > > >>> > > -- David >>> > > >>> > > On 10-May-2015 3:58 AM, --SekeRob wrote: >>> > > > >>> > > > Just wondering, with files over 4GB and a 64 bit lib introduced, is >>> it not a CMS >>> > > > project requirement to run on a 64 bit OS? >>> > > > >>> > > > >>> > > >>> > > _______________________________________________ >>> > > boinc_alpha mailing list >>> > > boinc_al...@ssl.berkeley.edu <mailto:boinc_al...@ssl.berkeley.edu> >>> <mailto:boinc_al...@ssl.berkeley.edu <mailto:boinc_al...@ssl.berkeley.edu>> >>> > > http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_alpha >>> > > To unsubscribe, visit the above URL and >>> > > (near bottom of page) enter your email address. >>> >>> > > >>> > > >>> > > >>> > > >>> > > _______________________________________________ >>> > > boinc_alpha mailing list >>> > > boinc_al...@ssl.berkeley.edu <mailto:boinc_al...@ssl.berkeley.edu> >>> <mailto:boinc_al...@ssl.berkeley.edu <mailto:boinc_al...@ssl.berkeley.edu>> >>> > > http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_alpha >>> > > To unsubscribe, visit the above URL and >>> > > (near bottom of page) enter your email address. >>> > > >>> > > >>> > >>> > _______________________________________________ >>> > boinc_alpha mailing list >>> > boinc_al...@ssl.berkeley.edu <mailto:boinc_al...@ssl.berkeley.edu> >>> > http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_alpha >>> > To unsubscribe, visit the above URL and >>> > (near bottom of page) enter your email address. >>> >>> _______________________________________________ >>> boinc_alpha mailing list >>> boinc_al...@ssl.berkeley.edu <mailto:boinc_al...@ssl.berkeley.edu> >>> http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_alpha >>> To unsubscribe, visit the above URL and >>> (near bottom of page) enter your email address. >>> >>> >> > > > > ------------------------------------------------------------------------------------ > Avast logo <http://www.avast.com/> > > This email has been checked for viruses by Avast antivirus software. > www.avast.com <http://www.avast.com/> > > _______________________________________________ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address. _______________________________________________ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
