Re: [boinc_dev] Can we do shared memory with no disk usage?
Gesendet: Montag, 17. Juni 2013 um 21:29 Uhr Von: David Anderson da...@ssl.berkeley.edu An: boinc_dev@ssl.berkeley.edu Betreff: Re: [boinc_dev] Can we do shared memory with no disk usage? In situations were BOINC is causing unexpectedly large ( 1 GB/hour) disk I/O, we need to figure out the source of the I/O. -- David The situation goes away with the boinc-app-seti package's binary instead of the official SETI clients, which is not configured for the graphics. I confirmed this on two machines, one of which beging the laptop that had so severe problems before. Example for WCG's clean energy (1 client) together with SETI-no-graphics (2 clients): iotop: 6046 idle boinc 0.00 B/s 1300.60 K/s 0.00 % 0.00 % ../../projects/www.worldcommunitygrid.org/wcgrid_cep2_qchem~4.C28H14S5Se.18.3.bp86.svp.n.opt 4038401046153527445 8411 0 6043 idle boinc 0.00 B/s 408.16 B/s 0.00 % 0.00 % ../../projects/www.worldcommunitygrid.org/wcgrid_cep2_6.40_~-exec wcgrid_cep2_qchem_prod_linux.x86 -float 1 -stop 43200 a 1MB/s write by the WCG, the two non-graphical SETI are just silent. The same I observed with the no-graphics Milkyway@Home of Debian. I then tried POEM and Yoyo@Home, which was seen in iotop, albeit performing equally and always a tolerable 1KB/s rate. I'll go and try geeting the original situation back with a self-compiled graphics-enabled SETI over the course of next week. Cheers, Steffen On 17-Jun-2013 11:14 AM, Steffen Möller wrote: Hello, I presume nobody wants to have the user play too much with any folders. I liked the all on a ramdisk idea, but BOINC occupies more than 10GB on the machine(s) in question, and it will similarly ask for that much (too much with today's common 16 or 24 GB setups when adding the memory for the apps) also for other setups. The checkpointing interval I understood (asked around) to be ignored by the client apps of a quite a few projects. Mine is set to some 7200 seconds (two hours) and the IO does not decrease. It would not be my prime target for optimisation. Could it be the graphics? We once observed that SETI without configuration for graphics is a few %ages faster than the same client with the same compilation options but graphics-savvy, even though no graphical client was run. If much IO was used for that, this might be an explanation. I cannot test this at the moment. Cheers, Steffen Gesendet: Montag, 17. Juni 2013 um 17:20 Uhr Von: Eric J Korpela korp...@ssl.berkeley.edu An: Steffen Möller steffen_moel...@gmx.de Cc: boinc_dev@ssl.berkeley.edu boinc_dev@ssl.berkeley.edu Betreff: Re: [boinc_dev] Can we do shared memory with no disk usage? We considered using memory mapped files for the checkpoint state information in SETI@home, but decided that it was virtually impossible to guarantee synchronization on exit. And, of course, the problem with using a ram disk for checkpoints that the checkpoint data disappears if power is lost. But if you want checkpoints to not occur, there is a preference for that... Tasks checkpoint to disk at most every: 60 seconds You can set it to 8 hours. But you'll also want to choose suspend to memory. Of course the output of the app (which depending on the app might be more frequent than checkpoints) will still spin up the disks. If you really want to operate from a ram disk, just copy your project directory to /dev/shm and link it into /var/lib/boinc/projects. You'll still need to back it up occasionally for those times when the power goes out. ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address. ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address. ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
Re: [boinc_dev] Can we do shared memory with no disk usage?
Hello, Gesendet: Mittwoch, 19. Juni 2013 um 21:39 Uhr Von: Nicolás Alvarez nicolas.alva...@gmail.com 2013/6/18 Heinz-Bernd Eggenstein heinz-bernd.eggenst...@aei.mpg.de: Hi all, Interesting. I guess it would be useful to run BOINC on a dedicated partition (e.g. ext hard disk/ USB stick) to isolate BOINC's contribution to the stats. Or drop iostat and use iotop instead, which gives realtime I/O stats per *thread*. Many thanks for the suggestion. Done it. It is not the boinc client but the scientific applications that make up for the IO. But that was not so much disputed. The remaining question is about how much of that IO is due to the * file based shared memory * status updates * checkpointing * the respective app's intrinsic purpose To help the interpretation of iotop, please feel reminded that the average persecond rate in KB/s for an average 2GB/s is (2*1024*1024*1024)/(60*60*1024) [1] 582.5422 Averaged over 10 seconds, I get typically values between 50 and 150KB/s Total DISK READ : 0.00 B/s | Total DISK WRITE : 55.31 K/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 68.83 K/s TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND 2337 be/4 root0.00 B/s0.00 B/s 0.00 % 0.06 % [flush-8:0] 4095 idle boinc 0.00 B/s7.56 K/s 0.00 % 0.05 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3111937 4068 idle boinc 0.00 B/s4.77 K/s 0.00 % 0.03 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3160216 4675 idle boinc 0.00 B/s9.15 K/s 0.00 % 0.02 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3030191 3589 idle boinc 0.00 B/s9.95 K/s 0.00 % 0.01 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3194037 4401 idle boinc 0.00 B/s4.38 K/s 0.00 % 0.01 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3075156 4132 idle boinc 0.00 B/s6.76 K/s 0.00 % 0.00 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3764396 4133 idle boinc 0.00 B/s 407.43 B/s 0.00 % 0.00 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3764396 18744 idle boinc 0.00 B/s 1222.28 B/s 0.00 % 0.00 % boinc --check_all_logins --redirectio --dir /var/lib/boinc-client 1022 idle boinc 0.00 B/s6.37 K/s 0.00 % 0.00 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3231298 3869 idle boinc 0.00 B/s4.77 K/s 0.00 % 0.00 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3180570 4096 idle boinc 0.00 B/s0.00 B/s 0.00 % 0.00 % ../../projects/boinc.bakerlab.org_rosetta/minirosetta_3~watchdog -run::rng mt19937 -constant_seed -jran 3111937 and while run over 10 minutes, iostat gives $ date; iostat -h -m; sleep 600; date; iostat -h -m Thu Jun 20 12:04:04 CEST 2013 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 3.27 0.02 0.35 29254 668009 Thu Jun 20 12:14:04 CEST 2013 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 3.27 0.02 0.36 29254 668359 So, over 10 minutes, we still have a 35 MB/(60s) = 597KB/s and the ever reported 2 GB/h. One of my first actions after reading the initial iostat output (not shown) was to have the ext4 formatted root partition to receive the extra option commit=60. The noatime was already set. All data presented here were taken after that change. The file mapping of shared memory seems to be generally accepted as a possible IO trap, as seen e.g. here http://serverfault.com/questions/361032/high-disk-i-o-when-cache-is-used . It seems a bit like the scientific applications have some bits here and there to tune their IO behaviour. Running Milkyway 0.18 for instance, I see no IO for the clients at all while they went from 5% to 27%. I have locally implemented a patch for (presumed) diskless shared memory. You can have a look over here http://anonscm.debian.org/gitweb/?p=pkg-boinc/boinc.git;a=blob;f=debian/patches/mmap_mem_only.patch;hb=HEAD Again: it is not applied in Debian's regular BOINC packages but needs to be commented in in debian/patches/series. I'll report if it works and about respective effects on SETI (may required a recompilation) next week. Best, Steffen ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom
Re: [boinc_dev] Can we do shared memory with no disk usage?
It is easy to set check pointing to a longer time. There is already a setting available for this. -Original Message- From: boinc_dev [mailto:boinc_dev-boun...@ssl.berkeley.edu] On Behalf Of Steffen Möller Sent: Monday, June 17, 2013 4:04 AM Cc: boinc_dev@ssl.berkeley.edu Subject: Re: [boinc_dev] Can we do shared memory with no disk usage? Gesendet: Montag, 17. Juni 2013 um 08:46 Uhr Von: David Anderson da...@ssl.berkeley.edu Manager/client communication uses TCP - no disk I/O. So the possible sources of large disk I/O are: 1) checkpoint or output file generation by apps 2) writing of client_state.xml (or maybe other files) by the BOINC client The checkpointing and the client_state.xml for my 24/7 machines I would very much like to just switch off or have updated only every hour or have their update initiated only upon request by the boinc client. 3) client/app communication via memory-mapped files. According to my calculations, this should generate less than 1 MB/Hour per task. Note: we use memory-mapped files (mmap) instead of pure shared memory segments (shmget) because Mac OS X has a system-wide limit of 32 shared-mem segments, and some Linux systems have limits. Maybe there's a way to configure memory-mapped files to not write back to the disk file, but I can't find one. It should be the shm_open with mmap together, i.e. just substituting the call to open that BOINC currently performs with shm_open. From http://pubs.opengroup.org/onlinepubs/009695399/functions/shm_open.html fd = shm_open(/myregion, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR); rptr = mmap(NULL, sizeof(struct region), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); Should be disk-free, with /myregion only having a symbolic meaning. Here http://stackoverflow.com/questions/4836863/shared-memory-or-mmap-linux-c-c-ipc I also found a reference to an interesting IPv6 to localhost with multicast idea, No idea if that is applicable for BOINC. The trend is more towards open_shm+mmap. Many thanks for the swift reply Steffen Can someone investigate which of these is the source of the large I/O? -- David On 16-Jun-2013 11:03 AM, Steffen Möller wrote: Hello, iostat gives rather drastic values for the amount of data that is written to disk by BOINC and/or its applications. Some good fellow once crafted a but report about it http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=636075 and my own reply was not overly helpful at the time. To reduce the IO overhead is certainly helpful. Reduced latency is one thing, but with an outreach to the mobile world there are many SSD / flash device users who care so much about write endurance. It kept bugging me, and quite a while back it came to me that this may not be because of the application's work but instead be mere overhead of a file based communication between the app and the boinc-client / boinc-manager. I just never got around chasing this up, also having read so often about communication done via shared memory, which should not need much IO, and if so, then not with disk devices but something like tmpfs. Let's see how it is done, I just thought. From what I found, there are two functions to create shared memory in BOINC, both in lib/shmem.cpp. One is through create_shmem which internally uses shmget and should be just fine. The other is create_shmem_mmap which internally uses mmap - which can be memory-only or not. The early mmap allowed the memory only (anonymous) sharing only for forks of the same process. For anything else one needs to pass a regular file descriptor to communicate to mmap from/to where to get/write the data. Newer years brought the function shm_open (http://linux.die.net/man/3/shm_open), which creates an entry in /dev/shm if I got this right, and allows forwarding this fd for a complete in memory-only communication with a (pseudo-)file-mapping mmap. In today's BOINC code, mmap is called with a file descriptor created with open (no typo, it is from boinc/lib/std_fixes.h), which itself calles open64 as defined in /usr/include/fcntl.h (?) and expects to create a regular file. I admit not to know too much about the consequences of a memory-only communication for BOINC. It is not more than a couple of megs every second indicating the status of the applications, right? So not too much memory would be taken. With checkpointing performed independently, this could then work just fine. Is there something I did not see? I am otherwise much tempted to address it and see how far I get. Some extra thinking is required to maintain a compatibility between the BOINC client and older statically linked applications. With Debian we have the applications dynamically linking to the same BOINC library as the BOINC client. Promising enough? Please direct me
Re: [boinc_dev] Can we do shared memory with no disk usage?
___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
Re: [boinc_dev] Can we do shared memory with no disk usage?
2013/6/18 Heinz-Bernd Eggenstein heinz-bernd.eggenst...@aei.mpg.de: Hi all, Interesting. I guess it would be useful to run BOINC on a dedicated partition (e.g. ext hard disk/ USB stick) to isolate BOINC's contribution to the stats. Or drop iostat and use iotop instead, which gives realtime I/O stats per *thread*. -- Nicolás ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
Re: [boinc_dev] Can we do shared memory with no disk usage?
Gesendet: Montag, 17. Juni 2013 um 21:29 Uhr Von: David Anderson da...@ssl.berkeley.edu An: boinc_dev@ssl.berkeley.edu Betreff: Re: [boinc_dev] Can we do shared memory with no disk usage? In situations were BOINC is causing unexpectedly large ( 1 GB/hour) disk I/O, we need to figure out the source of the I/O. -- David My little centrino laptop had SETI (official client) run over night. $ iostat -h -m Linux 3.8-1-rt-amd64 (Toshiba) 06/18/2013 _x86_64_(2 CPU) ... Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 33.80 0.17 0.39 3040 6859 $ date Tue Jun 18 00:30:06 CEST 2013 $ iostat -h -m ... Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 65.32 0.07 0.98 3387 45737 $ date Tue Jun 18 08:36:50 CEST 2013 Looking only at the last column, in those 8 hours this were (45737-6859)/1024 [1] 37.9668 GB and consequently (45737-6859)/1024/8 [1] 4.74585 GB/h another machine, running about 10 clients in parallel (all Rosetta, one WCG), had a bit less of IO ... here iostat was run twice with 1h interim sleep twin1a:~ $ date ; iostat -m -h Mon Jun 17 23:34:12 CEST 2013 Linux 3.2.0-3-amd64 (twin1a)06/17/2013 _x86_64_(24 CPU) Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 3.08 0.02 0.30 29091 504228 $ sleep 3600 ; date ; iostat -m -h Tue Jun 18 00:34:26 CEST 2013 .. Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 3.09 0.02 0.30 29091 506276 So this is about 2 GB per hour written and nothing read And yet another machine, same hardware, mostly einstein with 1 Rosetta and 1 WCG twin1b:~ $ date; iostat -h -m ; sleep 3600 ; date ; iostat -h -m Mon Jun 17 23:58:16 CEST 2013 ... Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 1.67 0.00 0.12 758482377118 Tue Jun 18 00:58:16 CEST 2013 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 1.67 0.00 0.12 758482379914 Which means another 2 GB per hour. The machines were not running anything else but BOINC, the laptop had firefox open in the background. No BOINC graphical clients run anywhere. iostat comes with the sysstat package, for anyone out there to try. The laptop only has 1G of mem, which may lead to some swapping and account for some of the IO. Still, to me it looks mostly like some process writing a lot of status information that is not read by anyone. The missing reads for the big machines I might want to explain by large IO buffers ... there certainly have been a couple of uploads that should have caused some read, otherwise. Cheers, Steffen ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
Re: [boinc_dev] Can we do shared memory with no disk usage?
Hi all, Interesting. I guess it would be useful to run BOINC on a dedicated partition (e.g. ext hard disk/ USB stick) to isolate BOINC's contribution to the stats. How often is the client_state.xml file updated? It can grow to enormous sizes if you have a huge number of waiting tasks or unreported finished tasks (stderr outputs !!). Cheers HB - Heinz-Bernd Eggenstein Max Planck Institute for Gravitational Physics Callinstrasse 38 D-30167 Hannover, Germany Tel.: +49-511-762-19466 (Room 037) From: Steffen Möller steffen_moel...@gmx.de To: boinc_dev@ssl.berkeley.edu boinc_dev@ssl.berkeley.edu, Date: 06/18/2013 09:03 AM Subject:Re: [boinc_dev] Can we do shared memory with no disk usage? Sent by:boinc_dev boinc_dev-boun...@ssl.berkeley.edu Gesendet: Montag, 17. Juni 2013 um 21:29 Uhr Von: David Anderson da...@ssl.berkeley.edu An: boinc_dev@ssl.berkeley.edu Betreff: Re: [boinc_dev] Can we do shared memory with no disk usage? In situations were BOINC is causing unexpectedly large ( 1 GB/hour) disk I/O, we need to figure out the source of the I/O. -- David My little centrino laptop had SETI (official client) run over night. $ iostat -h -m Linux 3.8-1-rt-amd64 (Toshiba) 06/18/2013 _x86_64_(2 CPU) ... Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 33.80 0.17 0.39 3040 6859 $ date Tue Jun 18 00:30:06 CEST 2013 $ iostat -h -m ... Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 65.32 0.07 0.98 3387 45737 $ date Tue Jun 18 08:36:50 CEST 2013 Looking only at the last column, in those 8 hours this were (45737-6859)/1024 [1] 37.9668 GB and consequently (45737-6859)/1024/8 [1] 4.74585 GB/h another machine, running about 10 clients in parallel (all Rosetta, one WCG), had a bit less of IO ... here iostat was run twice with 1h interim sleep twin1a:~ $ date ; iostat -m -h Mon Jun 17 23:34:12 CEST 2013 Linux 3.2.0-3-amd64 (twin1a)06/17/2013 _x86_64_(24 CPU) Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 3.08 0.02 0.30 29091 504228 $ sleep 3600 ; date ; iostat -m -h Tue Jun 18 00:34:26 CEST 2013 .. Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 3.09 0.02 0.30 29091 506276 So this is about 2 GB per hour written and nothing read And yet another machine, same hardware, mostly einstein with 1 Rosetta and 1 WCG twin1b:~ $ date; iostat -h -m ; sleep 3600 ; date ; iostat -h -m Mon Jun 17 23:58:16 CEST 2013 ... Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 1.67 0.00 0.12 758482377118 Tue Jun 18 00:58:16 CEST 2013 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 1.67 0.00 0.12 758482379914 Which means another 2 GB per hour. The machines were not running anything else but BOINC, the laptop had firefox open in the background. No BOINC graphical clients run anywhere. iostat comes with the sysstat package, for anyone out there to try. The laptop only has 1G of mem, which may lead to some swapping and account for some of the IO. Still, to me it looks mostly like some process writing a lot of status information that is not read by anyone. The missing reads for the big machines I might want to explain by large IO buffers ... there certainly have been a couple of uploads that should have caused some read, otherwise. Cheers, Steffen ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address. ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
Re: [boinc_dev] Can we do shared memory with no disk usage?
How much I/O do these machines do when BOINC is not running? 4 GB/hr is about 1 MB/sec. I can't think of any BOINC activity that would write this much. We need to look at the system call trace to get more info. -- David On 18-Jun-2013 12:03 AM, Steffen Möller wrote: Gesendet: Montag, 17. Juni 2013 um 21:29 Uhr Von: David Anderson da...@ssl.berkeley.edu An: boinc_dev@ssl.berkeley.edu Betreff: Re: [boinc_dev] Can we do shared memory with no disk usage? In situations were BOINC is causing unexpectedly large ( 1 GB/hour) disk I/O, we need to figure out the source of the I/O. -- David My little centrino laptop had SETI (official client) run over night. $ iostat -h -m Linux 3.8-1-rt-amd64 (Toshiba) 06/18/2013 _x86_64_(2 CPU) ... Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 33.80 0.17 0.39 3040 6859 $ date Tue Jun 18 00:30:06 CEST 2013 $ iostat -h -m ... Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 65.32 0.07 0.98 3387 45737 $ date Tue Jun 18 08:36:50 CEST 2013 Looking only at the last column, in those 8 hours this were (45737-6859)/1024 [1] 37.9668 GB and consequently (45737-6859)/1024/8 [1] 4.74585 GB/h another machine, running about 10 clients in parallel (all Rosetta, one WCG), had a bit less of IO ... here iostat was run twice with 1h interim sleep twin1a:~ $ date ; iostat -m -h Mon Jun 17 23:34:12 CEST 2013 Linux 3.2.0-3-amd64 (twin1a)06/17/2013 _x86_64_(24 CPU) Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 3.08 0.02 0.30 29091 504228 $ sleep 3600 ; date ; iostat -m -h Tue Jun 18 00:34:26 CEST 2013 .. Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 3.09 0.02 0.30 29091 506276 So this is about 2 GB per hour written and nothing read And yet another machine, same hardware, mostly einstein with 1 Rosetta and 1 WCG twin1b:~ $ date; iostat -h -m ; sleep 3600 ; date ; iostat -h -m Mon Jun 17 23:58:16 CEST 2013 ... Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 1.67 0.00 0.12 758482377118 Tue Jun 18 00:58:16 CEST 2013 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 1.67 0.00 0.12 758482379914 Which means another 2 GB per hour. The machines were not running anything else but BOINC, the laptop had firefox open in the background. No BOINC graphical clients run anywhere. iostat comes with the sysstat package, for anyone out there to try. The laptop only has 1G of mem, which may lead to some swapping and account for some of the IO. Still, to me it looks mostly like some process writing a lot of status information that is not read by anyone. The missing reads for the big machines I might want to explain by large IO buffers ... there certainly have been a couple of uploads that should have caused some read, otherwise. Cheers, Steffen ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address. ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
Re: [boinc_dev] Can we do shared memory with no disk usage?
The state file is written mostly when jobs start and end. (A long time ago it was written on each checkpoint, but we changed this). -- David On 18-Jun-2013 1:29 AM, Heinz-Bernd Eggenstein wrote: How often is the client_state.xml file updated? It can grow to enormous sizes if you have a huge number of waiting tasks or unreported finished tasks (stderr outputs !!). ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
Re: [boinc_dev] Can we do shared memory with no disk usage?
Gesendet: Montag, 17. Juni 2013 um 08:46 Uhr Von: David Anderson da...@ssl.berkeley.edu Manager/client communication uses TCP - no disk I/O. So the possible sources of large disk I/O are: 1) checkpoint or output file generation by apps 2) writing of client_state.xml (or maybe other files) by the BOINC client The checkpointing and the client_state.xml for my 24/7 machines I would very much like to just switch off or have updated only every hour or have their update initiated only upon request by the boinc client. 3) client/app communication via memory-mapped files. According to my calculations, this should generate less than 1 MB/Hour per task. Note: we use memory-mapped files (mmap) instead of pure shared memory segments (shmget) because Mac OS X has a system-wide limit of 32 shared-mem segments, and some Linux systems have limits. Maybe there's a way to configure memory-mapped files to not write back to the disk file, but I can't find one. It should be the shm_open with mmap together, i.e. just substituting the call to open that BOINC currently performs with shm_open. From http://pubs.opengroup.org/onlinepubs/009695399/functions/shm_open.html fd = shm_open(/myregion, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR); rptr = mmap(NULL, sizeof(struct region), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); Should be disk-free, with /myregion only having a symbolic meaning. Here http://stackoverflow.com/questions/4836863/shared-memory-or-mmap-linux-c-c-ipc I also found a reference to an interesting IPv6 to localhost with multicast idea, No idea if that is applicable for BOINC. The trend is more towards open_shm+mmap. Many thanks for the swift reply Steffen Can someone investigate which of these is the source of the large I/O? -- David On 16-Jun-2013 11:03 AM, Steffen Möller wrote: Hello, iostat gives rather drastic values for the amount of data that is written to disk by BOINC and/or its applications. Some good fellow once crafted a but report about it http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=636075 and my own reply was not overly helpful at the time. To reduce the IO overhead is certainly helpful. Reduced latency is one thing, but with an outreach to the mobile world there are many SSD / flash device users who care so much about write endurance. It kept bugging me, and quite a while back it came to me that this may not be because of the application's work but instead be mere overhead of a file based communication between the app and the boinc-client / boinc-manager. I just never got around chasing this up, also having read so often about communication done via shared memory, which should not need much IO, and if so, then not with disk devices but something like tmpfs. Let's see how it is done, I just thought. From what I found, there are two functions to create shared memory in BOINC, both in lib/shmem.cpp. One is through create_shmem which internally uses shmget and should be just fine. The other is create_shmem_mmap which internally uses mmap - which can be memory-only or not. The early mmap allowed the memory only (anonymous) sharing only for forks of the same process. For anything else one needs to pass a regular file descriptor to communicate to mmap from/to where to get/write the data. Newer years brought the function shm_open (http://linux.die.net/man/3/shm_open), which creates an entry in /dev/shm if I got this right, and allows forwarding this fd for a complete in memory-only communication with a (pseudo-)file-mapping mmap. In today's BOINC code, mmap is called with a file descriptor created with open (no typo, it is from boinc/lib/std_fixes.h), which itself calles open64 as defined in /usr/include/fcntl.h (?) and expects to create a regular file. I admit not to know too much about the consequences of a memory-only communication for BOINC. It is not more than a couple of megs every second indicating the status of the applications, right? So not too much memory would be taken. With checkpointing performed independently, this could then work just fine. Is there something I did not see? I am otherwise much tempted to address it and see how far I get. Some extra thinking is required to maintain a compatibility between the BOINC client and older statically linked applications. With Debian we have the applications dynamically linking to the same BOINC library as the BOINC client. Promising enough? Please direct me. Cheers, Steffen ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address. ___ boinc_dev mailing list
Re: [boinc_dev] Can we do shared memory with no disk usage?
We considered using memory mapped files for the checkpoint state information in SETI@home, but decided that it was virtually impossible to guarantee synchronization on exit. And, of course, the problem with using a ram disk for checkpoints that the checkpoint data disappears if power is lost. But if you want checkpoints to not occur, there is a preference for that... Tasks checkpoint to disk at most every: 60 seconds You can set it to 8 hours. But you'll also want to choose suspend to memory. Of course the output of the app (which depending on the app might be more frequent than checkpoints) will still spin up the disks. If you really want to operate from a ram disk, just copy your project directory to /dev/shm and link it into /var/lib/boinc/projects. You'll still need to back it up occasionally for those times when the power goes out. On Mon, Jun 17, 2013 at 1:04 AM, Steffen Möller steffen_moel...@gmx.dewrote: Gesendet: Montag, 17. Juni 2013 um 08:46 Uhr Von: David Anderson da...@ssl.berkeley.edu Manager/client communication uses TCP - no disk I/O. So the possible sources of large disk I/O are: 1) checkpoint or output file generation by apps 2) writing of client_state.xml (or maybe other files) by the BOINC client The checkpointing and the client_state.xml for my 24/7 machines I would very much like to just switch off or have updated only every hour or have their update initiated only upon request by the boinc client. 3) client/app communication via memory-mapped files. According to my calculations, this should generate less than 1 MB/Hour per task. Note: we use memory-mapped files (mmap) instead of pure shared memory segments (shmget) because Mac OS X has a system-wide limit of 32 shared-mem segments, and some Linux systems have limits. Maybe there's a way to configure memory-mapped files to not write back to the disk file, but I can't find one. It should be the shm_open with mmap together, i.e. just substituting the call to open that BOINC currently performs with shm_open. From http://pubs.opengroup.org/onlinepubs/009695399/functions/shm_open.html fd = shm_open(/myregion, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR); rptr = mmap(NULL, sizeof(struct region), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); Should be disk-free, with /myregion only having a symbolic meaning. Here http://stackoverflow.com/questions/4836863/shared-memory-or-mmap-linux-c-c-ipc I also found a reference to an interesting IPv6 to localhost with multicast idea, No idea if that is applicable for BOINC. The trend is more towards open_shm+mmap. Many thanks for the swift reply Steffen Can someone investigate which of these is the source of the large I/O? -- David On 16-Jun-2013 11:03 AM, Steffen Möller wrote: Hello, iostat gives rather drastic values for the amount of data that is written to disk by BOINC and/or its applications. Some good fellow once crafted a but report about it http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=636075 and my own reply was not overly helpful at the time. To reduce the IO overhead is certainly helpful. Reduced latency is one thing, but with an outreach to the mobile world there are many SSD / flash device users who care so much about write endurance. It kept bugging me, and quite a while back it came to me that this may not be because of the application's work but instead be mere overhead of a file based communication between the app and the boinc-client / boinc-manager. I just never got around chasing this up, also having read so often about communication done via shared memory, which should not need much IO, and if so, then not with disk devices but something like tmpfs. Let's see how it is done, I just thought. From what I found, there are two functions to create shared memory in BOINC, both in lib/shmem.cpp. One is through create_shmem which internally uses shmget and should be just fine. The other is create_shmem_mmap which internally uses mmap - which can be memory-only or not. The early mmap allowed the memory only (anonymous) sharing only for forks of the same process. For anything else one needs to pass a regular file descriptor to communicate to mmap from/to where to get/write the data. Newer years brought the function shm_open ( http://linux.die.net/man/3/shm_open), which creates an entry in /dev/shm if I got this right, and allows forwarding this fd for a complete in memory-only communication with a (pseudo-)file-mapping mmap. In today's BOINC code, mmap is called with a file descriptor created with open (no typo, it is from boinc/lib/std_fixes.h), which itself calles open64 as defined in /usr/include/fcntl.h (?) and expects to create a regular file. I admit not to know too much about the consequences of a memory-only communication for
Re: [boinc_dev] Can we do shared memory with no disk usage?
Hello, I presume nobody wants to have the user play too much with any folders. I liked the all on a ramdisk idea, but BOINC occupies more than 10GB on the machine(s) in question, and it will similarly ask for that much (too much with today's common 16 or 24 GB setups when adding the memory for the apps) also for other setups. The checkpointing interval I understood (asked around) to be ignored by the client apps of a quite a few projects. Mine is set to some 7200 seconds (two hours) and the IO does not decrease. It would not be my prime target for optimisation. Could it be the graphics? We once observed that SETI without configuration for graphics is a few %ages faster than the same client with the same compilation options but graphics-savvy, even though no graphical client was run. If much IO was used for that, this might be an explanation. I cannot test this at the moment. Cheers, Steffen Gesendet: Montag, 17. Juni 2013 um 17:20 Uhr Von: Eric J Korpela korp...@ssl.berkeley.edu An: Steffen Möller steffen_moel...@gmx.de Cc: boinc_dev@ssl.berkeley.edu boinc_dev@ssl.berkeley.edu Betreff: Re: [boinc_dev] Can we do shared memory with no disk usage? We considered using memory mapped files for the checkpoint state information in SETI@home, but decided that it was virtually impossible to guarantee synchronization on exit. And, of course, the problem with using a ram disk for checkpoints that the checkpoint data disappears if power is lost. But if you want checkpoints to not occur, there is a preference for that... Tasks checkpoint to disk at most every: 60 seconds You can set it to 8 hours. But you'll also want to choose suspend to memory. Of course the output of the app (which depending on the app might be more frequent than checkpoints) will still spin up the disks. If you really want to operate from a ram disk, just copy your project directory to /dev/shm and link it into /var/lib/boinc/projects. You'll still need to back it up occasionally for those times when the power goes out. ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
Re: [boinc_dev] Can we do shared memory with no disk usage?
In situations were BOINC is causing unexpectedly large ( 1 GB/hour) disk I/O, we need to figure out the source of the I/O. -- David On 17-Jun-2013 11:14 AM, Steffen Möller wrote: Hello, I presume nobody wants to have the user play too much with any folders. I liked the all on a ramdisk idea, but BOINC occupies more than 10GB on the machine(s) in question, and it will similarly ask for that much (too much with today's common 16 or 24 GB setups when adding the memory for the apps) also for other setups. The checkpointing interval I understood (asked around) to be ignored by the client apps of a quite a few projects. Mine is set to some 7200 seconds (two hours) and the IO does not decrease. It would not be my prime target for optimisation. Could it be the graphics? We once observed that SETI without configuration for graphics is a few %ages faster than the same client with the same compilation options but graphics-savvy, even though no graphical client was run. If much IO was used for that, this might be an explanation. I cannot test this at the moment. Cheers, Steffen Gesendet: Montag, 17. Juni 2013 um 17:20 Uhr Von: Eric J Korpela korp...@ssl.berkeley.edu An: Steffen Möller steffen_moel...@gmx.de Cc: boinc_dev@ssl.berkeley.edu boinc_dev@ssl.berkeley.edu Betreff: Re: [boinc_dev] Can we do shared memory with no disk usage? We considered using memory mapped files for the checkpoint state information in SETI@home, but decided that it was virtually impossible to guarantee synchronization on exit. And, of course, the problem with using a ram disk for checkpoints that the checkpoint data disappears if power is lost. But if you want checkpoints to not occur, there is a preference for that... Tasks checkpoint to disk at most every: 60 seconds You can set it to 8 hours. But you'll also want to choose suspend to memory. Of course the output of the app (which depending on the app might be more frequent than checkpoints) will still spin up the disks. If you really want to operate from a ram disk, just copy your project directory to /dev/shm and link it into /var/lib/boinc/projects. You'll still need to back it up occasionally for those times when the power goes out. ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address. ___ boinc_dev mailing list boinc_dev@ssl.berkeley.edu http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
