Re: [ccp4bb] I compressed my images by ~ a factor of two, and they load and process in mosflm faster
Yes, this is exactly what I meant. If the data are amenable (which was addressed in the previous discussion with reference to diffraction images) and there is a suitable lossless compression/expansion algorithm, then on most modern computers it is faster to read the compressed data from disk and expand it in RAM, rather than directly read the uncompressed image from a magnetic plate. Of course this depends on all sorts of factors such as the speed of the disk, the compression ratio, the CPU(s) clock speed, if the decompression can be done in parallel, how much calculation the decompression requires, and so on. Bill's example is nice because the compression is transparent, so no extra work needs to be done by developers. However, this is one for Macs only. I'd like to know whether integration runs faster using CBF images with the decompression overhead of CBFlib compared with reading the same data in uncompressed form on standard hardware (whatever that means). Cheers David -Original Message- From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Andrew Purkiss-Trew Sent: 18 September 2009 21:52 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] I compressed my images by ~ a factor of two, and they load and process in mosflm faster The current bottleneck with file systems is the speed of getting data on or off the magnetic surface. So filesystem compression helps, as less data needs to be physically written or read per image. The CPU time spent compressing the data is less than the time saved in writing less data to the surface. I would be interested to see if the speed up is the same with a solid state drive, as there is near 'random access' here, unlike with a magnetic drive where the seek time is one of the bottlenecks. For example, mechanical hard drives are limited to about 130MB/s, whereas SSDs can already manage 200MB/s (faster than a first generation SATA interface at 150MB/s can cope with and one of the drivers behind the 2nd (300MB/s) and 3rd generation (600MB/s) SATA intefaces). The large size of our image files should make them ideal for use with SSDs. Quoting James Holton jmhol...@lbl.gov: I think it important to point out that despite the subject line, Dr. Scott's statement was: I think they process a bit faster too Strangely enough, this has not convinced me to re-format my RAID array with an new file system nor re-write all my software to support yet another new file format. I guess I am just lazy that way. Has anyone measured the speed increase? Have macs become I/O-bound again? In any case, I think it is important to remember that there are good reasons for leaving image file formats uncompressed. Probably the most important is the activation barrier to new authors writing new programs that read them. fread() is one thing, but finding the third-party code for a particular compression algorithm, navigating a CVS repository and linking to a library are quite another! This is actually quite a leap for those of us who never had any formal training in computer science. Personally, I still haven't figured out how to read pck images, as it is much easier to write jiffy programs for uncompressed data. For example, if all you want to do is extract a group of pixels (such as a spot), then you have to decompress the whole image! In computer speak: fseek() is rendered useless by compression. This could be why Mar opted not to use the pck compression for their newer CCD-based detectors? That said, compressed file systems do appear particularly attractive if space is limiting. Apparently HFS can do it, but what about other operating systems? Does anyone have experience with a Linux file system that both supports compression and doesn't get corrupted easily? -James Holton MAD Scientist Graeme Winter wrote: Hi David, If the data compression is carefully chosen you are right: lossless jpeg2000 compression on diffraction images works very well, but is a spot slow. The CBF compression using the byte offset method is a little less good at compression put massively faster... as you point out, this is the one used in the pilatus images. I recall that the .pck format used for the MAR image plates had the same property - it was quicker to read in a compressed image that the raw equivalent. So... once everyone is using the CBF standard for their images, with native lossless compression, it'll save a fair amount in disk space (=£/$), make life easier for people and - perhaps most importantly - save a lot of data transfer time. Now the funny thing with this is that if we compress the images before we store them, the compression implemented in the file system will be less effective... oh well, can't win em all... Cheers, Graeme 2009/9/18 Waterman, David (DLSLtd,RAL,DIA) david.water...@diamond.ac.uk: Just to comment on this, my friend in the computer game
[ccp4bb] How to dissolve palmitoyl-DHAP
Hi everybody, This is my first post in this maininglist. As title I need to dissolve this fatty acid in order to add to my protein to try to crystallize together... I tried with DMSO (but nothing) and with polar solvents but I think they will destroy my protein...Any suggestion? Thank you Simone Nenci
[ccp4bb] Postdoctoral Position - University of Puerto Rico, Rio Piedras Campus
A postdoctoral research position is immediately available in the laboratory of Dr. Eric R. Schreiter, at the University of Puerto Rico - Rio Piedras Campus, to study the effects of S-nitrosylation on protein structure and function at the molecular level. Applicants should possess a PhD in a relevant field (chemistry, biochemistry, molecular biology, structural biology, etc.) and ideally have experience in structural characterization of macromolecules using x-ray crystallography. Experience in protein biochemistry and other biophysical techniques for characterizing proteins is also highly desirable. Excellent communication skills, motivation and the ability to work as part of a team are required. The PI's laboratory is equipped with state of the art instrumentation for macromolecular x-ray crystallography, including a Rigaku RU-H3R X-ray generator with a CCD detector and X-stream 2000 low temperature system. The Rio Piedras Campus of the University of Puerto Rico is its flagship campus for basic science research and is only two miles away from the Medical Sciences Campus, accessible by train. It is located in San Juan, Puerto Rico, which offers a warm climate year-round and an abundance of outdoor activities in close proximity. Puerto Rico uses both Spanish and English as official languages, but laboratory business is conducted primarily in English. Interested individuals should submit a CV, a summary of research achievements, and names of three references to: Dr. Eric Schreiter (eschrei...@vmail.uprrp.edu)
Re: [ccp4bb] I compressed my images by ~ a factor of two, and they load and process in mosflm faster
Hi Not a typical run, but I just got these on my Macbook pro from a 320 image 1.5Å myoglobin dataset, collected on a Q315 - [macf3c-4:~/test/cbf] harry% cd cbf [macf3c-4:~/test/cbf/cbf] harry% time mosflm integrate integrate.lp 445.355u 27.951s 8:38.57 91.2% 0+0k 1+192io 41pf+0w [macf3c-4:~/test/cbf/cbf] harry% cd ../original [macf3c-4:~/test/cbf/original] harry% time mosflm integrate integrate.lp 279.331u 18.691s 8:05.76 61.3% 0+0k 0+240io 16pf+0w I am somewhat surprised at this. Since I wasn't running anything else, I'm also a little surprised that, although the user times above are so different, so are the percentages of the elapsed clock times. Herb may be able to comment more knowledgeably. I don't have my Snow Leopard box here so can't compare the ditto'd files just at the moment. On 21 Sep 2009, at 13:26, Waterman, David (DLSLtd,RAL,DIA) wrote: Yes, this is exactly what I meant. If the data are amenable (which was addressed in the previous discussion with reference to diffraction images) and there is a suitable lossless compression/ expansion algorithm, then on most modern computers it is faster to read the compressed data from disk and expand it in RAM, rather than directly read the uncompressed image from a magnetic plate. Of course this depends on all sorts of factors such as the speed of the disk, the compression ratio, the CPU(s) clock speed, if the decompression can be done in parallel, how much calculation the decompression requires, and so on. Bill's example is nice because the compression is transparent, so no extra work needs to be done by developers. However, this is one for Macs only. I'd like to know whether integration runs faster using CBF images with the decompression overhead of CBFlib compared with reading the same data in uncompressed form on standard hardware (whatever that means). Cheers David -Original Message- From: CCP4 bulletin board [mailto:ccp...@jiscmail.ac.uk] On Behalf Of Andrew Purkiss-Trew Sent: 18 September 2009 21:52 To: CCP4BB@JISCMAIL.AC.UK Subject: Re: [ccp4bb] I compressed my images by ~ a factor of two, and they load and process in mosflm faster The current bottleneck with file systems is the speed of getting data on or off the magnetic surface. So filesystem compression helps, as less data needs to be physically written or read per image. The CPU time spent compressing the data is less than the time saved in writing less data to the surface. I would be interested to see if the speed up is the same with a solid state drive, as there is near 'random access' here, unlike with a magnetic drive where the seek time is one of the bottlenecks. For example, mechanical hard drives are limited to about 130MB/s, whereas SSDs can already manage 200MB/s (faster than a first generation SATA interface at 150MB/s can cope with and one of the drivers behind the 2nd (300MB/s) and 3rd generation (600MB/s) SATA intefaces). The large size of our image files should make them ideal for use with SSDs. Quoting James Holton jmhol...@lbl.gov: I think it important to point out that despite the subject line, Dr. Scott's statement was: I think they process a bit faster too Strangely enough, this has not convinced me to re-format my RAID array with an new file system nor re-write all my software to support yet another new file format. I guess I am just lazy that way. Has anyone measured the speed increase? Have macs become I/O-bound again? In any case, I think it is important to remember that there are good reasons for leaving image file formats uncompressed. Probably the most important is the activation barrier to new authors writing new programs that read them. fread() is one thing, but finding the third-party code for a particular compression algorithm, navigating a CVS repository and linking to a library are quite another! This is actually quite a leap for those of us who never had any formal training in computer science. Personally, I still haven't figured out how to read pck images, as it is much easier to write jiffy programs for uncompressed data. For example, if all you want to do is extract a group of pixels (such as a spot), then you have to decompress the whole image! In computer speak: fseek() is rendered useless by compression. This could be why Mar opted not to use the pck compression for their newer CCD-based detectors? That said, compressed file systems do appear particularly attractive if space is limiting. Apparently HFS can do it, but what about other operating systems? Does anyone have experience with a Linux file system that both supports compression and doesn't get corrupted easily? -James Holton MAD Scientist Graeme Winter wrote: Hi David, If the data compression is carefully chosen you are right: lossless jpeg2000 compression on diffraction images works very well, but is a spot slow. The CBF compression using the byte offset method is a
[ccp4bb] iMosflm indexing error
Hello all,
I am attempting to use the iMosflm 1.0.3 GUI with the test dataset available
from the webpage. I am able to load the dataset without any problems.
However, when I click on Indexing the program displays the error below.
Clicking OK just results in a spinning E in the upper right hand corner
while the program says Finding spots on image hg_084.mar1600 I'm not sure
what it did with image one. There are no spots selected in the Image
Display.
Looking at the mosflm.lp file it appears that spots were found for both
image 1 and 84 successfully. Perhaps this is a problem with the GUI not
updating?
Any help would be appreciated. It's been a while since I played with
iMosflm but this didn't happen when I went through the tutorial with
previous versions.
Regards,
Mike
Michael S. Murray, Ph.D.
National Institute of Environmental Health Sciences
MD E3-01
P.O. Box 12233
Research Triangle Park, NC 27709
Phone: (919) 541-0268
syntax error in expression 627.0 * : premature end of expression
syntax error in expression 627.0 * : premature end of expression
while executing
expr $y_pixels * [$::session getParameterValue yscale]
(auto arm line 23)
invoked from within
switch -- $a_type {
auto {
set type $a_type
set x_mm [lindex $args 4]
set y_mm [lindex $args 5]
if {1 || ([lindex $args 3] == ...
while constructing object ::Spotlist::spot0 in ::Spot::constructor
(body line 7)
invoked from within
Spot \#auto auto $image_height $pixel_size $origin $axis_order $x $y
$intensity $stdev
(object ::Indexwizard::spotlist0 method ::Spotlist::readSpotFile
body line 35)
invoked from within
readSpotFile [lindex $args 1]
while constructing object ::Indexwizard::spotlist0 in
::Spotlist::constructor (body line 23)
invoked from within
Spotlist \#auto file $l_image [$l_image makeAuxiliaryFileName spt
$::mosflm_directory]
(object ::.c.body.indexing method
::Indexwizard::processSpotfindingResults body line 55)
invoked from within
[.c component indexing] processSpotfindingResults $dom
(object ::Mosflm::m method ::Mosflm::processMessage body line 92)
invoked from within
::Mosflm::m processMessage
(in namespace inscope ::Mosflm script line 1)
invoked from within
namespace inscope ::Mosflm {::Mosflm::m processMessage}
[ccp4bb] Postdoc position for crystallographic studies of protein-DNA complexes
Postdoctoral position - University of Minnesota A postdoctoral position will be open in January 2010 to conduct crystallographic studies of protein-DNA complexes. Our laboratory studies structure and function of various DNA-processing enzymes primarily using x-ray crystallography. Current research projects include retroviral integrases, tyrosine family of site-specific recombinases, and Holliday-junction resolvases. Successful candidate should have research experience (and Ph.D.) in biochemistry, biophysics, or similar disciplines. Prior experience with x-ray crystallography or other structural biology techniques is preferred. The University of Minnesota at Twin Cities offers superb research environment with state of the art x-ray diffraction and robotic crystallization facilities. If interested, please send a CV and contact information for references to Hideki Aihara. Hideki Aihara (aihar...@umn.edu) http://www.cbs.umn.edu/BMBB/faculty/Aihara.H.shtml Department of Biochemistry, Molecular Biology and Biophysics University of Minnesota
Re: [ccp4bb] Format issue with TLSIN/TLSOUT files
Also, TLS parameters can be converted to conventional B-factors with both an isotropic and an anisotropic component which can then represented in conventional PDB format. PHENIX does this automatically, but I'm sure other software can perform the conversion as well. --Paul
