Dear all --
I cannot remember exactly, but I thought we had a long discussion on
the rightness of using compressed images, especially when considering
the loss of information while doing so. What was the conclusion of
the debate again? (sorry, too lazy to dig in the archives).
-- Leo --
On 18 Sep 2009, at 23:50, Graeme Winter wrote:
Hi David,
If the data compression is carefully chosen you are right: lossless
jpeg2000 compression on diffraction images works very well, but is a
spot slow. The CBF compression using the byte offset method is a
little less good at compression put massively faster... as you point
out, this is the one used in the pilatus images. I recall that the
.pck format used for the MAR image plates had the same property - it
was quicker to read in a compressed image that the raw equivalent.
So... once everyone is using the CBF standard for their images, with
native lossless compression, it'll save a fair amount in disk space
(=£/$), make life easier for people and - perhaps most importantly -
save a lot of data transfer time.
Now the funny thing with this is that if we compress the images before
we store them, the compression implemented in the file system will be
less effective... oh well, can't win em all...
Cheers,
Graeme
2009/9/18 Waterman, David (DLSLtd,RAL,DIA)
<[email protected]>:
Just to comment on this, my friend in the computer game industry
insists
that compression begets speed in almost all data handling situations.
This will be worth bearing in mind as we start to have more fine-
sliced
Pilatus 6M (or similar) datasets to deal with.
Cheers,
David.
-----Original Message-----
From: CCP4 bulletin board [mailto:[email protected]] On Behalf Of
William G. Scott
Sent: 17 September 2009 22:48
To: [email protected]
Subject: [ccp4bb] I compressed my images by ~ a factor of two, and
they
load and process in mosflm faster
If you have OS X 10.6, this will impress your friends and save you
some
disk space:
% du -h -d 1 mydata
3.5G mydata
mv mydata mydata.1
sudo ditto --hfsCompression mydata.1 mydata rm -rf mydata.1
% du -h -d 1 mydata
1.8G mydata
This does hfs filesystem compression, so the images are still
recognized
by mosflm, et al. I think they process a bit faster too, because
half
the information is packed into the resource fork.
This e-mail and any attachments may contain confidential,
copyright and or privileged material, and are for the use of the
intended addressee only. If you are not the intended addressee or
an authorised recipient of the addressee please notify us of
receipt by returning the e-mail and do not use, copy, retain,
distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the
individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any
attachments are free from viruses and we cannot accept liability
for any damage which you may sustain as a result of software
viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in
England and Wales with its registered office at Diamond House,
Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11
0DE, United Kingdom
------------------------------------------------------------
Chavas Leonard, Ph.D.
Assistant Professor
------------------------------------------------------------
Structural Biology Research Center
Photon Factory
High Energy Research Organization (KEK)
305-0801 Tsukuba Oho 1-1
Japan
------------------------------------------------------------
Tel: +81(0)29-864-5642 (4901)
Fax: +81(0)29-864-2801
e-mail: [email protected]
------------------------------------------------------------
Science Advisory Board (BIT Life Sciences)
Editorial Board (JAA)
------------------------------------------------------------
http://pfweis.kek.jp/~leo
