For those who may not have made it through all the CCP4bb postings in
October-December 2011 on archiving raw images, I have posted a summary at the
IUCR Diffraction Data Deposition Working Group forum page
http://forums.iucr.org/viewforum.php?f=21 in which I have attempted to list
the unique
Message-
From: James Holton [mailto:jmhol...@lbl.gov]
Sent: 27 October 2011 06:04
To: Nave, Colin (DLSLtd,RAL,DIA)
Cc: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
In the spirit of supporting a can do attitude, I have decided to try
and frame the binary images or no images
/
Diamond Light Source, Chilton, Didcot, Oxon, OX11 0DE, U.K.
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Matthew
BOWLER
Sent: 26 October 2011 10:03
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
The archiving of all raw data and subsequently making
On Wednesday, 26 October 2011, James Holton wrote:
Of course, if we are willing to relax the requirement of validation and
curation, this could be a whole lot easier. In fact, there is already
an image deposition infrastructure in place! It is called TARDIS:
http://tardis.edu.au/
-
De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de Colin
Nave
Enviado el: jueves, 27 de octubre de 2011 0:49
Para: CCP4BB@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images
Dear George, Martin
I don't understand the point that one is throwing away
Hi James,
1) thanks for sending and email in this thread longer than mine, I was
worried I had killed it... ;)
2) you say:
Of course, if we are willing to relax the requirement of validation and
curation, this could be a whole lot easier. In fact, there is already
an image deposition
Spanish National Research Council
www.csic.es
-Mensaje original-
De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de Colin
Nave
Enviado el: jueves, 27 de octubre de 2011 0:49
Para: CCP4BB@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images
Dear
===
(Below, the message John was referring to in his request for clarification)
--
Date: Wed, 26 Oct 2011 15:29:32 +0100
From: Gerard Bricogne g...@globalphasing.com
Subject: Re: [ccp4bb] IUCr committees, depositing images
To: CCP4BB@JISCMAIL.AC.UK
Dear John
National Research Council
www.csic.es
http://www.csic.es/ Sin título-1
De: David Waterman [mailto:dgwater...@gmail.com]
Enviado el: jueves, 27 de octubre de 2011 12:05
Para: Martin M. Ripoll
CC: CCP4BB@jiscmail.ac.uk
Asunto: Re: [ccp4bb] IUCr committees, depositing images
I agree with Colin
This discussion of image deposition and archival has certainly been
illuminating. While there have been clear directions on how the process would
actually work, I am becoming increasingly curious on why it should be done
(outside of threatening non publication, social acceptance, funding, etc).
@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images
Dear George, Martin
I don't understand the point that one is throwing away information by
storing in frames. If the frames have sufficiently fine intervals (given
by
some sampling theorem
On Thu, 2011-10-27 at 15:36 +0200, George M. Sheldrick wrote:
In non-continuous mode, the goniometer has
to accelerate at the start of a frame and decellerate at the end, then
wait for
the frame to be read.
Someone should be able to confirm this, but I was under impression that
at
A few responses regarding TARDIS:
While it's true that TARDIS.edu.au is just an index to files hosted elsewhere,
a new solution, MyTARDIS has been developed as a repository that holds
information on diffraction (and other kinds of) data. The idea being that
facilities, labs or institutions
Since when has the cost of any project been limited by the cost of
hardware? Someone has to /implement /this -- and make a career out of
it; thunderingly absent from this thread has been the chorus of
volunteers who will write the grant.
phx
On 25/10/2011 21:10, Herbert J. Bernstein wrote:
Hi James,
Just to pick up on your point about the Pilatus detectors. Yesterday
in 2 hours of giving a beamline a workout (admittedly with Thaumatin)
we acquired 400 + GB of data*. Now I appreciate that this is not
really routine operation, but it does raise an interesting point - if
you have
The archiving of all raw data and subsequently making it public is
something that the large facilities are currently debating whether to
do. Here at the ESRF we store user data for only 6 months (and I
believe that it is available longer on tape) and we already have trouble
with capacity. My
This raises an important point. The new continuous readout detectors such as the
Pilatus for beamlines or the Bruker Photon for in-house use enable the crystal
to
be rotated at constant velocity, eliminating the mechanical errors associated
with
'stop and go' data collection. Storing their data
Dear James,
Good analysis! You bring up important points.
On 10/24/11 23:56, James Holton wrote:
The Pilatus is fast, but or decades now we have had detectors that can read
out in ~1s. This means that you can collect a typical ~100 image dataset in a
few minutes (if flux is not limiting).
Dear Frank,
re 'who will write the grant?'.
This is not as easy as it sounds, would that it were!
There are two possible business plans:-
Option 1. Specifically for MX is the PDB as the first and foremost
candidate to seek such additional funds for full diffraction data
deposition for each
Dear John and colleagues,
There seem to be a set a centrifugal forces at play within this thread
that are distracting us from a sensible path of concrete action by throwing
decoys in every conceivable direction, e.g.
* Pilatus detectors spew out such a volume of data that we can't
Could you perhaps use the principle of capture storage that is used by
wild-life photographers with high-speed cameras?
The principle is that the movie is written to the same area of memory,
jumping back to the beginning when it is full (this part is not essential,
but it makes the principle
I just want to jump in to state that I am ALL FOR the notion of
depositing the images that go with the structure factors and the
refined structure.
Through the years, I have been interviewing folks about the strange
satellite diffraction they saw, but ignored,
used the mains that they could
Is anyone seriously questioning whether we should archive the images
used for published structures? That amount of space is trivial, could
be implemented just as another link in the PDB website, and would be
really helpful in some cases. One person could set it up in a day! You
could just make it
Cool - we've found our volunteer!!
On 26/10/2011 17:28, Jacob Keller wrote:
Is anyone seriously questioning whether we should archive the images
used for published structures? That amount of space is trivial, could
be implemented just as another link in the PDB website, and would be
really
Touche! But alas, I have no access to the PDB's server, so...
JPK
On Wed, Oct 26, 2011 at 11:54 AM, Frank von Delft
frank.vonde...@sgc.ox.ac.uk wrote:
Cool - we've found our volunteer!!
On 26/10/2011 17:28, Jacob Keller wrote:
Is anyone seriously questioning whether we should archive the
Dear Colleagues,
Gerard strikes a very useful note in pleading for a can-do
approach. Part of going from can-do to actually-done
is to make realistic estimates of the costs of doing and
then to adjust plans appropriately to do what can be afforded
now and to work towards doing as much of what
On Oct 26, 2011, at 9:59 AM, Patrick Shaw Stewart wrote:
The principle is that the movie is written to the same area of memory,
jumping back to the beginning when it is full (this part is not essential,
but it makes the principle clear). Then, when the photographer takes his
finger off
be as ubiquitous but her
argument is along the same lines.
Regards
Colin
-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Herbert
J. Bernstein
Sent: 26 October 2011 18:55
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
Dear
: miércoles, 26 de octubre de 2011 11:52
Para: CCP4BB@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images
This raises an important point. The new continuous readout detectors such as
the
Pilatus for beamlines or the Bruker Photon for in-house use enable the
crystal to
be rotated
. Bernstein
Sent: 26 October 2011 18:55
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
Dear Colleagues,
Gerard strikes a very useful note in pleading for a can-do
approach. Part of going from can-do to actually-done
is to make realistic estimates of the costs
-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Martin M.
Ripoll
Sent: 26 October 2011 22:50
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
Dear George, dear all,
I was just trying to summarize my point of view regarding this important
: Gerard Bricogne [mailto:g...@globalphasing.com]
Sent: 26 October 2011 23:23
To: Nave, Colin (DLSLtd,RAL,DIA)
Cc: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
Dear Colin,
Thank you for accepting the heavy burden of responsibility your
colleagues have thrown onto your
Dear James,
This is technically ingenious stuff.
Perhaps it could be applied to help the 'full archive challenge' ie containing
many data sets that will never lead to publication/ database deposition?
However for the latter,publication/deposition, subset you would surely not
'tamper' with the
On Oct 24, 2011, at 3:56 PM, James Holton wrote:
The PDB only gets about 8000 depositions per year
Just to put this into dollars. If each dataset is about 17 GB in size, then
that's about 14 TB of storage that needs to come online every year to store the
raw data for every structure. A two
To be fair to those concerned about cost, a more conservative estimate
from the NSF RDLM workshop last summer in Princeton is $1,000 to $3,000
per terabyte per year for long term storage allowing for overhead in
moderate-sized institutions such as the PDB. Larger entities, such
as Google are
James Holton wrote:
I go to a lot of methods meetings, and it pains me to see the most
brilliant minds in the field starved for interesting data sets. The
problem is that it is very easy to get people to send you data that is
so bad that it can't be solved by any software imaginable (I've got
On Tue, Oct 25, 2011 at 3:20 PM, Pete Meyer pame...@mcw.edu wrote:
This is probably an idea that has already been tried (or discarded as
unsuitable for reasons that don't occur to me at the moment) - but why not
start with good crystals (such as lysozyme) and deliberately make them
worse?
The Pilatus is fast, but or decades now we have had detectors that can
read out in ~1s. This means that you can collect a typical ~100 image
dataset in a few minutes (if flux is not limiting). Since there are
~150 beamlines currently operating around the world and they are open
about 200
John Helliwell points out to me that it might be useful to know what MX
crystallographic data researchers in different countries are already
expected to deposit or save. He notes that research funding agencies in
the UK expect researchers to preserve their raw experimental data for at
least 5
Of Tom Peat
Sent: 18 October 2011 23:29
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images
If we are talking schemes, here is another one that we use that might be
considered:
Date/person/project/barcode/well#/crystal#
At the Australian synchrotron, a directory is automatically
Has anyone raided the point that while archiving is good, it will only
be generally useful if the image HEADERS are informative and use a
comprehensible format - and the data base is documented...
Eleanor
On 10/19/2011 10:45 AM, Alun Ashton wrote:
‘Short’ bit:
Hi Eleanor,
So far I have managed to lurk on this one - keeping an eye on things
but not getting involved. However this has prompted me to respond!
Has anyone raided the point that while archiving is good, it will only be
generally useful if the image HEADERS are informative and use a
Dear Colleagues,
Following on from my posting to the CCP4bb of yesterday an IUCr Forum
has been set up for Public input on the diffraction data deposition
future. Thus this Forum will record an organised set of inputs for
future reference. Instructions on how to register at this Forum can be
found
Some crystals are hard to make, so storing all the data the best way to get
reproducibility. On the other hand, no one needs more images of lysozyme. So
using the same standard for every deposition doesn't sound right.
The discussion should be held on the basis of overall cost to the research
Perhaps this kind of discussion that should be continued on the IUCr
forums, once people have had a chance to register. However, to answer
these particular points:
On Tue, 2011-10-18 at 10:52 +, Chris Morris wrote:
Some crystals are hard to make, so storing all the data the best way to get
I could not agree less. There is constant development of the software for
refinement that allow to do things that were not
possible or were not necessary in the past such as intelligent refinement of
occupancies of mutually exclusive sites, entities and conformations.
I frequently remeasure
...@post.tau.ac.il]
Sent: Tuesday, October 18, 2011 1:40 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] IUCr committees, depositing images
I could not agree less. There is constant development of the software for
refinement that allow to do things that were not
possible or were not necessary
@JISCMAIL.AC.UK] on behalf of Felix
Frolow [mbfro...@post.tau.ac.il]
Sent: Tuesday, October 18, 2011 1:40 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] IUCr committees, depositing images
I could not agree less. There is constant development of the software
for refinement that allow to do
For the record, the amount of disk storage space per unit cost has doubled
every 14 months for the last 30 years. It's an exponential relationship:
www.mkomo.com/cost-per-gigabyte
So data generated at a very high rate today, will be trivial to store in the
near future. That's not to say it is
Dear Enrico,
Please don't get me wrong: what you are saying is not incorrect, but it
is only half the story.
On Tue, 2011-10-18 at 15:13 +0200, Enrico Stura wrote:
With improving techniques, we should always be making progress!
Yes, of course!
If we are trying to answer a biological question
: [ccp4bb] IUCr committees, depositing images
Dear All,
Since John has responded to the issues raised by Tom and several responders,
and he has just described the overall goal of IUCr DDD WG (Diffraction Data
Deposition Working Group), I will just second on this important issue
particularly
given that:
- storage is becoming cheaper exponentially,
- computer power is increasing exponentially,
I think there is no reason to not store all images used for solving a structure
- linked to the pdb entry and properly annotated with beam centre, lambda,
pixel order and all other necessary
Dear Enrico, Frank and colleagues,
I am glad to have suggested that everyone's views on this issue should
be aired out on this BB rather than sent off-list to an IUCr committee
member: this is much more interactive and thought-provoking.
There would seem to be clear biases in some of
On Tue, 2011-10-18 at 18:17 +0100, Gerard Bricogne wrote:
it would be of the greatest value to that
investigator to be able to double-check how reliable some features of
that
structure (especially its ligands) actually are.
Certainly, one could argue here that the current PDB policy that
18, 2011 12:01 pm
Subject: Re: [ccp4bb] IUCr committees, depositing images
One more consideration:
Since organization is not one of my greatest talents, I would be absolutely
delighted if a databank took over the burden of archiving my raw data for me.
Phoebe
What do you consider good ? r.m.s.d of 2.5 Å ? fatal for drug design.
On Oct 18, 2011, at 12:19 PM, Enrico Stura wrote:
Dear Peter,
How many crystallographers does it take to transform bad data into good
data?
None, you need a modeller. Only a modeller can give you a structure with
perfect
Original message
Date: Tue, 18 Oct 2011 18:17:14 +0100
From: CCP4 bulletin board CCP4BB@JISCMAIL.AC.UK (on behalf of Gerard
Bricogne g...@globalphasing.com)
Subject: Re: [ccp4bb] IUCr committees, depositing images
To: CCP4BB@JISCMAIL.AC.UK
Dear Enrico, Frank and colleagues
On the deposition of raw data:
I recommend to the committee that before it convenes again, every member
should go collect some data on a beamline with a Pilatus detector [feel
free to join us at Diamond]. Because by the probable time any
recommendations actually emerge, most beamlines will
One other question (for both key issues described): what exactly is the
problem the committees are aiming to address?
Because I can't help noticing that Tom's email did not spark an on-list
discussion; do people actually feel either are issues? Isn't the more
burning problem how best to
On the deposition of raw data:
Committees, wherever you are!
I guess that abstaining from storing the raw diffraction data in the form of
frames is not very wise
whatever its size is. I regret that for some PDB entries I do not have
diffraction data (needless to say that authors
do not
Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Frank
von Delft
Sent: Sunday, October 16, 2011 12:00 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] IUCr committees, depositing images
One other question (for both key issues described): what exactly is the
problem
Wasn't that already implemented in Phenix ?
Jürgen
On Oct 16, 2011, at 4:20 PM, Bernhard Rupp (Hofkristallrat a.D.) wrote:
REPROCESS_PDB
..
Jürgen Bosch
Johns Hopkins University
Bloomberg School of Public Health
Department of Biochemistry Molecular Biology
Johns Hopkins
On the technical feasibility of storage of original data:
Sure, running Pilatus for an olympic record, we will go home with several T of
data after 24 h (will we?).
Yes, we do already. I just checked the number of images from PILATUS 6M
we have collected so far this year : ca. 1.7 millions.
As in reprocess completely from images, I meant.
From: Bosch, Juergen [mailto:jubo...@jhsph.edu]
Sent: Sunday, October 16, 2011 1:31 PM
To: hofkristall...@gmail.com
Cc: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] IUCr committees, depositing images
Wasn't that already implemented
On Oct 16, 2011, at 4:42 PM, Guenter Fritz wrote:
In the end for the determination of the structure only a few datasets
will be necessary. This means maybe 0.5 Tb of compressed data to
deposit. I don't think this is too much.
I think those 0.5 TB will be not essential, the more interesting
Ø Do you mean to reprocess determination of I and sig(I) from the
diffraction images automatically??? Or just to get an access to the raw
data?
Reprocessing the images with the to-be-developed new software that will
process the information in the data in full, and then using the
On Oct 16, 2011, at 4:18 PM, Bosch, Juergen wrote:
The rest is comparable to a collection of stamps, although with the benefit
as BR mentioned of adding an additional hurdle/layer to falsifying structures.
Not if you are interested in scattering that falls between reciprocal lattice
maxima,
Ø Not if you are interested in scattering that falls between reciprocal
lattice maxima, or if you want to preserve the possibility of applying
future data reduction packages.
Yep, this is exactly what I expressed in my original statement:
diffuse solvent contributions, commensurate and
I'm simultaneously embarrassed to have not read the thread to completion before
replying, and also totally pumped that I would answer a question the same way
as Bernhard Rupp!
On Oct 16, 2011, at 7:34 PM, Bernhard Rupp (Hofkristallrat a.D.) wrote:
Ø Not if you are interested in scattering
On 17/10/2011 01:52, Wladek Minor wrote:
Frank,
This is serious problem for biologists. There is a structure with
ligand. The same data were re-interpreted and people did not find the
ligand. This re-interpretation is not really valid until we will look
into diffraction data. Biologist
We overestimate the value of individual structures because we're human :)
If a problem is important enough that one structure makes or breaks the
case, a sensible thing to do would be to get more structures and strive to
obtain some other flavor of pertinent information by methods that are
71 matches
Mail list logo
