Re: [ccp4bb] raw data deposition

[Petr Kolenko]

Dear colleagues,

my opinion is that we should develop methods or approaches to validate
!processing! of raw data. If this is possible. We have many validation
tools for structure refinement, but no tool to validate data
processing. In case we have this tools, there is no need to deposit
diffraction images (2-5GB instead of 10 MB). I think.
Of course, how to validate this? This might be topic for a new
discussion. I am sure, that in the very beginning of crystallography,
there were no tools to validate the structures as well. I am also sure
that some opinions may arise today. (Online server, where one can
upload the images with log files from processing?)
We should concentrate more on quality of our outcome, than on storage
of these data.

Petr


2011/10/28 Katherine Sippel :
> Generally during these rigorous bb debates I prefer to stay silent and
> absorb all the information possible so that I can make an informed decision
> later on. I fear that I am compelled to contribute in this instance. In
> regards to the "does this make a difference in the biological interpretation
> stage" issue, I can state that it does. In my comparatively miniscule career
> I have run into this issue three times. The first two address Adrian's
> point...
>
>> And (b) even if they do, is this continual improvement even worthwhile?  I
>>  am always depressed at how little a model changes from an initial build to
>> the final one, even when the rfree drops from 35 to 23. All that work! - and
>> my biological interpretation would have been almost the same at the
>> beginning as at the end.
>
>
> In one instance I adopted an orphaned structure and ran it through a
> slightly more advanced refinement protocol (on the same structure factors)
> and ended up with a completely different story than the one I started with
> [1]. Another researcher in my grad lab identified mis-oriented catalytic
> residues in an existing structure from EDS server maps which affects the
> biochemistry of the catalytic mechanism [2].
>
> In another case I decided that I would reprocess some images that I had
> originally indexed and scaled in my "Ooo buttons clicky clicky" stage of
> learning crystallography and the improved structure factors revealed a
> alternate conformations for both a critical loop and ligand orientation [3].
>
> And this was all in the last 4 years. So I would posit that the answer is
> yes there are significant biological insights to be had with the capacity to
> reassess data in any form.
>
> Katherine
>
> [1] J Phys Chem Lett. 2010 Oct 7;1(19):2898-2902
> [2] Acta Crystallogr D Biol Crystallogr. 2009 Mar;65(Pt 3):294-6.
> [3] Manuscript in progress
>
> 
> Katherine Sippel, PhD
> Postdoctoral Associate
> Baylor College of Medicine
>



-- 
Petr Kolenko
kole...@imc.cas.cz
http://kolda.webz.cz

Re: [ccp4bb] raw data deposition

[Katherine Sippel]

Generally during these rigorous bb debates I prefer to stay silent and
absorb all the information possible so that I can make an informed decision
later on. I fear that I am compelled to contribute in this instance. In
regards to the "does this make a difference in the biological interpretation
stage" issue, I can state that it does. In my comparatively miniscule career
I have run into this issue three times. The first two address Adrian's
point...

And (b) even if they do, is this continual improvement even worthwhile?  I
>  am always depressed at how little a model changes from an initial build to
> the final one, even when the rfree drops from 35 to 23. All that work! - and
> my biological interpretation would have been almost the same at the
> beginning as at the end.
>

In one instance I adopted an orphaned structure and ran it through a
slightly more advanced refinement protocol (on the same structure factors)
and ended up with a completely different story than the one I started with
[1]. Another researcher in my grad lab identified mis-oriented catalytic
residues in an existing structure from EDS server maps which affects the
biochemistry of the catalytic mechanism [2].

In another case I decided that I would reprocess some images that I had
originally indexed and scaled in my "Ooo buttons clicky clicky" stage of
learning crystallography and the improved structure factors revealed a
alternate conformations for both a critical loop and ligand orientation [3].

And this was all in the last 4 years. So I would posit that the answer is
yes there are significant biological insights to be had with the capacity to
reassess data in any form.

Katherine

[1] J Phys Chem Lett. 2010 Oct 7;1(19):2898-2902
[2] Acta Crystallogr D Biol Crystallogr. 2009 Mar;65(Pt 3):294-6.
[3] Manuscript in progress


Katherine Sippel, PhD
Postdoctoral Associate
Baylor College of Medicine

Re: [ccp4bb] raw data deposition

[James Stroud]

On Oct 27, 2011, at 5:22 PM, Francis E Reyes wrote:
> So I ask again, are there literature examples where reevaluation of the 
> crystallographic data has directly resulted in new biological insights into 
> the system being modeled?

This is a poor criterion on which to base any conclusions or decisions. We can 
blame the lack of examples on unavailability of the data.

Right now, I'd love to get my hands on the raw images for a particular cryoEM 
data set, but they are not available--only the maps. But the maps assume one 
symmetry and I have a hypothesis that the true symmetry is different. I could 
test my hypothesis by reprocessing the data were it available.

James

Re: [ccp4bb] Weird blob appears

[Artem Evdokimov]

Weird. It's pretty much exactly what a nonbranched sugar (triose, in this
case) would look like.

were the dimensions consistent with three sugars? were there any collisions
with backbone or side chains inside/near the mystery density?
It really looks like maltotriose... is the protein a sugar binder by any
chance?

Artem
On Thu, Oct 27, 2011 at 8:46 PM, Jacob Keller <
j-kell...@fsm.northwestern.edu> wrote:

> Blob is gone--something funny happened, I guess. I went back to using
> the original mtz from scala, removed and replaced a bunch of waters,
> and no more worm! I can't really figure it out, and wish I knew
> exactly what happened, but I think I am just going to non-chalantly
> move along.
>
> Jacob
>
>
> On Thu, Oct 27, 2011 at 8:41 PM, Yuri Pompeu  wrote:
> > Jacob,
> > By simply looking at the figures you show, it does look like you have
> some type of long, maybe polymeric, molecule bound.
> > With that being said:
> > 1- It is in the symmetry axis so maybe be a little noisy there
> > 2-If you are in doubt about it being real or not check the density and
> how it fits into your protein and (symm. related neighbours of course). If
> the desity appears to be in position for some real interactions- i.e, there
> are some peaks at H-bond distances of polar groups etc...- it may be real.
> If its all random and through your protein atoms, probably not
> > Keep in mind alternate conformers
> > HTH
> >
> >
>
>
>
> --
> ***
> Jacob Pearson Keller
> Northwestern University
> Medical Scientist Training Program
> email: j-kell...@northwestern.edu
> ***
>

Re: [ccp4bb] IUCr committees, depositing images

[Ashley Buckle]

A few responses regarding TARDIS:

While it's true that TARDIS.edu.au is just an index to files hosted elsewhere, 
a new solution, MyTARDIS has been developed as a repository that holds 
information on diffraction (and other kinds of) data. The idea being that 
facilities, labs or institutions take responsibility by hosting their own 
instance of MyTARDIS on their own storage with the eventual view to have 
entries released publicly and indexed by TARDIS.edu.au.

An example of this is the MyTARDIS experiment released here:

http://mytardis.its.monash.edu.au/experiment/view/636/

Also accessible in the tardis.edu.au entry here:

http://hdl.handle.net/102.100.100/6984

Both point to the exact same file location.

We also have a mechanism in which MyTARDIS has been placed at the Australian 
Synchrotron and linked to beamlines there to automatically store diffraction 
data locally. This data is then routed to MyTARDIS at a user's local 
university, where applicable. From here the workflow to publication takes 
similar form as that shown above.

It's open source licensed and deployable today:
http://code.google.com/p/mytardis/

With this as the installation documentation:
http://mytardis.readthedocs.org/en/latest/install.html

And here is how new instrument data can reach MyTARDIS:
http://packages.python.org/MyTARDIS/ingesting.html#mets
http://packages.python.org/MyTARDIS/ref/mets-format.html

This function has seen deployments at the Australian neutron facility ANSTO and 
in Microscopy/Microanalysis labs, too.

In response to James Holton's suggestion of getting PDB to accept TARDIS id's, 
I spoke with Kim Henrick  4 years ago if it would be possible simply to put a 
TARDIS id/url/handle in a REMARK, so the coordinates could be associated with 
the raw data that produced them.  Kim was very positive that this should 
happen, and told me that it would be discussed in a wwPDB advisory board 
meeting, but I never heard anything back. In the meantime we have managed to 
get TARDIS URL's in several papers, usually in supplementary information.  The 
editors/publishers didn't really object but didnt really take much notice I 
think either!  

TARDIS development is an active area of my group and now that we are 
contributing to the IUCr efforts (thanks to John Helliwell), I'm optimistic 
that things are moving forward.  We discussed some of the points raised here in 
the Acta Cryst TARDIS paper a couple of years ago 
(http://scripts.iucr.org/cgi-bin/paper?S0907444908015540) particularly the 
usefulness of raw data availability for methods development, so I'm really 
happy to see continued active discussion in this area!

Cheers
Ashley Buckle 
Steve Androulakis

On 27/10/2011, at 6:25 PM, Ethan Merritt wrote:

> On Wednesday, 26 October 2011, James Holton wrote:
>> Of course, if we are willing to relax the requirement of validation and 
>> curation, this could be a whole lot easier.  In fact, there is already 
>> an image deposition infrastructure in place!  It is called TARDIS:
>> 
>> http://tardis.edu.au/
>> 
>> Perhaps the best way forward would be for "the PDB" to introduce a new 
>> field for one or more TARDIS ids in a PDB deposition?  It would be 
>> optional at the first, but no doubt required in the future.
> 
> As I understand it, TARDIS is just an indexing system.
> You're still on your own to actually store the images.
> The TARDIS setup will cough up the information that your images
> are stored on a machine named pony.lbl.gov, or at least they were 
> at the time you registered them for indexing, where they supposedly
> can be retrieved using tag #XYZ.
> 
> But so far as I know you would still be at the mercy of pony.lbl.gov
> going up in flames, or being renamed twinkle.lbl.gov, or being 
> decommissioned when the next budget crunch hits.
> 
> For that matter, I don't know what the provision or expectation
> is that anyone outside your institution could see or access the
> machine holding the set of files that TARDIS told them were there.
> 
> If I've got this wrong, perhaps Ashley Buckle can chime in with
> an update on TARDIS.
> 
>   Ethan


Associate Professor Ashley M Buckle
NHMRC Senior Research Fellow
The Department of Biochemistry and Molecular Biology,
Faculty of Medicine
Building 77
Monash University, Clayton, Vic 3800
Australia

http://www.med.monash.edu.au/biochem/staff/abuckle.html
iChat: ashley.buc...@me.com
skype: ashley.buckle
Tel: (613) 9902 9313 (office)
Fax : (613) 9902 9500
mobile: 0430 913031

[ccp4bb] refmac 5.6 ccp4 6.2.0

[Kenneth A. Satyshur]

Has anyone had problems with Refmac 5.6? I tried refining our stucture at 1.24 
A,
aniso with H in riding position and it just exploded! I get error in distances 
such as

    Standard  External   All
    Bonds:  3270 0  3270
   Angles:  4923 0  4923
  Chirals:   214 0   214
   Planes:   368 0   368
 Torsions:   957 0   957
---

 Number of reflections in file  90428
 Number of reflections read  90428


 CGMAT cycle number =  1

     Bond distance outliers 


Bond distance deviations from the ideal >10.000Sigma will be monitored

A  5 PRO C   A - A  5 PRO HB1 A mod.= 3.344 id.= 1.329 dev= -2.015 
sig.= 0.014
A  5 PRO C   B - A  5 PRO HB1 B mod.= 2.997 id.= 1.329 dev= -1.668 
sig.= 0.014
A  5 PRO HB1 A - A  5 PRO HG1 A mod.= 2.292 id.= 1.458 dev= -0.834 
sig.= 0.021
A  5 PRO HB1 A - A  6 LEU HD23A mod.= 7.407 id.= 0.860 dev= -6.547 
sig.= 0.020
A  5 PRO HB1 B - A  5 PRO HG1 B mod.= 2.247 id.= 1.458 dev= -0.789 
sig.= 0.021
A  5 PRO HB1 B - A  6 LEU HD23B mod.= 6.529 id.= 0.860 dev= -5.669 
sig.= 0.020
A  5 PRO HG1 A - A  5 PRO HD1 A mod.= 2.267 id.= 0.980 dev= -1.287 
sig.= 0.020
A  5 PRO HG1 A - A  6 LEU N   A mod.= 4.860 id.= 1.530 dev= -3.330 
sig.= 0.020
A  5 PRO HG1 A - A  6 LEU HD21A mod.= 6.129 id.= 1.525 dev= -4.604 
sig.= 0.021
A  5 PRO HG1 B - A  5 PRO HD1 B mod.= 2.236 id.= 0.980 dev= -1.256 
sig.= 0.020
A  5 PRO HG1 B - A  6 LEU N   B mod.= 4.922 id.= 1.530 dev= -3.392 
sig.= 0.020
A  5 PRO HG1 B - A  6 LEU HD21B mod.= 6.664 id.= 1.525 dev= -5.139 
sig.= 0.021
A  6 LEU N   A - A  6 LEU CA  A mod.= 1.467 id.= 0.970 dev= -0.497 
sig.= 0.020
A  6 LEU N   A - A  6 LEU HA  A mod.= 2.005 id.= 0.970 dev= -1.035 
sig.= 0.020
A  6 LEU N   A - A  6 LEU CB  A mod.= 2.497 id.= 1.530 dev= -0.967 
sig.= 0.020
A  6 LEU N   B - A  6 LEU CA  B mod.= 1.469 id.= 0.970 dev= -0.499 
sig.= 0.020
A  6 LEU N   B - A  6 LEU HA  B mod.= 2.032 id.= 0.970 dev= -1.062 
sig.= 0.020
A  6 LEU N   B - A  6 LEU CB  B mod.= 2.446 id.= 1.530 dev= -0.916 
sig.= 0.020
A  6 LEU CB  A - A  6 LEU HB2 A mod.= 0.969 id.= 1.521 dev=  0.552 
sig.= 0.020
A

Rfree goes form 17 to 28 and R from 15 to 25.
Coot map looks like a bunch of busted insect parts.


I use the exact same input using ccp4 6.1.13 and Refmac 5.5 and all is good. I 
am forced to use the
old ccp4 and refmac to publish. Rf 17 R 15. 
thanks

--
Kenneth A. Satyshur, M.S.,Ph.D.
Associate Scientist
University of Wisconsin
Madison, Wisconsin 53706
608-215-5207
<>

Re: [ccp4bb] Weird blob appears

[Jacob Keller]

Blob is gone--something funny happened, I guess. I went back to using
the original mtz from scala, removed and replaced a bunch of waters,
and no more worm! I can't really figure it out, and wish I knew
exactly what happened, but I think I am just going to non-chalantly
move along.

Jacob


On Thu, Oct 27, 2011 at 8:41 PM, Yuri Pompeu  wrote:
> Jacob,
> By simply looking at the figures you show, it does look like you have some 
> type of long, maybe polymeric, molecule bound.
> With that being said:
> 1- It is in the symmetry axis so maybe be a little noisy there
> 2-If you are in doubt about it being real or not check the density and how it 
> fits into your protein and (symm. related neighbours of course). If the 
> desity appears to be in position for some real interactions- i.e, there are 
> some peaks at H-bond distances of polar groups etc...- it may be real. If its 
> all random and through your protein atoms, probably not
> Keep in mind alternate conformers
> HTH
>
>



-- 
***
Jacob Pearson Keller
Northwestern University
Medical Scientist Training Program
email: j-kell...@northwestern.edu
***

Re: [ccp4bb] Weird blob appears

[Yuri Pompeu]

Jacob,
By simply looking at the figures you show, it does look like you have some type 
of long, maybe polymeric, molecule bound.
With that being said:
1- It is in the symmetry axis so maybe be a little noisy there
2-If you are in doubt about it being real or not check the density and how it 
fits into your protein and (symm. related neighbours of course). If the desity 
appears to be in position for some real interactions- i.e, there are some peaks 
at H-bond distances of polar groups etc...- it may be real. If its all random 
and through your protein atoms, probably not
Keep in mind alternate conformers
HTH

Re: [ccp4bb] raw data deposition

[Francis E Reyes]

Thanks for bringing this up front Ed. Specifically bringing your second point 
to the forefront. Do we need to do it? Or to rephrase it more directly .. WHY 
do we need to do it? 

Answering why we need to do it will really help with compliance. Lest we not 
forget we are asking the general crystallography community (which encompasses a 
large variety of interests in competition with the interest to archive the 
actual images) to go an additional step and provide detailed metadata (among 
other things). Of course you could force the community into compliance but I'm 
pretty sure we can motivate behavior without threats. 

So I ask again, are there literature examples where reevaluation of the 
crystallographic data has directly resulted in new biological insights into the 
system being modeled?



On Oct 27, 2011, at 3:27 PM, Ed Pozharski wrote:

> 1.  How to do it.
> 
> That is what the other thread is dealing with and my overall feeling is
> that difficulties have been largely exaggerated early on.  You are right
> that concrete steps can be taken.
> 
> 2.  Do we need to do it.
> 
> To me, it's no-brainer, but some responses seem to suggest not everyone
> is really on board.  Again, I am sure this has to be done, but consensus
> in this area is equally important.  

-
Francis E. Reyes M.Sc.
215 UCB
University of Colorado at Boulder

Re: [ccp4bb] raw data deposition

[Gerard Bricogne]

Dear Adrian,

 I too follow Voltaire, and your point of view nicely illustrates the
diversity of outlook and priorities between practitioners of our arcane art.

 I can only say that I have seen many cases where structural detail only
obtainable through hard work in phasing and/or refinement has produced
conclusions that had a significant impact on the ambient biological story,
and that this hard work would not have been possible if no one had cared
about continuing to develop ever better methods and software.


 With best wishes,
 
  Gerard.

--
On Fri, Oct 28, 2011 at 01:40:11AM +0300, Adrian Goldman wrote:
> Ok. This is my last post before I go to bed. Look at the opportunity cost of 
> this discussion alone - bright minds who should be solving structures or 
> developing algorithms - anything! Debating this. 
> 
> However - as someone else remarked will (a) anyone care about > 90% of the 
> structures in 50 years?  
> 
> And (b) even if they do, is this continual improvement even worthwhile?  I  
> am always depressed at how little a model changes from an initial build to 
> the final one, even when the rfree drops from 35 to 23. All that work! - and 
> my biological interpretation would have been almost the same at the beginning 
> as at the end. 
> 
> The structures aren't important in themselves. It's the story they tell. So 
> to me this is an effort to fix what ain't broke. 
> 
> Adrian
> 
> Sent from my iPhone
> 
> On 28 Oct 2011, at 01:15, Michel Fodje  wrote:
> 
> > Every dataset costs money to produce. Is it more cost effective to expect 
> > that those wishing to use the data repeat the expenditures by repeating the 
> > experiments?  To exaggerate the point, imagine a world without published 
> > research articles, would it be more expensive to do science or less? We 
> > should not simply dismiss an idea just because we think today that "640K is 
> > more memory than anyone will ever need"
> > 
> > 
> >> On Oct 27, 2011, at 3:47 PM, Adrian Goldman wrote:
> >> 2) I agree with Susan.  In a time of limited funding, is this the most 
> >> important use of money?
> > 

-- 

 ===
 * *
 * Gerard Bricogne g...@globalphasing.com  *
 * *
 * Global Phasing Ltd. *
 * Sheraton House, Castle Park Tel: +44-(0)1223-353033 *
 * Cambridge CB3 0AX, UK   Fax: +44-(0)1223-366889 *
 * *
 ===

Re: [ccp4bb] raw data deposition

[Adrian Goldman]

Ok. This is my last post before I go to bed. Look at the opportunity cost of 
this discussion alone - bright minds who should be solving structures or 
developing algorithms - anything! Debating this. 

However - as someone else remarked will (a) anyone care about > 90% of the 
structures in 50 years?  

And (b) even if they do, is this continual improvement even worthwhile?  I  am 
always depressed at how little a model changes from an initial build to the 
final one, even when the rfree drops from 35 to 23. All that work! - and my 
biological interpretation would have been almost the same at the beginning as 
at the end. 

The structures aren't important in themselves. It's the story they tell. So to 
me this is an effort to fix what ain't broke. 

Adrian

Sent from my iPhone

On 28 Oct 2011, at 01:15, Michel Fodje  wrote:

> Every dataset costs money to produce. Is it more cost effective to expect 
> that those wishing to use the data repeat the expenditures by repeating the 
> experiments?  To exaggerate the point, imagine a world without published 
> research articles, would it be more expensive to do science or less? We 
> should not simply dismiss an idea just because we think today that "640K is 
> more memory than anyone will ever need"
> 
> 
>> On Oct 27, 2011, at 3:47 PM, Adrian Goldman wrote:
>> 2) I agree with Susan.  In a time of limited funding, is this the most 
>> important use of money?
> 

Re: [ccp4bb] raw data deposition

[Gerard Bricogne]

Dear Nat,

 You are making an excellent point, that I would like to supplement with
another drawn from an intermediate stage between making compulsory the
deposition of coordinates (to which you are referring) and the discussion we
are having right now about moving towards the deposition of diffraction
images - namely, the deposition of "structure factor" data.

 At first that idea seemed to many to be just as far-fetched as the
current one is seen by many. I can remember an impassioned e-mail to this BB
by Gerard Kleywegt with subject line "SOS: save our structure factors!",
pleading the case for that deposition to be made cmpulsory so as to be make 
it possible to have as objective a picture as possible of the quality of the
electron density on which the model was based; and he went on to produce the
Electron Density Server, the usefulness of which few would now dispute.
There are probably few instances in which the EDS could be proven to have
led to "significant new biological insights", but it is undeniable that it
must have provided very useful means of checking deposited structures to see
whether there might be questionable bits in crucial regions, whereas
previously one would have had to believe indiscriminately everything that
was modelled.

 This structure factor deposition also led to the possibility of
large-scale testing of new developments in refinement algorithms which
played a huge role in helping improvements in those to be throroughly
evaluated, and the programs to be made robust. This led in turn to being
able to see more detail or more corrections in old pdb entries via the EDS,
culminating in such initiatives as PDB-redo that, if not revolutionising the
biological information content of the pdb, has certainly helped make its
contents much more assessable. Through the effect on the improvement of
refinement programs, it can be said that the greatest beneficiaries of the
deposition of structure factors yesterday are not so much the people who
deposited the associated structures at the time, but everyone who refines
structures today and will do so tomorrow with the much improved programs it
has helped produce.

 We are simply today at the logical next step, i.e. depositing the
images that the structure factors came from. For many reasons that have been
described by many people, images often contain much more information about
the reliability (or otherwise) of the structure factors derived from them (I
have repeatedly mentioned the corruption by reflexions from parasitic
lattices). Such images will not only provide the foodstuff for new
developments aimed at dealing better with the problem: once those
developments have taken place, more reliable data will be obtainable from
them, that may frequently clean up dubious features of the previous maps or
bring into question certain parts of the previous models. I think that
Adrian's rather dismissive comment that developers can get the job done from
a few scraps of bad images gleaned from colleagues in distress is simply a
sign of a lack of experience in developing software. 

 We should not, therefore, be too blinkered and ask only "What will it
do for my structure if I deposit my images", but instead ask "What will
depositing my images do to improve the processing and refinement programs of
tomorrow" (I am not trying to sound like JFK here ...). The answer is: an
awful lot! These improvements will then help everybody, including the
sceptical depositor in question in his or her next tough project; but as
usual they will be taken for granted by those who thought that depositing
images was a waste of time ... .

 I hope this elicits more comments from doubters and detractors: their
voices and arguments should certainly be heard.


 With best wishes,
 
  Gerard.

--
On Thu, Oct 27, 2011 at 02:11:28PM -0700, Nat Echols wrote:
> On Thu, Oct 27, 2011 at 1:47 PM, Adrian Goldman
> wrote:
> 
> > 1) this is not a matter of science, but science (internal) policy, and so
> > the majority actually SHOULD count.
> >
> 
> It's worth keeping in mind that there was once strong opposition to the
> current rules on PDB deposition - the best example I could find is here:
> 
> http://www.nature.com/nsmb/journal/v5/n6/pdf/nsb0698-407.pdf
> 
> Notably, nearly a third of scientists polled thought they should be allowed
> to publish without releasing coordinates.  If this had been a majority,
> should the journal editors have meekly submitted and allowed the old policy
> of 1-year holds to continue?  Admittedly, the issue of archiving raw images
> is not the same, since they are of much less use to the community, but it's
> a good example of why some opinions should be ignored.
> 
> -Nat

-- 

 ===
 * *
 * Gerard Bricogne g...@globalphasing.com  *
 *

Re: [ccp4bb] raw data deposition

[Michel Fodje]

Every dataset costs money to produce. Is it more cost effective to expect that 
those wishing to use the data repeat the expenditures by repeating the 
experiments?  To exaggerate the point, imagine a world without published 
research articles, would it be more expensive to do science or less? We should 
not simply dismiss an idea just because we think today that "640K is more 
memory than anyone will ever need"


> On Oct 27, 2011, at 3:47 PM, Adrian Goldman wrote:
> 2) I agree with Susan.  In a time of limited funding, is this the most 
> important use of money?

Re: [ccp4bb] raw data deposition

[Ed Pozharski]

Dear Adrian,

thank you - this is most helpful in assessing why we do or don't need to
deposit the raw data.

However:

> And let me say that, as this bb hardly reaches ALL practicing MM
> crystallographers, but only those with an interest in techniques, the
> results AND discussion are heavily skewed in favor of storage.  

Fair enough.  I am afraid we need another survey to settle this (oh,
no!).  But given the variety of questions that are asked on this bb it
is my expectation that the subset is quite representative.  I also
thought that "protein crystallographer" implies interest in techniques.
If those with interest in techniques are nowadays a minority... well,
let's just say it explains a lot.

> So - looking at my own navel - why would one, did I, not write until
> now?  There is in the bb a loud active (and my guess) minority whose
> opinions are already formed, so responding seems pointless.  It won't
> change anything and will just lead to opprobrium pouring down on my
> head.  That's one reason.

I can't, of course, pretend to be the spokesperson of the loud minority
(but it's surely true that I am occasionally loud and obnoxious).  To
summarize my personal feelings about the issue you raise, let me quote
Voltaire: 
"I do not agree with what you have to say, but I'll defend to the death
your right to say it."

> But let me say - and I voted 'no' as should be blindingly obvious -
> two more things.
> 1) this is not a matter of science, but science (internal) policy, and
> so the majority actually SHOULD count.

Agreed with the caveat that majority could be wrong.

> 2) I agree with Susan.  In a time of limited funding, is this the most
> important use of money?

This is an important point, but I suspect that a) most of the task can
be accomplished within existing framework (thus no extra personnel
costs) and b) the extra storage is really, really cheap these days -
even if I store all the data we collect, it is probably still less than
dewar shipping costs.

> But enough of the nattering nabobs of negativism! As such frame
> information is so valuable for future development efforts, I think all
> it would require would be an email to a local crystallographer working
> on an impossible problem, and I am sure it would be forthcoming.  For
> s/w development purposes, I can't believe that even a small fraction
> of the terabytes of frame data off the pilots is needed...

IMHO, this is not about developers getting data to work with (I am sure
they already have plenty).  It's about extending the retro-processing
concept pioneered by PDB-REDO to the integration/scaling.  And yes, it
is about preventing wishful overinterpretation.  While this is not about
raw data processing, will correcting  someone fitting an alpha helix
with multiple di-EG molecules change the course of history?  Of course
not.  It won't even change the main finding of that paper.  But I always
believed in getting the best structure possible under the circumstances
(and, of course, failed to live up to that standard).

But I digress, as usual.  Once again - thank you, I think it's very
important that these issues are discussed.  If the raw data deposition
is made mandatory (which I support), I'd like you to at least see my
reasoning, if not bring you over to the dark side.

Cheers,

Ed.  


-- 
"I'd jump in myself, if I weren't so good at whistling."
   Julian, King of Lemurs

Re: [ccp4bb] raw data deposition

[Jacob Keller]

Since this hasn't been brought up--there is the consideration that in
10 or more years maybe x-ray crystallography will be completely a
thing of the past, with some kind of massively-superior modality
taking over. Of course there is no way to bank on this, but I am
wondering whether this is something to consider or not. Do we really
think people will still be crystallizing proteins in 50 years, or far
less, looking up structures determined in 2011? Has anybody recently
used the original myoglobin structure?

JPK


On Thu, Oct 27, 2011 at 4:55 PM, Michel Fodje
 wrote:
> We store raw data for two main reasons:
> a)  We currently use only a fraction of the information actually contained in 
> raw images and extraction of that fraction can be improved. Destroying the 
> data means
> - we lose the extra information, and make future research in some areas 
> either impossible or more costly
> - we make it more difficult to improve current data reduction methods
> b)  Raw data is the best way to independently validate a published structure 
> and prevent fraud.
>
> The majority of crystallographers already recognize these truths. That is why 
> almost all of them do keep backups of their data even after structures have 
> been published.
>
> To those still against making data public I would ask a simple question:  
> Would you object to providing the raw data from a published structure if such 
> data were available and you did not have to bear an unreasonable 
> inconvenience in the process? My guess is that most crystallographers are 
> reasonable scientists and such a "Poll" will probably result in ~100% "Yes" 
> and ~0% "No". I'm I wrong?
>
> The real issue then is how do we make the data available in such a way that 
> the inconvenience (if any) to all the stake-holders is reasonable.  Some 
> great ideas have already been advanced.
>
> In the short-term,  we could start by using the fact that synchrotron 
> facilities already store raw data for a period. However, a lot of data is 
> collected which is not published. Given the limited disk space, it may be 
> useful to know exactly which datasets result in a publication and should be 
> kept for an extended period. If a unique ID (such as the DOI suggestion) is 
> provided to every dataset and required during deposition/publication, then 
> synchrotron facilities can preserve only those datasets which have been 
> published after a given "grace" period. Combined with a central Meta-data 
> server similar to TARDIS, such a system could be developed in a relatively 
> short period of time, while longer term central storage ideas are worked out.
>
> Again the best solution is going to be one which requires the least amount of 
> effort from crystallographers. In fact, I can see a system in which the 
> experiment metadata for a PDB entry/dataset comes directly from the 
> synchrotron facility during deposition so that users simply provide a unique 
> dataset ID and the experimental details are pre-filled for them.
>
> Of course the above completely ignores home sources.
>
>
> /Michel
>> -Original Message-
>> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of D
>> Bonsor
>> Sent: October-27-11 3:10 PM
>> To: CCP4BB@JISCMAIL.AC.UK
>> Subject: Re: [ccp4bb] raw data deposition
>>
>> Why should we store images?
>>
>> From most of the posts it seems to aid in software development. If that is
>> the case, there should be a Failed Protein Databank (FPDB) where people
>> could upload datasets which they cannot solve. This would aid software
>> development and allow someone else to have ago at solving the structure.
>>
>> If it is for historical reasons, how can someone decide whether their
>> structure is historical? I would propose that images should be uploaded for a
>> protein or protein-complex that has never be solved before. That way the
>> images are there if that structure does become historical.
>>
>> The question is not whether or not images should be uploaded but who
>> would use the images that were uploaded.
>>
>> For example, people who use crystallography as a tool to aid in
>> characterization of their protein, would probably not look at images for 
>> 99.5%
>> of other protein datasets, and they probably would not look at images for a
>> protein that is related to their own protein. They are more interested in the
>> final structure. I too would probably not be interested in reprocessing and
>> solving a structure again when I can easily access the final product already.
>



-- 
***
Jacob Pearson Keller
Northwestern University
Medical Scientist Training Program
email: j-kell...@northwestern.edu
***

[ccp4bb] Continuous/shutterless readout detectors - preserving the information content. Was IUCr committees, depositing images

[Colin Nave]

Altering the thread title so " IUCr committees, depositing images" can be kept 
on topic

George is right to raise the point about pixels not being read out at the same 
time. In principle of course this could be corrected for. Although the Bruker 
and PILATUS systems both allow shutterless operation the readout effects will 
be different (PILATUS is photon counting).

George raised this issue of preserving the information and this is essentially 
a sampling issue. One point is that the detectors do not actually sample the 
data. They average over the area of a pixel and the phi rotation width. In 
general, this average only corresponds to the value at the centre of a voxel 
(with x,y and phi coordinates) if the intensity varies linearly across the 
voxel. If this is not the case, it would cause errors in profile fitting.

Allowing for this, what is the appropriate sampling to preserve the 
information? Fine structure is present in the reflection profiles (in x, y and 
phi) for crystals at cryo-temperature (e.g. J. Lovelace, C. R. Murphy, R. Pahl, 
K. Brister and G. E. O. Borgstahl J. Appl. Cryst. (2006)). If one wants to get 
the maximum benefit from profile fitting then small phi ranges (and small 
pixels) are required. In principle the values should relate to the angular 
divergence of the incident beam so that information is not loss. This is one 
limit (an issue which Martin raised). Of course one can reduce the beam 
divergences until one hits the limit given by sample effects but the numbers 
then get silly. Despite this, I think there would be benefits, for example 
relating to the issue of whether one can separate out different cell dimensions 
(I am not going to let go of this point!).

If one sets this up (at the moment it would be possible in phi at least) one 
might find that the present software cannot handle it so yes, George's point 
about this could well be valid.

Colin


-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of George M. 
Sheldrick
Sent: 27 October 2011 14:37
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images

There are two further complications. In non-continuous mode, the goniometer has
to accelerate at the start of a frame and decellerate at the end, then wait for
the frame to be read. So even if the shutter always functions perfectly, my 
intuition tells me that it must be more accurate to rotate at constant speed 
(however my intuition is often wrong). Secondly, in continuous mode, usually 
not all pixels are read out at precisely the same time.

George

On Thu, Oct 27, 2011 at 11:05:01AM +0100, David Waterman wrote:
> I agree with Colin here. Framing is simply a process of sampling an original
> signal at some 'frequency' (related to the phi-width of each frame). At some
> point, delta phi is small enough that the original signal is oversampled, and
> can be reconstructed _within the bounds of noise_. Beyond that point I see no
> advantage to sampling finer - and certainly not going to the limit of
> representing your data in some unframed continuous readout form.
> 
> Perhaps I am missing something, and I realise this is another OT diversion 
> from
> this most fruitful of threads.
> 
> Cheers
> 
> -- David
> 
> 
> On 27 October 2011 08:55, Martin M. Ripoll  wrote:
> 
> Dear Colin,
> 
> I think you understood perfectly what George was saying regarding the loss
> of information, but he will probably answer better than I.
> 
> In any case, and for the ones that did not understand it, what George was
> telling is related to the fact that a data collection made with a
> continuous
> crystal rotation contains more information than when this information is
> transformed into frames... The loss of information that we are referring 
> to
> has the same meaning as when we calculate electron density maps with
> different grid sizes. The finer the grid, the greater is the information 
> on
> the map.
> 
> But you are right saying that the shorter the interval between produced
> frames, the lower the loss of information. However, the procedure that you
> are suggesting should have some limits... otherwise the amount of
> information would grow dramatically.
> 
> All the best,
> Martin
> 
> Dr. Martin Martinez-Ripoll
> Research Professor
> xmar...@iqfr.csic.es
> Department of Crystallography & Structural Biology
> www.xtal.iqfr.csic.es
> Telf.: +34 917459550
> Consejo Superior de Investigaciones Cient ficas
> Spanish National Research Council
> www.csic.es
> 
> 
> -Mensaje original-
> De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de Colin
> Nave
> Enviado el: jueves, 27 de octubre de 2011 0:49
> Para: CCP4BB@JISCMAIL.AC.UK
> Asunto: Re: [ccp4bb] IUCr committees, depositing images
> 
> Dear George, Martin
> 
> I don't u

Re: [ccp4bb] raw data deposition

[Michel Fodje]

We store raw data for two main reasons:
a)  We currently use only a fraction of the information actually contained in 
raw images and extraction of that fraction can be improved. Destroying the data 
means 
- we lose the extra information, and make future research in some areas either 
impossible or more costly
- we make it more difficult to improve current data reduction methods
b)  Raw data is the best way to independently validate a published structure 
and prevent fraud.

The majority of crystallographers already recognize these truths. That is why 
almost all of them do keep backups of their data even after structures have 
been published.   

To those still against making data public I would ask a simple question:  Would 
you object to providing the raw data from a published structure if such data 
were available and you did not have to bear an unreasonable inconvenience in 
the process? My guess is that most crystallographers are reasonable scientists 
and such a "Poll" will probably result in ~100% "Yes" and ~0% "No". I'm I wrong?

The real issue then is how do we make the data available in such a way that the 
inconvenience (if any) to all the stake-holders is reasonable.  Some great 
ideas have already been advanced. 

In the short-term,  we could start by using the fact that synchrotron 
facilities already store raw data for a period. However, a lot of data is 
collected which is not published. Given the limited disk space, it may be 
useful to know exactly which datasets result in a publication and should be 
kept for an extended period. If a unique ID (such as the DOI suggestion) is 
provided to every dataset and required during deposition/publication, then 
synchrotron facilities can preserve only those datasets which have been 
published after a given "grace" period. Combined with a central Meta-data 
server similar to TARDIS, such a system could be developed in a relatively 
short period of time, while longer term central storage ideas are worked out.

Again the best solution is going to be one which requires the least amount of 
effort from crystallographers. In fact, I can see a system in which the 
experiment metadata for a PDB entry/dataset comes directly from the synchrotron 
facility during deposition so that users simply provide a unique dataset ID and 
the experimental details are pre-filled for them.

Of course the above completely ignores home sources.


/Michel
> -Original Message-
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of D
> Bonsor
> Sent: October-27-11 3:10 PM
> To: CCP4BB@JISCMAIL.AC.UK
> Subject: Re: [ccp4bb] raw data deposition
> 
> Why should we store images?
> 
> From most of the posts it seems to aid in software development. If that is
> the case, there should be a Failed Protein Databank (FPDB) where people
> could upload datasets which they cannot solve. This would aid software
> development and allow someone else to have ago at solving the structure.
> 
> If it is for historical reasons, how can someone decide whether their
> structure is historical? I would propose that images should be uploaded for a
> protein or protein-complex that has never be solved before. That way the
> images are there if that structure does become historical.
> 
> The question is not whether or not images should be uploaded but who
> would use the images that were uploaded.
> 
> For example, people who use crystallography as a tool to aid in
> characterization of their protein, would probably not look at images for 99.5%
> of other protein datasets, and they probably would not look at images for a
> protein that is related to their own protein. They are more interested in the
> final structure. I too would probably not be interested in reprocessing and
> solving a structure again when I can easily access the final product already.

Re: [ccp4bb] raw data deposition

[Thomas C. Terwilliger]

Why should we store images?

There are many reasons why storing images can be useful, but one is the
ability to re-analyze the data for a structure, or for all structures, in
a systematic and improved way.

I imagine that in a few years the PDB-REDO approach to rebuilding
structures will be extended to complete redetermination of all structures
on a regular basis.  The resulting structures will continuously improve,
and each new redetermination will be stored so that a static view can be
referenced.

The data for these structures will remain constant, the interpretation
will change (presumably an  improvement).

The logical continuation of this approach will be to move back from merged
data to unmerged data, and then to raw images.  Surely we will develop
improved methods for analysis of images, and structures (or perhaps
details of structures) will improve.

Surely also some structures that were determined with less than optimal
care today will become accurate structures in this way.

-Tom T


  The raw data for
>> Why should we store images?
>>
>> From most of the posts it seems to aid in software development. If that is
>> the case, there should be a Failed Protein Databank (FPDB) where people
>> could upload datasets which they cannot solve. This would aid software
>> development and allow someone else to have ago at solving the structure.
>>
>> If it is for historical reasons, how can someone decide whether their
>> structure is historical? I would propose that images should be uploaded
>> for a protein or protein-complex that has never be solved before. That way
>> the images are there if that structure does become historical.
>>
>> The question is not whether or not images should be uploaded but who would
>> use the images that were uploaded.
>>
>> For example, people who use crystallography as a tool to aid in
>> characterization of their protein, would probably not look at images for
>> 99.5% of other protein datasets, and they probably would not look at
>> images for a protein that is related to their own protein. They are more
>> interested in the final structure. I too would probably not be interested
>> in reprocessing and solving a structure again when I can easily access the
>> final product already.
>>

Re: [ccp4bb] raw data deposition

[Ed Pozharski]

This is my response to Gerard, originally off-list, but which he feels
needs to be made public.

Dear Gerard,

1.  I think any opinion (collective or individual) by now is affected by
the ongoing discussion.
2.  I am not sure how this would make the discussion less public.
3.  Yes, we should continue to seek consensus, but perhaps it may be
useful to see if the consensus already exists.  Or if the proposition of
storing the raw data (which I personally support, but haven't always)
faces strong headwinds.
4.  Any online poll is always skewed towards the people who care enough
about the issue.  My concern was that I don't really know how many
people support this.  Granted, the right decision is not always
supported by the majority and "democracy" has its obvious limits, but
what we gain by this is some idea as to where people actually stand.

My hope is that the poll would show that most people support the
deposition of raw images.  This should presumably help your argument
(which, again, I wholeheartedly support) that this has to be done.  If
it shows the opposite... well, then we have the work to do to convince
them.  And perhaps listen to what their arguments are.

I think there are two questions regarding the raw data deposition (not
necessarily in that order):

1.  How to do it.

That is what the other thread is dealing with and my overall feeling is
that difficulties have been largely exaggerated early on.  You are right
that concrete steps can be taken.

2.  Do we need to do it.

To me, it's no-brainer, but some responses seem to suggest not everyone
is really on board.  Again, I am sure this has to be done, but consensus
in this area is equally important.  

HTH,

Ed.
On Thu, 2011-10-27 at 17:25 +0100, Gerard Bricogne wrote:
> Dear Ed,
> 
>  I am really puzzled by this initiative. It assumes that there is a
> pre-formed "collective opinion" out there, independent from and unaffected
> by the exchanges of views that have taken place on this BB, that would be
> worth more than the conclusions we might reach by pursuing these exchanges.
> 
>  The thread you are obviously deciding to dissociate yourself from was
> initiated in response to a suggestion that views on this topic would
> usefully be aired publicly on this BB rather than posted off-list to Tom
> Terwilliger, who immediately agreed that this was a good idea and has been
> very supportive of this discussion.
> 
>  Shouldn't we continue to try and put our heads together to reach a
> consensus, rather than collect opinions that may be little more than prior
> prejudices?
> 
>  What shall we gain by such a vote? I may be misunderstanding what you
> have in mind, of course :-) .
> 
> 
>  With best wishes,
>  
>   Gerard.
> 
> --
> On Thu, Oct 27, 2011 at 12:08:24PM -0400, Ed Pozharski wrote:
> > I am curious as to what the collective opinion on the raw data
> > deposition actually is across the cross-section of the macromolecular
> > crystallography community subscribed to the bb.  So, if you have a
> > second and a formed opinion on the idea of the depositions of the raw
> > data, please vote here
> > 
> > http://tinyurl.com/3qlwwsh
> > 
> > I'll post the results as soon as they look settled.
> > 
> > Cheers,
> > 
> > Ed.
> > 
> > -- 
> > "Hurry up before we all come back to our senses!"
> >Julian, King of Lemurs
> 

-- 
Oh, suddenly throwing a giraffe into a volcano to make water is crazy?
Julian, King of Lemurs

Re: [ccp4bb] raw data deposition

[Nat Echols]

On Thu, Oct 27, 2011 at 1:47 PM, Adrian Goldman
wrote:

> 1) this is not a matter of science, but science (internal) policy, and so
> the majority actually SHOULD count.
>

It's worth keeping in mind that there was once strong opposition to the
current rules on PDB deposition - the best example I could find is here:

http://www.nature.com/nsmb/journal/v5/n6/pdf/nsb0698-407.pdf

Notably, nearly a third of scientists polled thought they should be allowed
to publish without releasing coordinates.  If this had been a majority,
should the journal editors have meekly submitted and allowed the old policy
of 1-year holds to continue?  Admittedly, the issue of archiving raw images
is not the same, since they are of much less use to the community, but it's
a good example of why some opinions should be ignored.

-Nat

Re: [ccp4bb] raw data deposition

[D Bonsor]

Why should we store images?

From most of the posts it seems to aid in software development. If that is the 
case, there should be a Failed Protein Databank (FPDB) where people could 
upload datasets which they cannot solve. This would aid software development 
and allow someone else to have ago at solving the structure.

If it is for historical reasons, how can someone decide whether their structure 
is historical? I would propose that images should be uploaded for a protein or 
protein-complex that has never be solved before. That way the images are there 
if that structure does become historical.  

The question is not whether or not images should be uploaded but who would use 
the images that were uploaded.

For example, people who use crystallography as a tool to aid in 
characterization of their protein, would probably not look at images for 99.5% 
of other protein datasets, and they probably would not look at images for a 
protein that is related to their own protein. They are more interested in the 
final structure. I too would probably not be interested in reprocessing and 
solving a structure again when I can easily access the final product already.

Re: [ccp4bb] raw data deposition

[Craig A. Bingman]

I strongly suspect that it is much more cost effective to have the PDB archive 
a unit of data than it is to have it archived at the lab or department level.  
So I suspect that more money will be available for doing science if we turn 
over archival responsibilities for image data to the kind folks at the PDB, who 
really know what they are doing and capture efficiencies of scale that are out 
of reach for most labs.  

On Oct 27, 2011, at 3:47 PM, Adrian Goldman wrote:

> 2) I agree with Susan.  In a time of limited funding, is this the most 
> important use of money?

Re: [ccp4bb] raw data deposition

[Adrian Goldman]

Um, I have thought about entering this thread at least a dozen times.  I've 
started several comments and stopped all of them.

First, I am with the silent majority who doesn't think this data storage is a 
good idea (or not a good enough idea) but who hasn't responded till now.  And 
let me say that, as this bb hardly reaches ALL practicing MM crystallographers, 
but only those with an interest in techniques, the results AND discussion are 
heavily skewed in favor of storage.  At least that's what I think.

So - looking at my own navel - why would one, did I, not write until now?  
There is in the bb a loud active (and my guess) minority whose opinions are 
already formed, so responding seems pointless.  It won't change anything and 
will just lead to opprobrium pouring down on my head.  That's one reason.  

But let me say - and I voted 'no' as should be blindingly obvious - two more 
things.
1) this is not a matter of science, but science (internal) policy, and so the 
majority actually SHOULD count.
2) I agree with Susan.  In a time of limited funding, is this the most 
important use of money?
This point was made in a news-and views I recently read but cannot find despite 
an hour of searching - we as a species are not good at judging the opportunity 
costs implicit in choices.  There are plenty implicit in this choice, would it 
not, for instance, be MUCH more useful to finally get the modellers to release 
their source code?

But enough of the nattering nabobs of negativism! As such frame information is 
so valuable for future development efforts, I think all it would require would 
be an email to a local crystallographer working on an impossible problem, and I 
am sure it would be forthcoming.  For s/w development purposes, I can't believe 
that even a small fraction of the terabytes of frame data off the pilots is 
needed...

Adrian Goldman

Sent from my iPad

On 27 Oct 2011, at 21:17, Ed Pozharski  wrote:

> Dear Garib,
> 
> I am afraid clarification is in order.
> 
> Firstly, the results are available here
> 
> https://docs.google.com/spreadsheet/ccc?key=0Ahe0ET6Vsx-kdHh4cjdLZGZrSEpUOG9kV2hkb3ZXNHc
> 
> Click Form->Show summary to see the pie chart.  This is so you don't
> need to vote again to see the results (and please, don't vote more than
> once anyway!).  In my past experience, the results get more or less
> final in a day or two or once the number of responses reaches ~300.
> 
> Secondly, it was not my intent to provide a "democracy-based argument".
> Majority is often wrong.
> 
> Thirdly, it was not my intent to bias the results by carefully crafting
> misleading/confusing options.  Just disregard the part past "No".  Or
> provide you own reasons using the "Other" - I personally find that
> category the most interesting.
> 
> Fourthly, my intent was to separate the discussion of "how to do it"
> from "should we do it".  I disagree with Garib somewhat that this is
> purely scientific question, and perhaps it is open to some opinion.  The
> proposed changes will affect everyone (albeit in minor way), and my
> ultimate intent is not to impose democracy but rather, as Jacob pointed
> out, to potentially give voice to the silent faction.  Garib is right
> that we should approach the question scientifically, but it's important
> to know if the issue is at all controversial.  (In a strange way, the
> smaller the minority is on either side the more important it seems to me
> personally that every effort is made to assure that its position is well
> understood).
> 
> Hope this clarifies things,
> 
> Ed.
> 
> On Thu, 2011-10-27 at 18:05 +0100, Garib N Murshudov wrote:
>> 
>> 
>> I never thought that science should be done democratically. (Note, I
>> voted to see results. Otherwise results are invisible). It would be
>> unimaginable to decide by majority vote that a particular equation  or
>> theory is valid (e.g. relativity theory).  I thought that storing data
>> is a scientific question and should be tackled scientifically. You
>> provide evidence, proof or proof of principle. 
>> The most important question is repeatability of the experiment.
>> Question is: how far should we go? I know that there is at least one
>> case of overmerged data in the pdb. This particular question could be
>> solved (only partially) if you deposit unmerged data, with images it
>> is solved completely. Overmerging means averaging structures, thus
>> losing differences between them (biologically important or not).
>> Overmerging could be over translation (superlattice), rotation (higher
>> space group) or both.
>> 
>> 
>> Has anybody ever done systematic analysis of pdb (even better data
>> sets collected on one of the synchrotrons) to see the seriousness of
>> the problem? I suspect the problem is much more serious than it is
>> perceived.
>> 
>> Before you provide sufficient evidence everybody will have their
>> opinion.
>> 
>> 
>> Garib
>> 
>> 
>> 
>> On 27 Oct 2011, at 17:08, Ed Pozharski wrote:
>> 
>>> I am curiou

Re: [ccp4bb] Weird blob appears

[Jacob Keller]

Part of what is bothering me is that the density showed up at one
exact point in the refinement, and I am currently testing what exactly
it was that changed things. To me, the blob almost looks like a mask
of the molecule, and there is very little 2Fo density, so that's
weird. I am also really bothered that the R value is so low, and yet
there's this huge blob. Something seems buggy to me. Also a
clarification: the blob runs into a two-fold axis in the middle, so
things get dodgy there too. I will let you know what I find out.

JPK




On Thu, Oct 27, 2011 at 2:53 PM, David Briggs  wrote:
> I agree with Rafael,
>
> From those pictures it looks like a sugar chain - maybe 2-3
> saccharides in a row.
>
> HTH
>
> D
> 
> David C. Briggs PhD
> Father, Structural Biologist and Sceptic
> 
> University of Manchester E-mail:
> david.c.bri...@manchester.ac.uk
> 
> http://manchester.academia.edu/DavidBriggs (v.sensible)
> http://xtaldave.wordpress.com/ (sensible)
> http://xtaldave.posterous.com/ (less sensible)
> Twitter: @xtaldave
> Skype: DocDCB
> 
>
>
>
> On 27 October 2011 17:27, Jacob Keller  wrote:
>> Dear Crystallographers,
>>
>> In the course of a reasonably smooth refinement, all of a sudden there
>> is a huge worm-hole-type blob in the electron density (see pics). Has
>> anyone seen this before? Is it some effect of the refinement
>> over-fitting or something? For some background, resolution is 2.2Ang
>> and there are 4mol/ASU, but one of them is markedly worse in terms of
>> density than the rest, and appears to be in a non-crucial part of the
>> lattice, as if just sitting in one of the interstices. R/Rfree is
>> ~19.5/24.5.
>>
>> Jacob
>>
>> --
>> ***
>> Jacob Pearson Keller
>> Northwestern University
>> Medical Scientist Training Program
>> email: j-kell...@northwestern.edu
>> ***
>>
>



-- 
***
Jacob Pearson Keller
Northwestern University
Medical Scientist Training Program
email: j-kell...@northwestern.edu
***

Re: [ccp4bb] Weird blob appears

[David Briggs]

I agree with Rafael,

>From those pictures it looks like a sugar chain - maybe 2-3
saccharides in a row.

HTH

D

David C. Briggs PhD
Father, Structural Biologist and Sceptic

University of Manchester E-mail:
david.c.bri...@manchester.ac.uk

http://manchester.academia.edu/DavidBriggs (v.sensible)
http://xtaldave.wordpress.com/ (sensible)
http://xtaldave.posterous.com/ (less sensible)
Twitter: @xtaldave
Skype: DocDCB




On 27 October 2011 17:27, Jacob Keller  wrote:
> Dear Crystallographers,
>
> In the course of a reasonably smooth refinement, all of a sudden there
> is a huge worm-hole-type blob in the electron density (see pics). Has
> anyone seen this before? Is it some effect of the refinement
> over-fitting or something? For some background, resolution is 2.2Ang
> and there are 4mol/ASU, but one of them is markedly worse in terms of
> density than the rest, and appears to be in a non-crucial part of the
> lattice, as if just sitting in one of the interstices. R/Rfree is
> ~19.5/24.5.
>
> Jacob
>
> --
> ***
> Jacob Pearson Keller
> Northwestern University
> Medical Scientist Training Program
> email: j-kell...@northwestern.edu
> ***
>

Re: [ccp4bb] raw data deposition

[Ed Pozharski]

Sorry, the results in a pie-chart form are available here (but the
spreadsheet may be useful too if you want to see what is meant by
"other")

https://docs.google.com/spreadsheet/viewanalytics?hl=en_US&formkey=dHh4cjdLZGZrSEpUOG9kV2hkb3ZXNHc6MQ


-- 
Oh, suddenly throwing a giraffe into a volcano to make water is crazy?
Julian, King of Lemurs

Re: [ccp4bb] frm2frm

[Byram, Susan]

This is a reply to the group with a resolution of the frm2frm question. Frm2frm 
is a utility program which is part of the Bruker PROTEUM suite of software. It 
converts Bruker CCD frame images so they can be processed by external programs 
eg MOSFLM.We can provide a demo copy of the software suite upon request, so 
that a user can process Bruker CCD data collected at another site.

We learned that Khuchtumur has access to the PROTEUM suite and has located 
frm2frm, so his question is resolved.  It was very helpful to see the post on 
ccp4.
Best regards
Sue Byram

..
Susan K. Byram
Business Manager
Crystallographic Systems
Bruker AXS Inc.
5465 East Cheryl Parkway
Madison, WI 53711 USA
Toll-free tel.: (800) 234-XRAY
Tel.:   (608) 276-3041
Cell:   (608) 347-6035
Fax:(608) 276-3006
sue.by...@bruker-axs.com

www.bruker-axs.com
..


From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of 
khuchtumur bumerdene
Sent: Wednesday, October 26, 2011 12:07 AM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] frm2frm

Hello,

Does anyone know where I could download frm2frm utility from Bruker? Is it even 
possible to do so?



The information contained in this email is confidential. It is intended solely 
for the addressee. Access to this email by anyone else is unauthorized. If you 
are not the intended recipient, any form of disclosure, reproduction, 
distribution or any action taken or refrained from in reliance on it, is 
prohibited and may be unlawful. Please notify the sender immediately.

[ccp4bb] PDB header info wrong.

[Ian Tickle]

Hi, currently Refmac writes the wrong info for the low resolution
cutoff, for example:

REMARK   3   PROGRAM : REFMAC 5.6.0119
REMARK   3  DATA USED IN REFINEMENT.
REMARK   3   RESOLUTION RANGE HIGH (ANGSTROMS) :   1.00
REMARK   3   RESOLUTION RANGE LOW  (ANGSTROMS) :  84.35

In contrast Shel-X puts in:

REMARK   3   RESOLUTION RANGE LOW  (ANGSTROMS) : 10.00

The mtzdump of the MTZ file shows:

   1 NONE   -83   0  0  100.00-27.0 27.0  84.35   1.00   H  H
   2 NONE 0  97  0  100.00 54.9 54.9  84.35   1.00   H  K
   3 ASC  0  83  0  100.00 31.0 31.0  84.35   1.00   H  L
   4 NONE0.0 1.0 0  100.00 0.95 0.95  84.35   1.00   I  FREE
   5 NONE0.0  2071.8   435   99.82   125.22   125.22  10.00   1.00   F  FP
   6 NONE0.990.0   435   99.82 2.73 2.73  10.00   1.00
  Q  SIGFP

So obviously what has happened is that the Free R flags have been
generated to the high resolution cutoff and the low-res reflections,
i.e. below 10 Ang only have HKL & FREE defined.  Somewhere along the
line (this is data from the PDB) the low-res F's were lost.  So IMO
one cannot claim that these data were "used" in refinement.  Getting
tthe low-res cutoff wrong has a big effect on the electron density
that one expects for a given resolution range!  It seems to me that it
would be preferable to omit information rather than risk putting in
the wrong value, at least then one would know to go find the info
somewhere else.

Cheers

-- Ian

Re: [ccp4bb] raw data deposition

[Ed Pozharski]

Dear Garib,

I am afraid clarification is in order.

Firstly, the results are available here

https://docs.google.com/spreadsheet/ccc?key=0Ahe0ET6Vsx-kdHh4cjdLZGZrSEpUOG9kV2hkb3ZXNHc

Click Form->Show summary to see the pie chart.  This is so you don't
need to vote again to see the results (and please, don't vote more than
once anyway!).  In my past experience, the results get more or less
final in a day or two or once the number of responses reaches ~300.

Secondly, it was not my intent to provide a "democracy-based argument".
Majority is often wrong.

Thirdly, it was not my intent to bias the results by carefully crafting
misleading/confusing options.  Just disregard the part past "No".  Or
provide you own reasons using the "Other" - I personally find that
category the most interesting.

Fourthly, my intent was to separate the discussion of "how to do it"
from "should we do it".  I disagree with Garib somewhat that this is
purely scientific question, and perhaps it is open to some opinion.  The
proposed changes will affect everyone (albeit in minor way), and my
ultimate intent is not to impose democracy but rather, as Jacob pointed
out, to potentially give voice to the silent faction.  Garib is right
that we should approach the question scientifically, but it's important
to know if the issue is at all controversial.  (In a strange way, the
smaller the minority is on either side the more important it seems to me
personally that every effort is made to assure that its position is well
understood).

Hope this clarifies things,

Ed.

On Thu, 2011-10-27 at 18:05 +0100, Garib N Murshudov wrote:
> 
> 
> I never thought that science should be done democratically. (Note, I
> voted to see results. Otherwise results are invisible). It would be
> unimaginable to decide by majority vote that a particular equation  or
> theory is valid (e.g. relativity theory).  I thought that storing data
> is a scientific question and should be tackled scientifically. You
> provide evidence, proof or proof of principle. 
> The most important question is repeatability of the experiment.
>  Question is: how far should we go? I know that there is at least one
> case of overmerged data in the pdb. This particular question could be
> solved (only partially) if you deposit unmerged data, with images it
> is solved completely. Overmerging means averaging structures, thus
> losing differences between them (biologically important or not).
> Overmerging could be over translation (superlattice), rotation (higher
> space group) or both.
> 
> 
> Has anybody ever done systematic analysis of pdb (even better data
> sets collected on one of the synchrotrons) to see the seriousness of
> the problem? I suspect the problem is much more serious than it is
> perceived.
> 
> Before you provide sufficient evidence everybody will have their
> opinion.
> 
> 
> Garib
> 
> 
> 
> On 27 Oct 2011, at 17:08, Ed Pozharski wrote:
> 
> > I am curious as to what the collective opinion on the raw data
> > deposition actually is across the cross-section of the
> > macromolecular
> > crystallography community subscribed to the bb.  So, if you have a
> > second and a formed opinion on the idea of the depositions of the
> > raw
> > data, please vote here
> > 
> > http://tinyurl.com/3qlwwsh
> > 
> > I'll post the results as soon as they look settled.
> > 
> > Cheers,
> > 
> > Ed.
> > 
> > -- 
> > "Hurry up before we all come back to our senses!"
> >   Julian, King of Lemurs
> > 
> 
> Garib N Murshudov 
> Structural Studies Division
> MRC Laboratory of Molecular Biology
> Hills Road 
> Cambridge 
> CB2 0QH UK
> Email: ga...@mrc-lmb.cam.ac.uk 
> Web http://www.mrc-lmb.cam.ac.uk
> 
> 
> 
> 
> 
> 

-- 
Edwin Pozharski, PhD, Assistant Professor
University of Maryland, Baltimore
--
When the Way is forgotten duty and justice appear;
Then knowledge and wisdom are born along with hypocrisy.
When harmonious relationships dissolve then respect and devotion arise;
When a nation falls to chaos then loyalty and patriotism are born.
--   / Lao Tse /

Re: [ccp4bb] raw data deposition

[Garib N Murshudov]

I never thought that science should be done democratically. (Note, I voted to 
see results. Otherwise results are invisible). It would be unimaginable to 
decide by majority vote that a particular equation  or theory is valid (e.g. 
relativity theory).  I thought that storing data is a scientific question and 
should be tackled scientifically. You provide evidence, proof or proof of 
principle. 
The most important question is repeatability of the experiment.  Question is: 
how far should we go? I know that there is at least one case of overmerged data 
in the pdb. This particular question could be solved (only partially) if you 
deposit unmerged data, with images it is solved completely. Overmerging means 
averaging structures, thus losing differences between them (biologically 
important or not). Overmerging could be over translation (superlattice), 
rotation (higher space group) or both.

Has anybody ever done systematic analysis of pdb (even better data sets 
collected on one of the synchrotrons) to see the seriousness of the problem? I 
suspect the problem is much more serious than it is perceived.

Before you provide sufficient evidence everybody will have their opinion.

Garib


On 27 Oct 2011, at 17:08, Ed Pozharski wrote:

> I am curious as to what the collective opinion on the raw data
> deposition actually is across the cross-section of the macromolecular
> crystallography community subscribed to the bb.  So, if you have a
> second and a formed opinion on the idea of the depositions of the raw
> data, please vote here
> 
> http://tinyurl.com/3qlwwsh
> 
> I'll post the results as soon as they look settled.
> 
> Cheers,
> 
> Ed.
> 
> -- 
> "Hurry up before we all come back to our senses!"
>   Julian, King of Lemurs

Garib N Murshudov 
Structural Studies Division
MRC Laboratory of Molecular Biology
Hills Road 
Cambridge 
CB2 0QH UK
Email: ga...@mrc-lmb.cam.ac.uk 
Web http://www.mrc-lmb.cam.ac.uk

Re: [ccp4bb] raw data deposition

[Jacob Keller]

In medical school, I found out that there could be a large population
in a class which was completely lost or completely disagreed with what
was being said, but there was only silence. When the lecturer would
pose a question, it would take a painful silence before anyone in the
100+ student class would hazard an answer (one can only speculate why
this was, but so be it.) Even a yes-no show of hands would yield only
~10% participation. Too much commitment? Anyway, the solution for many
lecturers was to pass out "clickers," which allowed the vote
anonymously for a given answer in a multiple choice question. Here we
have Ed's version of clickers for the ccp4bb. The problem is that
clickers are not very articulate--they just click!

I fully agree that those who disagree should articulate their
opinions--maybe subscribe anonymously if necessary? I don't like it
that people are so stymied, but this seems to be the way things are,
so the question is how to work with it.

Jacob

On Thu, Oct 27, 2011 at 11:48 AM, Gerard Bricogne
 wrote:
> Dear Jacob,
>
>     I agree, of course, with the goal of giving everyone a voice, but
> knowing that 40% of the voters find storing images a waste of time falls
> short of knowing why they think so and taking their arguments into account.
> Disagreeing without saying why when a topic is being actively discussed is a
> position that does not contribute anything very constructive.
>
>     I think there should be an extra category of answer that would be
> "I don't care", so that people who have no opinion do not get confused with
> those who have an articulate position against the proposal, and wh should
> then articulate it!
>
>
>     With best wishes,
>
>          Gerard.
>
> --
> On Thu, Oct 27, 2011 at 11:30:20AM -0500, Jacob Keller wrote:
>> One thing that the poll is useful for is something I find surprising:
>> ~40% when I checked found storing images a waste of time. So, I guess
>> this might be useful for finding the "silent [significant] minority."
>> Why not have those folks chime in about why they think this is
>> useless, even to store images of solved datasets?
>>
>> JPK
>>
>> On Thu, Oct 27, 2011 at 11:25 AM, Gerard Bricogne
>>  wrote:
>> > Dear Ed,
>> >
>> >     I am really puzzled by this initiative. It assumes that there is a
>> > pre-formed "collective opinion" out there, independent from and unaffected
>> > by the exchanges of views that have taken place on this BB, that would be
>> > worth more than the conclusions we might reach by pursuing these exchanges.
>> >
>> >     The thread you are obviously deciding to dissociate yourself from was
>> > initiated in response to a suggestion that views on this topic would
>> > usefully be aired publicly on this BB rather than posted off-list to Tom
>> > Terwilliger, who immediately agreed that this was a good idea and has been
>> > very supportive of this discussion.
>> >
>> >     Shouldn't we continue to try and put our heads together to reach a
>> > consensus, rather than collect opinions that may be little more than prior
>> > prejudices?
>> >
>> >     What shall we gain by such a vote? I may be misunderstanding what you
>> > have in mind, of course :-) .
>> >
>> >
>> >     With best wishes,
>> >
>> >          Gerard.
>> >
>> > --
>> > On Thu, Oct 27, 2011 at 12:08:24PM -0400, Ed Pozharski wrote:
>> >> I am curious as to what the collective opinion on the raw data
>> >> deposition actually is across the cross-section of the macromolecular
>> >> crystallography community subscribed to the bb.  So, if you have a
>> >> second and a formed opinion on the idea of the depositions of the raw
>> >> data, please vote here
>> >>
>> >> http://tinyurl.com/3qlwwsh
>> >>
>> >> I'll post the results as soon as they look settled.
>> >>
>> >> Cheers,
>> >>
>> >> Ed.
>> >>
>> >> --
>> >> "Hurry up before we all come back to our senses!"
>> >>                            Julian, King of Lemurs
>> >
>> > --
>> >
>> >     ===
>> >     *                                                             *
>> >     * Gerard Bricogne                     g...@globalphasing.com  *
>> >     *                                                             *
>> >     * Global Phasing Ltd.                                         *
>> >     * Sheraton House, Castle Park         Tel: +44-(0)1223-353033 *
>> >     * Cambridge CB3 0AX, UK               Fax: +44-(0)1223-366889 *
>> >     *                                                             *
>> >     ===
>> >
>>
>>
>>
>> --
>> ***
>> Jacob Pearson Keller
>> Northwestern University
>> Medical Scientist Training Program
>> email: j-kell...@northwestern.edu
>> ***
>



-- 
***
Jacob Pearson Keller
Northwestern University
Medical Scientist Training Program
email: j-kell...

Re: [ccp4bb] raw data deposition

[Gerard Bricogne]

Dear Jacob,

 I agree, of course, with the goal of giving everyone a voice, but
knowing that 40% of the voters find storing images a waste of time falls
short of knowing why they think so and taking their arguments into account.
Disagreeing without saying why when a topic is being actively discussed is a
position that does not contribute anything very constructive. 

 I think there should be an extra category of answer that would be
"I don't care", so that people who have no opinion do not get confused with
those who have an articulate position against the proposal, and wh should
then articulate it!


 With best wishes,
 
  Gerard.

--
On Thu, Oct 27, 2011 at 11:30:20AM -0500, Jacob Keller wrote:
> One thing that the poll is useful for is something I find surprising:
> ~40% when I checked found storing images a waste of time. So, I guess
> this might be useful for finding the "silent [significant] minority."
> Why not have those folks chime in about why they think this is
> useless, even to store images of solved datasets?
> 
> JPK
> 
> On Thu, Oct 27, 2011 at 11:25 AM, Gerard Bricogne
>  wrote:
> > Dear Ed,
> >
> >     I am really puzzled by this initiative. It assumes that there is a
> > pre-formed "collective opinion" out there, independent from and unaffected
> > by the exchanges of views that have taken place on this BB, that would be
> > worth more than the conclusions we might reach by pursuing these exchanges.
> >
> >     The thread you are obviously deciding to dissociate yourself from was
> > initiated in response to a suggestion that views on this topic would
> > usefully be aired publicly on this BB rather than posted off-list to Tom
> > Terwilliger, who immediately agreed that this was a good idea and has been
> > very supportive of this discussion.
> >
> >     Shouldn't we continue to try and put our heads together to reach a
> > consensus, rather than collect opinions that may be little more than prior
> > prejudices?
> >
> >     What shall we gain by such a vote? I may be misunderstanding what you
> > have in mind, of course :-) .
> >
> >
> >     With best wishes,
> >
> >          Gerard.
> >
> > --
> > On Thu, Oct 27, 2011 at 12:08:24PM -0400, Ed Pozharski wrote:
> >> I am curious as to what the collective opinion on the raw data
> >> deposition actually is across the cross-section of the macromolecular
> >> crystallography community subscribed to the bb.  So, if you have a
> >> second and a formed opinion on the idea of the depositions of the raw
> >> data, please vote here
> >>
> >> http://tinyurl.com/3qlwwsh
> >>
> >> I'll post the results as soon as they look settled.
> >>
> >> Cheers,
> >>
> >> Ed.
> >>
> >> --
> >> "Hurry up before we all come back to our senses!"
> >>                            Julian, King of Lemurs
> >
> > --
> >
> >     ===
> >     *                                                             *
> >     * Gerard Bricogne                     g...@globalphasing.com  *
> >     *                                                             *
> >     * Global Phasing Ltd.                                         *
> >     * Sheraton House, Castle Park         Tel: +44-(0)1223-353033 *
> >     * Cambridge CB3 0AX, UK               Fax: +44-(0)1223-366889 *
> >     *                                                             *
> >     ===
> >
> 
> 
> 
> -- 
> ***
> Jacob Pearson Keller
> Northwestern University
> Medical Scientist Training Program
> email: j-kell...@northwestern.edu
> ***

Re: [ccp4bb] raw data deposition

[Ethan Merritt]

On Thursday, October 27, 2011 09:30:20 am Jacob Keller wrote:

> One thing that the poll is useful for is something I find surprising:
> ~40% when I checked found storing images a waste of time. So, I guess
> this might be useful for finding the "silent [significant] minority."
> Why not have those folks chime in about why they think this is
> useless, even to store images of solved datasets?

That kind of misses the point that the images may be of more value
to others (software developers, TDS wonks, Gloria's incommensurate
lattice hunt, ...) than they are to the person who originally collected them.  

So "I don't need them" is different from "there is no point in saving them".

Ethan

-- 
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
University of Washington, Seattle 98195-7742

Re: [ccp4bb] raw data deposition

[Susan Lea]

I think the key is that the questions asks "is a waste of money".
In a straightened funding time it may just be that storing the raw images in 
addition to the processed
data doesn't float to the top of the list of "things that must be done whatever 
else happens in science".


Something can be desirable but just not come above a funding barrier.




Susan



Prof. Susan M. Lea  tel: +44 1865 275181

Professor of Chemical Pathology  & Co-Director of the James Martin Vaccine 
Design Institute
Sir William Dunn School of Pathology, Oxford OX1 3RE UK

Tutorial Fellow @ Brasenose College, Oxford OX1 4AJ


From: CCP4 bulletin board [CCP4BB@JISCMAIL.AC.UK] On Behalf Of Jacob Keller 
[j-kell...@fsm.northwestern.edu]
Sent: 27 October 2011 17:30
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] raw data deposition

One thing that the poll is useful for is something I find surprising:
~40% when I checked found storing images a waste of time. So, I guess
this might be useful for finding the "silent [significant] minority."
Why not have those folks chime in about why they think this is
useless, even to store images of solved datasets?

JPK

On Thu, Oct 27, 2011 at 11:25 AM, Gerard Bricogne
 wrote:
> Dear Ed,
>
> I am really puzzled by this initiative. It assumes that there is a
> pre-formed "collective opinion" out there, independent from and unaffected
> by the exchanges of views that have taken place on this BB, that would be
> worth more than the conclusions we might reach by pursuing these exchanges.
>
> The thread you are obviously deciding to dissociate yourself from was
> initiated in response to a suggestion that views on this topic would
> usefully be aired publicly on this BB rather than posted off-list to Tom
> Terwilliger, who immediately agreed that this was a good idea and has been
> very supportive of this discussion.
>
> Shouldn't we continue to try and put our heads together to reach a
> consensus, rather than collect opinions that may be little more than prior
> prejudices?
>
> What shall we gain by such a vote? I may be misunderstanding what you
> have in mind, of course :-) .
>
>
> With best wishes,
>
>  Gerard.
>
> --
> On Thu, Oct 27, 2011 at 12:08:24PM -0400, Ed Pozharski wrote:
>> I am curious as to what the collective opinion on the raw data
>> deposition actually is across the cross-section of the macromolecular
>> crystallography community subscribed to the bb.  So, if you have a
>> second and a formed opinion on the idea of the depositions of the raw
>> data, please vote here
>>
>> http://tinyurl.com/3qlwwsh
>>
>> I'll post the results as soon as they look settled.
>>
>> Cheers,
>>
>> Ed.
>>
>> --
>> "Hurry up before we all come back to our senses!"
>>Julian, King of Lemurs
>
> --
>
> ===
> * *
> * Gerard Bricogne g...@globalphasing.com  *
> * *
> * Global Phasing Ltd. *
> * Sheraton House, Castle Park Tel: +44-(0)1223-353033 *
> * Cambridge CB3 0AX, UK   Fax: +44-(0)1223-366889 *
> * *
> ===
>



--
***
Jacob Pearson Keller
Northwestern University
Medical Scientist Training Program
email: j-kell...@northwestern.edu
***

Re: [ccp4bb] Weird blob appears

[Rafael Molina]

Maybe sugars?

El 10/27/11 6:27 PM, Jacob Keller escribió:

Dear Crystallographers,

In the course of a reasonably smooth refinement, all of a sudden there
is a huge worm-hole-type blob in the electron density (see pics). Has
anyone seen this before? Is it some effect of the refinement
over-fitting or something? For some background, resolution is 2.2Ang
and there are 4mol/ASU, but one of them is markedly worse in terms of
density than the rest, and appears to be in a non-crucial part of the
lattice, as if just sitting in one of the interstices. R/Rfree is
~19.5/24.5.

Jacob




--
---
Rafael Molina, PhD

Structural Biology and Biocomputing Programme

CNIO - Centro Nacional de Investigaciones Oncológicas
C/ Melchor Fernández Almagro, 3, E-28029 Madrid (España)

www.cnio.es
---


**NOTA DE CONFIDENCIALIDAD** Este correo electrónico, y en su caso los ficheros 
adjuntos, pueden contener información protegida para el uso exclusivo de su 
destinatario. Se prohíbe la distribución, reproducción o cualquier otro tipo de 
transmisión por parte de otra persona que no sea el destinatario. Si usted 
recibe por error este correo, se ruega comunicarlo al remitente y borrar el 
mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may 
contain confidential and privileged information for the sole use of the 
designated recipient named above. Distribution, reproduction or any other use 
of this transmission by any party other than the intended recipient is 
prohibited. If you are not the intended recipient please contact the sender and 
delete all copies.

Re: [ccp4bb] raw data deposition

[Jacob Keller]

One thing that the poll is useful for is something I find surprising:
~40% when I checked found storing images a waste of time. So, I guess
this might be useful for finding the "silent [significant] minority."
Why not have those folks chime in about why they think this is
useless, even to store images of solved datasets?

JPK

On Thu, Oct 27, 2011 at 11:25 AM, Gerard Bricogne
 wrote:
> Dear Ed,
>
>     I am really puzzled by this initiative. It assumes that there is a
> pre-formed "collective opinion" out there, independent from and unaffected
> by the exchanges of views that have taken place on this BB, that would be
> worth more than the conclusions we might reach by pursuing these exchanges.
>
>     The thread you are obviously deciding to dissociate yourself from was
> initiated in response to a suggestion that views on this topic would
> usefully be aired publicly on this BB rather than posted off-list to Tom
> Terwilliger, who immediately agreed that this was a good idea and has been
> very supportive of this discussion.
>
>     Shouldn't we continue to try and put our heads together to reach a
> consensus, rather than collect opinions that may be little more than prior
> prejudices?
>
>     What shall we gain by such a vote? I may be misunderstanding what you
> have in mind, of course :-) .
>
>
>     With best wishes,
>
>          Gerard.
>
> --
> On Thu, Oct 27, 2011 at 12:08:24PM -0400, Ed Pozharski wrote:
>> I am curious as to what the collective opinion on the raw data
>> deposition actually is across the cross-section of the macromolecular
>> crystallography community subscribed to the bb.  So, if you have a
>> second and a formed opinion on the idea of the depositions of the raw
>> data, please vote here
>>
>> http://tinyurl.com/3qlwwsh
>>
>> I'll post the results as soon as they look settled.
>>
>> Cheers,
>>
>> Ed.
>>
>> --
>> "Hurry up before we all come back to our senses!"
>>                            Julian, King of Lemurs
>
> --
>
>     ===
>     *                                                             *
>     * Gerard Bricogne                     g...@globalphasing.com  *
>     *                                                             *
>     * Global Phasing Ltd.                                         *
>     * Sheraton House, Castle Park         Tel: +44-(0)1223-353033 *
>     * Cambridge CB3 0AX, UK               Fax: +44-(0)1223-366889 *
>     *                                                             *
>     ===
>



-- 
***
Jacob Pearson Keller
Northwestern University
Medical Scientist Training Program
email: j-kell...@northwestern.edu
***

Re: [ccp4bb] raw data deposition

[Gerard Bricogne]

Dear Ed,

 I am really puzzled by this initiative. It assumes that there is a
pre-formed "collective opinion" out there, independent from and unaffected
by the exchanges of views that have taken place on this BB, that would be
worth more than the conclusions we might reach by pursuing these exchanges.

 The thread you are obviously deciding to dissociate yourself from was
initiated in response to a suggestion that views on this topic would
usefully be aired publicly on this BB rather than posted off-list to Tom
Terwilliger, who immediately agreed that this was a good idea and has been
very supportive of this discussion.

 Shouldn't we continue to try and put our heads together to reach a
consensus, rather than collect opinions that may be little more than prior
prejudices?

 What shall we gain by such a vote? I may be misunderstanding what you
have in mind, of course :-) .


 With best wishes,
 
  Gerard.

--
On Thu, Oct 27, 2011 at 12:08:24PM -0400, Ed Pozharski wrote:
> I am curious as to what the collective opinion on the raw data
> deposition actually is across the cross-section of the macromolecular
> crystallography community subscribed to the bb.  So, if you have a
> second and a formed opinion on the idea of the depositions of the raw
> data, please vote here
> 
> http://tinyurl.com/3qlwwsh
> 
> I'll post the results as soon as they look settled.
> 
> Cheers,
> 
> Ed.
> 
> -- 
> "Hurry up before we all come back to our senses!"
>Julian, King of Lemurs

-- 

 ===
 * *
 * Gerard Bricogne g...@globalphasing.com  *
 * *
 * Global Phasing Ltd. *
 * Sheraton House, Castle Park Tel: +44-(0)1223-353033 *
 * Cambridge CB3 0AX, UK   Fax: +44-(0)1223-366889 *
 * *
 ===

Re: [ccp4bb] IUCr committees, depositing images

[Ed Pozharski]

On Thu, 2011-10-27 at 15:36 +0200, George M. Sheldrick wrote:
> In non-continuous mode, the goniometer has
> to accelerate at the start of a frame and decellerate at the end, then
> wait for
> the frame to be read.

Someone should be able to confirm this, but I was under impression that
at synchrotrons the acceleration/decelaration occurs outside
phi_start-phi_end range  to allow for constant speed when the shutter
opens.

 
-- 
"I'd jump in myself, if I weren't so good at whistling."
   Julian, King of Lemurs

Re: [ccp4bb] Off-topic: DSF thermo cycler low temp limit

[Morten Groftehauge]

Lots of systems go to 4C. Even a cheapo one like the Applied Biosystems
StepOnePlus.

-Morten Groftehauge

On 27 October 2011 03:17, Jürgen Bosch  wrote:

> CD spec with Pelletier is an option too
>
> Jürgen
>
> ..
> Jürgen Bosch
> Johns Hopkins Bloomberg School of Public Health
> Department of Biochemistry & Molecular Biology
> Johns Hopkins Malaria Research Institute
> 615 North Wolfe Street, W8708
> Baltimore, MD 21205
> Phone: +1-410-614-4742
> Lab:  +1-410-614-4894
> Fax:  +1-410-955-3655
> http://web.mac.com/bosch_lab/
>
> On Oct 26, 2011, at 19:08, Reginald McNulty  wrote:
>
> We are trying to determine Tm value of rather unstable proteins with a Tm
> in
> the mid 20 C range using DSF/thermofluor.  Our Stratagene thermocycler has
> a
> low temp limit of 25 C (Peltier).  I called the company and they said it's
> a
> 'hardware limit' that cannot be changed.
>
> 1) Has anyone been able to 'hotwire' the MX3000/3005 to go below this
> limit?
>
> 2) Are there other thermofluor machines that allow lower starting
> temperatures (say 4, 10 or 15 C)?
>
> Best regards,
> -Reggie
>
>
>
>

Re: [ccp4bb] Refinement of a structure with intermolecular disulfide bond

[Tim Gruene]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Dear Jiyuan Ke,

the SSBOND
(http://www.ccp4.ac.uk/html/refmac5/files/coordinates.html#pdb_ssbond)
does allow symmetry related molecules, so it seems that is the header
entry you are looking for.

Tim

On 10/27/2011 03:56 PM, Ke, Jiyuan wrote:
> Dear All,
> 
> I have a question regarding refinement of a crystal structure with
> intermolecular disulfide bond. There is one monomer in an asymmetric
> unit. The biological assembly for the molecule is a disulfide bond
> linked dimer. And the disulfide bond is on a 2-fold crystallographic
> symmetry. During refinement using refmac, how do I tell the program
> to keep the disulfide bond? Currently two S atoms are pushing away
> from each other after refmac refinement. Thanks!
> 
> Jiyuan Ke, Ph.D. Research Scientist Van Andel Research Institute 333
> Bostwick Ave NE Grand Rapids, MI 49503
> 
> 
> The information transmitted is intended only for the person or entity
> to which it is addressed and may contain confidential and/or
> privileged material. Any review, retransmission, dissemination or
> other use of, or taking of any action in reliance upon, this
> information by persons or entities other than the intended recipient
> is prohibited. If you received this in error, please contact the
> sender and delete the material from any computer.
> 

- -- 
- --
Dr Tim Gruene
Institut fuer anorganische Chemie
Tammannstr. 4
D-37077 Goettingen

GPG Key ID = A46BEE1A

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFOqWS8UxlJ7aRr7hoRApjWAKDXJMiZO2TvGpp3xK9vxAO0iFZYAwCfYnYp
YiHr2Ix5wup4WTCY/k/YlNU=
=aVDP
-END PGP SIGNATURE-

[ccp4bb] Refinement of a structure with intermolecular disulfide bond

[Ke, Jiyuan]

Dear All,

I have a question regarding refinement of a crystal structure with 
intermolecular disulfide bond. There is one monomer in an asymmetric unit. The 
biological assembly for the molecule is a disulfide bond linked dimer. And the 
disulfide bond is on a 2-fold crystallographic symmetry. During refinement 
using refmac, how do I tell the program to keep the disulfide bond? Currently 
two S atoms are pushing away from each other after refmac refinement. Thanks!

Jiyuan Ke, Ph.D.
Research Scientist
Van Andel Research Institute
333 Bostwick Ave NE
Grand Rapids, MI 49503


The information transmitted is intended only for the person or entity to which 
it is addressed and may contain confidential and/or privileged material. Any 
review, retransmission, dissemination or other use of, or taking of any action 
in reliance upon, this information by persons or entities other than the 
intended recipient is prohibited. If you received this in error, please contact 
the sender and delete the material from any computer.

Re: [ccp4bb] IUCr committees, depositing images

[George M. Sheldrick]

There are two further complications. In non-continuous mode, the goniometer has
to accelerate at the start of a frame and decellerate at the end, then wait for
the frame to be read. So even if the shutter always functions perfectly, my 
intuition tells me that it must be more accurate to rotate at constant speed 
(however my intuition is often wrong). Secondly, in continuous mode, usually 
not all pixels are read out at precisely the same time.

George

On Thu, Oct 27, 2011 at 11:05:01AM +0100, David Waterman wrote:
> I agree with Colin here. Framing is simply a process of sampling an original
> signal at some 'frequency' (related to the phi-width of each frame). At some
> point, delta phi is small enough that the original signal is oversampled, and
> can be reconstructed _within the bounds of noise_. Beyond that point I see no
> advantage to sampling finer - and certainly not going to the limit of
> representing your data in some unframed continuous readout form.
> 
> Perhaps I am missing something, and I realise this is another OT diversion 
> from
> this most fruitful of threads.
> 
> Cheers
> 
> -- David
> 
> 
> On 27 October 2011 08:55, Martin M. Ripoll  wrote:
> 
> Dear Colin,
> 
> I think you understood perfectly what George was saying regarding the loss
> of information, but he will probably answer better than I.
> 
> In any case, and for the ones that did not understand it, what George was
> telling is related to the fact that a data collection made with a
> continuous
> crystal rotation contains more information than when this information is
> transformed into frames... The loss of information that we are referring 
> to
> has the same meaning as when we calculate electron density maps with
> different grid sizes. The finer the grid, the greater is the information 
> on
> the map.
> 
> But you are right saying that the shorter the interval between produced
> frames, the lower the loss of information. However, the procedure that you
> are suggesting should have some limits... otherwise the amount of
> information would grow dramatically.
> 
> All the best,
> Martin
> 
> Dr. Martin Martinez-Ripoll
> Research Professor
> xmar...@iqfr.csic.es
> Department of Crystallography & Structural Biology
> www.xtal.iqfr.csic.es
> Telf.: +34 917459550
> Consejo Superior de Investigaciones Cient ficas
> Spanish National Research Council
> www.csic.es
> 
> 
> -Mensaje original-
> De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de Colin
> Nave
> Enviado el: jueves, 27 de octubre de 2011 0:49
> Para: CCP4BB@JISCMAIL.AC.UK
> Asunto: Re: [ccp4bb] IUCr committees, depositing images
> 
> Dear George, Martin
> 
> I don't understand the point that one is throwing away information by
> storing in frames. If the frames have sufficiently fine intervals (given 
> by
> some sampling theorem consideration) I can't see how one loses 
> information.
> Can one of you explain?
> Thanks
> Colin
> 
> 
> 
> -Original Message-
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of
> Martin
> M. Ripoll
> Sent: 26 October 2011 22:50
> To: ccp4bb
> Subject: Re: [ccp4bb] IUCr committees, depositing images
> 
> Dear George, dear all,
> 
> I was just trying to summarize my point of view regarding this important
> issue when I got your e-mail, that reflects exactly my own opinion!
> 
> Martin
> 
> Dr. Martin Martinez-Ripoll
> Research Professor
> xmar...@iqfr.csic.es
> Department of Crystallography & Structural Biology
> www.xtal.iqfr.csic.es
> Telf.: +34 917459550
> Consejo Superior de Investigaciones Cient ficas
> Spanish National Research Council
> www.csic.es
> 
> 
> 
> -Mensaje original-
> De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de George
> M. Sheldrick
> Enviado el: mi rcoles, 26 de octubre de 2011 11:52
> Para: CCP4BB@JISCMAIL.AC.UK
> Asunto: Re: [ccp4bb] IUCr committees, depositing images
> 
> This raises an important point. The new continuous readout detectors such
> as
> the
> Pilatus for beamlines or the Bruker Photon for in-house use enable the
> crystal to
> be rotated at constant velocity, eliminating the mechanical errors
> associated with
> 'stop and go' data collection. Storing their data in 'frames' is an
> artifical
> construction that is currently required for the established data
> integration
> programs but is in fact throwing away information. Maybe in 10 years time
> 'frames'
> will be as obsolete as punched cards!
> 
> George
> 
> On Wed, Oct 26, 2011 at 09:39:40AM +0100, Graeme Winter wrote:
> > Hi James,
> >
> > Just 

Re: [ccp4bb] IUCr committees, depositing images

[John R Helliwell]

Dear Gerard,
Thankyou indeed for this clear and already detailed plan. As well as
for MX SR datasets, ~75% of the total, this plan can readily be
extended to eg SR SAXS and SANS.


For the benefit of CCP4bb participants just to mention that the IUCr
DDD WG is seeking 'can do' solutions across a broad spread; we have
already put two in place. Firstly the IUCr Commissions are supplying
exemplar datasets and associated metadata, and this is happening; this
will no doubt involve some communities getting clarity and agreement
re the data and metadata required. Secondly, we are looking with IUCr
Journals into whether archiving of data with the literature might be
feasible e.g. for chemical  crystallography. Nb chemical
crystallography is a huge field and the largest fraction of IUCr
Journals by publications number. The 'synchrotrons administering doi
plan' largely does not satisfy this major need as the largest fraction
of these datasets are measured in house. However University
institutional repositories do host data sets and I will check with the
University of Manchester one if there is a size limit because this
would again avoiding moving the data around unduly. The Journals would
then 'only' have to deal with cases of universities where there is no
such institutional repository.

Re your suggestion that there is a 'Moving goalposts' afoot eg due to
the pixel detector; let's be clear, the IUCr DDD WG and myself in
particular already know about, and have used, the pixel detector, and
indeed the Working Group is not called the Diffraction Data Images
Working Group! The principle of 'archiving data with literature' is
the key principle, whatever device is used in the measurements. This
principle is an immoveable goalpost.

Greetings,
John
Prof John R Helliwell DSc



On Thu, Oct 27, 2011 at 11:14 AM, Gerard Bricogne
 wrote:
> Dear all,
>
>     In response to my message below, posted yesterday afternoon (GMT+1) I
> received an off-list message from John Helliwell asking me to clarify what I
> had meant by "already stored information" when referring to the outline of a
> feasibility study envisaged by the IUCr Forum. I answered him, with Cc to
> Tom Terwilliger and Brian McMahon. Tom then wrote to me to suggest that I
> post my reply to this BB, and John agreed to it - so I include it below.
>
>     I hope it will serve to illustrate the possibility of undertaking
> something concrete right away. It was written hurriedly, and is not meant to
> be a bullet-proof proposal.
>
>
>     With best wishes,
>
>          Gerard.
>
> ===
> Dear John,
>
>     Thank you for this message off-list.
>
>     What I was referring to is the combination of two ideas that emerged
> during our discussions in Madrid as the basis for a pilot project - not yet
> a final solution:
>
>     (1) the identification of every dataset collected on a synchrotron and
> stored locally could be implemented by assigning it a unique doi - that was
> discussed with Alun, the chap from the Australian synchrotron, Brian, John
> Westbrook etc. at the meeting;
>
>     (2) since the synchrotrons already store all their user data for some
> time, and as several of them intend to make them publicly available after a
> statutory period, it seemed a rather minor and inexpensive extra step to
> promote the small fraction of datasets actually associated with a
> publication to on-line storage, still at synchrotron sources, for the
> purpose of making them accessible to the public as a sort of extension of
> the present structure factor file.
>
>     This initial experiment would therefore involve "federating" the local
> storage facilities of a few synchrotrons interested in participating in this
> pilot project (or feasibility study), with a special treatment for those
> datasets, but nothing that would be inordinately onerous in terms of extra
> costs. The relevant doi would then appear in the pdb entry and would enable
> interested users to access the raw images for their own purposes. The public
> disclosure of image sets would also be at the option of the authors of a pdb
> entry, not compulsory at this stage.
>
>     Starting things in this way would enable a number of critical
> procedures to be tested and of key parameters to be estimated, and in
> particular to start storing those datasets before that storage can be passed
> on to a more central entity, and before deposition is made compulsory by the
> journals' own policies.
>
>     I hope this is not too confused a picture. Its main ingredients are
>
>     1. a number of synchrotrons willing to put in the extra resources into
> assigning those unique doi's to datasets collected there, which they archive
> anyway, and into making those that are eventually associated with a
> publication accessible online in a reasonably interactive manner; this is
> not a description of the void set, as both the Australian synchrotron and
> Diamond e

Re: [ccp4bb] IUCr committees, depositing images

[Francis E Reyes]

This discussion of image deposition and archival has certainly been 
illuminating. While there have been clear directions on how the process would 
actually work, I am becoming increasingly curious on why it should be done 
(outside of threatening non publication, social acceptance, funding, etc).  I 
think a huge challenge is to convince the users that filling out the metadata 
with as much detail as possible is a worthwhile endeavor.


Specifically, are there documented cases where reinterpretation of an MX 
dataset  (the raw images) has resulted in new biological insight into the 
system being modeled?


If possible I'd like to compile a list (off list) for my own education (though 
I can certainly report my results to the BB if the interest is there).

(Ok I have a selfish reason as well as I will be considering the reanalysis of 
some really old datasets that proved troublesome for a colleague and am looking 
for small hints of wisdom)


Some that come to mind for me:

Meyer et al. Structure of the 12-subunit RNA polymerase II refined with the aid 
of anomalous diffraction data. J Biol Chem (2009) vol. 284 (19) pp. 12933-9

Wang. Inclusion of weak high-resolution X-ray data for improvement of a group 
II intron structure. Acta Crystallogr D Biol Crystallogr (2010) vol. 66 (Pt 9) 
pp. 988-1000

F



-
Francis E. Reyes M.Sc.
215 UCB
University of Colorado at Boulder

[ccp4bb] Job opening for a structural biologist to join the “Biocrystallography and Structural Biology of Therapeutic Targets” group at the Institute for the Biology and Chemistry of Proteins, Lyon in

[Nushin Aghajari]

Job opening for a structural biologist to join the “Biocrystallography 
and Structural Biology of Therapeutic Targets” group at the Institute 
for the Biology and Chemistry of Proteins, Lyon in France.


A 2 year postdoctoral position with support from the French National 
Research Agency (ANR) is available to study regulatory mechanisms of 
BMP-1/tolloid like proteinases ((BTPs; also known as procollagen 
C-proteinases) which are enzymes implicated in a number of key events 
during tissue morphogenesis and tissue repair. The research project 
proposed aims at a better understanding of the mechanism by which 
specific enhancer proteins regulate BTP activity, by solving the 3D 
structure of the complex of the substrate:enhancer complex.
Our studies involve a multidisciplinary approach, which combines 
molecular- and structural biology along with classical biochemical 
procedures.
A Ph.D. in biochemistry or biophysics, and a solid experience in 
crystallization and X-ray structure determination are required.
Background knowledge in protein expression and purification is highly 
desirable.


Salary in the range 30-35 k€, depending on experience.

The position is immediately available and interested candidates should 
send their CV, a letter of interest and contact information of 3 
references to Dr. Nushin Aghajari; n.aghaj...@ibcp.fr and Dr. David 
Hulmes; d.hul...@ibcp.fr

Re: [ccp4bb] IUCr committees, depositing images

[Martin M. Ripoll]

Dear David, 

 

As you probably have read in my previous message to Colin, I also agreed
with him. 

 

I just wanted to clarify (and obviously not to him) what George said. It was
just a comment and not a lesson! Sorry if my message was misunderstood!

 

Martin



Dr. Martin Martinez-Ripoll

Research Professor

  xmar...@iqfr.csic.es

Department of Crystallography & Structural Biology

  www.xtal.iqfr.csic.es

Telf.: +34 917459550

Consejo Superior de Investigaciones Científicas

Spanish National Research Council

www.csic.es

  Sin título-1

 

De: David Waterman [mailto:dgwater...@gmail.com] 
Enviado el: jueves, 27 de octubre de 2011 12:05
Para: Martin M. Ripoll
CC: CCP4BB@jiscmail.ac.uk
Asunto: Re: [ccp4bb] IUCr committees, depositing images

 

I agree with Colin here. Framing is simply a process of sampling an original
signal at some 'frequency' (related to the phi-width of each frame). At some
point, delta phi is small enough that the original signal is oversampled,
and can be reconstructed _within the bounds of noise_. Beyond that point I
see no advantage to sampling finer - and certainly not going to the limit of
representing your data in some unframed continuous readout form.

 

Perhaps I am missing something, and I realise this is another OT diversion
from this most fruitful of threads.

 

Cheers

-- David



On 27 October 2011 08:55, Martin M. Ripoll  wrote:

Dear Colin,

I think you understood perfectly what George was saying regarding the loss
of information, but he will probably answer better than I.

In any case, and for the ones that did not understand it, what George was
telling is related to the fact that a data collection made with a continuous
crystal rotation contains more information than when this information is
transformed into frames... The loss of information that we are referring to
has the same meaning as when we calculate electron density maps with
different grid sizes. The finer the grid, the greater is the information on
the map.

But you are right saying that the shorter the interval between produced
frames, the lower the loss of information. However, the procedure that you
are suggesting should have some limits... otherwise the amount of
information would grow dramatically.

All the best,

Martin

Dr. Martin Martinez-Ripoll
Research Professor
xmar...@iqfr.csic.es
Department of Crystallography & Structural Biology
www.xtal.iqfr.csic.es
Telf.: +34 917459550  
Consejo Superior de Investigaciones Científicas
Spanish National Research Council
www.csic.es


-Mensaje original-

De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de Colin
Nave
Enviado el: jueves, 27 de octubre de 2011 0:49

Para: CCP4BB@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images

Dear George, Martin

I don't understand the point that one is throwing away information by
storing in frames. If the frames have sufficiently fine intervals (given by
some sampling theorem consideration) I can't see how one loses information.
Can one of you explain?
Thanks
Colin



-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Martin
M. Ripoll
Sent: 26 October 2011 22:50
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images

Dear George, dear all,

I was just trying to summarize my point of view regarding this important
issue when I got your e-mail, that reflects exactly my own opinion!

Martin

Dr. Martin Martinez-Ripoll
Research Professor
xmar...@iqfr.csic.es
Department of Crystallography & Structural Biology
www.xtal.iqfr.csic.es
Telf.: +34 917459550  
Consejo Superior de Investigaciones Científicas
Spanish National Research Council
www.csic.es



-Mensaje original-
De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de George
M. Sheldrick
Enviado el: miércoles, 26 de octubre de 2011 11:52
Para: CCP4BB@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images

This raises an important point. The new continuous readout detectors such as
the
Pilatus for beamlines or the Bruker Photon for in-house use enable the
crystal to
be rotated at constant velocity, eliminating the mechanical errors
associated with
'stop and go' data collection. Storing their data in 'frames' is an
artifical
construction that is currently required for the established data integration
programs but is in fact throwing away information. Maybe in 10 years time
'frames'
will be as obsolete as punched cards!

George

On Wed, Oct 26, 2011 at 09:39:40AM +0100, Graeme Winter wrote:
> Hi James,
>
> Just to pick up on your point about the Pilatus detectors. Yesterday
> in 2 hours of giving a beamline a workout (admittedly with Thaumatin)
> we acquired 400 + GB of data*. Now I appreciate that this is not
> really routine operation,

Re: [ccp4bb] IUCr committees, depositing images

[Gerard Bricogne]

Dear all, 

 In response to my message below, posted yesterday afternoon (GMT+1) I
received an off-list message from John Helliwell asking me to clarify what I
had meant by "already stored information" when referring to the outline of a
feasibility study envisaged by the IUCr Forum. I answered him, with Cc to
Tom Terwilliger and Brian McMahon. Tom then wrote to me to suggest that I
post my reply to this BB, and John agreed to it - so I include it below. 

 I hope it will serve to illustrate the possibility of undertaking
something concrete right away. It was written hurriedly, and is not meant to
be a bullet-proof proposal.


 With best wishes,
 
  Gerard.

===
Dear John,

 Thank you for this message off-list.
 
 What I was referring to is the combination of two ideas that emerged
during our discussions in Madrid as the basis for a pilot project - not yet
a final solution:

 (1) the identification of every dataset collected on a synchrotron and
stored locally could be implemented by assigning it a unique doi - that was
discussed with Alun, the chap from the Australian synchrotron, Brian, John
Westbrook etc. at the meeting;

 (2) since the synchrotrons already store all their user data for some
time, and as several of them intend to make them publicly available after a
statutory period, it seemed a rather minor and inexpensive extra step to
promote the small fraction of datasets actually associated with a
publication to on-line storage, still at synchrotron sources, for the
purpose of making them accessible to the public as a sort of extension of
the present structure factor file.

 This initial experiment would therefore involve "federating" the local
storage facilities of a few synchrotrons interested in participating in this
pilot project (or feasibility study), with a special treatment for those
datasets, but nothing that would be inordinately onerous in terms of extra
costs. The relevant doi would then appear in the pdb entry and would enable
interested users to access the raw images for their own purposes. The public 
disclosure of image sets would also be at the option of the authors of a pdb
entry, not compulsory at this stage.

 Starting things in this way would enable a number of critical
procedures to be tested and of key parameters to be estimated, and in
particular to start storing those datasets before that storage can be passed
on to a more central entity, and before deposition is made compulsory by the
journals' own policies.

 I hope this is not too confused a picture. Its main ingredients are
 
 1. a number of synchrotrons willing to put in the extra resources into
assigning those unique doi's to datasets collected there, which they archive
anyway, and into making those that are eventually associated with a
publication accessible online in a reasonably interactive manner; this is
not a description of the void set, as both the Australian synchrotron and
Diamond expressed interest;

 2. a fraction of the creators of MX pdb entries willing to have the
images associated to a pdb entry made available to the public in this way;

 3. a participation of the PDB and the IUCr in the supervision, or even
coordination, of this pilot project, with a view to examining the logistics
involved and coming up with realistic, "evidence-based" estimates of costs
and benefits, as the basis for subsequently making the hard decisions about
gathering the necessary resources to implement the best form of archiving
(central vs. federated), and deciding whether to make the deposition of raw
data compulsory or not.

 The expression "evidence-based" is crucial here: without starting to do
something on a small scale with a subset of willing partners, we will
continue to discuss things "in abstracto" for ever, as has been happening in
some of the contributions to this thread. The true costs will only be known
by actually doing something, and this is even more true about the benefits,
i.e. some developers showing that re-exploiting the raw images with new
processing approaches can deliver better results than those originally
deposited. With such evidence, getting the extra support needed would be
much more likely than if it was applied for in the present situation, where
only opinions are available. So the "let's do it" argument seems to me
irresistible.

 I hope, of course, that I haven't misinterpreted nor overinterpreted
what was discussed in Madrid. It was a pity that so many of the influential
people with various commission-related titles were only able to stay for the
beginning of that meeting, that was quite formal and lacking in any form of
ground-breaking discussion. I can guarantee that I was *increasingly awake*
as the meeting proceeded, and that the principles I outlined above were
actually discussed, even if some of them might have been only partially
recorded

Re: [ccp4bb] IUCr committees, depositing images

[David Waterman]

I agree with Colin here. Framing is simply a process of sampling an original
signal at some 'frequency' (related to the phi-width of each frame). At some
point, delta phi is small enough that the original signal is oversampled,
and can be reconstructed _within the bounds of noise_. Beyond that point I
see no advantage to sampling finer - and certainly not going to the limit of
representing your data in some unframed continuous readout form.

Perhaps I am missing something, and I realise this is another OT diversion
from this most fruitful of threads.

Cheers

-- David


On 27 October 2011 08:55, Martin M. Ripoll  wrote:

> Dear Colin,
>
> I think you understood perfectly what George was saying regarding the loss
> of information, but he will probably answer better than I.
>
> In any case, and for the ones that did not understand it, what George was
> telling is related to the fact that a data collection made with a
> continuous
> crystal rotation contains more information than when this information is
> transformed into frames... The loss of information that we are referring to
> has the same meaning as when we calculate electron density maps with
> different grid sizes. The finer the grid, the greater is the information on
> the map.
>
> But you are right saying that the shorter the interval between produced
> frames, the lower the loss of information. However, the procedure that you
> are suggesting should have some limits... otherwise the amount of
> information would grow dramatically.
>
> All the best,
> Martin
> 
> Dr. Martin Martinez-Ripoll
> Research Professor
> xmar...@iqfr.csic.es
> Department of Crystallography & Structural Biology
> www.xtal.iqfr.csic.es
> Telf.: +34 917459550
> Consejo Superior de Investigaciones Científicas
> Spanish National Research Council
> www.csic.es
>
>
> -Mensaje original-
> De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de Colin
> Nave
> Enviado el: jueves, 27 de octubre de 2011 0:49
> Para: CCP4BB@JISCMAIL.AC.UK
> Asunto: Re: [ccp4bb] IUCr committees, depositing images
>
> Dear George, Martin
>
> I don't understand the point that one is throwing away information by
> storing in frames. If the frames have sufficiently fine intervals (given by
> some sampling theorem consideration) I can't see how one loses information.
> Can one of you explain?
> Thanks
> Colin
>
>
>
> -Original Message-
> From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of
> Martin
> M. Ripoll
> Sent: 26 October 2011 22:50
> To: ccp4bb
> Subject: Re: [ccp4bb] IUCr committees, depositing images
>
> Dear George, dear all,
>
> I was just trying to summarize my point of view regarding this important
> issue when I got your e-mail, that reflects exactly my own opinion!
>
> Martin
> 
> Dr. Martin Martinez-Ripoll
> Research Professor
> xmar...@iqfr.csic.es
> Department of Crystallography & Structural Biology
> www.xtal.iqfr.csic.es
> Telf.: +34 917459550
> Consejo Superior de Investigaciones Científicas
> Spanish National Research Council
> www.csic.es
>
>
>
> -Mensaje original-
> De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de George
> M. Sheldrick
> Enviado el: miércoles, 26 de octubre de 2011 11:52
> Para: CCP4BB@JISCMAIL.AC.UK
> Asunto: Re: [ccp4bb] IUCr committees, depositing images
>
> This raises an important point. The new continuous readout detectors such
> as
> the
> Pilatus for beamlines or the Bruker Photon for in-house use enable the
> crystal to
> be rotated at constant velocity, eliminating the mechanical errors
> associated with
> 'stop and go' data collection. Storing their data in 'frames' is an
> artifical
> construction that is currently required for the established data
> integration
> programs but is in fact throwing away information. Maybe in 10 years time
> 'frames'
> will be as obsolete as punched cards!
>
> George
>
> On Wed, Oct 26, 2011 at 09:39:40AM +0100, Graeme Winter wrote:
> > Hi James,
> >
> > Just to pick up on your point about the Pilatus detectors. Yesterday
> > in 2 hours of giving a beamline a workout (admittedly with Thaumatin)
> > we acquired 400 + GB of data*. Now I appreciate that this is not
> > really routine operation, but it does raise an interesting point - if
> > you have loaded a sample and centred it, collected test shots and
> > decided it's not that great, why not collect anyway as it may later
> > prove to be useful?
> >
> > Bzzt. 2 minutes or less later you have a full data set, and barely
> > even time to go get a cup of tea.
> >
> > This does to some extent move the goalposts, as you can acquire far
> > more data than you need. You never know, you may learn something
> > interesting from it - perhaps it has different symmetry or packing?
> > What it does mean is if we can have a method of tagging this data
> > there may be massively more opportunity to get also-ran data sets for
> > methods de

Re: [ccp4bb] IUCr committees, depositing images

[Alun Ashton]

Hi James,

1) thanks for sending and email in this thread longer than mine, I was 
"worried" I had killed it... ;)

2)  you say:
> Of course, if we are willing to relax the requirement of validation and
> curation, this could be a whole lot easier.  In fact, there is already
> an image deposition infrastructure in place!  It is called TARDIS:
> 
> http://tardis.edu.au/
> 
> Perhaps the best way forward would be for "the PDB" to introduce a new
> field for one or more TARDIS ids in a PDB deposition?  It would be
> optional at the first, but no doubt required in the future.

And Ethan has just beat me to this point, from the Tardis web site, first 
paragraph last sentence:
"Storage was and remains federated, meaning the public index, TARDIS.edu.au 
contains no data itself and merely points to data stored in external labs and 
institutions."

So is the raw data even public? In that sense I think what were supposed to 
adopt in European facilities is ODI's which link accordingly to the facilities 
own suppository, and how that is developed and where you put it is down to 
regional preferences. It can obviously be large, small, publicly accessible, 
shared between friends or private. To be effective it will need to be around 
for a long time. 

But to build on the success of model for the 'PDB' I would agree that someone 
should pay to have someone host this, and at least in the first instance make 
sure PDB structures have their raw data available and accessible to all. Tardis 
is in the right direction as a catalogue, ICAT in the EU is a simpler DB, ISPyB 
would need a bit of work to get data sharable through the interfaces, but I 
think it is a richer data structure. 

Standardisation of the data would also be good, its amazing what a simple word 
checker can do to your emails these days

BTW we pay the equivalent to about a weeks beamtime on one beamline for 200TB a 
year data storage and access to it for the whole facility, and someone else to 
take the pain of hosting it 

Alun
___
Alun Ashton, alun.ash...@diamond.ac.uk Tel: +44 1235 778404
Scientific Software Team Leader,  http://www.diamond.ac.uk/
Diamond Light Source, Chilton, Didcot, Oxon, OX11 0DE, U.K.

[ccp4bb] PhD Position / Helmholtz Centre for Infection Research

[Andrea Scrima]

Applications are invited for a PhD position in Protein Crystallography and
Biochemistry at the Helmholtz-Centre for Infection Research, Braunschweig,
Germany.





Topic:

Structural characterization of protein complexes involved in the regulation
of autophagy





Methods:

protein expression and purification,  X-ray crystallography, biophysical
methods (ITC, fluorescence, Biacore...), molecular biology





The Helmholtz Centre for Infection Research:

More than 600 people are employed at the Helmholtz Centre for Infection
Research (HZI) in Braunschweig, a research institution in the Helmholtz
Association of German Research Centres funded jointly by the Federal
Republic of Germany and the Federal State of Lower Saxony. The Department
of Molecular and Structural Biology offers excellent working conditions,
state-of-the-art technical platforms and equipment (including bacterial,
insect cell and mammalian protein expression systems, access to rotating
anode X-ray source and synchrotron), a lively scientific community and
excellent support for the development of personal soft-skills. The Centre’s
task is to undertake biomedical research in the field of infection biology
to expand our knowledge of the fundamental mechanisms behind medically
relevant infectious diseases with the aim of providing the key for the
development of new therapeutics and vaccines.





For further information and details of how to apply, please visit:

http://www.helmholtz-hzi.de/de/job_und_karriere/bachelor_master_und_doktorarbeiten/stellenangebote/masters_and_thesis_detail/job/details/532011-1/





Application deadline:

06.11.2011

Re: [ccp4bb] IUCr committees, depositing images

[Martin M. Ripoll]

Dear Colin,

I think you understood perfectly what George was saying regarding the loss
of information, but he will probably answer better than I. 

In any case, and for the ones that did not understand it, what George was
telling is related to the fact that a data collection made with a continuous
crystal rotation contains more information than when this information is
transformed into frames... The loss of information that we are referring to
has the same meaning as when we calculate electron density maps with
different grid sizes. The finer the grid, the greater is the information on
the map. 

But you are right saying that the shorter the interval between produced
frames, the lower the loss of information. However, the procedure that you
are suggesting should have some limits... otherwise the amount of
information would grow dramatically.

All the best,
Martin

Dr. Martin Martinez-Ripoll
Research Professor
xmar...@iqfr.csic.es
Department of Crystallography & Structural Biology
www.xtal.iqfr.csic.es
Telf.: +34 917459550
Consejo Superior de Investigaciones Científicas
Spanish National Research Council
www.csic.es


-Mensaje original-
De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de Colin
Nave
Enviado el: jueves, 27 de octubre de 2011 0:49
Para: CCP4BB@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images

Dear George, Martin

I don't understand the point that one is throwing away information by
storing in frames. If the frames have sufficiently fine intervals (given by
some sampling theorem consideration) I can't see how one loses information.
Can one of you explain?
Thanks
Colin



-Original Message-
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Martin
M. Ripoll
Sent: 26 October 2011 22:50
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images

Dear George, dear all,

I was just trying to summarize my point of view regarding this important
issue when I got your e-mail, that reflects exactly my own opinion!

Martin

Dr. Martin Martinez-Ripoll
Research Professor
xmar...@iqfr.csic.es
Department of Crystallography & Structural Biology
www.xtal.iqfr.csic.es
Telf.: +34 917459550
Consejo Superior de Investigaciones Científicas
Spanish National Research Council
www.csic.es



-Mensaje original-
De: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] En nombre de George
M. Sheldrick
Enviado el: miércoles, 26 de octubre de 2011 11:52
Para: CCP4BB@JISCMAIL.AC.UK
Asunto: Re: [ccp4bb] IUCr committees, depositing images

This raises an important point. The new continuous readout detectors such as
the
Pilatus for beamlines or the Bruker Photon for in-house use enable the
crystal to 
be rotated at constant velocity, eliminating the mechanical errors
associated with
'stop and go' data collection. Storing their data in 'frames' is an
artifical
construction that is currently required for the established data integration
programs but is in fact throwing away information. Maybe in 10 years time
'frames' 
will be as obsolete as punched cards!

George

On Wed, Oct 26, 2011 at 09:39:40AM +0100, Graeme Winter wrote:
> Hi James,
> 
> Just to pick up on your point about the Pilatus detectors. Yesterday
> in 2 hours of giving a beamline a workout (admittedly with Thaumatin)
> we acquired 400 + GB of data*. Now I appreciate that this is not
> really routine operation, but it does raise an interesting point - if
> you have loaded a sample and centred it, collected test shots and
> decided it's not that great, why not collect anyway as it may later
> prove to be useful?
> 
> Bzzt. 2 minutes or less later you have a full data set, and barely
> even time to go get a cup of tea.
> 
> This does to some extent move the goalposts, as you can acquire far
> more data than you need. You never know, you may learn something
> interesting from it - perhaps it has different symmetry or packing?
> What it does mean is if we can have a method of tagging this data
> there may be massively more opportunity to get also-ran data sets for
> methods development types. What it also means however is that the cost
> of curating this data is then an order of magnitude higher.
> 
> Also moving it around is also rather more painful.
> 
> Anyhow, I would try to avoid dismissing the effect that new continuous
> readout detectors will have on data rates, from experience it is
> pretty substantial.
> 
> Cheerio,
> 
> Graeme
> 
> *by "data" here what I mean is images, rather than information which
> is rather more time consuming to acquire. I would argue you get that
> from processing / analysing the data...
> 
> On 24 October 2011 22:56, James Holton  wrote:
> > The Pilatus is fast, but or decades now we have had detectors that can
read
> > out in ~1s.  This means that you can collect a typical ~100 image
dataset in
> > a few minutes (if flux is not limiting).  Since there are ~150 beamlines

Re: [ccp4bb] COOT not "connected" to PHENIX

[Pavel Afonine]

Hi,


This is why the right way to distribute *nix software is to distribute
> software itself and ask the end-user to get all the dependencies (not
> that hard these days).



it is a wonderful idea, apart from two tiny nuances: 1) many have no idea
about what the command line is, and 2) asking users going through the
trouble of installing dependencies will scare them away from the
software. Personally, I do not use programs that do not work right out of
the box or take more than a minute to make it run.

So, I would rather let the majority use whatever works, and let a few gurus
count versions of Python in various software and how much space they take.

Pavel

Re: [ccp4bb] IUCr committees, depositing images

[Ethan Merritt]

On Wednesday, 26 October 2011, James Holton wrote:
> Of course, if we are willing to relax the requirement of validation and 
> curation, this could be a whole lot easier.  In fact, there is already 
> an image deposition infrastructure in place!  It is called TARDIS:
> 
> http://tardis.edu.au/
> 
> Perhaps the best way forward would be for "the PDB" to introduce a new 
> field for one or more TARDIS ids in a PDB deposition?  It would be 
> optional at the first, but no doubt required in the future.

As I understand it, TARDIS is just an indexing system.
You're still on your own to actually store the images.
The TARDIS setup will cough up the information that your images
are stored on a machine named pony.lbl.gov, or at least they were 
at the time you registered them for indexing, where they supposedly
can be retrieved using tag #XYZ.

But so far as I know you would still be at the mercy of pony.lbl.gov
going up in flames, or being renamed twinkle.lbl.gov, or being 
decommissioned when the next budget crunch hits.

For that matter, I don't know what the provision or expectation
is that anyone outside your institution could see or access the
machine holding the set of files that TARDIS told them were there.

If I've got this wrong, perhaps Ashley Buckle can chime in with
an update on TARDIS.

Ethan

Re: [ccp4bb] IUCr committees, depositing images

[Alun Ashton]

Sorry Matt, some large facilities do already keep all their raw and processed 
data. And I think the EU grant you mention to coordinate this is 
http://www.pan-data.eu  soon to be odi, includes the ESRF :), don't your 
computing people tell you anything ?! :)

PanData have a meeting in early November and they (ahem we) are already in 
touch with the working group and will formalise that soon after the meeting.

Alun
___
Alun Ashton, alun.ash...@diamond.ac.uk Tel: +44 1235 778404
Scientific Software Team Leader,  http://www.diamond.ac.uk/
Diamond Light Source, Chilton, Didcot, Oxon, OX11 0DE, U.K.
From: CCP4 bulletin board [mailto:CCP4BB@JISCMAIL.AC.UK] On Behalf Of Matthew 
BOWLER
Sent: 26 October 2011 10:03
To: ccp4bb
Subject: Re: [ccp4bb] IUCr committees, depositing images

The archiving of all raw data and subsequently making it public is something 
that the large facilities are currently debating whether to do.  Here at the 
ESRF we store user data for only 6 months (and I believe that it is available 
longer on tape) and we already have trouble with capacity.  My personal view is 
that facilities should take the lead on this - for MX we already have a very 
good archiving system - ISPyB - also running at Diamond.  ISPyB stores lots of 
meta data and jpgs of the raw images but not the images themselves but a link 
to the location of the data with an option to download if still available.  My 
preferred option would be to store all academically funded data and then make 
it publicly available after say 2-5 years (this will no doubt spark another 
debate on time limits, special dispensation etc).  What needs to be thought 
about is how to order the data and how to make sure that the correct meta data 
are stored with each data set - this will rely heavily on user input at the 
time of the experiment rather than gathering together data sets for depositions 
much later.  As already mentioned, this type of resource could be extremely 
useful for developers and also as a general scientific resource.  Smells like 
an EU grant to me. Cheers, Matt.


On 26/10/2011 10:21, Frank von Delft wrote:
Since when has the cost of any project been limited by the cost of hardware?  
Someone has to implement this -- and make a career out of it;  thunderingly 
absent from this thread has been the chorus of volunteers who will write the 
grant.
phx





--

Matthew Bowler

Structural Biology Group

European Synchrotron Radiation Facility

B.P. 220, 6 rue Jules Horowitz

F-38043 GRENOBLE CEDEX

FRANCE

===

Tel: +33 (0) 4.76.88.29.28

Fax: +33 (0) 4.76.88.29.04



http://go.esrf.eu/MX

http://go.esrf.eu/Bowler

===