Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[James Holton]

Ahh, waters.  Where would structure-related debate be without them?

I see. So if your default refinement procedure is to add an unspecified 
number of waters, then yes Rwork might not be all that useful, as it 
will depend on how the building goes.


Again, it all depends on what you want your data to do.  If you are 
looking for subtle difference features, such as a bound ligand, 
mutation, etc. then clear identification of weak density should be your 
"score".  So, I say:

1) pick some weak density
2) omit the model under it
3) refine to convergence
4) measure the Fo-Fc difference peak
  I tend to use the MAPMAN "peek" function for this, but I'm sure there 
are other ways.  I say use the SAME starting model each time, one where 
you have built in the obvious waters, ions, etc, but borderline or 
otherwise inconsistent ones, leave them out.  Then pick a low-lying 
feature as your test density.


Do not use molecular replacement. Use pointless with your starting model 
on "xyzin" to re-index the data so that it matches the model. Super fast 
and easy to do. No origin issues, and it doesn't modify the pdb.


Aside:  I actually have a program for removing unneeded waters I call 
"watershed".  It is not fast, but it is thorough, and you only need to 
do it for your reference structure. You will need these programs:

https://github.com/fraser-lab/holton_scripts/tree/master/watershed
https://github.com/fraser-lab/holton_scripts/blob/master/converge_refmac.com
 a pdb, an mtz, and a file called refmac_opts.txt that contains all the 
refmac5 keywords you want to use (if any).  You will also want a lot of 
CPUs, and the script works with the PBS, and SGE clusters I have access 
to (and I'm working on Slurm). What watershed does is delete waters one 
at a time and re-refine to convergence.  Also, as a control, you want to 
refine the starting structure for the same number of cycles. Each "minus 
one water" structure gets its own CPU. Once everything settles, you look 
at the final Rwork values. If deleting a water ends up making Rwork 
better? ... then you probably shouldn't have built it in the first 
place. That water is evil and must go. After throwing out the worst 
water, you now have a new starting point. In some published structures 
more than 100 waters can be eliminated this way. Almost always brings 
Rwork and Rfree closer together, even though Rfree does not enter into 
any automated decisions.


 Using simulated data (where I know the ground truth) I find the 
watershed procedure tends to un-do all the horrible things that happen 
after you get over-aggressive and stuff waters into every little peak 
you see. Eventually, as you add more noise waters, Rfree starts to go 
up, and the map starts to look less like the ground truth, but Rwork 
keeps going down the more waters you add.  What watershed does pretty 
reliably is bring you back to the point just before where Rfree started 
to take a turn for the worse, and you can do this without ever looking 
at Rfree!


Of course, it is always better to not put in bad waters in the first 
place, but sometimes its hard to tell.


Anyway, I suggest using a watershed-ed model as your reference.

Hope that is helpful in some way?

-James Holton
MAD Scientist


On 11/2/2021 5:01 PM, Murpholino Peligro wrote:

That's exactly what I am doing...
citing David...

"I expect the A and B data sets to be quite similar, but I would like 
to evaluate which protocol was "better", and I want to do this 
quickly, ideally looking at a single number."


and

"I do want to find a way to assess the various tweaks I can try in 
data processing for a single case"


Why not do all those things with Rwork?
I thought that comparing the R-free rather than the R-work was going 
to be easier Because last week the structure was dehydrated So 
the refinement program added "strong waters" and due to a thousand or 
so extra reflections I could have a dozen or so extra waters and the 
difference in R-work value between protocols due to extra waters was 
going to be a little bit more difficult to compare. I have now the 
final structure so I could very well compare the R-work doing another 
round of refinement, maybe randomizing adps at the beginning or 
something.


Thanks a lot.









El lun, 1 de nov. de 2021 a la(s) 03:22, David Waterman 
(dgwater...@gmail.com) escribió:


Hi James,

What you wrote makes lots of sense. I had not heard about Rsleep,
so that looks like interesting reading, thanks.

I have often used Rfree as a simple tool to compare two protocols.
If I am not actually optimising against Rfree but just using it
for a one-off comparison then that is okay, right?

Let's say I have two data processing protocols, A and B. Between
these I might be exploring some difference in options within one
data processing program, perhaps different geometry refinement
parameters, or scaling options. I expect the A and B data sets to
be 

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Ian Tickle]

Hi Clemens

OK so you're saying use only the reflections that are in common between all
datasets and keep the parameterisation the same.  There are clearly two
distinctly different ways in which datasets can differ: either a different
set of indices due to different resolution cut-offs or completeness, and/or
as you say different sets of values of Iobs/Fobs for the same set of
indices.  In practice of course it's very likely to be a mixture of the two
cases.  Clearly your suggestion could be of help in identifying the "best"
dataset in the latter case, but not in the former case, since it's not even
looking at the reflections that are not in common, so there's no way of
knowing whether they would have improved the map had they been used
(without obviously eye-balling the map!).

I just have an uncomfortable feeling about using a data-derived model to
judge the quality of the data: it sounds like a circular argument to me!
IMO there should be two completely distinct steps: first decide on the
"best" data using data-related criteria alone (Rmeas, CChalf, resolution,
whatever), then with this "best" dataset decide on the "best" model using
the model selection metrics (Rfree, free likelihood and their relatives).

Very interesting discussion!

Cheers

-- Ian


On Thu, 4 Nov 2021 at 16:00, Clemens Vonrhein 
wrote:

> On Wed, Nov 03, 2021 at 12:54:00PM +, Ian Tickle wrote:
> > Suppose you had to compare two datasets differing only in their
> > high-resolution cut-offs.  Now Rwork, Rfree and Rall will inevitably have
> > higher values at the high d* end, which means that if you apply a cut-off
> > at the high d* end all the overall R values will get smaller, so use of
> any
> >  R value as a data-quality metric will tend to result in selection of the
> > lowest-resolution dataset from your candidate set, which may well give a
> lower
> > quality map: not what you want!
>
> Couldn't that be avoided by using the common set of reflection data
> present in both the A and B dataset (from David's example - or even
> across a whole series A-Z)? When at the same time we keep the same
> model parametrisation (i.e. not adding altConfs or waters, keeping
> same TLS definitions etc) it might be useful to compare different data
> processing procedures via the R{free,all,work} during refinement.
>
> As far as I can see, the only difference would be the actual values of
> F and sig(F) (or I and sig(I)) attached to the identical set of Miller
> indices ... right?
>
> Cheers
>
> Clemens
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Clemens Vonrhein]

On Wed, Nov 03, 2021 at 12:54:00PM +, Ian Tickle wrote:
> Suppose you had to compare two datasets differing only in their
> high-resolution cut-offs.  Now Rwork, Rfree and Rall will inevitably have
> higher values at the high d* end, which means that if you apply a cut-off
> at the high d* end all the overall R values will get smaller, so use of any
>  R value as a data-quality metric will tend to result in selection of the
> lowest-resolution dataset from your candidate set, which may well give a lower
> quality map: not what you want!

Couldn't that be avoided by using the common set of reflection data
present in both the A and B dataset (from David's example - or even
across a whole series A-Z)? When at the same time we keep the same
model parametrisation (i.e. not adding altConfs or waters, keeping
same TLS definitions etc) it might be useful to compare different data
processing procedures via the R{free,all,work} during refinement.

As far as I can see, the only difference would be the actual values of
F and sig(F) (or I and sig(I)) attached to the identical set of Miller
indices ... right?

Cheers

Clemens



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Ian Tickle]

Hi, whilst I completely concur with James that Rfree is not a suitable
metric for this purpose for all the reasons he mentioned, it's not clear to
me that Rwork is much better.  If you really want to go down that route, even
better would be Rall, i.e. ignoring the free R flags, though I realise that
some programs use the test set to obtain unbiased sigmaA values, so that
may not be practical.  IMO Rall is hardly better for this purpose anyway.

None of the refinement R values are truly suitable for this purpose,
basically because they are strictly only "model selection" metrics (e.g.
see https://en.wikipedia.org/wiki/Model_selection?wprov=sfla1), i.e.
metrics used to select a mathematical model (a specific parameterisation of
the atomic model such as overall B, isotropic Bs, TLS, anisotropic tensors
etc.) from a set of candidate models, importantly assuming that the
identical data are used in each comparison.  Obviously, if both data and
model are changed one cannot sensibly perform comparisons.  Also because of
the danger of overfitting one would obviously not use Rwork or Rall for model
selection, rather Rfree in cross-validation.

Suppose you had to compare two datasets differing only in their
high-resolution cut-offs.  Now Rwork, Rfree and Rall will inevitably have
higher values at the high d* end, which means that if you apply a cut-off
at the high d* end all the overall R values will get smaller, so use of any
 R value as a data-quality metric will tend to result in selection of the
lowest-resolution dataset from your candidate set, which may well give a lower
quality map: not what you want!

I was faced with exactly this problem when designing an automated pipeline
to accept large numbers of fragment-screening datasets coming from the
synchrotrons, which tend to output a number of datasets for each crystal
processed in different ways.  If only the synchrotrons made the decision
for me and gave me only "the best" dataset (either unmerged or scaled and
merged) for each crystal !  So I needed a quick-and-dirty solution, and
obviously no atomic model was available at that stage in the processing, so
refinement R values were out of the question as a metric anyway.

All I did as a quick-and-dirty solution was to take the highest resolution
dataset from each crystal as the "best" one after applying appropriate standard
cut-offs based on mean I/sd(I), completeness and Rmeas.  So basically I was
using resolution as a data-quality metric, which makes a lot of sense to
me.  However, given a refined atomic model then the slow-and-clean method
would clearly be to examine the density, as others have suggested.

Cheers

-- Ian

On Wed, 3 Nov 2021, 00:05 Murpholino Peligro,  wrote:

> That's exactly what I am doing...
> citing David...
>
> "I expect the A and B data sets to be quite similar, but I would like to
> evaluate which protocol was "better", and I want to do this quickly,
> ideally looking at a single number."
>
> and
>
> "I do want to find a way to assess the various tweaks I can try in data
> processing for a single case"
>
> Why not do all those things with Rwork?
> I thought that comparing the R-free rather than the R-work was going to be
> easier Because last week the structure was dehydrated So the
> refinement program added "strong waters" and due to a thousand or so extra
> reflections I could have a dozen or so extra waters and the difference in
> R-work value between protocols due to extra waters was going to be a little
> bit more difficult to compare. I have now the final structure so I could
> very well compare the R-work doing another round of refinement, maybe
> randomizing adps at the beginning or something.
>
> Thanks a lot.
>
>
>
>
>
>
>
>
>
> El lun, 1 de nov. de 2021 a la(s) 03:22, David Waterman (
> dgwater...@gmail.com) escribió:
>
>> Hi James,
>>
>> What you wrote makes lots of sense. I had not heard about Rsleep, so that
>> looks like interesting reading, thanks.
>>
>> I have often used Rfree as a simple tool to compare two protocols. If I
>> am not actually optimising against Rfree but just using it for a one-off
>> comparison then that is okay, right?
>>
>> Let's say I have two data processing protocols, A and B. Between these I
>> might be exploring some difference in options within one data processing
>> program, perhaps different geometry refinement parameters, or scaling
>> options. I expect the A and B data sets to be quite similar, but I would
>> like to evaluate which protocol was "better", and I want to do this
>> quickly, ideally looking at a single number. I don't like I/sigI because I
>> don't trust the sigmas, CC1/2 is often noisy, and I'm totally sworn off
>> merging R statistics for these purposes. I tend to use Rfree as an
>> easily-available metric, independent from the data processing program and
>> the merging stats. It also allows a comparison of A and B in terms of the
>> "product" of crystallography, namely the refined structure. In this I am
>> lucky 

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Murpholino Peligro]

That's exactly what I am doing...
citing David...

"I expect the A and B data sets to be quite similar, but I would like to
evaluate which protocol was "better", and I want to do this quickly,
ideally looking at a single number."

and

"I do want to find a way to assess the various tweaks I can try in data
processing for a single case"

Why not do all those things with Rwork?
I thought that comparing the R-free rather than the R-work was going to be
easier Because last week the structure was dehydrated So the
refinement program added "strong waters" and due to a thousand or so extra
reflections I could have a dozen or so extra waters and the difference in
R-work value between protocols due to extra waters was going to be a little
bit more difficult to compare. I have now the final structure so I could
very well compare the R-work doing another round of refinement, maybe
randomizing adps at the beginning or something.

Thanks a lot.









El lun, 1 de nov. de 2021 a la(s) 03:22, David Waterman (
dgwater...@gmail.com) escribió:

> Hi James,
>
> What you wrote makes lots of sense. I had not heard about Rsleep, so that
> looks like interesting reading, thanks.
>
> I have often used Rfree as a simple tool to compare two protocols. If I am
> not actually optimising against Rfree but just using it for a one-off
> comparison then that is okay, right?
>
> Let's say I have two data processing protocols, A and B. Between these I
> might be exploring some difference in options within one data processing
> program, perhaps different geometry refinement parameters, or scaling
> options. I expect the A and B data sets to be quite similar, but I would
> like to evaluate which protocol was "better", and I want to do this
> quickly, ideally looking at a single number. I don't like I/sigI because I
> don't trust the sigmas, CC1/2 is often noisy, and I'm totally sworn off
> merging R statistics for these purposes. I tend to use Rfree as an
> easily-available metric, independent from the data processing program and
> the merging stats. It also allows a comparison of A and B in terms of the
> "product" of crystallography, namely the refined structure. In this I am
> lucky because I'm not trying to solve a structure. I may be looking at
> lysozyme or proteinase K: something where I can download a pretty good
> approximation to the truth from the PDB.
>
> So, what I do is process the data by A and process by B, ensure the data
> sets have the same free set, then refine to convergence (or at least, a lot
> of cycles) starting from a PDB structure. I then evaluate A vs B in terms
> of Rfree, though without an error bar on Rfree I don't read too much into
> small differences.
>
> Does this procedure seem sound? Perhaps it could be improved by randomly
> jiggling the atoms in the starting structure, in case the PDB deposition
> had already followed an A- or B-like protocol. Perhaps the whole approach
> is suspect. Certainly I wouldn't want to generalise by saying that A or B
> is better in all cases, but I do want to find a way to assess the various
> tweaks I can try in data processing for a single case.
>
> Any thoughts? I appreciate the wisdom of the BB here.
>
> Cheers
>
> -- David
>
>
> On Fri, 29 Oct 2021 at 15:50, James Holton  wrote:
>
>>
>> Well, of all the possible metrics you could use to asses data quality
>> Rfree is probably the worst one.  This is because it is a cross-validation
>> metric, and cross-validations don't work if you use them as an optimization
>> target. You can try, and might even make a little headway, but then your
>> free set is burnt. If you have a third set of observations, as suggested
>> for Rsleep (doi:10.1107/S0907444907033458), then you have a chance at
>> another round of cross-validation. Crystallographers don't usually do this,
>> but it has become standard practice in machine learning (training=Rwork,
>> validation=Rfree and testing=Rsleep).
>>
>> So, unless you have an Rsleep set, any time you contemplate doing a bunch
>> of random things and picking the best Rfree ... don't.  Just don't.  There
>> madness lies.
>>
>> What happens after doing this is you will be initially happy about your
>> lower Rfree, but everything you do after that will make it go up more than
>> it would have had you not performed your Rfree optimization. This is
>> because the changes in the data that made Rfree randomly better was
>> actually noise, and as the structure becomes more correct it will move away
>> from that noise. It's always better to optimize on something else, and then
>> check your Rfree as infrequently as possible. Remember it is the control
>> for your experiment. Never mix your positive control with your sample.
>>
>> As for the best metric to assess data quality?  Well, what are you doing
>> with the data? There are always compromises in data processing and
>> reduction that favor one application over another.  If this is a "I just
>> want the structure" project, then score on the 

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[James Holton]

Hi David,

Why not do all those things with Rwork? It is much less noisy than 
Rfree. Have you ever seen a case in such analysis where Rwork didn't 
tell you the same thing Rfree did?  If so, did you believe the difference?


Once when I was playing with lossy image compression if I picked just 
the right compression ratio I could get slightly better Rfree. But that 
is not something I'd recommend as a good idea.


-James Holton
MAD Scientist

On 11/1/2021 2:22 AM, David Waterman wrote:

Hi James,

What you wrote makes lots of sense. I had not heard about Rsleep, so 
that looks like interesting reading, thanks.


I have often used Rfree as a simple tool to compare two protocols. If 
I am not actually optimising against Rfree but just using it for a 
one-off comparison then that is okay, right?


Let's say I have two data processing protocols, A and B. Between these 
I might be exploring some difference in options within one data 
processing program, perhaps different geometry refinement parameters, 
or scaling options. I expect the A and B data sets to be quite 
similar, but I would like to evaluate which protocol was "better", and 
I want to do this quickly, ideally looking at a single number. I don't 
like I/sigI because I don't trust the sigmas, CC1/2 is often noisy, 
and I'm totally sworn off merging R statistics for these purposes. I 
tend to use Rfree as an easily-available metric, independent from the 
data processing program and the merging stats. It also allows a 
comparison of A and B in terms of the "product" of crystallography, 
namely the refined structure. In this I am lucky because I'm not 
trying to solve a structure. I may be looking at lysozyme or 
proteinase K: something where I can download a pretty good 
approximation to the truth from the PDB.


So, what I do is process the data by A and process by B, ensure the 
data sets have the same free set, then refine to convergence (or at 
least, a lot of cycles) starting from a PDB structure. I then evaluate 
A vs B in terms of Rfree, though without an error bar on Rfree I don't 
read too much into small differences.


Does this procedure seem sound? Perhaps it could be improved by 
randomly jiggling the atoms in the starting structure, in case the PDB 
deposition had already followed an A- or B-like protocol. Perhaps the 
whole approach is suspect. Certainly I wouldn't want to generalise by 
saying that A or B is better in all cases, but I do want to find a way 
to assess the various tweaks I can try in data processing for a single 
case.


Any thoughts? I appreciate the wisdom of the BB here.

Cheers

-- David


On Fri, 29 Oct 2021 at 15:50, James Holton  wrote:


Well, of all the possible metrics you could use to asses data
quality Rfree is probably the worst one.  This is because it is a
cross-validation metric, and cross-validations don't work if you
use them as an optimization target. You can try, and might even
make a little headway, but then your free set is burnt. If you
have a third set of observations, as suggested for Rsleep
(doi:10.1107/S0907444907033458), then you have a chance at another
round of cross-validation. Crystallographers don't usually do
this, but it has become standard practice in machine learning
(training=Rwork, validation=Rfree and testing=Rsleep).

So, unless you have an Rsleep set, any time you contemplate doing
a bunch of random things and picking the best Rfree ... don't. 
Just don't.  There madness lies.

What happens after doing this is you will be initially happy about
your lower Rfree, but everything you do after that will make it go
up more than it would have had you not performed your Rfree
optimization. This is because the changes in the data that made
Rfree randomly better was actually noise, and as the structure
becomes more correct it will move away from that noise. It's
always better to optimize on something else, and then check your
Rfree as infrequently as possible. Remember it is the control for
your experiment. Never mix your positive control with your sample.

As for the best metric to assess data quality?  Well, what are you
doing with the data? There are always compromises in data
processing and reduction that favor one application over another. 
If this is a "I just want the structure" project, then score on
the resolution where CC1/2 hits your favorite value. For some that
is 0.5, others 0.3. I tend to use 0.0 so I can cut it later
without re-processing. Whatever you do just make it consistent.

If its for anomalous, score on CCanom or if that's too noisy the
Imean/sigma in the lowest-angle resolution or highest-intensity
bin. This is because for anomalous you want to minimize relative
error. The end-all-be-all of anomalous signal strength is the
phased anomalous difference Fourier. You need phases to do one,
but if you have a structure just omit an 

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[David Waterman]

Hi James,

What you wrote makes lots of sense. I had not heard about Rsleep, so that
looks like interesting reading, thanks.

I have often used Rfree as a simple tool to compare two protocols. If I am
not actually optimising against Rfree but just using it for a one-off
comparison then that is okay, right?

Let's say I have two data processing protocols, A and B. Between these I
might be exploring some difference in options within one data processing
program, perhaps different geometry refinement parameters, or scaling
options. I expect the A and B data sets to be quite similar, but I would
like to evaluate which protocol was "better", and I want to do this
quickly, ideally looking at a single number. I don't like I/sigI because I
don't trust the sigmas, CC1/2 is often noisy, and I'm totally sworn off
merging R statistics for these purposes. I tend to use Rfree as an
easily-available metric, independent from the data processing program and
the merging stats. It also allows a comparison of A and B in terms of the
"product" of crystallography, namely the refined structure. In this I am
lucky because I'm not trying to solve a structure. I may be looking at
lysozyme or proteinase K: something where I can download a pretty good
approximation to the truth from the PDB.

So, what I do is process the data by A and process by B, ensure the data
sets have the same free set, then refine to convergence (or at least, a lot
of cycles) starting from a PDB structure. I then evaluate A vs B in terms
of Rfree, though without an error bar on Rfree I don't read too much into
small differences.

Does this procedure seem sound? Perhaps it could be improved by randomly
jiggling the atoms in the starting structure, in case the PDB deposition
had already followed an A- or B-like protocol. Perhaps the whole approach
is suspect. Certainly I wouldn't want to generalise by saying that A or B
is better in all cases, but I do want to find a way to assess the various
tweaks I can try in data processing for a single case.

Any thoughts? I appreciate the wisdom of the BB here.

Cheers

-- David


On Fri, 29 Oct 2021 at 15:50, James Holton  wrote:

>
> Well, of all the possible metrics you could use to asses data quality
> Rfree is probably the worst one.  This is because it is a cross-validation
> metric, and cross-validations don't work if you use them as an optimization
> target. You can try, and might even make a little headway, but then your
> free set is burnt. If you have a third set of observations, as suggested
> for Rsleep (doi:10.1107/S0907444907033458), then you have a chance at
> another round of cross-validation. Crystallographers don't usually do this,
> but it has become standard practice in machine learning (training=Rwork,
> validation=Rfree and testing=Rsleep).
>
> So, unless you have an Rsleep set, any time you contemplate doing a bunch
> of random things and picking the best Rfree ... don't.  Just don't.  There
> madness lies.
>
> What happens after doing this is you will be initially happy about your
> lower Rfree, but everything you do after that will make it go up more than
> it would have had you not performed your Rfree optimization. This is
> because the changes in the data that made Rfree randomly better was
> actually noise, and as the structure becomes more correct it will move away
> from that noise. It's always better to optimize on something else, and then
> check your Rfree as infrequently as possible. Remember it is the control
> for your experiment. Never mix your positive control with your sample.
>
> As for the best metric to assess data quality?  Well, what are you doing
> with the data? There are always compromises in data processing and
> reduction that favor one application over another.  If this is a "I just
> want the structure" project, then score on the resolution where CC1/2 hits
> your favorite value. For some that is 0.5, others 0.3. I tend to use 0.0 so
> I can cut it later without re-processing.  Whatever you do just make it
> consistent.
>
> If its for anomalous, score on CCanom or if that's too noisy the
> Imean/sigma in the lowest-angle resolution or highest-intensity bin. This
> is because for anomalous you want to minimize relative error. The
> end-all-be-all of anomalous signal strength is the phased anomalous
> difference Fourier. You need phases to do one, but if you have a structure
> just omit an anomalous scatterer of interest, refine to convergence, and
> then measure the peak height at the position of the omitted anomalous
> atom.  Instructions for doing anomalous refinement in refmac5 are here:
>
> https://www2.mrc-lmb.cam.ac.uk/groups/murshudov/content/refmac/refmac_keywords.html
>
> If you're looking for a ligand you probably want isomorphism, and in that
> case refining with a reference structure looking for low Rwork is not a bad
> strategy. This will tend to select for crystals containing a molecule that
> looks like the one you are refining.  But be careful! If it is an 

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[James Holton]

Well, of all the possible metrics you could use to asses data quality 
Rfree is probably the worst one.  This is because it is a 
cross-validation metric, and cross-validations don't work if you use 
them as an optimization target. You can try, and might even make a 
little headway, but then your free set is burnt. If you have a third set 
of observations, as suggested for Rsleep 
(doi:10.1107/S0907444907033458), then you have a chance at another round 
of cross-validation. Crystallographers don't usually do this, but it has 
become standard practice in machine learning (training=Rwork, 
validation=Rfree and testing=Rsleep).


So, unless you have an Rsleep set, any time you contemplate doing a 
bunch of random things and picking the best Rfree ... don't.  Just 
don't.  There madness lies.


What happens after doing this is you will be initially happy about your 
lower Rfree, but everything you do after that will make it go up more 
than it would have had you not performed your Rfree optimization. This 
is because the changes in the data that made Rfree randomly better was 
actually noise, and as the structure becomes more correct it will move 
away from that noise. It's always better to optimize on something else, 
and then check your Rfree as infrequently as possible. Remember it is 
the control for your experiment. Never mix your positive control with 
your sample.


As for the best metric to assess data quality?  Well, what are you doing 
with the data? There are always compromises in data processing and 
reduction that favor one application over another.  If this is a "I just 
want the structure" project, then score on the resolution where CC1/2 
hits your favorite value. For some that is 0.5, others 0.3. I tend to 
use 0.0 so I can cut it later without re-processing. Whatever you do 
just make it consistent.


If its for anomalous, score on CCanom or if that's too noisy the 
Imean/sigma in the lowest-angle resolution or highest-intensity bin. 
This is because for anomalous you want to minimize relative error. The 
end-all-be-all of anomalous signal strength is the phased anomalous 
difference Fourier. You need phases to do one, but if you have a 
structure just omit an anomalous scatterer of interest, refine to 
convergence, and then measure the peak height at the position of the 
omitted anomalous atom.  Instructions for doing anomalous refinement in 
refmac5 are here:

https://www2.mrc-lmb.cam.ac.uk/groups/murshudov/content/refmac/refmac_keywords.html

If you're looking for a ligand you probably want isomorphism, and in 
that case refining with a reference structure looking for low Rwork is 
not a bad strategy. This will tend to select for crystals containing a 
molecule that looks like the one you are refining.  But be careful! If 
it is an apo structure your ligand-bound crystals will have higher Rwork 
due to the very difference density you are looking for.


But if its the same data just being processed in different ways, first 
make a choice about what you are interested in, and then optimize on 
that.  just don't optimize on Rfree!


-James Holton
MAD Scientist


On 10/27/2021 8:44 AM, Murpholino Peligro wrote:
Let's say I ran autoproc with different combinations of options for a 
specific dataset, producing dozens of different (but not so different) 
mtz files...
Then I ran phenix.refine with the same options for the same structure 
but with all my mtz zoo

What would be the best metric to say "hey this combo works the best!"?
R-free?
Thanks

M. Peligro



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 







To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Clemens Vonrhein]

On Thu, Oct 28, 2021 at 06:28:05PM +0530, Shipra Bijpuria wrote:
> I would first look at the dataset stats and define a resolution range
> mainly based on I/sigI >1 and cc1/2 >0.5. Based on this, would take the
> good resolution datasets only.

Some probably obvious word of caution here: these (quite sensible)
suggestions will depend hugely on factors like (1) binning, (2)
definition of a particular data quality metric and sometimes (3) the
treatment of Friedel pairs in computing these values. When comparing
seemingly identical metrics (as labelled) from different
programs/pipelines you have to be careful and aware of the various
differences in implementation. If one is sure that these values are
computed in the same way with the same binning, comparisons will be
possible, yes.

Also, a lot (most?) datasets are poorly described with a single
resolution value, even it it is rather convenient for sorting ;-)

Just thought to add this for the record for future generations of
ccp4bb-archive searching crystallographers ;-)

Clemens



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Shipra Bijpuria]

I would first look at the dataset stats and define a resolution range
mainly based on I/sigI >1 and cc1/2 >0.5. Based on this, would take the
good resolution datasets only.

Further for comparing these mtz after refinement, I personally prefer
looking at the electron density maps rather than just going with R values.
For this I normally select two or three regions where I see weak density
map and then compare different datasets.

Shipra

On Wed, 27 Oct 2021 at 9:15 PM, Murpholino Peligro 
wrote:

> Let's say I ran autoproc with different combinations of options for a
> specific dataset, producing dozens of different (but not so different) mtz
> files...
> Then I ran phenix.refine with the same options for the same structure but
> with all my mtz zoo
> What would be the best metric to say "hey this combo works the best!"?
> R-free?
> Thanks
>
> M. Peligro
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Eleanor Dodson]

THis is always a difficult decision. More commonly I have worried about the
best resolution cut off.
Judge on high Rmerges?  Keep the overall R value acceptably low?? etc etc..

I always come back to the map - is it sharper with the extra data? Is more
unmodelled solvent showing up? etc..
But these decisions cannot be easily assessed by anyone else so I just keep
as much information as I can , down to I/SigI ~ 1 , and let the R value
climb..
However in the end it is a personal decision..

I wonder if the COOT validation of density fit could be used as a numerical
tool..
What do you think Paul?
Eleanor

On Thu, 28 Oct 2021 at 08:46, Clemens Vonrhein 
wrote:

> Hi,
>
> An alternative is to assess the quality of the phases (which are the
> result of the model refined against the data) via radiation-damage
> maps (what we call "F(early)-F(late) maps"). Assuming you collected
> high enough multiplicity data, autoPROC (which is what you used for
> processing, if I understood you right) would create those extra
> amplitudes in the output MTZ files if possible. Refinement with BUSTER
> would automatically compute and analyse those raditation damage maps
> for "interesting" sites, like decarboxylation, Cys-SG/Met-SD damage
> etc. You could then assume that the cleanest/strongest indications of
> such radiation damage arise when using the best phases (i.e. best
> model - as a result of best data to refine against).
>
> For examples see e.g.
>
>
> https://www.globalphasing.com/buster/wiki/index.cgi?Covid19ReRefineRadDam
>   https://www.globalphasing.com/buster/wiki/index.cgi?Covid19ReRefine5VZR
>
> and also
>
>   https://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/msg49077.html
>
> This would all be scriptable I guess.
>
> Cheers
>
> Clemens
>
> PS: there are only really two files coming out of autoPROC, namely
> truncate-unique.mtz (traditional, isotropic analysis) and
> staraniso_alldata-unique.mtz (anisotropic analysis via STARANISO).
>
>
> On Wed, Oct 27, 2021 at 12:58:53PM -0500, Murpholino Peligro wrote:
> > So... how can I get a metric for noise in electron density maps?
> > First thing that occurred to me
> > open in coot and do validate->difference map peaks-> get number of peaks
> > (is this scriptable?)
> > or
> > Second
> > phenix.real_space_correlation detail=residue file.pdb file.mtz
> >
> >
> > Thanks again
> >
> > El mié, 27 de oct. de 2021 a la(s) 10:52, vincent Chaptal (
> > vincent.chap...@ibcp.fr) escribió:
> >
> > > Hi,
> > >
> > > It's hard to find a single metric...
> > > Ultimately, the quality of electron density maps, lower noise in fo-fc?
> > >
> > > Best
> > > Vincent
> > >
> > > Le 27/10/2021 à 17:44, Murpholino Peligro a écrit :
> > >
> > > Let's say I ran autoproc with different combinations of options for a
> > > specific dataset, producing dozens of different (but not so different)
> mtz
> > > files...
> > > Then I ran phenix.refine with the same options for the same structure
> but
> > > with all my mtz zoo
> > > What would be the best metric to say "hey this combo works the best!"?
> > > R-free?
> > > Thanks
> > >
> > > M. Peligro
> > >
> > > --
> > >
> > > To unsubscribe from the CCP4BB list, click the following link:
> > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> > >
> > >
> > > --
> > >
> > > Vincent Chaptal, PhD
> > >
> > > Director of GdR APPICOM
> > >
> > > Drug Resistance and Membrane Proteins Lab
> > >
> > >
> > > MMSB -UMR5086
> > >
> > > 7 passage du Vercors
> > >
> > > 69007 LYON
> > >
> > > FRANCE
> > >
> > > +33 4 37 65 29 01
> > >
> > > http://www.appicom.cnrs.fr
> > >
> > > http://mmsb.cnrs.fr/en/
> > >
> > >
> > >
> > > --
> > >
> > > To unsubscribe from the CCP4BB list, click the following link:
> > > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> > >
> >
> > 
> >
> > To unsubscribe from the CCP4BB list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> >
> > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> available at https://www.jiscmail.ac.uk/policyandsecurity/
>
> --
>
> *--
> * Clemens Vonrhein, Ph.D. vonrhein AT GlobalPhasing DOT com
> * Global Phasing Ltd., Sheraton House, Castle Park
> * Cambridge CB3 0AX, UK   www.globalphasing.com
> *--
>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are
> available at 

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Clemens Vonrhein]

Hi,

An alternative is to assess the quality of the phases (which are the
result of the model refined against the data) via radiation-damage
maps (what we call "F(early)-F(late) maps"). Assuming you collected
high enough multiplicity data, autoPROC (which is what you used for
processing, if I understood you right) would create those extra
amplitudes in the output MTZ files if possible. Refinement with BUSTER
would automatically compute and analyse those raditation damage maps
for "interesting" sites, like decarboxylation, Cys-SG/Met-SD damage
etc. You could then assume that the cleanest/strongest indications of
such radiation damage arise when using the best phases (i.e. best
model - as a result of best data to refine against).

For examples see e.g.

  https://www.globalphasing.com/buster/wiki/index.cgi?Covid19ReRefineRadDam
  https://www.globalphasing.com/buster/wiki/index.cgi?Covid19ReRefine5VZR

and also

  https://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/msg49077.html

This would all be scriptable I guess.

Cheers

Clemens

PS: there are only really two files coming out of autoPROC, namely
truncate-unique.mtz (traditional, isotropic analysis) and
staraniso_alldata-unique.mtz (anisotropic analysis via STARANISO).


On Wed, Oct 27, 2021 at 12:58:53PM -0500, Murpholino Peligro wrote:
> So... how can I get a metric for noise in electron density maps?
> First thing that occurred to me
> open in coot and do validate->difference map peaks-> get number of peaks
> (is this scriptable?)
> or
> Second
> phenix.real_space_correlation detail=residue file.pdb file.mtz
> 
> 
> Thanks again
> 
> El mié, 27 de oct. de 2021 a la(s) 10:52, vincent Chaptal (
> vincent.chap...@ibcp.fr) escribió:
> 
> > Hi,
> >
> > It's hard to find a single metric...
> > Ultimately, the quality of electron density maps, lower noise in fo-fc?
> >
> > Best
> > Vincent
> >
> > Le 27/10/2021 à 17:44, Murpholino Peligro a écrit :
> >
> > Let's say I ran autoproc with different combinations of options for a
> > specific dataset, producing dozens of different (but not so different) mtz
> > files...
> > Then I ran phenix.refine with the same options for the same structure but
> > with all my mtz zoo
> > What would be the best metric to say "hey this combo works the best!"?
> > R-free?
> > Thanks
> >
> > M. Peligro
> >
> > --
> >
> > To unsubscribe from the CCP4BB list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> >
> >
> > --
> >
> > Vincent Chaptal, PhD
> >
> > Director of GdR APPICOM
> >
> > Drug Resistance and Membrane Proteins Lab
> >
> >
> > MMSB -UMR5086
> >
> > 7 passage du Vercors
> >
> > 69007 LYON
> >
> > FRANCE
> >
> > +33 4 37 65 29 01
> >
> > http://www.appicom.cnrs.fr
> >
> > http://mmsb.cnrs.fr/en/
> >
> >
> >
> > --
> >
> > To unsubscribe from the CCP4BB list, click the following link:
> > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> >
> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> 
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing 
> list hosted by www.jiscmail.ac.uk, terms & conditions are available at 
> https://www.jiscmail.ac.uk/policyandsecurity/

-- 

*--
* Clemens Vonrhein, Ph.D. vonrhein AT GlobalPhasing DOT com
* Global Phasing Ltd., Sheraton House, Castle Park 
* Cambridge CB3 0AX, UK   www.globalphasing.com
*--



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

[ccp4bb] AW: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Schreuder, Herman /DE]

Dear Murpholino,

What was the reason of trying all these different processing methods? I, and I 
guess most other crystallographers will process the data using a standard 
procedure and if the results are good, will not try a myriad of other 
processing methods. If it is to get most out of a poorly diffracting crystal, I 
would go for the data set with the best resolution, completeness and 
statistics. If it is to better see a poorly defined ligand, you have to be 
careful. You may be selecting the data set who's noise best resembles the 
ligand, which you dearly want to see, but which is not there. The twilight 
server provides lots of examples of what will happen then. So also here, I 
would select the data set based on resolution, completeness and statistics. 
Alternatively, you could see which data set gives the best Rfree after a few 
rounds of refinement.

Best,
Herman

Von: CCP4 bulletin board  Im Auftrag von Murpholino 
Peligro
Gesendet: Mittwoch, 27. Oktober 2021 19:59
An: CCP4BB@JISCMAIL.AC.UK
Betreff: Re: [ccp4bb] what would be the best metric to asses the quality of a 
mtz file?

So... how can I get a metric for noise in electron density maps?
First thing that occurred to me
open in coot and do validate->difference map peaks-> get number of peaks
(is this scriptable?)
or
Second
phenix.real_space_correlation detail=residue file.pdb file.mtz


Thanks again

El mié, 27 de oct. de 2021 a la(s) 10:52, vincent Chaptal 
(vincent.chap...@ibcp.fr) escribió:
Hi,

It's hard to find a single metric...
Ultimately, the quality of electron density maps, lower noise in fo-fc?

Best
Vincent
Le 27/10/2021 à 17:44, Murpholino Peligro a écrit :
Let's say I ran autoproc with different combinations of options for a specific 
dataset, producing dozens of different (but not so different) mtz files...
Then I ran phenix.refine with the same options for the same structure but with 
all my mtz zoo
What would be the best metric to say "hey this combo works the best!"?
R-free?
Thanks

M. Peligro



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

--

Vincent Chaptal, PhD

Director of GdR APPICOM

Drug Resistance and Membrane Proteins Lab



MMSB -UMR5086

7 passage du Vercors

69007 LYON

FRANCE

+33 4 37 65 29 01

http://www.appicom.cnrs.fr

http://mmsb.cnrs.fr/en/





To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Phil Jeffrey]

CCP4's PEAKMAX program would be quite scriptable.

Phil

On 10/27/21 1:58 PM, Murpholino Peligro wrote:

So... how can I get a metric for noise in electron density maps?
First thing that occurred to me
open in coot and do validate->difference map peaks-> get number of peaks
(is this scriptable?)
or
Second
phenix.real_space_correlation detail=residue file.pdb file.mtz


Thanks again





To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Murpholino Peligro]

So... how can I get a metric for noise in electron density maps?
First thing that occurred to me
open in coot and do validate->difference map peaks-> get number of peaks
(is this scriptable?)
or
Second
phenix.real_space_correlation detail=residue file.pdb file.mtz


Thanks again

El mié, 27 de oct. de 2021 a la(s) 10:52, vincent Chaptal (
vincent.chap...@ibcp.fr) escribió:

> Hi,
>
> It's hard to find a single metric...
> Ultimately, the quality of electron density maps, lower noise in fo-fc?
>
> Best
> Vincent
>
> Le 27/10/2021 à 17:44, Murpholino Peligro a écrit :
>
> Let's say I ran autoproc with different combinations of options for a
> specific dataset, producing dozens of different (but not so different) mtz
> files...
> Then I ran phenix.refine with the same options for the same structure but
> with all my mtz zoo
> What would be the best metric to say "hey this combo works the best!"?
> R-free?
> Thanks
>
> M. Peligro
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>
>
> --
>
> Vincent Chaptal, PhD
>
> Director of GdR APPICOM
>
> Drug Resistance and Membrane Proteins Lab
>
>
> MMSB -UMR5086
>
> 7 passage du Vercors
>
> 69007 LYON
>
> FRANCE
>
> +33 4 37 65 29 01
>
> http://www.appicom.cnrs.fr
>
> http://mmsb.cnrs.fr/en/
>
>
>
> --
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Re: [ccp4bb] what would be the best metric to asses the quality of a mtz file?

[vincent Chaptal]

Hi,

It's hard to find a single metric...
Ultimately, the quality of electron density maps, lower noise in fo-fc?

Best
Vincent

Le 27/10/2021 à 17:44, Murpholino Peligro a écrit :
Let's say I ran autoproc with different combinations of options for a 
specific dataset, producing dozens of different (but not so different) 
mtz files...
Then I ran phenix.refine with the same options for the same structure 
but with all my mtz zoo

What would be the best metric to say "hey this combo works the best!"?
R-free?
Thanks

M. Peligro



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1 





--

Vincent Chaptal, PhD

Director of GdR APPICOM

Drug Resistance and Membrane Proteins Lab


MMSB -UMR5086

7 passage du Vercors

69007 LYON

FRANCE

+33 4 37 65 29 01

http://www.appicom.cnrs.fr

http://mmsb.cnrs.fr/en/





To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

[ccp4bb] what would be the best metric to asses the quality of a mtz file?

[Murpholino Peligro]

Let's say I ran autoproc with different combinations of options for a
specific dataset, producing dozens of different (but not so different) mtz
files...
Then I ran phenix.refine with the same options for the same structure but
with all my mtz zoo
What would be the best metric to say "hey this combo works the best!"?
R-free?
Thanks

M. Peligro



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/