Re: [ccp4bb] What refinement programs are fully Open Source?

[Winter, Graeme (DLSLtd,RAL,LSCI)]

Dear Tim,

While I agree that the papers describing XDS are excellent (and acknowledge 
precisely this every time I talk about DIALS!) I am certain that the actual 
implementation of XDS deviates significantly from the mathematical descriptions 
in the paper for the simple reason that writing data processing programs is 
hard and XDS is good.

I think the spirit behind the original question is that the opportunity exists 
to inspect every calculation which took you from your measured data to whatever 
the science result is - in the general case I would imagine the number of 
people who _do_ inspect is vanishingly small, but there are no fundamental 
limitations to this. Having things be open is however independent of them being 
good :-) - I would not conflate these ideas

I also recognise the distinction between free and open source highlighted in 
the thread - with xia2 and DIALS we have been fortunate to get funding from 
Diamond Light Source, CCP4, Lawrence Berkeley National Lab, the EU, NIH and 
Wellcome Trust (with enthusiastic letters of support from many who are on this 
list) which puts us in the position of being able to make the product free to 
all end users as well as making the source code available for inspection - 
someone needs to pay the bills, ultimately, so I want to thank the various 
funding agencies for this, as well as end users for citations and letters of 
support.

Best wishes Graeme

On 7 May 2020, at 21:32, Tim Gruene 
mailto:tim.gru...@univie.ac.at>> wrote:

Dear Pietro,

to add a bit to Ethan's warning, you might even consider XDS open
source: although the very source is not freely downloadable, it is so
well documented that one could rewrite the program just from its
documentation. At least, this is how the program SAINT started, as
far as I now.

Best regards,
Tim

On Thu, 7 May 2020 17:18:38 +
"Roversi, Pietro (Dr.)" mailto:pr...@leicester.ac.uk>> 
wrote:

Thank you Ethan for taking the the time to answer and explain.
Yes I am sure I have asked a vague and imprecise question.

Practically, I am going to point to xia2 for data processing:
https://www.ccp4.ac.uk/newsletters/newsletter48/articles/Xia2/manual.html

and hope it is "Open Source enough" - without too much scrutiny on
dependencies?

So, what about a refinement suite of programs that is "just as Open
Source" as xia2 is for data processing?

Unless this second message of mine is making my re-drafted question
worse than the original one .

with best wishes,

Pietro


Pietro Roversi

Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/

LISCB Wellcome Trust ISSF Fellow

https://le.ac.uk/liscb/research-groups/pietro-roversi


Leicester Institute of Structural and Chemical Biology
Department of Molecular and Cell Biology, University of Leicester
Henry Wellcome Building
Lancaster Road, Leicester, LE1 7HB
England, United Kingdom

Skype: roversipietro
Mobile phone  +44 (0) 7927952047
Tel. +44 (0)116 2297237




From: Ethan A Merritt 
Sent: 07 May 2020 18:08
To: Roversi, Pietro (Dr.) 
Cc: CCP4BB@jiscmail.ac.uk 
Subject: Re: [ccp4bb] What refinement programs are fully Open Source?

On Thursday, 7 May 2020 09:34:13 PDT Roversi, Pietro (Dr.) wrote:
Dear all,

we are in the editorial stages of a manuscript that I submitted to
Wellcome Open Research for publication.

The journal/editor ask us to list fully Open Source alternatives to
the pieces of software we used, for example for data processing and
refinement.

What refinement programs are fully Open Source?

There are recurring battles and philosophical fractures over what
exactly "open source" means, either in practice or aspirationally.
You would do well to provide a definition before asking people for
suggestions that meet your criteria.

At one point the Open Source Foundation (OSF) claimed to have the
authority to declare something was or was not "open source" and kept
lists of approved code, but their definition was in conflict with
guidelines from other places including funding agencies [*].  Also
the OSF itself seems to have largely disappeared from view, so maybe
that's a bad place to start.

There are at least two fracture lines in this battle.
The one created by people who feel a need to distinguish between
"free/libre code" and "open code",  and the one created by people
whose main concern is "documentation and claims are not enough;
I need to see the code actually used for the calculations reported in
this work".
Then there's the concern mostly of interest to corporate legal
departments "can we use this in our commercial products".

   Ethan (coding veteran with scars from this battle)


[*] it was also in conflict with the ordinary English language meaning
of "open" and "source", which didn't help any.



Thanks!

Pietro


Pietro Roversi

Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/

LISCB Wellcome Trust ISSF Fellow

Re: [ccp4bb] What refinement programs are fully Open Source?

[Ethan A Merritt]

On Thursday, 7 May 2020 13:58:55 PDT David Waterman wrote:
> Dear Ethan,
> 
> The copy of xia2 that I have on my machine says at the top of the source
> > # Copyright (C) 2013 David Waterman
> > #
> > # This code is distributed under the terms and conditions of the
> > # CCP4 Program Suite Licence Agreement as a CCP4 Application.
> 
> 
> This surprised me greatly. I wondered if perhaps you were referring to the
> ccp4i2 GUI for xia2, for which I must admit responsibility. However, I
> can't find "Copyright (C) 2013 David Waterman" anywhere within the current
> release of CCP4, so I am perplexed.

I sincerely apologize for inferring more than I should have from that snippet.
My search may indeed have picked up a wrapper rather than the program
itself.  Here is what I did:

head `locate bin/xia2`

> I would like to make it clear that xia2 is not distributed under the terms
> of the CCP4 licence. It has a permissive free software licence, namely
> BSD-3, which must surely conform to any reasonable definition of Open
> Source. See here https://github.com/xia2/xia2/blob/master/LICENSE
> 
> While xia2 has certain modes that run non-open software, Graeme's point is
> that the default mode runs DIALS, which is BSD-3 as well.
>
> -- David

The larger point was that xia2 is not so much a computational code by
itself as an open-ended framework for multiple codes that are not
uniformly licensed or available for code inspection/modification.
I hope I have that right!  

Ethan

-- 
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, Seattle 98195-7742



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] What refinement programs are fully Open Source?

[David Waterman]

Dear Ethan,

The copy of xia2 that I have on my machine says at the top of the source
> # Copyright (C) 2013 David Waterman
> #
> # This code is distributed under the terms and conditions of the
> # CCP4 Program Suite Licence Agreement as a CCP4 Application.


This surprised me greatly. I wondered if perhaps you were referring to the
ccp4i2 GUI for xia2, for which I must admit responsibility. However, I
can't find "Copyright (C) 2013 David Waterman" anywhere within the current
release of CCP4, so I am perplexed.

I would like to make it clear that xia2 is not distributed under the terms
of the CCP4 licence. It has a permissive free software licence, namely
BSD-3, which must surely conform to any reasonable definition of Open
Source. See here https://github.com/xia2/xia2/blob/master/LICENSE

While xia2 has certain modes that run non-open software, Graeme's point is
that the default mode runs DIALS, which is BSD-3 as well.

-- David


On Thu, 7 May 2020 at 18:56, Ethan A Merritt  wrote:

> On Thursday, 7 May 2020 10:18:38 PDT Roversi, Pietro (Dr.) wrote:
> > Thank you Ethan for taking the the time to answer and explain.
> > Yes I am sure I have asked a vague and imprecise question.
> >
> > Practically, I am going to point to xia2 for data processing:
> >
> https://www.ccp4.ac.uk/newsletters/newsletter48/articles/Xia2/manual.html
> >
> > and hope it is "Open Source enough" - without too much scrutiny on
> dependencies?
>
> I may misundertand the full scope of xia2, but I believe it is a scripted
> pipeline
> that invokes other programs.  Those other programs individually may have
> their own very different licensing or distribution or use restrictions.
> The user guide you linked to says
> if you use xia2 in published work please include the references
> for the programs it has used, which are printed at the end of the
> output.
>
> The copy of xia2 that I have on my machine says at the top of the source
>
> # Copyright (C) 2013 David Waterman
> #
> # This code is distributed under the terms and conditions of the
> # CCP4 Program Suite Licence Agreement as a CCP4 Application.
>
> So I guess the first question is whether the CCP4 License Agreement
> meets your definition of Open Source.
>
> The next question would be what programs did xia2 choose to run?
> Some of these may meet your criteria for Open Source, others not.
> For example, does xia2 invoke shelx?  Does shelx meet your criteria?
>
> Ethan
>
>
>
> >
> > So, what about a refinement suite of programs that is "just as Open
> Source" as xia2 is for data processing?
> >
> > Unless this second message of mine is making my re-drafted question
> worse than the original one .
> >
> > with best wishes,
> >
> > Pietro
> >
> >
> > Pietro Roversi
> >
> > Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
> >
> > LISCB Wellcome Trust ISSF Fellow
> >
> > 
> https://le.ac.uk/liscb/research-groups/pietro-roversi
> >
> >
> > Leicester Institute of Structural and Chemical Biology
> > Department of Molecular and Cell Biology, University of Leicester
> > Henry Wellcome Building
> > Lancaster Road, Leicester, LE1 7HB
> > England, United Kingdom
> >
> > Skype: roversipietro
> > Mobile phone  +44 (0) 7927952047
> > Tel. +44 (0)116 2297237
> >
> >
> >
> > 
> > From: Ethan A Merritt 
> > Sent: 07 May 2020 18:08
> > To: Roversi, Pietro (Dr.) 
> > Cc: CCP4BB@jiscmail.ac.uk 
> > Subject: Re: [ccp4bb] What refinement programs are fully Open Source?
> >
> > On Thursday, 7 May 2020 09:34:13 PDT Roversi, Pietro (Dr.) wrote:
> > > Dear all,
> > >
> > > we are in the editorial stages of a manuscript that I submitted to
> Wellcome Open Research for publication.
> > >
> > > The journal/editor ask us to list fully Open Source alternatives to
> the pieces of software we used, for example for data processing and
> refinement.
> > >
> > > What refinement programs are fully Open Source?
> >
> > There are recurring battles and philosophical fractures over what exactly
> > "open source" means, either in practice or aspirationally.
> > You would do well to provide a definition before asking people for
> > suggestions that meet your criteria.
> >
> > At one point the Open Source Foundation (OSF) claimed to have the
> authority
> > to declare something was or was not "open source" and kept lists of
> > approved code, but their definition was in conflict with guidelines from
> > other places including funding agencies [*].  Also the OSF itself seems
> to
> > have largely disappeared from view, so maybe that's a bad place to start.
> >
> > There are at least two fracture lines in this battle.
> > The one created by people who feel a need to distinguish between
> > "free/libre code" and "open code",  and the one created by people
> > whose main concern is "documentation and claims are not enough;
> > I need to see the code actually used for the calculations reported in
> > this 

Re: [ccp4bb] What refinement programs are fully Open Source?

[Tim Gruene]

Dear Pietro,

to add a bit to Ethan's warning, you might even consider XDS open
source: although the very source is not freely downloadable, it is so
well documented that one could rewrite the program just from its
documentation. At least, this is how the program SAINT started, as
far as I now.

Best regards,
Tim

On Thu, 7 May 2020 17:18:38 +
"Roversi, Pietro (Dr.)"  wrote:

> Thank you Ethan for taking the the time to answer and explain.
> Yes I am sure I have asked a vague and imprecise question.
> 
> Practically, I am going to point to xia2 for data processing:
> https://www.ccp4.ac.uk/newsletters/newsletter48/articles/Xia2/manual.html
> 
> and hope it is "Open Source enough" - without too much scrutiny on
> dependencies?
> 
> So, what about a refinement suite of programs that is "just as Open
> Source" as xia2 is for data processing?
> 
> Unless this second message of mine is making my re-drafted question
> worse than the original one .
> 
> with best wishes,
> 
> Pietro
> 
> 
> Pietro Roversi
> 
> Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
> 
> LISCB Wellcome Trust ISSF Fellow
> 
> https://le.ac.uk/liscb/research-groups/pietro-roversi
> 
> 
> Leicester Institute of Structural and Chemical Biology
> Department of Molecular and Cell Biology, University of Leicester
> Henry Wellcome Building
> Lancaster Road, Leicester, LE1 7HB
> England, United Kingdom
> 
> Skype: roversipietro
> Mobile phone  +44 (0) 7927952047
> Tel. +44 (0)116 2297237
> 
> 
> 
> 
> From: Ethan A Merritt 
> Sent: 07 May 2020 18:08
> To: Roversi, Pietro (Dr.) 
> Cc: CCP4BB@jiscmail.ac.uk 
> Subject: Re: [ccp4bb] What refinement programs are fully Open Source?
> 
> On Thursday, 7 May 2020 09:34:13 PDT Roversi, Pietro (Dr.) wrote:
> > Dear all,
> >
> > we are in the editorial stages of a manuscript that I submitted to
> > Wellcome Open Research for publication.
> >
> > The journal/editor ask us to list fully Open Source alternatives to
> > the pieces of software we used, for example for data processing and
> > refinement.
> >
> > What refinement programs are fully Open Source?  
> 
> There are recurring battles and philosophical fractures over what
> exactly "open source" means, either in practice or aspirationally.
> You would do well to provide a definition before asking people for
> suggestions that meet your criteria.
> 
> At one point the Open Source Foundation (OSF) claimed to have the
> authority to declare something was or was not "open source" and kept
> lists of approved code, but their definition was in conflict with
> guidelines from other places including funding agencies [*].  Also
> the OSF itself seems to have largely disappeared from view, so maybe
> that's a bad place to start.
> 
> There are at least two fracture lines in this battle.
> The one created by people who feel a need to distinguish between
> "free/libre code" and "open code",  and the one created by people
> whose main concern is "documentation and claims are not enough;
> I need to see the code actually used for the calculations reported in
> this work".
> Then there's the concern mostly of interest to corporate legal
> departments "can we use this in our commercial products".
> 
> Ethan (coding veteran with scars from this battle)
> 
> 
> [*] it was also in conflict with the ordinary English language meaning
> of "open" and "source", which didn't help any.
> 
> 
> >
> > Thanks!
> >
> > Pietro
> >
> >
> > Pietro Roversi
> >
> > Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
> >
> > LISCB Wellcome Trust ISSF Fellow
> >
> > https://le.ac.uk/liscb/research-groups/pietro-roversi
> >
> >
> > Leicester Institute of Structural and Chemical Biology
> > Department of Molecular and Cell Biology, University of Leicester
> > Henry Wellcome Building
> > Lancaster Road, Leicester, LE1 7HB
> > England, United Kingdom
> >
> > Skype: roversipietro
> > Mobile phone  +44 (0) 7927952047
> > Tel. +44 (0)116 2297237
> >
> >
> >
> > 
> >
> > To unsubscribe from the CCP4BB list, click the following link:
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DCCP4BB%26A%3D1data=02%7C01%7Cpr159%40leicester.ac.uk%7Cf8cc2fb23bb84707d7a708d7f2a96338%7Caebecd6a31d44b0195ce8274afe853d9%7C0%7C0%7C637244681673138009sdata=nfhtEWy2FS96MYwaHsT4qoQi%2BMbPetQdTwfAf3FDjGg%3Dreserved=0
> >  
> 
> 
> --
> Ethan A Merritt
> Biomolecular Structure Center,  K-428 Health Sciences Bldg
> MS 357742,   University of Washington, Seattle 98195-7742
> 
> 
> 
> 

[ccp4bb] Fwd: Re: What refinement programs are fully Open Source?

[Helen Ginn]

Hi Pietro!

As for free and open-source (GPLv3 license) refinement software, I am 
developing Vagabond.


https://vagabond.hginn.co.uk

It is based on refining bond torsion angles in an ensemble of related 
structures which also captures flexibility (no B factors!). There are no 
restraints, only constraints. This allows a significant side-step of the 
parametrisation problem and reduces the level of overfitting in the map 
(which is more widespread than I first realised).


It is in the middle of development, very young and therefore not as 
polished or refined as other (atomistic) refinement software, and is 
currently best used as an intermediate tool, when you have confusing and 
muddy electron density, to create a less overfitting-biased map to build 
into and then continue with your favourite other refinement software.


git clone https://www.github.com/helenginn/vagabond.git

Typing this into a command line will demonstrate how open source it is!

Helen

---

On Thursday, 7 May 2020 09:34:13 PDT Roversi, Pietro (Dr.) wrote:

Dear all,

we are in the editorial stages of a manuscript that I submitted to 
Wellcome Open Research for publication.


The journal/editor ask us to list fully Open Source alternatives to the 
pieces of software we used, for example for data processing and 
refinement.


What refinement programs are fully Open Source?

Thanks!

Pietro

Pietro Roversi




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] What refinement programs are fully Open Source?

[Ethan A Merritt]

On Thursday, 7 May 2020 10:18:38 PDT Roversi, Pietro (Dr.) wrote:
> Thank you Ethan for taking the the time to answer and explain.
> Yes I am sure I have asked a vague and imprecise question.
> 
> Practically, I am going to point to xia2 for data processing:
> https://www.ccp4.ac.uk/newsletters/newsletter48/articles/Xia2/manual.html
> 
> and hope it is "Open Source enough" - without too much scrutiny on 
> dependencies?

I may misundertand the full scope of xia2, but I believe it is a scripted 
pipeline
that invokes other programs.  Those other programs individually may have
their own very different licensing or distribution or use restrictions.
The user guide you linked to says
if you use xia2 in published work please include the references 
for the programs it has used, which are printed at the end of the
output.

The copy of xia2 that I have on my machine says at the top of the source

# Copyright (C) 2013 David Waterman
#
# This code is distributed under the terms and conditions of the
# CCP4 Program Suite Licence Agreement as a CCP4 Application.

So I guess the first question is whether the CCP4 License Agreement
meets your definition of Open Source.

The next question would be what programs did xia2 choose to run?
Some of these may meet your criteria for Open Source, others not.
For example, does xia2 invoke shelx?  Does shelx meet your criteria?

Ethan



> 
> So, what about a refinement suite of programs that is "just as Open Source" 
> as xia2 is for data processing?
> 
> Unless this second message of mine is making my re-drafted question worse 
> than the original one .
> 
> with best wishes,
> 
> Pietro
> 
> 
> Pietro Roversi
> 
> Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
> 
> LISCB Wellcome Trust ISSF Fellow
> 
> https://le.ac.uk/liscb/research-groups/pietro-roversi
> 
> 
> Leicester Institute of Structural and Chemical Biology
> Department of Molecular and Cell Biology, University of Leicester
> Henry Wellcome Building
> Lancaster Road, Leicester, LE1 7HB
> England, United Kingdom
> 
> Skype: roversipietro
> Mobile phone  +44 (0) 7927952047
> Tel. +44 (0)116 2297237
> 
> 
> 
> 
> From: Ethan A Merritt 
> Sent: 07 May 2020 18:08
> To: Roversi, Pietro (Dr.) 
> Cc: CCP4BB@jiscmail.ac.uk 
> Subject: Re: [ccp4bb] What refinement programs are fully Open Source?
> 
> On Thursday, 7 May 2020 09:34:13 PDT Roversi, Pietro (Dr.) wrote:
> > Dear all,
> >
> > we are in the editorial stages of a manuscript that I submitted to Wellcome 
> > Open Research for publication.
> >
> > The journal/editor ask us to list fully Open Source alternatives to the 
> > pieces of software we used, for example for data processing and refinement.
> >
> > What refinement programs are fully Open Source?
> 
> There are recurring battles and philosophical fractures over what exactly
> "open source" means, either in practice or aspirationally.
> You would do well to provide a definition before asking people for
> suggestions that meet your criteria.
> 
> At one point the Open Source Foundation (OSF) claimed to have the authority
> to declare something was or was not "open source" and kept lists of
> approved code, but their definition was in conflict with guidelines from
> other places including funding agencies [*].  Also the OSF itself seems to
> have largely disappeared from view, so maybe that's a bad place to start.
> 
> There are at least two fracture lines in this battle.
> The one created by people who feel a need to distinguish between
> "free/libre code" and "open code",  and the one created by people
> whose main concern is "documentation and claims are not enough;
> I need to see the code actually used for the calculations reported in
> this work".
> Then there's the concern mostly of interest to corporate legal
> departments "can we use this in our commercial products".
> 
> Ethan (coding veteran with scars from this battle)
> 
> 
> [*] it was also in conflict with the ordinary English language meaning
> of "open" and "source", which didn't help any.
> 
> 
> >
> > Thanks!
> >
> > Pietro
> >
> >
> > Pietro Roversi
> >
> > Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
> >
> > LISCB Wellcome Trust ISSF Fellow
> >
> > https://le.ac.uk/liscb/research-groups/pietro-roversi
> >
> >
> > Leicester Institute of Structural and Chemical Biology
> > Department of Molecular and Cell Biology, University of Leicester
> > Henry Wellcome Building
> > Lancaster Road, Leicester, LE1 7HB
> > England, United Kingdom
> >
> > Skype: roversipietro
> > Mobile phone  +44 (0) 7927952047
> > Tel. +44 (0)116 2297237
> >
> >

Re: [ccp4bb] What refinement programs are fully Open Source?

[Winter, Graeme (DLSLtd,RAL,LSCI)]

Hi Pietro,

To be precise, xia2 is a tool (which is open source under any definition - BSD 
licensed) for running data processing programs on your behalf. If you run with 
DIALS for indexing, integration and scaling (which are licensed the same way, 
and uses cctbx which is also open source) then it is true that you have run 
essentially a full stack of open source to get you from images to scaled and 
merged intensities

All of this is available on GitHub

All the best Graeme

On 7 May 2020, at 18:18, Roversi, Pietro (Dr.) 
mailto:pr...@leicester.ac.uk>> wrote:

Thank you Ethan for taking the the time to answer and explain.
Yes I am sure I have asked a vague and imprecise question.

Practically, I am going to point to xia2 for data processing:
https://www.ccp4.ac.uk/newsletters/newsletter48/articles/Xia2/manual.html

and hope it is "Open Source enough" - without too much scrutiny on dependencies?

So, what about a refinement suite of programs that is "just as Open Source" as 
xia2 is for data processing?

Unless this second message of mine is making my re-drafted question worse than 
the original one .

with best wishes,

Pietro

Pietro Roversi
Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
LISCB Wellcome Trust ISSF Fellow
https://le.ac.uk/liscb/research-groups/pietro-roversi


Leicester Institute of Structural and Chemical Biology
Department of Molecular and Cell Biology, University of Leicester
Henry Wellcome Building
Lancaster Road, Leicester, LE1 7HB
England, United Kingdom

Skype: roversipietro
Mobile phone  +44 (0) 7927952047
Tel. +44 (0)116 2297237




From: Ethan A Merritt mailto:merr...@uw.edu>>
Sent: 07 May 2020 18:08
To: Roversi, Pietro (Dr.) mailto:pr...@leicester.ac.uk>>
Cc: CCP4BB@jiscmail.ac.uk 
mailto:CCP4BB@jiscmail.ac.uk>>
Subject: Re: [ccp4bb] What refinement programs are fully Open Source?

On Thursday, 7 May 2020 09:34:13 PDT Roversi, Pietro (Dr.) wrote:
> Dear all,
>
> we are in the editorial stages of a manuscript that I submitted to Wellcome 
> Open Research for publication.
>
> The journal/editor ask us to list fully Open Source alternatives to the 
> pieces of software we used, for example for data processing and refinement.
>
> What refinement programs are fully Open Source?

There are recurring battles and philosophical fractures over what exactly
"open source" means, either in practice or aspirationally.
You would do well to provide a definition before asking people for
suggestions that meet your criteria.

At one point the Open Source Foundation (OSF) claimed to have the authority
to declare something was or was not "open source" and kept lists of
approved code, but their definition was in conflict with guidelines from
other places including funding agencies [*].  Also the OSF itself seems to
have largely disappeared from view, so maybe that's a bad place to start.

There are at least two fracture lines in this battle.
The one created by people who feel a need to distinguish between
"free/libre code" and "open code",  and the one created by people
whose main concern is "documentation and claims are not enough;
I need to see the code actually used for the calculations reported in
this work".
Then there's the concern mostly of interest to corporate legal
departments "can we use this in our commercial products".

Ethan (coding veteran with scars from this battle)


[*] it was also in conflict with the ordinary English language meaning
of "open" and "source", which didn't help any.


>
> Thanks!
>
> Pietro
>
>
> Pietro Roversi
>
> Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
>
> LISCB Wellcome Trust ISSF Fellow
>
> https://le.ac.uk/liscb/research-groups/pietro-roversi
>
>
> Leicester Institute of Structural and Chemical Biology
> Department of Molecular and Cell Biology, University of Leicester
> Henry Wellcome Building
> Lancaster Road, Leicester, LE1 7HB
> England, United Kingdom
>
> Skype: roversipietro
> Mobile phone  +44 (0) 7927952047
> Tel. +44 (0)116 2297237
>
>
>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DCCP4BB%26A%3D1data=02%7C01%7Cpr159%40leicester.ac.uk%7Cf8cc2fb23bb84707d7a708d7f2a96338%7Caebecd6a31d44b0195ce8274afe853d9%7C0%7C0%7C637244681673138009sdata=nfhtEWy2FS96MYwaHsT4qoQi%2BMbPetQdTwfAf3FDjGg%3Dreserved=0
>


--
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, 

Re: [ccp4bb] What refinement programs are fully Open Source?

[Dale Tronrud]

   Could you point out where in this document the license is described?

   While not particularly relevant to your question, TNT is a reasonable
example of what Ethan is talking about.  TNT was always "open source"
since all source code was distributed to every user.  It has never been
"free", however.  To get it one had to agree to the license from the
University of Oregon and, if a for-profit organization, pay money.

Dale Tronrud

On 5/7/2020 10:18 AM, Roversi, Pietro (Dr.) wrote:
> Thank you Ethan for taking the the time to answer and explain.
> Yes I am sure I have asked a vague and imprecise question.
> 
> Practically, I am going to point to xia2 for data processing:
> https://www.ccp4.ac.uk/newsletters/newsletter48/articles/Xia2/manual.html
> 
> and hope it is "Open Source enough" - without too much scrutiny on
> dependencies?
> 
> So, what about a refinement suite of programs that is "just as Open
> Source" as xia2 is for data processing?
> 
> Unless this second message of mine is making my re-drafted question
> worse than the original one .
> 
> with best wishes,
> 
> Pietro
> 
> Pietro Roversi
> 
> Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
> 
> LISCB Wellcome Trust ISSF Fellow
> 
> https://le.ac.uk/liscb/research-groups/pietro-roversi
> 
> 
> Leicester Institute of Structural and Chemical Biology
> Department of Molecular and Cell Biology, University of Leicester
> Henry Wellcome Building
> Lancaster Road, Leicester, LE1 7HB
> England, United Kingdom
> 
> Skype: roversipietro
> Mobile phone  +44 (0) 7927952047
> Tel. +44 (0)116 2297237
> 
> 
> 
> *From:* Ethan A Merritt 
> *Sent:* 07 May 2020 18:08
> *To:* Roversi, Pietro (Dr.) 
> *Cc:* CCP4BB@jiscmail.ac.uk 
> *Subject:* Re: [ccp4bb] What refinement programs are fully Open Source?
>  
> On Thursday, 7 May 2020 09:34:13 PDT Roversi, Pietro (Dr.) wrote:
>> Dear all,
>> 
>> we are in the editorial stages of a manuscript that I submitted to Wellcome 
>> Open Research for publication.
>> 
>> The journal/editor ask us to list fully Open Source alternatives to the 
>> pieces of software we used, for example for data processing and refinement.
>> 
>> What refinement programs are fully Open Source?
> 
> There are recurring battles and philosophical fractures over what exactly
> "open source" means, either in practice or aspirationally.
> You would do well to provide a definition before asking people for
> suggestions that meet your criteria. 
> 
> At one point the Open Source Foundation (OSF) claimed to have the authority
> to declare something was or was not "open source" and kept lists of
> approved code, but their definition was in conflict with guidelines from
> other places including funding agencies [*].  Also the OSF itself seems to
> have largely disappeared from view, so maybe that's a bad place to start.
> 
> There are at least two fracture lines in this battle.
> The one created by people who feel a need to distinguish between
> "free/libre code" and "open code",  and the one created by people
> whose main concern is "documentation and claims are not enough;
> I need to see the code actually used for the calculations reported in
> this work".
> Then there's the concern mostly of interest to corporate legal
> departments "can we use this in our commercial products".
> 
>     Ethan (coding veteran with scars from this battle)
> 
> 
> [*] it was also in conflict with the ordinary English language meaning
> of "open" and "source", which didn't help any.
> 
> 
>> 
>> Thanks!
>> 
>> Pietro
>> 
>> 
>> Pietro Roversi
>> 
>> Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
>> 
>> LISCB Wellcome Trust ISSF Fellow
>> 
>> https://le.ac.uk/liscb/research-groups/pietro-roversi
>> 
>> 
>> Leicester Institute of Structural and Chemical Biology
>> Department of Molecular and Cell Biology, University of Leicester
>> Henry Wellcome Building
>> Lancaster Road, Leicester, LE1 7HB
>> England, United Kingdom
>> 
>> Skype: roversipietro
>> Mobile phone  +44 (0) 7927952047
>> Tel. +44 (0)116 2297237
>> 
>> 
>> 
>> 
>> 
>> To unsubscribe from the CCP4BB list, click the following link:
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DCCP4BB%26A%3D1data=02%7C01%7Cpr159%40leicester.ac.uk%7Cf8cc2fb23bb84707d7a708d7f2a96338%7Caebecd6a31d44b0195ce8274afe853d9%7C0%7C0%7C637244681673138009sdata=nfhtEWy2FS96MYwaHsT4qoQi%2BMbPetQdTwfAf3FDjGg%3Dreserved=0
>> 
> 
> 
> -- 
> Ethan A Merritt
> Biomolecular Structure 

Re: [ccp4bb] What refinement programs are fully Open Source?

[Roversi, Pietro (Dr.)]

Thank you Ethan for taking the the time to answer and explain.
Yes I am sure I have asked a vague and imprecise question.

Practically, I am going to point to xia2 for data processing:
https://www.ccp4.ac.uk/newsletters/newsletter48/articles/Xia2/manual.html

and hope it is "Open Source enough" - without too much scrutiny on dependencies?

So, what about a refinement suite of programs that is "just as Open Source" as 
xia2 is for data processing?

Unless this second message of mine is making my re-drafted question worse than 
the original one .

with best wishes,

Pietro


Pietro Roversi

Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/

LISCB Wellcome Trust ISSF Fellow

https://le.ac.uk/liscb/research-groups/pietro-roversi


Leicester Institute of Structural and Chemical Biology
Department of Molecular and Cell Biology, University of Leicester
Henry Wellcome Building
Lancaster Road, Leicester, LE1 7HB
England, United Kingdom

Skype: roversipietro
Mobile phone  +44 (0) 7927952047
Tel. +44 (0)116 2297237




From: Ethan A Merritt 
Sent: 07 May 2020 18:08
To: Roversi, Pietro (Dr.) 
Cc: CCP4BB@jiscmail.ac.uk 
Subject: Re: [ccp4bb] What refinement programs are fully Open Source?

On Thursday, 7 May 2020 09:34:13 PDT Roversi, Pietro (Dr.) wrote:
> Dear all,
>
> we are in the editorial stages of a manuscript that I submitted to Wellcome 
> Open Research for publication.
>
> The journal/editor ask us to list fully Open Source alternatives to the 
> pieces of software we used, for example for data processing and refinement.
>
> What refinement programs are fully Open Source?

There are recurring battles and philosophical fractures over what exactly
"open source" means, either in practice or aspirationally.
You would do well to provide a definition before asking people for
suggestions that meet your criteria.

At one point the Open Source Foundation (OSF) claimed to have the authority
to declare something was or was not "open source" and kept lists of
approved code, but their definition was in conflict with guidelines from
other places including funding agencies [*].  Also the OSF itself seems to
have largely disappeared from view, so maybe that's a bad place to start.

There are at least two fracture lines in this battle.
The one created by people who feel a need to distinguish between
"free/libre code" and "open code",  and the one created by people
whose main concern is "documentation and claims are not enough;
I need to see the code actually used for the calculations reported in
this work".
Then there's the concern mostly of interest to corporate legal
departments "can we use this in our commercial products".

Ethan (coding veteran with scars from this battle)


[*] it was also in conflict with the ordinary English language meaning
of "open" and "source", which didn't help any.


>
> Thanks!
>
> Pietro
>
>
> Pietro Roversi
>
> Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
>
> LISCB Wellcome Trust ISSF Fellow
>
> https://le.ac.uk/liscb/research-groups/pietro-roversi
>
>
> Leicester Institute of Structural and Chemical Biology
> Department of Molecular and Cell Biology, University of Leicester
> Henry Wellcome Building
> Lancaster Road, Leicester, LE1 7HB
> England, United Kingdom
>
> Skype: roversipietro
> Mobile phone  +44 (0) 7927952047
> Tel. +44 (0)116 2297237
>
>
>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DCCP4BB%26A%3D1data=02%7C01%7Cpr159%40leicester.ac.uk%7Cf8cc2fb23bb84707d7a708d7f2a96338%7Caebecd6a31d44b0195ce8274afe853d9%7C0%7C0%7C637244681673138009sdata=nfhtEWy2FS96MYwaHsT4qoQi%2BMbPetQdTwfAf3FDjGg%3Dreserved=0
>


--
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, Seattle 98195-7742





To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] What refinement programs are fully Open Source?

[Ethan A Merritt]

On Thursday, 7 May 2020 09:34:13 PDT Roversi, Pietro (Dr.) wrote:
> Dear all,
> 
> we are in the editorial stages of a manuscript that I submitted to Wellcome 
> Open Research for publication.
> 
> The journal/editor ask us to list fully Open Source alternatives to the 
> pieces of software we used, for example for data processing and refinement.
> 
> What refinement programs are fully Open Source?

There are recurring battles and philosophical fractures over what exactly
"open source" means, either in practice or aspirationally.
You would do well to provide a definition before asking people for
suggestions that meet your criteria.  

At one point the Open Source Foundation (OSF) claimed to have the authority
to declare something was or was not "open source" and kept lists of 
approved code, but their definition was in conflict with guidelines from
other places including funding agencies [*].  Also the OSF itself seems to
have largely disappeared from view, so maybe that's a bad place to start.

There are at least two fracture lines in this battle.
The one created by people who feel a need to distinguish between
"free/libre code" and "open code",  and the one created by people
whose main concern is "documentation and claims are not enough;
I need to see the code actually used for the calculations reported in 
this work".
Then there's the concern mostly of interest to corporate legal
departments "can we use this in our commercial products".

Ethan (coding veteran with scars from this battle)


[*] it was also in conflict with the ordinary English language meaning
of "open" and "source", which didn't help any.


> 
> Thanks!
> 
> Pietro
> 
> 
> Pietro Roversi
> 
> Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/
> 
> LISCB Wellcome Trust ISSF Fellow
> 
> https://le.ac.uk/liscb/research-groups/pietro-roversi
> 
> 
> Leicester Institute of Structural and Chemical Biology
> Department of Molecular and Cell Biology, University of Leicester
> Henry Wellcome Building
> Lancaster Road, Leicester, LE1 7HB
> England, United Kingdom
> 
> Skype: roversipietro
> Mobile phone  +44 (0) 7927952047
> Tel. +44 (0)116 2297237
> 
> 
> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
> 


-- 
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, Seattle 98195-7742



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

[ccp4bb] What refinement programs are fully Open Source?

[Roversi, Pietro (Dr.)]

Dear all,

we are in the editorial stages of a manuscript that I submitted to Wellcome 
Open Research for publication.

The journal/editor ask us to list fully Open Source alternatives to the pieces 
of software we used, for example for data processing and refinement.

What refinement programs are fully Open Source?

Thanks!

Pietro


Pietro Roversi

Lecturer (Teaching and Research) https://le.ac.uk/natural-sciences/

LISCB Wellcome Trust ISSF Fellow

https://le.ac.uk/liscb/research-groups/pietro-roversi


Leicester Institute of Structural and Chemical Biology
Department of Molecular and Cell Biology, University of Leicester
Henry Wellcome Building
Lancaster Road, Leicester, LE1 7HB
England, United Kingdom

Skype: roversipietro
Mobile phone  +44 (0) 7927952047
Tel. +44 (0)116 2297237





To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] Dose in diffraction patterns?

[James Holton]

One more correction:

For example, if you see an average pixel value of 20 photons on a 
Pilatus 6M, then that is P=120e6 photons.  If that was a t=0.1 s 
exposure from a sample 100 microns thick, then the beamline flux was 
about 1e12 photons/s.  Note that this is the flux after any attenuation, 
not before.



On 5/7/2020 9:06 AM, James Holton wrote:
Ah!  I did that last formula wrong.  Never do algebra in your head 
without checking. It should be:


The equation then becomes:

f = P/t/L/1.2e-5

Where 1.2e-5 = 0.2 cm^2/g * 1.2 g/cm^3 * 1e-4 cm/micron * 50%, f=flux 
and t=exposure (as above).


For example, if you see an average pixel value of 20 photons on a 
Pilatus 6M, then that is P=12e6 photons.  If that was a t=0.1 s 
exposure from a sample 100 microns thick, then the beamline flux was 
about 1e12 photons/s.  Note that this is the flux after any 
attenuation, not before.


Oh, and if you want a reference for that 2000 ph/um^2 = 1 Gy rule, it 
is here:

https://doi.org/10.1107/S0909049509004361

And, of course, if you are lucky enough to have accurate flux, size 
and shape information for the beam and sample, plus chemical 
composition the most accurate dose you'll get from raddose-3D: 
https://www.raddo.se/


-James Holton
MAD Scientist

On 5/6/2020 4:04 PM, James Holton wrote:

In general?  No.

I believe a few places put "flux" into the header, but as Andreas 
just mentioned that is only one of the bits of information you need 
to calculate dose.  If all you want is a rough estimate, then the 
numbers you need are:

f = flux (photons/s)
t = exposure time (s)
w = wavelength (A)
a = beam area (um^2)

The dose (D) to a sample of protein/water/plastic under a given beam 
will be roughly:

D = f*t*w^2/a/2000  (in Gy)

For example: a crystal under a square 100 um x 100 um beam at 1 A 
wavelength with flux 1e12 ph/s will get 1 MGy dose in 20 s.


The 2000/w^2 is a fudge factor that fits the true curve of metal-free 
protein crystals to within 15% for the wavelength range 0.5 < w < 3.  
The error induced by not knowing if the beam was round or square and 
just multiplying together the width and height to get the area (a), 
is 21%.  The error from not realizing you had 100 mM uranium in your 
sample is about a factor of two at 1 A.  Smaller concentrations and 
lighter atoms have less impact on accuracy.


If you don't know the flux, or beam size, you can try looking them up 
at http://biosync.sbkb.org/ . I scraped these for my little dose 
calculator here:

bl831.als.lbl.gov/xtallife.html
Some of the biosync numbers are more accurate than others, however, 
depending on how often beamline scientists remember to update the 
site, and how well they know themselves.  And attenuation is not 
always written into the header either.


In a pinch, you can estimate the flux by the total number of photons 
on the image (P).  This is assuming that you know the sample 
thickness (L) in microns.  You must also assume that the total 
scattering cross section of the atoms in the sample is close to that 
of oxygen (0.2 cm^2/g), that the sample density is 1.2 g/cm^3 and 
that about 50% of the scattered photons reach the detector.  None of 
these are terrible assumptions. The equation then becomes:


f = P/t/L*1.2e-5

Where 1.2e-5 = 0.2 cm^2/g * 1.2 g/cm^3 * 1e-4 cm/micron * 50%, f=flux 
and t=exposure (as above).


Getting P from a pixel array is easy: you just add up all the pixel 
values.  From a CCD you want to be careful to subtract the baseline 
value from each pixel first (40 on ADSC, 10 on Mar/Rayonix), and then 
divide by the "gain", which near 1 A is ~1 on Mar/Rayonix, 0.6 for 
ADSC Q315 (swbin) and 1.8 for Q315r (hwbin).  A few considerations, 
yes, but it can be a good sanity check.


-James Holton
MAD Scientist


On 5/5/2020 11:48 AM, Murpholino Peligro wrote:
Do diffraction patterns publicly accessible contain information 
about the x-ray absorbed dose?




Thanks



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1










To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] Dose in diffraction patterns?

[James Holton]

Ah!  I did that last formula wrong.  Never do algebra in your head 
without checking. It should be:


The equation then becomes:

f = P/t/L/1.2e-5

Where 1.2e-5 = 0.2 cm^2/g * 1.2 g/cm^3 * 1e-4 cm/micron * 50%, f=flux 
and t=exposure (as above).


For example, if you see an average pixel value of 20 photons on a 
Pilatus 6M, then that is P=12e6 photons.  If that was a t=0.1 s exposure 
from a sample 100 microns thick, then the beamline flux was about 1e12 
photons/s.  Note that this is the flux after any attenuation, not before.


Oh, and if you want a reference for that 2000 ph/um^2 = 1 Gy rule, it is 
here:

https://doi.org/10.1107/S0909049509004361

And, of course, if you are lucky enough to have accurate flux, size and 
shape information for the beam and sample, plus chemical composition the 
most accurate dose you'll get from raddose-3D: https://www.raddo.se/


-James Holton
MAD Scientist

On 5/6/2020 4:04 PM, James Holton wrote:

In general?  No.

I believe a few places put "flux" into the header, but as Andreas just 
mentioned that is only one of the bits of information you need to 
calculate dose.  If all you want is a rough estimate, then the numbers 
you need are:

f = flux (photons/s)
t = exposure time (s)
w = wavelength (A)
a = beam area (um^2)

The dose (D) to a sample of protein/water/plastic under a given beam 
will be roughly:

D = f*t*w^2/a/2000  (in Gy)

For example: a crystal under a square 100 um x 100 um beam at 1 A 
wavelength with flux 1e12 ph/s will get 1 MGy dose in 20 s.


The 2000/w^2 is a fudge factor that fits the true curve of metal-free 
protein crystals to within 15% for the wavelength range 0.5 < w < 3.  
The error induced by not knowing if the beam was round or square and 
just multiplying together the width and height to get the area (a), is 
21%.  The error from not realizing you had 100 mM uranium in your 
sample is about a factor of two at 1 A.  Smaller concentrations and 
lighter atoms have less impact on accuracy.


If you don't know the flux, or beam size, you can try looking them up 
at http://biosync.sbkb.org/ . I scraped these for my little dose 
calculator here:

bl831.als.lbl.gov/xtallife.html
Some of the biosync numbers are more accurate than others, however, 
depending on how often beamline scientists remember to update the 
site, and how well they know themselves.  And attenuation is not 
always written into the header either.


In a pinch, you can estimate the flux by the total number of photons 
on the image (P).  This is assuming that you know the sample thickness 
(L) in microns.  You must also assume that the total scattering cross 
section of the atoms in the sample is close to that of oxygen (0.2 
cm^2/g), that the sample density is 1.2 g/cm^3 and that about 50% of 
the scattered photons reach the detector.  None of these are terrible 
assumptions. The equation then becomes:


f = P/t/L*1.2e-5

Where 1.2e-5 = 0.2 cm^2/g * 1.2 g/cm^3 * 1e-4 cm/micron * 50%, f=flux 
and t=exposure (as above).


Getting P from a pixel array is easy: you just add up all the pixel 
values.  From a CCD you want to be careful to subtract the baseline 
value from each pixel first (40 on ADSC, 10 on Mar/Rayonix), and then 
divide by the "gain", which near 1 A is ~1 on Mar/Rayonix, 0.6 for 
ADSC Q315 (swbin) and 1.8 for Q315r (hwbin).  A few considerations, 
yes, but it can be a good sanity check.


-James Holton
MAD Scientist


On 5/5/2020 11:48 AM, Murpholino Peligro wrote:
Do diffraction patterns publicly accessible contain information about 
the x-ray absorbed dose?




Thanks



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1








To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1