Re: [ccp4bb] MX data processing with GPUs??

[James Holton]

g difference between imaging 
and MX.

Imaging processing is limited by FFT performance, which needs floating point 
performance. Libraries for FFT on GPUs are standard and provided by hardware 
vendors, so it is easy to implement.

On the other hand MX algorithms for image processing, at least the one I know 
of, do only handful of FLOP per pixel and they will probably not benefit from 
GPU processing significantly, even if ported to such architecture – which would 
be also a non-negligible effort. So while it is not impossible to imagine 
GPU-accelerated MX software and hopefully people are working on this, it is not 
a low hanging fruit, like in case of GPU acceleration for imaging or cryo-EM.

On a side note if one could find a way to use machine learning for data 
processing and implement data processing pipeline in Tensorflow, then GPUs 
would pay off quickly.

Regarding Tim’s Raspberry Pi argument – it should be compared with Nvidia 
Jetson price, which is more or less RPi with GPU, and it won’t be actually that 
significant difference.

Best,
Filip


From: CCP4 bulletin board mailto:CCP4BB@JISCMAIL.AC.UK>> on behalf 
of Ana Carolina de Mattos Zeri mailto:ana.z...@lnls.br>>
Reply to: Ana Carolina de Mattos Zeri 
mailto:ana.z...@lnls.br>>
Date: Tuesday, 18 February 2020 at 20:58
To: "CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>" 
mailto:CCP4BB@JISCMAIL.AC.UK>>
Subject: [ccp4bb] MX data processing with GPUs??

Dear all
we have asked this of a few people, but the question remains:
does any of you have experienced/tried using GPU based software to treat MX 
data? for reducing or subsequent image analysis?
is it a lost battle?
how do you deal with the crescent amount of data we are facing, at Synchrotrons 
and XFELs?
Here at the Manaca beamline at Sirius we will continue to support CPU based 
software, but due to developments in the imaging beam lines, GPU machines are 
looking very attractive.
many thanks in advance for your thoughts,
all the best
Ana


Ana Carolina Zeri, PhD
Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
(19) 3518-2498
www.lnls.br<http://www.lnls.br/>
ana.z...@lnls.br<mailto:ana.z...@lnls.br>






Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais 
e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual 
consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você 
recebeu esta mensagem por engano, por favor avise o remetente e apague-a 
imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or 
privileged information. Observe its content carefully and consider possible 
querying to the sender before copying, disclosing or distributing it. If you 
have received this email by mistake, please notify the sender and delete it 
immediately.




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1


--
This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] MX data processing with GPUs??

[Kay Diederichs]

yte of data. It also needs to be highly 
>>> parallel, preferably without conditional (if/else) statements. Finally, 
>>> there is a variety of GPU architectures on the market and it is not exactly 
>>> obvious that code written for one GPU will be optimal on another one. So if 
>>> the code is based on a general purpose library, it will be easier to make 
>>> sure that it runs efficiently on all GPU hardware.
>>>
>>> I believe combination of these factors makes a big difference between 
>>> imaging and MX.
>>>
>>> Imaging processing is limited by FFT performance, which needs floating 
>>> point performance. Libraries for FFT on GPUs are standard and provided by 
>>> hardware vendors, so it is easy to implement.
>>>
>>> On the other hand MX algorithms for image processing, at least the one I 
>>> know of, do only handful of FLOP per pixel and they will probably not 
>>> benefit from GPU processing significantly, even if ported to such 
>>> architecture – which would be also a non-negligible effort. So while it is 
>>> not impossible to imagine GPU-accelerated MX software and hopefully people 
>>> are working on this, it is not a low hanging fruit, like in case of GPU 
>>> acceleration for imaging or cryo-EM.
>>>
>>> On a side note if one could find a way to use machine learning for data 
>>> processing and implement data processing pipeline in Tensorflow, then GPUs 
>>> would pay off quickly.
>>>
>>> Regarding Tim’s Raspberry Pi argument – it should be compared with Nvidia 
>>> Jetson price, which is more or less RPi with GPU, and it won’t be actually 
>>> that significant difference.
>>>
>>> Best,
>>> Filip
>>>
>>>
>>> From: CCP4 bulletin board 
>>> mailto:CCP4BB@JISCMAIL.AC.UK>> on behalf of Ana 
>>> Carolina de Mattos Zeri mailto:ana.z...@lnls.br>>
>>> Reply to: Ana Carolina de Mattos Zeri 
>>> mailto:ana.z...@lnls.br>>
>>> Date: Tuesday, 18 February 2020 at 20:58
>>> To: "CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>" 
>>> mailto:CCP4BB@JISCMAIL.AC.UK>>
>>> Subject: [ccp4bb] MX data processing with GPUs??
>>>
>>> Dear all
>>> we have asked this of a few people, but the question remains:
>>> does any of you have experienced/tried using GPU based software to treat MX 
>>> data? for reducing or subsequent image analysis?
>>> is it a lost battle?
>>> how do you deal with the crescent amount of data we are facing, at 
>>> Synchrotrons and XFELs?
>>> Here at the Manaca beamline at Sirius we will continue to support CPU based 
>>> software, but due to developments in the imaging beam lines, GPU machines 
>>> are looking very attractive.
>>> many thanks in advance for your thoughts,
>>> all the best
>>> Ana
>>>
>>>
>>> Ana Carolina Zeri, PhD
>>> Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
>>> Brazilian Synchrotron Light Laboratory (LNLS)
>>> Brazilian Center for Research in Energy and Materials (CNPEM)
>>> Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
>>> (19) 3518-2498
>>> www.lnls.br<http://www.lnls.br/>
>>> ana.z...@lnls.br<mailto:ana.z...@lnls.br>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Aviso Legal: Esta mensagem e seus anexos podem conter informações 
>>> confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e 
>>> considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou 
>>> distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o 
>>> remetente e apague-a imediatamente.
>>>
>>> Disclaimer: This email and its attachments may contain confidential and/or 
>>> privileged information. Observe its content carefully and consider possible 
>>> querying to the sender before copying, disclosing or distributing it. If 
>>> you have received this email by mistake, please notify the sender and 
>>> delete it immediately.
>>>
>>>
>>> 
>>>
>>> To unsubscribe from the CCP4BB list, click the following link:
>>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1
>>>
>>> 
>>>
>>> To unsubscribe from the CCP4BB list, click the following link:
>&g

Re: [ccp4bb] MX data processing with GPUs??

[Soisson, Stephen M]

We are doing more and more large sets of fragment and other ligand screening by 
x-ray these days, sometimes with the good folks at Diamond, sometimes 
elsewhere.  I only see this trend increasing (at least for us)  in the future 
as beamlines get even brighter and these types of screens can be done on 
microcrystals using robotics, etc.

My $0.02,

Steve

From: CCP4 bulletin board  On Behalf Of Frank Von Delft
Sent: Wednesday, February 19, 2020 8:52 AM
To: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] MX data processing with GPUs??

EXTERNAL EMAIL – Use caution with any links or file attachments.
Since Kay asks:

Do all of us really want and need to collect from thousands of crystals every 
synchrotron day? Are all of us really producing that many crystals? Who is?
I can't help to respond:  even if not "all of us", we certainly are!
https://www.diamond.ac.uk/Instruments/Mx/Fragment-Screening.html


We're now at <2min/crystal, including X-ray centring and a full dataset - so 
along with all Diamond's wizards, thank you Developers for making data 
processing magically fast and parallizable!

While we're at it, next deadline for proposals is April 1st:

https://www.diamond.ac.uk/Instruments/Mx/Fragment-Screening/Applying-for-XChem.html

(Apologies the shameless hijacking of the the thread...)



On 19/02/2020 12:21, Kay Diederichs wrote:

Dear Ana,



it is easy to ask the question (and I've been asked several times), but 
somewhat difficult to answer. To add to Graeme's excellent explanations:



- all developers of MX processing software have seriously considered to 
implement their algorithms on GPUs, and have decided that the effort (which is 
very significant) is not worth it, in terms of benefit for users and developers 
(who should pay them for the effort? - after all, this would not result in a 
highly cited publication!). We are aware of the fact that they are much faster 
than CPUs for specific types of calculations that are most useful for images 
where each pixel is treated in the same way - but these types of (potentially 
highly parallel) calculations do not represent a large fraction of where a MX 
data processing program spends its time, and even worse, the parallel and 
serial parts of the calculation alternate in fast succession (XDS has on the 
order of 10 parallel regions, none of which dominates the CPU time). 
Ultimately, it is the serial fraction of a program that determines its 
potential speed-up, due to Amdahl's law.



- MX data processing programs (at least XDS and DIALS) already exploit 
parallelism by using multiple CPUs at the same time; the current version of XDS 
can in principle use up to 99*99=9801 processors, and 60 machines each running 
60 threads (see below) would process a 360° dataset composed of 0.1° frames 
within seconds, if DELPHI=6.



- the recent Ryzen Threadripper 3XXX series CPUs have a significantly better 
cost/performance ratio than other processor families. A TR 3970X workstation 
can be bought for less than 5000€, and offers 64 threads. Graeme mentions AMD 
Rome; this is the server variant. The data transfer would become the 
bottleneck; to me, a cluster of workstations each equipped with two 10Gb ports 
looks attractive.



Finally, I have the feeling that speed in data collection and processing is 
over-rated. I get the impression that some (many?) people think they should 
collect a data set as quickly as the machine permits. But they may not be aware 
of the fact that the quality of the data is then not optimal. Going 10 times 
slower, and reducing the transmission to 10%, gives a resonable safety margin.



Further questions arise - does every crap crystal have to be put into the beam? 
And does every crap data set have to be processed? Do all of us really want and 
need to collect from thousands of crystals every synchrotron day? Are all of us 
really producing that many crystals? Who is? (you probably realize my lack of 
imagination by now)

I know that people who build and run synchrotron beamlines have a different 
perspective, concerning these questions, than their users. Some common sense, 
and a lot of discussion, would  benefit our community more than resorting to 
technological "solutions".



best wishes,

Kay



On Wed, 19 Feb 2020 08:08:40 +, Winter, Graeme (DLSLtd,RAL,LSCI) 
<mailto:graeme.win...@diamond.ac.uk> wrote:



Dear Ana,



To follow up on the contributions from others, there are some particular 
annoyances with MX processing which differentiate it from other “big data” or 
imaging problems.



In tomographic reconstruction you have a big block of data which needs to (as a 
simplistic approximation) be transformed by a bunch of trigonometric functions 
to another big block of data. The shape of the calculation is the same 
independent of the data itself, and overall this represents a massively 
parallel computationally expensive problem, which makes it worth the cost of 
getting t

Re: [ccp4bb] MX data processing with GPUs??

[Frank Von Delft]

n terms of GPU, the data need to 
typically pass through main memory three times - for spot finding you need to 
look at every pixel, and integration typically needs to load full frames to 
extract the profiles and then fit them (the shoebox regions can be cached 
between these, but they still need to pass in and out of the CPU). Since moving 
data in and out of memory is expensive and GPU memory is expensive this is a 
problem. For reference, a typical Eiger 16M data set uncompressed needs about 
half a terabyte of RAM (7,200 * 18 megapixels * 4 bytes) so in memory 
processing presents real challenges. The image analysis calculations themselves 
are typically rather light weight floating point work (e.g. summed area table 
calculations) without a lot of trigonometry.

All this, combined with the annoying habit of using words like “if” and “for” 
in the code (which kills GPU calculations dead) mean that even for spot finding 
it’s not worth the effort of moving the data into a GPU - we DIALS folks looked 
into this a couple of years back with a specialist from NVIDIA.

For what it’s worth we have spent some time looking at this here at Diamond, 
where we have a certain interest in speedy processing of MX data and the 
current (2020/02) best bang for buck appears to be AMD Rome.

We as a community have a challenge with keeping up with high data rate 
beamlines at both synchrotrons and FELs - I feel it is important to keep an eye 
on emerging technology and make best use of it (and share experiences of using 
it!) but we should also keep in mind that the processing done in MX is actually 
rather well defined and mathematical at its heart. It is very unlikely that 
deep learning will help with the mathematical challenges we face [1] as we know 
exactly the calculations we need to do (which are very well documented in the 
literature, thank you to everyone who has written these up over the years) and 
instead a clear focus on making the maths fast is needed.

Up to the point where someone comes up with a completely new way of looking at 
the data, of course. I’m sure someone out there is looking at this :-)

On the topic of raspberry pi machines ;-) these are fun but I would hate to 
look at the interconnect necessary to get enough boards to work together to 
keep up with a single AMD Rome box…

best wishes Graeme

[1] with the possible exception of classifying individual found spots and other 
niche areas


On 19 Feb 2020, at 07:04, Leonarski Filip Karol (PSI) 
mailto:filip.leonar...@psi.ch><mailto:filip.leonar...@psi.ch><mailto:filip.leonar...@psi.ch>>
 wrote:

Dear Ana,

To benefit from GPU architecture, over CPU, the algorithm needs to do quite 
significant number crunching – i.e. do at least certain number of floating 
point operations (FLOP) per one byte of data. It also needs to be highly 
parallel, preferably without conditional (if/else) statements. Finally, there 
is a variety of GPU architectures on the market and it is not exactly obvious 
that code written for one GPU will be optimal on another one. So if the code is 
based on a general purpose library, it will be easier to make sure that it runs 
efficiently on all GPU hardware.

I believe combination of these factors makes a big difference between imaging 
and MX.

Imaging processing is limited by FFT performance, which needs floating point 
performance. Libraries for FFT on GPUs are standard and provided by hardware 
vendors, so it is easy to implement.

On the other hand MX algorithms for image processing, at least the one I know 
of, do only handful of FLOP per pixel and they will probably not benefit from 
GPU processing significantly, even if ported to such architecture – which would 
be also a non-negligible effort. So while it is not impossible to imagine 
GPU-accelerated MX software and hopefully people are working on this, it is not 
a low hanging fruit, like in case of GPU acceleration for imaging or cryo-EM.

On a side note if one could find a way to use machine learning for data 
processing and implement data processing pipeline in Tensorflow, then GPUs 
would pay off quickly.

Regarding Tim’s Raspberry Pi argument – it should be compared with Nvidia 
Jetson price, which is more or less RPi with GPU, and it won’t be actually that 
significant difference.

Best,
Filip


From: CCP4 bulletin board 
mailto:CCP4BB@JISCMAIL.AC.UK><mailto:CCP4BB@JISCMAIL.AC.UK><mailto:CCP4BB@JISCMAIL.AC.UK>>
 on behalf of Ana Carolina de Mattos Zeri 
mailto:ana.z...@lnls.br><mailto:ana.z...@lnls.br><mailto:ana.z...@lnls.br>>
Reply to: Ana Carolina de Mattos Zeri 
mailto:ana.z...@lnls.br><mailto:ana.z...@lnls.br><mailto:ana.z...@lnls.br>>
Date: Tuesday, 18 February 2020 at 20:58
To: 
"CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK><mailto:CCP4BB@JISCMAIL.AC.UK><mailto:CCP4BB@JISCMAIL.AC.UK>"
 
mailto:CCP4BB@JISCMAIL.AC.UK><mailto:CCP4BB@JISCMAIL.AC.UK><m

Re: [ccp4bb] MX data processing with GPUs??

[David Schuller]

e analysis calculations themselves 
are typically rather light weight floating point work (e.g. summed area table 
calculations) without a lot of trigonometry.

All this, combined with the annoying habit of using words like “if” and “for” 
in the code (which kills GPU calculations dead) mean that even for spot finding 
it’s not worth the effort of moving the data into a GPU - we DIALS folks looked 
into this a couple of years back with a specialist from NVIDIA.

For what it’s worth we have spent some time looking at this here at Diamond, 
where we have a certain interest in speedy processing of MX data and the 
current (2020/02) best bang for buck appears to be AMD Rome.

We as a community have a challenge with keeping up with high data rate 
beamlines at both synchrotrons and FELs - I feel it is important to keep an eye 
on emerging technology and make best use of it (and share experiences of using 
it!) but we should also keep in mind that the processing done in MX is actually 
rather well defined and mathematical at its heart. It is very unlikely that 
deep learning will help with the mathematical challenges we face [1] as we know 
exactly the calculations we need to do (which are very well documented in the 
literature, thank you to everyone who has written these up over the years) and 
instead a clear focus on making the maths fast is needed.

Up to the point where someone comes up with a completely new way of looking at 
the data, of course. I’m sure someone out there is looking at this :-)

On the topic of raspberry pi machines ;-) these are fun but I would hate to 
look at the interconnect necessary to get enough boards to work together to 
keep up with a single AMD Rome box…

best wishes Graeme

[1] with the possible exception of classifying individual found spots and other 
niche areas


On 19 Feb 2020, at 07:04, Leonarski Filip Karol (PSI) 
mailto:filip.leonar...@psi.ch>> wrote:

Dear Ana,

To benefit from GPU architecture, over CPU, the algorithm needs to do quite 
significant number crunching – i.e. do at least certain number of floating 
point operations (FLOP) per one byte of data. It also needs to be highly 
parallel, preferably without conditional (if/else) statements. Finally, there 
is a variety of GPU architectures on the market and it is not exactly obvious 
that code written for one GPU will be optimal on another one. So if the code is 
based on a general purpose library, it will be easier to make sure that it runs 
efficiently on all GPU hardware.

I believe combination of these factors makes a big difference between imaging 
and MX.

Imaging processing is limited by FFT performance, which needs floating point 
performance. Libraries for FFT on GPUs are standard and provided by hardware 
vendors, so it is easy to implement.

On the other hand MX algorithms for image processing, at least the one I know 
of, do only handful of FLOP per pixel and they will probably not benefit from 
GPU processing significantly, even if ported to such architecture – which would 
be also a non-negligible effort. So while it is not impossible to imagine 
GPU-accelerated MX software and hopefully people are working on this, it is not 
a low hanging fruit, like in case of GPU acceleration for imaging or cryo-EM.

On a side note if one could find a way to use machine learning for data 
processing and implement data processing pipeline in Tensorflow, then GPUs 
would pay off quickly.

Regarding Tim’s Raspberry Pi argument – it should be compared with Nvidia 
Jetson price, which is more or less RPi with GPU, and it won’t be actually that 
significant difference.

Best,
Filip


From: CCP4 bulletin board mailto:CCP4BB@JISCMAIL.AC.UK>> on behalf 
of Ana Carolina de Mattos Zeri mailto:ana.z...@lnls.br>>
Reply to: Ana Carolina de Mattos Zeri 
mailto:ana.z...@lnls.br>>
Date: Tuesday, 18 February 2020 at 20:58
To: "CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>" 
mailto:CCP4BB@JISCMAIL.AC.UK>>
Subject: [ccp4bb] MX data processing with GPUs??

Dear all
we have asked this of a few people, but the question remains:
does any of you have experienced/tried using GPU based software to treat MX 
data? for reducing or subsequent image analysis?
is it a lost battle?
how do you deal with the crescent amount of data we are facing, at Synchrotrons 
and XFELs?
Here at the Manaca beamline at Sirius we will continue to support CPU based 
software, but due to developments in the imaging beam lines, GPU machines are 
looking very attractive.
many thanks in advance for your thoughts,
all the best
Ana


Ana Carolina Zeri, PhD
Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
(19) 3518-2498
www.lnls.br<http://www.lnls.br/>
ana.z...@lnls.br<mailto:ana.z...@lnls.br>






Aviso Legal: Esta mensag

Re: [ccp4bb] MX data processing with GPUs??

[Kay Diederichs]

nt work (e.g. summed 
>area table calculations) without a lot of trigonometry.
>
>All this, combined with the annoying habit of using words like “if” and “for” 
>in the code (which kills GPU calculations dead) mean that even for spot 
>finding it’s not worth the effort of moving the data into a GPU - we DIALS 
>folks looked into this a couple of years back with a specialist from NVIDIA.
>
>For what it’s worth we have spent some time looking at this here at Diamond, 
>where we have a certain interest in speedy processing of MX data and the 
>current (2020/02) best bang for buck appears to be AMD Rome.
>
>We as a community have a challenge with keeping up with high data rate 
>beamlines at both synchrotrons and FELs - I feel it is important to keep an 
>eye on emerging technology and make best use of it (and share experiences of 
>using it!) but we should also keep in mind that the processing done in MX is 
>actually rather well defined and mathematical at its heart. It is very 
>unlikely that deep learning will help with the mathematical challenges we face 
>[1] as we know exactly the calculations we need to do (which are very well 
>documented in the literature, thank you to everyone who has written these up 
>over the years) and instead a clear focus on making the maths fast is needed.
>
>Up to the point where someone comes up with a completely new way of looking at 
>the data, of course. I’m sure someone out there is looking at this :-)
>
>On the topic of raspberry pi machines ;-) these are fun but I would hate to 
>look at the interconnect necessary to get enough boards to work together to 
>keep up with a single AMD Rome box…
>
>best wishes Graeme
>
>[1] with the possible exception of classifying individual found spots and 
>other niche areas
>
>
>On 19 Feb 2020, at 07:04, Leonarski Filip Karol (PSI) 
>mailto:filip.leonar...@psi.ch>> wrote:
>
>Dear Ana,
>
>To benefit from GPU architecture, over CPU, the algorithm needs to do quite 
>significant number crunching – i.e. do at least certain number of floating 
>point operations (FLOP) per one byte of data. It also needs to be highly 
>parallel, preferably without conditional (if/else) statements. Finally, there 
>is a variety of GPU architectures on the market and it is not exactly obvious 
>that code written for one GPU will be optimal on another one. So if the code 
>is based on a general purpose library, it will be easier to make sure that it 
>runs efficiently on all GPU hardware.
>
>I believe combination of these factors makes a big difference between imaging 
>and MX.
>
>Imaging processing is limited by FFT performance, which needs floating point 
>performance. Libraries for FFT on GPUs are standard and provided by hardware 
>vendors, so it is easy to implement.
>
>On the other hand MX algorithms for image processing, at least the one I know 
>of, do only handful of FLOP per pixel and they will probably not benefit from 
>GPU processing significantly, even if ported to such architecture – which 
>would be also a non-negligible effort. So while it is not impossible to 
>imagine GPU-accelerated MX software and hopefully people are working on this, 
>it is not a low hanging fruit, like in case of GPU acceleration for imaging or 
>cryo-EM.
>
>On a side note if one could find a way to use machine learning for data 
>processing and implement data processing pipeline in Tensorflow, then GPUs 
>would pay off quickly.
>
>Regarding Tim’s Raspberry Pi argument – it should be compared with Nvidia 
>Jetson price, which is more or less RPi with GPU, and it won’t be actually 
>that significant difference.
>
>Best,
>Filip
>
>
>From: CCP4 bulletin board 
>mailto:CCP4BB@JISCMAIL.AC.UK>> on behalf of Ana 
>Carolina de Mattos Zeri mailto:ana.z...@lnls.br>>
>Reply to: Ana Carolina de Mattos Zeri 
>mailto:ana.z...@lnls.br>>
>Date: Tuesday, 18 February 2020 at 20:58
>To: "CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>" 
>mailto:CCP4BB@JISCMAIL.AC.UK>>
>Subject: [ccp4bb] MX data processing with GPUs??
>
>Dear all
>we have asked this of a few people, but the question remains:
>does any of you have experienced/tried using GPU based software to treat MX 
>data? for reducing or subsequent image analysis?
>is it a lost battle?
>how do you deal with the crescent amount of data we are facing, at 
>Synchrotrons and XFELs?
>Here at the Manaca beamline at Sirius we will continue to support CPU based 
>software, but due to developments in the imaging beam lines, GPU machines are 
>looking very attractive.
>many thanks in advance for your thoughts,
>all the best
>Ana
>
>
>Ana Carolina Zeri, PhD
>Manaca Beamline Coordinator (Macromolecu

Re: [ccp4bb] MX data processing with GPUs??

[Winter, Graeme (DLSLtd,RAL,LSCI)]

.

I believe combination of these factors makes a big difference between imaging 
and MX.

Imaging processing is limited by FFT performance, which needs floating point 
performance. Libraries for FFT on GPUs are standard and provided by hardware 
vendors, so it is easy to implement.

On the other hand MX algorithms for image processing, at least the one I know 
of, do only handful of FLOP per pixel and they will probably not benefit from 
GPU processing significantly, even if ported to such architecture – which would 
be also a non-negligible effort. So while it is not impossible to imagine 
GPU-accelerated MX software and hopefully people are working on this, it is not 
a low hanging fruit, like in case of GPU acceleration for imaging or cryo-EM.

On a side note if one could find a way to use machine learning for data 
processing and implement data processing pipeline in Tensorflow, then GPUs 
would pay off quickly.

Regarding Tim’s Raspberry Pi argument – it should be compared with Nvidia 
Jetson price, which is more or less RPi with GPU, and it won’t be actually that 
significant difference.

Best,
Filip


From: CCP4 bulletin board mailto:CCP4BB@JISCMAIL.AC.UK>> 
on behalf of Ana Carolina de Mattos Zeri 
mailto:ana.z...@lnls.br>>
Reply to: Ana Carolina de Mattos Zeri 
mailto:ana.z...@lnls.br>>
Date: Tuesday, 18 February 2020 at 20:58
To: "CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>" 
mailto:CCP4BB@JISCMAIL.AC.UK>>
Subject: [ccp4bb] MX data processing with GPUs??

Dear all
we have asked this of a few people, but the question remains:
does any of you have experienced/tried using GPU based software to treat MX 
data? for reducing or subsequent image analysis?
is it a lost battle?
how do you deal with the crescent amount of data we are facing, at Synchrotrons 
and XFELs?
Here at the Manaca beamline at Sirius we will continue to support CPU based 
software, but due to developments in the imaging beam lines, GPU machines are 
looking very attractive.
many thanks in advance for your thoughts,
all the best
Ana


Ana Carolina Zeri, PhD
Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
(19) 3518-2498
www.lnls.br<http://www.lnls.br/>
ana.z...@lnls.br<mailto:ana.z...@lnls.br>






Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais 
e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual 
consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você 
recebeu esta mensagem por engano, por favor avise o remetente e apague-a 
imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or 
privileged information. Observe its content carefully and consider possible 
querying to the sender before copying, disclosing or distributing it. If you 
have received this email by mistake, please notify the sender and delete it 
immediately.




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1



--

This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] MX data processing with GPUs??

[Winter, Graeme (DLSLtd,RAL,LSCI)]

orking on this, it is not 
a low hanging fruit, like in case of GPU acceleration for imaging or cryo-EM.

On a side note if one could find a way to use machine learning for data 
processing and implement data processing pipeline in Tensorflow, then GPUs 
would pay off quickly.

Regarding Tim’s Raspberry Pi argument – it should be compared with Nvidia 
Jetson price, which is more or less RPi with GPU, and it won’t be actually that 
significant difference.

Best,
Filip


From: CCP4 bulletin board mailto:CCP4BB@JISCMAIL.AC.UK>> 
on behalf of Ana Carolina de Mattos Zeri 
mailto:ana.z...@lnls.br>>
Reply to: Ana Carolina de Mattos Zeri 
mailto:ana.z...@lnls.br>>
Date: Tuesday, 18 February 2020 at 20:58
To: "CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>" 
mailto:CCP4BB@JISCMAIL.AC.UK>>
Subject: [ccp4bb] MX data processing with GPUs??

Dear all
we have asked this of a few people, but the question remains:
does any of you have experienced/tried using GPU based software to treat MX 
data? for reducing or subsequent image analysis?
is it a lost battle?
how do you deal with the crescent amount of data we are facing, at Synchrotrons 
and XFELs?
Here at the Manaca beamline at Sirius we will continue to support CPU based 
software, but due to developments in the imaging beam lines, GPU machines are 
looking very attractive.
many thanks in advance for your thoughts,
all the best
Ana


Ana Carolina Zeri, PhD
Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
(19) 3518-2498
www.lnls.br<http://www.lnls.br/>
ana.z...@lnls.br<mailto:ana.z...@lnls.br>






Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais 
e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual 
consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você 
recebeu esta mensagem por engano, por favor avise o remetente e apague-a 
imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or 
privileged information. Observe its content carefully and consider possible 
querying to the sender before copying, disclosing or distributing it. If you 
have received this email by mistake, please notify the sender and delete it 
immediately.




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1


-- 
This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] MX data processing with GPUs??

[Leonarski Filip Karol (PSI)]

Dear Ana,

To benefit from GPU architecture, over CPU, the algorithm needs to do quite 
significant number crunching – i.e. do at least certain number of floating 
point operations (FLOP) per one byte of data. It also needs to be highly 
parallel, preferably without conditional (if/else) statements. Finally, there 
is a variety of GPU architectures on the market and it is not exactly obvious 
that code written for one GPU will be optimal on another one. So if the code is 
based on a general purpose library, it will be easier to make sure that it runs 
efficiently on all GPU hardware.

I believe combination of these factors makes a big difference between imaging 
and MX.

Imaging processing is limited by FFT performance, which needs floating point 
performance. Libraries for FFT on GPUs are standard and provided by hardware 
vendors, so it is easy to implement.

On the other hand MX algorithms for image processing, at least the one I know 
of, do only handful of FLOP per pixel and they will probably not benefit from 
GPU processing significantly, even if ported to such architecture – which would 
be also a non-negligible effort. So while it is not impossible to imagine 
GPU-accelerated MX software and hopefully people are working on this, it is not 
a low hanging fruit, like in case of GPU acceleration for imaging or cryo-EM.

On a side note if one could find a way to use machine learning for data 
processing and implement data processing pipeline in Tensorflow, then GPUs 
would pay off quickly.

Regarding Tim’s Raspberry Pi argument – it should be compared with Nvidia 
Jetson price, which is more or less RPi with GPU, and it won’t be actually that 
significant difference.

Best,
Filip


From: CCP4 bulletin board  on behalf of Ana Carolina de 
Mattos Zeri 
Reply to: Ana Carolina de Mattos Zeri 
Date: Tuesday, 18 February 2020 at 20:58
To: "CCP4BB@JISCMAIL.AC.UK" 
Subject: [ccp4bb] MX data processing with GPUs??

Dear all
we have asked this of a few people, but the question remains:
does any of you have experienced/tried using GPU based software to treat MX 
data? for reducing or subsequent image analysis?
is it a lost battle?
how do you deal with the crescent amount of data we are facing, at Synchrotrons 
and XFELs?
Here at the Manaca beamline at Sirius we will continue to support CPU based 
software, but due to developments in the imaging beam lines, GPU machines are 
looking very attractive.
many thanks in advance for your thoughts,
all the best
Ana


Ana Carolina Zeri, PhD
Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
(19) 3518-2498
www.lnls.br<http://www.lnls.br>
ana.z...@lnls.br<mailto:ana.z...@lnls.br>






Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais 
e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual 
consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você 
recebeu esta mensagem por engano, por favor avise o remetente e apague-a 
imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or 
privileged information. Observe its content carefully and consider possible 
querying to the sender before copying, disclosing or distributing it. If you 
have received this email by mistake, please notify the sender and delete it 
immediately.



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] MX data processing with GPUs??

[Chun Hong Yoon]

Dear Ana,

how do you deal with the crescent amount of data we are facing, at
Synchrotrons and XFELs?
At LCLS, we plan to stream data to near-by supercomputing facilities for serial 
crystallography data processing. So we are developing code to fully utilize 
both CPUs and GPUs on these machines.

I agree with Tim that CPUs are more cost effective, but in my opinion GPUs are 
worth it if you plan to incorporate deep neural networks / compression / 
simulation etc in your data processing pipeline.

Best,
Chuck
--
Chunhong Yoon, Ph.D.

Head of LCLS Data Analytics
LCLS / SLAC National Accelerator Laboratory
2575 Sand Hill Road
Menlo Park, CA 94025
Bldg. 901, Rm. 132


On Feb 18, 2020, at 1:44 PM, Tim Gruene 
mailto:tim.gru...@univie.ac.at>> wrote:

Dear Ana,

you may want to check how many single board computers (like the Raspberry Pi
or true open source hardware platform) you can buy for a single GPU high end
graphics card.
It may be cheaper at better performance compared to GPU computing.

XDS (and possibly also DIALS) can distribute its job across ssh, hence you
only need the appropriate network connections.

XDS wouldn't run on Raspberry Pi, although it may not be difficult to port.

Best,
Tim


On Tuesday, February 18, 2020 8:48:17 PM CET Ana Carolina de Mattos Zeri
wrote:
Dear all
we have asked this of a few people, but the question remains:
does any of you have experienced/tried using GPU based software to treat MX
data? for reducing or subsequent image analysis? is it a lost battle?
how do you deal with the crescent amount of data we are facing, at
Synchrotrons and XFELs? Here at the Manaca beamline at Sirius we will
continue to support CPU based software, but due to developments in the
imaging beam lines, GPU machines are looking very attractive. many thanks
in advance for your thoughts,
all the best
Ana



Ana Carolina Zeri, PhD
Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
(19) 3518-2498
www.lnls.br>
ana.z...@lnls.br








Aviso Legal: Esta mensagem e seus anexos podem conter informa??es
confidenciais e/ou de uso restrito. Observe atentamente seu conte?do e
considere eventual consulta ao remetente antes de copi?-la, divulg?-la ou
distribu?-la. Se voc? recebeu esta mensagem por engano, por favor avise o
remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or
privileged information. Observe its content carefully and consider possible
querying to the sender before copying, disclosing or distributing it. If
you have received this email by mistake, please notify the sender and
delete it immediately.



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

--
--
Tim Gruene
Head of the Centre for X-ray Structure Analysis
Faculty of Chemistry
University of Vienna

Phone: +43-1-4277-70202

GPG Key ID = A46BEE1A



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1




To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

Re: [ccp4bb] MX data processing with GPUs??

[Tim Gruene]

Dear Ana,

you may want to check how many single board computers (like the Raspberry Pi 
or true open source hardware platform) you can buy for a single GPU high end 
graphics card. 
It may be cheaper at better performance compared to GPU computing.

XDS (and possibly also DIALS) can distribute its job across ssh, hence you 
only need the appropriate network connections. 

XDS wouldn't run on Raspberry Pi, although it may not be difficult to port.

Best,
Tim


On Tuesday, February 18, 2020 8:48:17 PM CET Ana Carolina de Mattos Zeri 
wrote:
> Dear all
> we have asked this of a few people, but the question remains:
> does any of you have experienced/tried using GPU based software to treat MX
> data? for reducing or subsequent image analysis? is it a lost battle?
> how do you deal with the crescent amount of data we are facing, at
> Synchrotrons and XFELs? Here at the Manaca beamline at Sirius we will
> continue to support CPU based software, but due to developments in the
> imaging beam lines, GPU machines are looking very attractive. many thanks
> in advance for your thoughts,
> all the best
> Ana
> 
> 
> 
> Ana Carolina Zeri, PhD
> Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
> Brazilian Synchrotron Light Laboratory (LNLS)
> Brazilian Center for Research in Energy and Materials (CNPEM)
> Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
> (19) 3518-2498
> www.lnls.br
> ana.z...@lnls.br
> 
> 
> 
> 
> 
> 
> 
> 
> Aviso Legal: Esta mensagem e seus anexos podem conter informa??es
> confidenciais e/ou de uso restrito. Observe atentamente seu conte?do e
> considere eventual consulta ao remetente antes de copi?-la, divulg?-la ou
> distribu?-la. Se voc? recebeu esta mensagem por engano, por favor avise o
> remetente e apague-a imediatamente.
> 
> Disclaimer: This email and its attachments may contain confidential and/or
> privileged information. Observe its content carefully and consider possible
> querying to the sender before copying, disclosing or distributing it. If
> you have received this email by mistake, please notify the sender and
> delete it immediately.
> 
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1

-- 
--
Tim Gruene
Head of the Centre for X-ray Structure Analysis
Faculty of Chemistry
University of Vienna

Phone: +43-1-4277-70202

GPG Key ID = A46BEE1A



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1


signature.asc
Description: This is a digitally signed message part.

[ccp4bb] MX data processing with GPUs??

[Ana Carolina de Mattos Zeri]

Dear all
we have asked this of a few people, but the question remains:
does any of you have experienced/tried using GPU based software to treat MX 
data? for reducing or subsequent image analysis?
is it a lost battle?
how do you deal with the crescent amount of data we are facing, at Synchrotrons 
and XFELs?
Here at the Manaca beamline at Sirius we will continue to support CPU based 
software, but due to developments in the imaging beam lines, GPU machines are 
looking very attractive.
many thanks in advance for your thoughts,
all the best
Ana



Ana Carolina Zeri, PhD
Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
(19) 3518-2498
www.lnls.br
ana.z...@lnls.br








Aviso Legal: Esta mensagem e seus anexos podem conter informa??es confidenciais 
e/ou de uso restrito. Observe atentamente seu conte?do e considere eventual 
consulta ao remetente antes de copi?-la, divulg?-la ou distribu?-la. Se voc? 
recebeu esta mensagem por engano, por favor avise o remetente e apague-a 
imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or 
privileged information. Observe its content carefully and consider possible 
querying to the sender before copying, disclosing or distributing it. If you 
have received this email by mistake, please notify the sender and delete it 
immediately.



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB=1