subject:"\[ccp4bb\] XDS"

Re: [ccp4bb] xds crashes

2022-11-29 Thread Demetres D. Leonidas

Dear Kay,

many thanks for your suggestions. We will look into this.

best

Demetres

On 11/29/2022 2:35 PM, Kay Diederichs wrote:

Dear Demetres,

I googled the Ubuntu 22.04 release notes and found
https://discourse.ubuntu.com/t/jammy-jellyfish-release-notes/24668 which says "We’ve enabled
the userspace OOMD service and are shipping the systemd-oomd package by default on the “Ubuntu
Desktop” flavour, to avoid overloaded systems and the need of the kernel’s OOM killer to kick in.
The OOMD status can be checked using oomctl." Then googling "Ubuntu 22.04 oomctl"
takes me e.g. to https://askubuntu.com/questions/1405021/firefox-closes-by-itself-in-ubuntu-22-04
and https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159 which confirms that there may be
a problem in Ubuntu 22.04 with processes being killed without good reason, and some hints on how to
configure your system to avoid that.

Maybe the systems-oomd package can simply be removed.

Hope that helps,
Kay

On Tue, 29 Nov 2022 10:25:17 +0200, Demetres D. Leonidas
wrote:

Dear Kay,

you are right and I do apologize for the misunderstanding. Point 1 and 2
are sufficient.

best

Demetres

On 11/29/2022 10:00 AM, Kay Diederichs wrote:

Dear Demetres,

sounds a bit drastic!
The second point alone would certainly suffice, because then no forking is done.

What kind of hardware is that?
I would expect that on a contemporary workstation you have 8GB of RAM or more;
that would allow for 8 or more processes.
Do you have a swapfile? If not, create one.

Please check out
https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Performance

Best wishes,
Kay

On Tue, 29 Nov 2022 09:34:49 +0200, Demetres D. Leonidas
wrote:

Dear Kay,

after some digging and help from Max Nanao we found that the problem
was the memory. It was fulling rapidly. The workaround was to:

1. unzip the cbf files prior to processing

2. use maximum number of jobs 1

3. use number of processors 4

4. run xds_par

Many thanks for all your help

Demetres

On 11/29/2022 9:03 AM, Kay Diederichs wrote:

Dear Demetres,

I agree with what James says: this is the operating system trying to to
prevent, in an over-zealous way, the
forking of too many or too big (in terms of memory) processes.
In other words, XDS hasn't changed, but the operating system changed - it
applies different limits to its processes.
Your friendly system administrator should try to find out exactly which
limit(s) changed, and reset the value(s).
Perhaps a hint is in the Release Notes of Ubuntu 22.04 ?

Pls report here when you've found it - currently you are the only one reporting
it.

Best wishes,
Kay

On Mon, 28 Nov 2022 12:06:43 -0800, James Holton wrote:

Sounds like your kernel might think "forkxds" is creating a "fork
bomb". Too many sub-processes firing off in too short a time triggers
this. On one of my systems, I fixed it by editing
/etc/security/limits.d/20-nproc.conf so that users are allowed ~10x more
pids than normal. This tends to prevent these mysterious "Killed" errors.

HTH?

-James Holton
MAD Scientist

On 11/28/2022 7:44 AM, Demetres D. Leonidas wrote:

Dear all,

I do not know if this is the right list and I would like to apologize
if it is not.

We have repeatedly experienced xds crashes in machines running ubuntu
22.04.1 with the following message repeated several times at the
INTEGRATE step

/usr/local/bin/forkxds: line 60: 3427 Done echo "$itask"

3428 Killed | $amain

We are trying to process data from P13 at EMBL-Hamburg.

We are running XDS version Jan 10, 2022 BUILT=20220820

Any ideas ?

Demetres

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/

--
---
Dr. Demetres D. Leonidas
Professor in Biochemistry
Department of Biochemistry & Biotechnology
University of Thessaly
Biopolis 41500 Larissa, Greece
-
Tel. +302410 565278
Tel. +302410 565297 (Lab)
Tel. +304210 565263 (X-ray)
Fax. +302410 565290
E-mail: ddleoni...@bio.uth.gr
ORCID ID orcid.org/-0002-3874-2523
---

--
Αυτό το email έχει ελεγχθεί για ιούς από το Avast antivirus.
www.avast.com

Re: [ccp4bb] xds crashes

2022-11-29 Thread Kay Diederichs

Dear Demetres,

I googled the Ubuntu 22.04 release notes and found 
https://discourse.ubuntu.com/t/jammy-jellyfish-release-notes/24668 which says 
"We’ve enabled the userspace OOMD service and are shipping the systemd-oomd 
package by default on the “Ubuntu Desktop” flavour, to avoid overloaded systems 
and the need of the kernel’s OOM killer to kick in. The OOMD status can be 
checked using oomctl." Then googling "Ubuntu 22.04 oomctl" takes me e.g. to 
https://askubuntu.com/questions/1405021/firefox-closes-by-itself-in-ubuntu-22-04
 and https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159 which 
confirms that there may be a problem in Ubuntu 22.04 with processes being 
killed without good reason, and some hints on how to configure your system to 
avoid that.

Maybe the systems-oomd package can simply be removed.

Hope that helps,
Kay

On Tue, 29 Nov 2022 10:25:17 +0200, Demetres D. Leonidas 
 wrote:

>Dear Kay,
>
>you are right and I do apologize for the misunderstanding. Point 1 and 2 
>are sufficient.
>
>best
>
>Demetres
>
>On 11/29/2022 10:00 AM, Kay Diederichs wrote:
>> Dear Demetres,
>>
>> sounds a bit drastic!
>> The second point alone would certainly suffice, because then no forking is 
>> done.
>>
>> What kind of hardware is that?
>> I would expect that on a contemporary workstation you have 8GB of RAM or 
>> more; that would allow for 8 or more processes.
>> Do you have a swapfile? If not, create one.
>>
>> Please check out 
>> https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Performance
>>
>> Best wishes,
>> Kay
>>
>>
>> On Tue, 29 Nov 2022 09:34:49 +0200, Demetres D. Leonidas 
>>  wrote:
>>
>>> Dear Kay,
>>>
>>> after some digging and help from Max Nanao we found that the problem
>>> was the memory.  It was fulling rapidly. The workaround was to:
>>>
>>> 1. unzip the cbf files prior to processing
>>>
>>> 2. use maximum number of jobs 1
>>>
>>> 3. use number of processors 4
>>>
>>> 4. run xds_par
>>>
>>> Many thanks for all your help
>>>
>>> Demetres
>>>
>>> On 11/29/2022 9:03 AM, Kay Diederichs wrote:
 Dear Demetres,

 I agree with what James says: this is the operating system trying to to 
 prevent, in an over-zealous way, the
 forking of too many or too big (in terms of memory) processes.
 In other words, XDS hasn't changed, but the operating system changed - it 
 applies different limits to its processes.
 Your friendly system administrator should try to find out exactly which 
 limit(s) changed, and reset the value(s).
 Perhaps a hint is in the Release Notes of Ubuntu 22.04 ?

 Pls report here when you've found it - currently you are the only one 
 reporting it.

 Best wishes,
 Kay

 On Mon, 28 Nov 2022 12:06:43 -0800, James Holton  wrote:

> Sounds like your kernel might think "forkxds" is creating a "fork
> bomb".  Too many sub-processes firing off in too short a time triggers
> this.  On one of my systems, I fixed it by editing
> /etc/security/limits.d/20-nproc.conf so that users are allowed ~10x more
> pids than normal.  This tends to prevent these mysterious "Killed" errors.
>
> HTH?
>
> -James Holton
> MAD Scientist
>
>
> On 11/28/2022 7:44 AM, Demetres D. Leonidas wrote:
>> Dear all,
>>
>> I do not know if this is the right list and I would like to apologize
>> if it is not.
>>
>> We have repeatedly experienced xds crashes in machines running ubuntu
>> 22.04.1 with the following message repeated several times at the
>> INTEGRATE step
>>
>> /usr/local/bin/forkxds: line 60:  3427 Done echo "$itask"
>>
>>     3428 Killed  | $amain
>>
>> We are trying to process data from P13 at EMBL-Hamburg.
>>
>> We are running XDS version Jan 10, 2022 BUILT=20220820
>>
>> Any ideas ?
>>
>> Demetres
>>
> 
>
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
>
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a 
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are 
> available at https://www.jiscmail.ac.uk/policyandsecurity/

 To unsubscribe from the CCP4BB list, click the following link:
 https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1

 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing 
 list hosted by www.jiscmail.ac.uk, terms & conditions are available at 
 https://www.jiscmail.ac.uk/policyandsecurity/
>>> -- 
>>> ---
>>> Dr. Demetres D. Leonidas
>>> Professor in Biochemistry
>>> Department of Biochemistry & Biotechnology
>>> University of

Re: [ccp4bb] xds crashes

2022-11-29 Thread ClAuS Flensburg

Dear Demetres,

On Tue, Nov 29, 2022 at 10:25:17AM +0200, Demetres D. Leonidas wrote:
> Dear Kay,
> 
> you are right and I do apologize for the misunderstanding. Point 1
> and 2 are sufficient.

You can probably avoid doing any of this by using the LIB= feature of
XDS on gzipped CBF files with the excellent xds-zcbf plugin [1].

If you have autoPROC available this will be used automatically [2].


Regards,

ClAuS & Clemens

[1] https://git.embl.de/nikolova/xds-zcbf
[2] https://www.globalphasing.com/autoproc

> 
> best
> 
> Demetres
> 
> On 11/29/2022 10:00 AM, Kay Diederichs wrote:
> >Dear Demetres,
> >
> >sounds a bit drastic!
> >The second point alone would certainly suffice, because then no forking is 
> >done.
> >
> >What kind of hardware is that?
> >I would expect that on a contemporary workstation you have 8GB of RAM or 
> >more; that would allow for 8 or more processes.
> >Do you have a swapfile? If not, create one.
> >
> >Please check out 
> >https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Performance
> >
> >Best wishes,
> >Kay
> >
> >
> >On Tue, 29 Nov 2022 09:34:49 +0200, Demetres D. Leonidas 
> > wrote:
> >
> >>Dear Kay,
> >>
> >>after some digging and help from Max Nanao we found that the problem
> >>was the memory.  It was fulling rapidly. The workaround was to:
> >>
> >>1. unzip the cbf files prior to processing
> >>
> >>2. use maximum number of jobs 1
> >>
> >>3. use number of processors 4
> >>
> >>4. run xds_par
> >>
> >>Many thanks for all your help
> >>
> >>Demetres
> >>
> >>On 11/29/2022 9:03 AM, Kay Diederichs wrote:
> >>>Dear Demetres,
> >>>
> >>>I agree with what James says: this is the operating system trying to to 
> >>>prevent, in an over-zealous way, the
> >>>forking of too many or too big (in terms of memory) processes.
> >>>In other words, XDS hasn't changed, but the operating system changed - it 
> >>>applies different limits to its processes.
> >>>Your friendly system administrator should try to find out exactly which 
> >>>limit(s) changed, and reset the value(s).
> >>>Perhaps a hint is in the Release Notes of Ubuntu 22.04 ?
> >>>
> >>>Pls report here when you've found it - currently you are the only one 
> >>>reporting it.
> >>>
> >>>Best wishes,
> >>>Kay
> >>>
> >>>
> >>>On Mon, 28 Nov 2022 12:06:43 -0800, James Holton  wrote:
> >>>
> Sounds like your kernel might think "forkxds" is creating a "fork
> bomb".  Too many sub-processes firing off in too short a time triggers
> this.  On one of my systems, I fixed it by editing
> /etc/security/limits.d/20-nproc.conf so that users are allowed ~10x more
> pids than normal.  This tends to prevent these mysterious "Killed" errors.
> 
> HTH?
> 
> -James Holton
> MAD Scientist
> 
> 
> On 11/28/2022 7:44 AM, Demetres D. Leonidas wrote:
> >Dear all,
> >
> >I do not know if this is the right list and I would like to apologize
> >if it is not.
> >
> >We have repeatedly experienced xds crashes in machines running ubuntu
> >22.04.1 with the following message repeated several times at the
> >INTEGRATE step
> >
> >/usr/local/bin/forkxds: line 60:  3427 Done echo "$itask"
> >
> >    3428 Killed  | $amain
> >
> >We are trying to process data from P13 at EMBL-Hamburg.
> >
> >We are running XDS version Jan 10, 2022 BUILT=20220820
> >
> >Any ideas ?
> >
> >Demetres
> >
> 
> 
> To unsubscribe from the CCP4BB list, click the following link:
> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> 
> This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a 
> mailing list hosted by www.jiscmail.ac.uk, terms & conditions are 
> available at https://www.jiscmail.ac.uk/policyandsecurity/
> >>>
> >>>
> >>>To unsubscribe from the CCP4BB list, click the following link:
> >>>https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB=1
> >>>
> >>>This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing 
> >>>list hosted by www.jiscmail.ac.uk, terms & conditions are available at 
> >>>https://www.jiscmail.ac.uk/policyandsecurity/
> >>-- 
> >>---
> >>Dr. Demetres D. Leonidas
> >>Professor in Biochemistry
> >>Department of Biochemistry & Biotechnology
> >>University of Thessaly
> >>Biopolis 41500 Larissa, Greece
> >>-
> >>Tel. +302410 565278
> >>Tel. +302410 565297 (Lab)
> >>Tel. +304210 565263 (X-ray)
> >>Fax. +302410 565290
> >>E-mail: ddleoni...@bio.uth.gr
> >>ORCID ID orcid.org/-0002-3874-2523
> >>---
> >>
> >>
> >>-- 
> >>  email   ??  ??  
> >>Avast antivirus.
> >>www.avast.com
> >>
>