Dear Stamatia,
If the molecules are processed completely independently by your code, it
may be simpler to split the SDF into chunks (e.g. with csplit in a bash
script) and then run separate instances of your python code on each chunk,
wait until all are finished and finally collate the output. Thus you can
avoid the problem altogether.
Best,
Michal
On Wed, 20 Feb 2019 at 11:28, Stamatia Zavitsanou <
stamatia.zavitsa...@oriel.ox.ac.uk> wrote:
> Hello everyone,
>
>
> We have been writing a script that searches though a large number of
> molecules within different files for a common substructure. To speed this
> up we have been attempting to run this script in parallel-see scripts
> below. However online the tutorial notes make reference to problems with
> using the SDMolSupplier in parallel, we were wondering what is the issue
> and how we could circumvent them to speed up some of our calculations.
>
>
> Non-parallel
>
>
> from __future__ import print_function
>
> from rdkit import Chem
>
> import os
>
> from progressbar import ProgressBar
>
> pbar=ProgressBar()
>
> matches = []
>
> directory = 'Q:\Data2'
>
> patt = Chem.MolFromSmarts('NC(N****NC=O)=O')
>
> for file in pbar(os.listdir(directory)):
>
> filename = os.fsdecode(file)
>
> if filename.endswith(".sdf"):
>
> f = os.path.join(directory,filename)
>
> suppl= Chem.SDMolSupplier(f)
>
> for mol in suppl:
>
> if mol is None: continue
>
> if mol.HasSubstructMatch(patt):
>
> matches.append(mol)
>
> w = Chem.SDWriter(r'C:\Users\tom.watts\Desktop\datasmarts4c.sdf')
>
> for m in matches: w.write(m)
>
> print(filename)
>
>
>
> Parallel
>
>
> pbar=ProgressBar()
>
> matches = []
>
> directory = 'E:\Data'
>
> patt = Chem.MolFromSmarts('NC(N****NC=O)=O')
>
> w = Chem.SDWriter(r'C:\Users\tom.watts\Desktop\SearchDataNonly.sdf')
>
> l=[]
>
> for file in pbar(os.listdir(directory)):
>
> filename = os.fsdecode(file)
>
> if filename.endswith(".sdf"):
>
> f = os.path.join(directory,filename)
>
> l.append(f)
>
> num_cores = multiprocessing.cpu_count()
>
> print(num_cores)
>
> lock = multiprocessing.Lock()
>
> def Search(i):
>
> suppl= Chem.SDMolSupplier(i)
>
> for mol in suppl:
>
> if mol is None: continue
>
> if mol.HasSubstructMatch(patt):
>
> matches.append(mol)
>
> return matches
>
> results = Parallel(n_jobs=20)(delayed(Search)(i) for i in l)
>
>
>
> We also wish to use a second script that opens one SDF file and then
> runs a loop over each molecule in the file. This is currently done
> serially and we were wondering if it could be made parallel.
>
>
>
> suppl = Chem.SDMolSupplier('Red3.sdf')
>
> *for* mol *in* suppl:
>
> patt = Chem.MolFromSmarts('NC(N)=O')
>
> num=mol.GetSubstructMatches(patt)
>
> logger.debug(Chem.MolToSmiles(mol))
>
> h=len(num)
>
> m3=Chem.AddHs(mol)
>
> cids =AllChem.EmbedMultipleConfs(m3, numConfs)
>
>
>
> Any comments can be useful.
>
>
> Thanks a lot,
>
> Stamatia Zavitsanou
> _______________________________________________
> Rdkit-discuss mailing list
> Rdkit-discuss@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/rdkit-discuss
>
_______________________________________________
Rdkit-discuss mailing list
Rdkit-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rdkit-discuss
