Hi Stamatia, Yes, SDMolSupplier is not thread safe. My guess is due to the nature of SDF file where a molecule record needs multiple lines and you do not know a-priory the number of lines per molecule in order to split the file to different threads/processes.
Given that your proposed approach is the preferred one. Process each SDF file and return matched molecules using a separate process. I would advice to use concurrent.futures ( https://docs.python.org/3/library/concurrent.futures.html) package instead of multiprocessing. As it provides an abstraction layer on top of multiprocessing. See the example on ProcessPoolExecutor. One important think to remember when returning the list of matched molecules make use you preserve the molecule objects ( https://rdkit.org/docs/GettingStartedInPython.html#preserving-molecules) as transferring data between processes in Python requires that the data to be picklable. Best, Christos Christos Kannas Cheminformatics Researcher & Software Developer [image: View Christos Kannas's profile on LinkedIn] <http://cy.linkedin.com/in/christoskannas> On Wed, 20 Feb 2019 at 11:28, Stamatia Zavitsanou < stamatia.zavitsa...@oriel.ox.ac.uk> wrote: > Hello everyone, > > > We have been writing a script that searches though a large number of > molecules within different files for a common substructure. To speed this > up we have been attempting to run this script in parallel-see scripts > below. However online the tutorial notes make reference to problems with > using the SDMolSupplier in parallel, we were wondering what is the issue > and how we could circumvent them to speed up some of our calculations. > > > Non-parallel > > > from __future__ import print_function > > from rdkit import Chem > > import os > > from progressbar import ProgressBar > > pbar=ProgressBar() > > matches = [] > > directory = 'Q:\Data2' > > patt = Chem.MolFromSmarts('NC(N****NC=O)=O') > > for file in pbar(os.listdir(directory)): > > filename = os.fsdecode(file) > > if filename.endswith(".sdf"): > > f = os.path.join(directory,filename) > > suppl= Chem.SDMolSupplier(f) > > for mol in suppl: > > if mol is None: continue > > if mol.HasSubstructMatch(patt): > > matches.append(mol) > > w = Chem.SDWriter(r'C:\Users\tom.watts\Desktop\datasmarts4c.sdf') > > for m in matches: w.write(m) > > print(filename) > > > > Parallel > > > pbar=ProgressBar() > > matches = [] > > directory = 'E:\Data' > > patt = Chem.MolFromSmarts('NC(N****NC=O)=O') > > w = Chem.SDWriter(r'C:\Users\tom.watts\Desktop\SearchDataNonly.sdf') > > l=[] > > for file in pbar(os.listdir(directory)): > > filename = os.fsdecode(file) > > if filename.endswith(".sdf"): > > f = os.path.join(directory,filename) > > l.append(f) > > num_cores = multiprocessing.cpu_count() > > print(num_cores) > > lock = multiprocessing.Lock() > > def Search(i): > > suppl= Chem.SDMolSupplier(i) > > for mol in suppl: > > if mol is None: continue > > if mol.HasSubstructMatch(patt): > > matches.append(mol) > > return matches > > results = Parallel(n_jobs=20)(delayed(Search)(i) for i in l) > > > > We also wish to use a second script that opens one SDF file and then > runs a loop over each molecule in the file. This is currently done > serially and we were wondering if it could be made parallel. > > > > suppl = Chem.SDMolSupplier('Red3.sdf') > > *for* mol *in* suppl: > > patt = Chem.MolFromSmarts('NC(N)=O') > > num=mol.GetSubstructMatches(patt) > > logger.debug(Chem.MolToSmiles(mol)) > > h=len(num) > > m3=Chem.AddHs(mol) > > cids =AllChem.EmbedMultipleConfs(m3, numConfs) > > > > Any comments can be useful. > > > Thanks a lot, > > Stamatia Zavitsanou > _______________________________________________ > Rdkit-discuss mailing list > Rdkit-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/rdkit-discuss >
_______________________________________________ Rdkit-discuss mailing list Rdkit-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/rdkit-discuss
