Hi all,
first some questions:
Can I set aromaticty of a Molecule manually? There is a setFlag(int flag_type)
method but did not find a list of flag_types.
That would prevent from having to perceive it each time a molecule is created
and if I look at Isomorphism Class, this flag has an effect on subgraph
matching. True?
Observations:
My search currently seems limited by data access and not the graph matching
itself. I was able to solve the blocking/freezing issue (had nothing to do with
threading just a logical error elsewhere) and using visualVm I can see that the
thread doing the database access is the most active one (running 100% of the
time). The graph matching thread is runnign only 20% of the time.
Using a BlockingQueue has benefits but not on memory usage in this case. I can
set it's size to 1 or 100 without an effect on memory consumption. As indicated
Jules probably due to delayed garbage collection. Or said otherwise when doing
a search with lots of hits you will always have a lot of IAtomContainer in
memory, regardless of your algorithm.
(this probably also explains why I see no benefit in using Isomorphism class
from cdk-1.3.7 over UniversialIsomorphismTester)
The conclusion is to either have more threads reading from the database or to
serialize all molecules to the database.
Best Regards,
Thomas
From: j.kerssemak...@cmbi.ru.nl
Date: Thu, 2 Dec 2010 15:44:55 +0100
Subject: Re: [Cdk-user] Substructure Searching, Fingerprints and cdk-1.3.7
Isomorphism Class
To: beginn...@hotmail.de
CC: cdk-user@lists.sourceforge.net
Hello Thomas (and other CDK-users of course)
- The aromaticity-flag could very well be in the fingerprint, I've never really
checked that to be honest. Does anyone else know?
The original reasoning for putting it in was that the aromaticity-detection is
'expensive' if you have to do it each time for a query, but it still is a
pretty good distinction (in a general metabolite database in any case, if your
database is 99% aromatic anyway, don't even bother ;-) ). We can eliminate
about half our dataset by that flag so it works pretty well
- The CDKHueckelAromaticityDetector does indeed modify the atomcontainer,
setting the ISAROMATIC flags for aromatic molecules. I don't know if it does
anything for the bond orders though..
- A serialized object is a native java object, which can be directly unpacked
into memory (fast). A molfile needs to be parsed by the molreader, which will
always be slower. Reading from database or from disk will probably not matter
that much in this case. varchars are stored in different data-blocks than the
rest of the row for (I think) all database systems, so the database is unlikely
to have these cached and therefore will still need to read them from disk
(though from a database file rather than a normal directory. The performance
will be about equal)
Of course, this doesn't apply if you have your complete database in memory,
which hsql does, I believe. Reading from memory is always way faster than
reading from disk.
- If you have optimised your method already to only load the atomcontainer
inside the for-loop, you will always only have one IAtomcontainer per thread in
memory(*) per running loop. If you limit the amount of queries running at the
same time, this should solve your problem. I'm horrible at threaded
programming, and from what I understand, so is every other human being, so my
guess is that the freezes/blocks you see stem from the threaded part, not from
the memory-overrun part.
(*): not strictly true, the default Java-servers only start removing things
from memory once memory becomes scarce, so there will probably be a few
atomcontainers from previous loop iterations lying around waiting to be tossed
away by the garbage collector.
- The list of often-occuring fragments will probably not do you much good. Such
a list is probably going to be bigger than your whole database after a few
days/weeks of user-interactions.
The cleaner, most often used method is (as you also suggest) to tell the user
"This query structure generated to many preliminary hits for this server to
handle, please be more specific."
You're welcome :-)
Best regards,
Jules
On 1 December 2010 17:54, Thomas Strunz <beginn...@hotmail.de> wrote:
Hi Jules,
thank you for this detailed explanations. I have some additional questions:
- Aromaticity flag: so you store a boolean value yes/No? What's the advantage
of this? or otherwise said should that not already be covered by the
fingerprint?
- CDKHueckelAromaticityDetector: does this modify the passed in IAtomContainer
or just returns true/false? (eg. in smsd.Isomorphism.init method set flag for
cleanAndConfigureMolecule to false if it was already done after reading the
molecule?
- About loading from molfiles: is it much slower than loading a serialized
molecule? Assuming the molfile is in the database.
I'm already working on a method on putting the molecules in a BlockingQueue,
e.g. have one thread read from the database and 1 (or more) others doing the
subgraph matching. Like that I could limit the amount of IAtomContainers in
memory. memory usage is reduced dramatically but still have issues (=test
freezes/blocks and does not return).
I agree that I need some additional steps but the example with benzene
mentioned remains. If the query fragment is very common, any additional steps
won't help because there actually are so many structures that match. My idea is
to add a table with such fragments and a table with Molecules that contain that
fragment. The fragment can be stored as canonical smiles or InchiKey (or an
other canonical form). The first step could then be to check if the query is
such a fragment and select all the matching Id's (Select with 1 join). So
subgraph matching can be avoided. This could be enhanced by automatically
adding query fragments that return to many hits ("the system learns").
It could also be done otherwise liek do fingerprint screen first and only do
this "fragment" screen if there are too many hits after fingerprinting.
A simpler approach would be to just always only return a defined number of hits
like max 500 and ask the user to more clearly define the query structure.
Thanks for your help,
Thomas
From: j.kerssemak...@cmbi.ru.nl
Date: Wed, 1 Dec 2010 12:10:28 +0100
Subject: Re: [Cdk-user] Substructure Searching, Fingerprints and cdk-1.3.7
Isomorphism Class
To: beginn...@hotmail.de
CC: cdk-user@lists.sourceforge.net
Hello All,
We've been using the CDK substructure search for a while now too in our biometa
database (http://cheminf.cmbi.ru.nl/cgi-bin/biometa/biometa.py?molecules%20jme).
Here is what we do to keep things manageable performanace-wise:
* Pre-calculate several statistics for all entries, namely:
- fingerprints
- an aromaticity flag
- amount of different elements
- number of rings (either aromatic or non-aromatic)
- total amount of atoms
- total amount of of atoms per element (in separate table of (molecule_id,
element_number, element_count))
* when querying, we calculate the same properties for the search-molecule and
then write a pretty long SQL-query that limits the results as much as possible
as cheaply as possible:
- first condition is the aromatic/non-aromatic flag (single flag comparison
--> cheapest you'll ever find)
- next condition is element-count, (a simple, cheap numerical '>='-comparison)
- then ringcount >=
- then the fingerprint comparison. We let the database do the logical AND and
==, because postgres has native bit-array operations, which our python-binding
(pgsql) can't handle because it doesn't understand bit-arrays (it converts them
to strings).
- finally a per-element atom-count comparison. This is last because in the
current set-up it has to join the element-count table to the molecule table,
which is probably slower.
* The mol-files for the resultant mol_id's are then concatenated into a large
sdf-file which is fed through a CDK SDFSubstructureFinder from a very old 2006
SVN version (dropped since, but I dare not replace it because everything will
probably implode if I try).
It's all a bit hack-ish, but it works fairly well since we don't have much
traffic. I never actually tested how the postgres query planner handles this,
but it is fairly smart.
Things that I would change if I rewrote it all today would be:
1) store serialized IAtomContainers in the database, to prevent having to
re-read the molfiles from disk every time
2) Use a faster substructure matcher (SMSD sounds good)
3) get rid of the element-count table, and rather add columns for our
most-prevalent atom-types to the molecule rows (namely: C,N,O,H,P). This avoids
the overhead of the join.
Thomas, as Nina already mentioned, you shouldn't load all the molecules before
the loop, rather load them one-by-one IN the loop. This means you only ever
need one molecule in-memory per query, which saves MASSIVELY on memory
requirements.
//Pseudocode example of what you have:
keyList = doTheSearch(); // get molecule ID's for potential candidates
map<key, molecule> theMap = manager.getMolecules(keyList); // This loads ALL
your molecules into memory, OUCH!!
for (key in map.keyset) {
mol = theMap.get(key);
if (searchtarget.isSubgraphOf(mol)) {
results.add(mol)
}
}
// Pseudocode for a more efficient way to do it (don't pre-load all molecules):
keyList = doTheSearch(); // get molecule ID's for potential candidates
for (key in keyList) {
mol = manager.getMolecule(key); // IMPROVEMENT: only load one molecule at a
time!
if (searchtarget.isSubgraphOf(mol)) {
results.add(mol)
}
}
Hope this helps,
Best regards,
Jules Kerssemakers
On 30 November 2010 21:19, Nina Jeliazkova <jeliazkova.n...@gmail.com> wrote:
Hi Thomas,
On 30 November 2010 21:58, Thomas Strunz <beginn...@hotmail.de> wrote:
Hi Nina,
I sure have more than 1 IAtomContainer in memory at time so I agree that might
be an issue but if screening lets say returns 1000 hits, 1000 subgraph matches
must be done and hence all the 1000 Molecules must be created first. So you
would suggest to read each one separatley from database after a subgraph match
returns?
What we are doing is getting database structure identifiers from prescreening
and reading structures one by one for subgraph matching. Few thousand of
IAtomContainers is fine for desktop application, but server side one could have
multiple queries at the same time and multiply the thousands to unreasonable
number.
A second issue is, if the query Molecule is a common fragment in the database,
let's assume benzene, and llike 80% of the fingerprints match, how do you
handle that and keep performance? subgraph matches on so mnay structures will
no perfrom well. How can you prevent that with very common substructures?
We have several levels of prescreening, fingerprints only are not sufficient
for reasonable performance. Also precalculated aromaticity flags to avoid
calculating that on the fly and caching of the final results. You could get an
overview from this poster from QSAR2010
http://www.ideaconsult.net/downloads/rhodes/posters/SMARTS.pdf .
Regards,
Nina
Regards,
Thomas
Just my two cents.
Besides prescreening, having minimum IAtomContainer objects in memory is the
key to performance. As less than one object doesn't make sense :) one
IATomContainer at a time is the best. Fingerprints can be pre-calculated and
no need to be loaded in-memory at all, let SQL do the prescreening.
We've been doing similar things (CDK, relational database, no cartridges) in
ambit (ambit.sourceforge.net) for quite few years already. There is
downloadable standalone application and a servlet container application war
file (to run your own service), as well as a running OpenTox REST services for
substructure searching , e.g.
https://ambit.uni-plovdiv.bg:8443/ambit2/query/smarts?search=c1ccccc1[Cl,Br,F]
http://apps.ideaconsult.net:8080/ambit2/query/smarts?search=c1ccccc1[Cl,Br,F,I]
Regards,
Nina
Regards,
Thomas
------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Cdk-user mailing list
Cdk-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/cdk-user
------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Cdk-user mailing list
Cdk-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/cdk-user
------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Cdk-user mailing list
Cdk-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/cdk-user
------------------------------------------------------------------------------
What happens now with your Lotus Notes apps - do you make another costly
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
_______________________________________________
Cdk-user mailing list
Cdk-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/cdk-user
