[appengine-java] Re: Storing query results in memcache
Level2 cache might be what you want, but anyway, just to add to the
dialog, I tinkered
with what's available in datanucleus-core v1.1.5.
Choices in datanucleus are: none, soft, and weak
(jcache is not available in any version, but see below for
customization)
In addition to those, one can make a class that implements
Level2Cache
by adopting the connectors that you see in the src code for
WeakLevel2Cache.java.
If implementing your own Level2Cache, you can, for example, use
Google's MapMaker in
Guava for finer control of expiration:
http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/collect/MapMaker.html
If you pursue a custom implementation, you'll have to edit
datanucleus-core-1.1.5.jar's plugin.xml to add the new class and name
for it.
http://www.datanucleus.org/extensions/level2_cache.html
http://grepcode.com/file_/repo1.maven.org/maven2/org.datanucleus/datanucleus-core/1.1.5/org/datanucleus/cache/WeakLevel2Cache.java/?v=source
Here's a quick check I used in development and production to assert
that the
level2 cache works in appengine (presumably by container, and not yet
distributed, not sure...)
/*
* Level 1 Cache - caching of instances within a
PersistenceManager
* Level 2 Cache - caching of instances within all
PersistenceManagers of a PersistenceManagerFactory,
*/
/*
* Add this to jdoconfig.xml or change yours for the level2
entries:
*
* persistence-manager-factory name=l2cache
property
name=javax.jdo.PersistenceManagerFactoryClass
value=org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory/
property name=javax.jdo.option.ConnectionURL
value=appengine/
property name=javax.jdo.option.NontransactionalRead
value=true/
property
name=javax.jdo.option.NontransactionalWrite value=true/
property name=javax.jdo.option.RetainValues
value=true/
property
name=datanucleus.appengine.autoCreateDatastoreTxns value=true/
property name=datanucleus.cache.level2 value=true/
property name=datanucleus.cache.level2.type
value=weak/
property name=datanucleus.cache.level1.type
value=weak/
/persistence-manager-factory
*/
PersistenceManagerFactory pmf = null;
PersistenceManager pm = null;
Transaction tx = null;
Event event1 = null;
try {
pmf = JDOHelper.getPersistenceManagerFactory(l2cache);
pm = pmf.getPersistenceManager();
tx = pm.currentTransaction();
DataStoreCache l2Cache = pmf.getDataStoreCache();
l2Cache.pinAll(true, Event.class);
tx.begin();
event1 = createEvent1();
pm.makePersistent(event1);
tx.commit();
} catch (Exception e) {
e.printStackTrace();
e.printStackTrace(out);
} finally {
if (tx.isActive()) {
tx.rollback();
}
if (pm != null) {
pm.close();
}
log.info(wrote 1 event to PM);
}
Level2Cache l2Cache =
((JDODataStoreCache)pmf.getDataStoreCache()).getLevel2Cache();
int nObjectsInCache = l2Cache.getSize();
int nPinnedObjects = l2Cache.getNumberOfPinnedObjects();
int nUnpinnedObjects = l2Cache.getNumberOfUnpinnedObjects();
log.info(tx is closed:);
log.info(number of objects in Level2Cache is +
nObjectsInCache);
log.info(number of pinned objects in Level2Cache is +
nPinnedObjects);
log.info(number of unpinned objects in Level2Cache is +
nUnpinnedObjects);
l2Cache.evictAll();
nObjectsInCache = l2Cache.getSize();
nPinnedObjects = l2Cache.getNumberOfPinnedObjects();
nUnpinnedObjects = l2Cache.getNumberOfUnpinnedObjects();
log.info(number of objects in Level2Cache is +
nObjectsInCache);
log.info(number of pinned objects in Level2Cache is +
nPinnedObjects);
log.info(number of unpinned objects in Level2Cache is +
nUnpinnedObjects);
On Oct 3, 5:14 am, nicanor.babula nicanor.bab...@gmail.com wrote:
Thanks for your help. I'll go with Ian's suggestion, storing arrays in the
memcache.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Storing query results in memcache
Thanks for your help. I'll go with Ian's suggestion, storing arrays in the memcache. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/8R5rRrRrb00J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Storing query results in memcache
use the level 2 cache? property name=datanucleus.cache.level2.type value=javax.cache / Objects are placed in the L2 cache when you commit() the transaction of a PersistenceManager. This means that you only have datastore- persisted objects in that cache. Also, if an object is deleted during a transaction then at commit it will be removed from the L2 cache if it is present. http://www.datanucleus.org/products/accessplatform/jdo/cache.html#level2 On Sep 29, 3:59 am, nicanor.babula nicanor.bab...@gmail.com wrote: Hi Ian, Thanks for sharing your strategy.. I am not sure if I understand exactly how could I use your system to solve my problem. Actually, the memcache concept is working nice solving similar, but simpler situations. I think I explained poorly my situation. Here it is: An application manages contacts. It actually has thousands of them. On the GUI side we have a listview, an editview and a createview. The list view has 4 columns, and is paged (30 records per page) and by default is sorted ascending by the first column. I can search the listview by issuing queries serverside (I want the results always ordered and paged accordingly). Now, a user clicks on new contact. The application goes into createView. Data input then save. Application goes to list view. The contact should have been in the first page, but it isn't. After a few seconds, it appears. The previous example with phone numbers is kind of the same issue, but with simpler queries: User modifies a contact by adding some phone numbers. Hits save and goes to list view, then he remembers that one number was wrong and goes back to editing the same contact. The phone numbers inserted seconds ago aren't there. In this case, (since for other reasons I the phone numbers entities cannot be children of the contact entity) every phone number entity has a property called contactId that points to the father contact. The solution employed in this simple case is storing in the memcache the phone numbers by using contact-based keys so that for example [code] memcache.get(MEMCACHE_phones_contactId_1234) [/code] would give me an all the phone numbers for contact having id=1234. Somehow (too long and boring to explain how) I manage to keep this array always up to date so when the user returns lightning-fast to the editview the newly inserted phone number are there, only that they are read from the memcache. Analogically, how could this be done for the fore-mentioned listview situation, where the query is much more complex? Hope I didn't confused you more.. Thanks, Cristian -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Storing query results in memcache
Hi Christian, If Nichole's tip doesn't help you, then for your list view, you could back it up with an array (in memcache?). Upon any create, edit, delete operation, you will have to test the relevant entity (using logic equivalent to your query) to see whether its entry in the array should be added, edited or removed. You would have to think about whether you want subsequent updates by other users or your system should be reflected at once or if a snapshot at first list view would suffice. (Note that for postings to this group, there is a delay of a few tens of seconds before a new post is visible in the list. That is OK for a technical forum, but I see that this is unacceptable for your app.) Good luck! Ian On Sep 30, 7:49 am, Nichole nichole.k...@gmail.com wrote: use the level 2 cache? property name=datanucleus.cache.level2.type value=javax.cache / Objects are placed in the L2 cache when you commit() the transaction of a PersistenceManager. This means that you only have datastore- persisted objects in that cache. Also, if an object is deleted during a transaction then at commit it will be removed from the L2 cache if it is present. http://www.datanucleus.org/products/accessplatform/jdo/cache.html#level2 On Sep 29, 3:59 am, nicanor.babula nicanor.bab...@gmail.com wrote: Hi Ian, Thanks for sharing your strategy.. I am not sure if I understand exactly how could I use your system to solve my problem. Actually, the memcache concept is working nice solving similar, but simpler situations. I think I explained poorly my situation. Here it is: An application manages contacts. It actually has thousands of them. On the GUI side we have a listview, an editview and a createview. The list view has 4 columns, and is paged (30 records per page) and by default is sorted ascending by the first column. I can search the listview by issuing queries serverside (I want the results always ordered and paged accordingly). Now, a user clicks on new contact. The application goes into createView. Data input then save. Application goes to list view. The contact should have been in the first page, but it isn't. After a few seconds, it appears. The previous example with phone numbers is kind of the same issue, but with simpler queries: User modifies a contact by adding some phone numbers. Hits save and goes to list view, then he remembers that one number was wrong and goes back to editing the same contact. The phone numbers inserted seconds ago aren't there. In this case, (since for other reasons I the phone numbers entities cannot be children of the contact entity) every phone number entity has a property called contactId that points to the father contact. The solution employed in this simple case is storing in the memcache the phone numbers by using contact-based keys so that for example [code] memcache.get(MEMCACHE_phones_contactId_1234) [/code] would give me an all the phone numbers for contact having id=1234. Somehow (too long and boring to explain how) I manage to keep this array always up to date so when the user returns lightning-fast to the editview the newly inserted phone number are there, only that they are read from the memcache. Analogically, how could this be done for the fore-mentioned listview situation, where the query is much more complex? Hope I didn't confused you more.. Thanks, Cristian -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Storing query results in memcache
Hi Ian, Thanks for sharing your strategy.. I am not sure if I understand exactly how could I use your system to solve my problem. Actually, the memcache concept is working nice solving similar, but simpler situations. I think I explained poorly my situation. Here it is: An application manages contacts. It actually has thousands of them. On the GUI side we have a listview, an editview and a createview. The list view has 4 columns, and is paged (30 records per page) and by default is sorted ascending by the first column. I can search the listview by issuing queries serverside (I want the results always ordered and paged accordingly). Now, a user clicks on new contact. The application goes into createView. Data input then save. Application goes to list view. The contact should have been in the first page, but it isn't. After a few seconds, it appears. The previous example with phone numbers is kind of the same issue, but with simpler queries: User modifies a contact by adding some phone numbers. Hits save and goes to list view, then he remembers that one number was wrong and goes back to editing the same contact. The phone numbers inserted seconds ago aren't there. In this case, (since for other reasons I the phone numbers entities cannot be children of the contact entity) every phone number entity has a property called contactId that points to the father contact. The solution employed in this simple case is storing in the memcache the phone numbers by using contact-based keys so that for example [code] memcache.get(MEMCACHE_phones_contactId_1234) [/code] would give me an all the phone numbers for contact having id=1234. Somehow (too long and boring to explain how) I manage to keep this array always up to date so when the user returns lightning-fast to the editview the newly inserted phone number are there, only that they are read from the memcache. Analogically, how could this be done for the fore-mentioned listview situation, where the query is much more complex? Hope I didn't confused you more.. Thanks, Cristian -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/3kOA7GAMTQUJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
