RE: caching proposals: SQLTransformer and Request
I just skimmed through the mailing list to find ideas of how to cache a pipeline starting with a request generator and later on passing data through the SQLTransformer. Did you or anybody else follow up on this idea? NB: Caching of the request generator would probably also need to cache request parameters passed not part as part of the URL. As usually only some request parameters are used in the pipeline, the sitemap element might list the request parameters to consider when generating cache key: map:generator type=request map:parameter name=request-data-used value=input1 map:parameter name=request-data-used value=input2 ... - A couple of things I'd like to do with Cocoon caching; let me know if this is crazy. 1. Add caching to the request generator. Many of my pipelines are transformations based upon the request, and since requestGenerator currently does not support caching, it means those transformations are always re-done (and often there is some sql at the end of the pipe which is really slow). I'd like to hash (or MD5?) the request string and use that as the cache key so that if I get the same one the pipeline knows it can skip over everything. 2. Add caching to the SqlTransformer. I know this sounds weird, but hear me out. Our database is modified infrequently, so usually returns the same data. There is a datestamp in a special table which indicates when the last time the database was updated. The SQL Transformer would remember the time of the last query. I would have a new parameter to the sql transformer to indicate when the data is dirty. map:transform type=sql map:parameter name=last-update-time value=cocoon:/lastUpdateTime.xml/ /map:transform LastUpdateTime looks like: Date20021005144321/Date (Or whatever the xml date format is, I forget). SQLGenerator would resolve cocoon:/lastUpdateTime.xml. I would then have a pipeline for lastUpdateTime.xml which would build it by querying my special table (but if you wanted, you could use some other mechanism to build it). SQLGenerator would compare the two dates and re-run the sql if it needed to. How does this sound? Steve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: caching proposals: SQLTransformer and Request
Christian,
When we've had the need to cache a query, we just throw the result (as a
DOM object) into the sesssion. I'm including the sample pipeline. I'll
typicailly aggregate the results of this with something else (usually a
dynamic query) and I'm all set. When I want to clear the cache, the
session attribute can be cleared.
(note, the getUserIdAction is something we wrote to grab the logged in
user out of the session. We use container based authentication, but
stash an object in the session for use in the sitemap).
We also ended up writing an action to invalidate a sesion object (used
to clear the cache).
map:pipeline internal-only=true
map:match pattern=assoc-list
map:act type=getUserIdAction
map:select type=sessionAttributeExists
map:parameter name=attribute-name
value=assoc-cache/
map:when test=true
map:generate type=session-attr
map:parameter name=attr-name
value=assoc-cache/
/map:generate
map:serialize type=xml/
/map:when
map:otherwise
map:generate src=xml/assoc_query.xml/
map:transform type=sql
map:parameter name=use-connection
value=mbrdb/
map:parameter name=userid value={userid}/
/map:transform
map:transform type=writeDOMsession
map:parameter name=dom-name
value=assoc-cache/
map:parameter name=dom-root-element
value=page/
/map:transform
map:serialize type=xml/
/map:otherwise
/map:select
/map:act
/map:match
David
Christian Kurz wrote:
I just skimmed through the mailing list to find ideas of how to cache a
pipeline starting with a request generator and later on passing data
through the SQLTransformer.
Did you or anybody else follow up on this idea?
NB: Caching of the request generator would probably also need to cache
request parameters passed not part as part of the URL. As usually only
some request parameters are used in the pipeline, the sitemap element
might list the request parameters to consider when generating cache key:
map:generator type=request
map:parameter name=request-data-used value=input1
map:parameter name=request-data-used value=input2
...
-
A couple of things I'd like to do with Cocoon caching; let me know if
this
is crazy.
1. Add caching to the request generator. Many of my pipelines are
transformations based upon the request, and since requestGenerator
currently
does not support caching, it means those transformations are always
re-done
(and often there is some sql at the end of the pipe which is really
slow).
I'd like to hash (or MD5?) the request string and use that as the cache
key
so that if I get the same one the pipeline knows it can skip over
everything.
2. Add caching to the SqlTransformer. I know this sounds weird, but
hear me
out. Our database is modified infrequently, so usually returns the same
data. There is a datestamp in a special table which indicates when the
last
time the database was updated. The SQL Transformer would remember the
time
of the last query. I would have a new parameter to the sql transformer
to
indicate when the data is dirty.
map:transform type=sql
map:parameter name=last-update-time
value=cocoon:/lastUpdateTime.xml/
/map:transform
LastUpdateTime looks like:
Date20021005144321/Date (Or whatever the xml date format is, I
forget).
SQLGenerator would resolve cocoon:/lastUpdateTime.xml.
I would then have a pipeline for lastUpdateTime.xml which would build it
by
querying my special table (but if you wanted, you could use some other
mechanism to build it).
SQLGenerator would compare the two dates and re-run the sql if it needed
to.
How does this sound?
Steve
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: caching proposals: SQLTransformer and Request
From: Stephen Ng [mailto:[EMAIL PROTECTED]] A couple of things I'd like to do with Cocoon caching; let me know if this is crazy. 1. Add caching to the request generator. Many of my pipelines are transformations based upon the request, and since requestGenerator currently does not support caching, it means those transformations are always re-done (and often there is some sql at the end of the pipe which is really slow). I'd like to hash (or MD5?) the request string and use that as the cache key so that if I get the same one the pipeline knows it can skip over everything. 2. Add caching to the SqlTransformer. I know this sounds weird, but hear me out. Our database is modified infrequently, so usually returns the same data. There is a datestamp in a special table which indicates when the last time the database was updated. The SQL Transformer would remember the time of the last query. I would have a new parameter to the sql transformer to indicate when the data is dirty. map:transform type=sql map:parameter name=last-update-time value=cocoon:/lastUpdateTime.xml/ /map:transform LastUpdateTime looks like: Date20021005144321/Date (Or whatever the xml date format is, I forget). SQLGenerator would resolve cocoon:/lastUpdateTime.xml. I would then have a pipeline for lastUpdateTime.xml which would build it by querying my special table (but if you wanted, you could use some other mechanism to build it). SQLGenerator would compare the two dates and re-run the sql if it needed to. How does this sound? May be you should consider different design, which is suited better for the problem? XSP pages with ESQL provide easy ability to program any caching behavior. Vadim Steve - Please check that your question has not already been answered in the FAQ before posting. http://xml.apache.org/cocoon/faqs.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: caching proposals: SQLTransformer and Request
May be you should consider different design, which is suited better for the problem? XSP pages with ESQL provide easy ability to program any caching behavior. What?!? You just convinced me to go from XSP/ESQL to SQLTransformer! The problem with ESQL is the Java recompilation, which is excessive when only the sql part is changing. Anyway, another way to solve my problem would be to simulate RequestGenerator using an XSP (have an xsp create the same format output as RequestGenerator). But this is really no different at all from rewriting RequestGenerator so it supports caching--the caching issues are exactly the same. So I think this means I should put caching into RequestGenerator. (As no-one has either protested or applauded, maybe not that many people are using RequestGenerator?) --Steve - Please check that your question has not already been answered in the FAQ before posting. http://xml.apache.org/cocoon/faqs.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: caching proposals: SQLTransformer and Request
RequestGenerator *should* cache. Give me a good reason why it shouldn't have caching! --Steve -Original Message- From: Stephen Ng [mailto:[EMAIL PROTECTED]] Sent: Friday, May 10, 2002 7:23 PM To: 'Vadim Gritsenko'; [EMAIL PROTECTED] Subject: RE: caching proposals: SQLTransformer and Request May be you should consider different design, which is suited better for the problem? XSP pages with ESQL provide easy ability to program any caching behavior. What?!? You just convinced me to go from XSP/ESQL to SQLTransformer! The problem with ESQL is the Java recompilation, which is excessive when only the sql part is changing. Anyway, another way to solve my problem would be to simulate RequestGenerator using an XSP (have an xsp create the same format output as RequestGenerator). But this is really no different at all from rewriting RequestGenerator so it supports caching--the caching issues are exactly the same. So I think this means I should put caching into RequestGenerator. (As no-one has either protested or applauded, maybe not that many people are using RequestGenerator?) --Steve - Please check that your question has not already been answered in the FAQ before posting. http://xml.apache.org/cocoon/faqs.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - Please check that your question has not already been answered in the FAQ before posting. http://xml.apache.org/cocoon/faqs.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
