hi, I am considering using the Java version of the Mapper (http:// code.google.com/p/appengine-mapreduce) for some batch processing, but would need to modify it from its out-of-the-box state. I have a couple of questions about the feasibility of what I am thinking about doing. Hopefully someone has been down this road already.
First, I need to operate on just a subset of entities of a given kind, retrieved by query. I think see how I could modify DatastoreInputFormat.java to do this, albeit in a hacky way. With the __scatter__ property now in place (I think?), would things shard properly with such a modification? Second: In the map action, I would like to retrieve and 'transactionally' modify another entity along with the map entity. This other entity is of a different kind and is not part of the same entity group. In general, a workaround would be to perform this second modification via a transactional task, launched from inside the transaction block that operates on the first entity. Would this work in the context of the map() method? I am guessing that map() is being run in a transactional context via the Mapper framework. Would a task launched from the map() method be treated as transactional? [Of course such a 'regular' task would be outside the mapper framework, but I don't see how this could be helped]. To further muddy the waters, what I would really like to do is synchronize 'batch puts' on the first mapped entity, with a transactional task updating a second entity along with each batch. Would this be doable w/ relatively minor modifications to the code? Conceptually I'd like to pass transactional task parameters, accumulated over a set of map calls, to the method performing the batch put, and have IT launch the transactional task. Finally, speed of processing is a big consideration for this particular task. I am guessing that the overhead of the mapper framework does slow things down some. (Of course it provides a lot of nice features in return). Does anyone have a sense of how much it slows things down vs. what a 'roll your own' approach might allow? -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
