Hi, I'm the main author of the Java side of appengine-mapreduce. Sorry for the delay in responding - I don't normally lurk this group, but Ikai brought to my attention.
We're happy to accept patches. We haven't been as good as we'd have liked about responding to them in a timely manner, since we've been buckling down on getting shuffle/reduce out, but no time like the present to try and correct that. I'm not entirely sure I understand the scope of the proposed patch. Are you thinking about adding filters at the DatastoreRecordReader level? It's not entirely clear to me that that provides a benefit over just applying the filter at the start of the map() function. Totally willing to believe I'm missing something, though. On a logistical note, for nontrivial contributions, we require a CLA from either you or your employer (depending on who owns the copyright for your work) before we can accept significant contributions. The relevant forms are at: http://code.google.com/legal/individual-cla-v1.0.html and http://code.google.com/legal/corporate-cla-v1.0.html. Feel free to email me privately if this is an issue. Thanks! -Fred On Nov 11, 1:06 am, Nacho Coloma <[email protected]> wrote: > I know, but I would prefer to discuss it first (offline is OK). I don't want > to start submitting patches without first confirming that > > * I got the concepts correctly (there are a lot of possible > implementations). > * The design is coherent with whatever direction the project is taking. > * The project is open to submissions of this kind. > > Otherwise, it's a waste of time for everyone. > > > > > > > > On Wed, Nov 10, 2010 at 6:58 PM, Maxim Veksler <[email protected]> wrote: > > Don't think Google would mind if you submit a patch, you have the source > > code under SVN available from here > >http://code.google.com/p/appengine-mapreduce/source/browse/ > > > On Wed, Nov 10, 2010 at 6:08 PM, Nacho Coloma <[email protected]> wrote: > > >> Is there any option to contribute to this project? I am using hacks to > >> do the filtering, and it seems easy to implement. > > >> I would love to discuss things first, but appengine-mapreduce seems > >> like a read-only project to me. It's fine but I would like to help it > >> move forward. > > >> On Nov 8, 10:11 pm, "Ikai Lan (Google)" > >> <[email protected]<ikai.l%[email protected]> > > >> wrote: > >> > There was talk about supporting it, but the priority is getting out the > >> > shuffle/reduce steps first. > > >> > appengine-mapreduce is an open source project. You can probably just > >> > edit/subclass DatastoreInputFormat.java and add the filter > >> functionality: > > >> >http://www.google.com/codesearch/p?hl=en#XwsseYUY0Ps/trunk/java/src/c. > >> .. > > >> > -- > >> > Ikai Lan > >> > Developer Programs Engineer, Google App Engine > >> > Blogger:http://googleappengine.blogspot.com > >> > Reddit:http://www.reddit.com/r/appengine > >> > Twitter:http://twitter.com/app_engine > > >> > On Mon, Nov 8, 2010 at 12:28 PM, Brad <[email protected]> > >> wrote: > >> > > Ikai, > > >> > > Are there any plans to support filter functionality in the future? I > >> > > have a similar situation where I want to iterate over a collection > >> > > that is big enough to be used by the mapper framework, but much > >> > > smaller than the actual collection size of all entities of that kind. > > >> > > Brad > > >> > > On Nov 8, 11:56 am, "Ikai Lan (Google)" > >> > > <[email protected]<ikai.l%[email protected]> > >> <ikai.l%[email protected] <ikai.l%[email protected]>> > > >> > > wrote: > >> > > > No, the mapper will go over everything. If the number of entities is > >> > > small, > >> > > > you are better off placing an indexed property on ONLY these fields, > >> then > >> > > > just iterating over them with a cursor and either a local client > >> making > >> > > > multiple HTTP calls to your App Engine app or chained task queues. > > >> > > > -- > >> > > > Ikai Lan > >> > > > Developer Programs Engineer, Google App Engine > >> > > > Blogger:http://googleappengine.blogspot.com > >> > > > Reddit:http://www.reddit.com/r/appengine > >> > > > Twitter:http://twitter.com/app_engine > > >> > > > On Sun, Nov 7, 2010 at 5:58 AM, aswath satrasala < > >> > > [email protected] > > >> > > > > wrote: > >> > > > > Hello, > >> > > > > I want to perform few aggregations on entities of a kind that are > >> > > filtered > >> > > > > by the property. > >> > > > > The entities I want to perform the aggregation is small compared > >> to the > >> > > > > complete entity set. > >> > > > > Is there any way to pass the filters to the mapper, so that only > >> those > >> > > > > entities are retrieved and aggregation is performed > > >> > > > > *Regards > >> > > > > -Aswath > >> > > > >http://vs-accounting.appspot.com > >> > > > > Accounting for Indian markets.* > > >> > > > > -- > >> > > > > You received this message because you are subscribed to the Google > >> > > Groups > >> > > > > "Google App Engine for Java" group. > >> > > > > To post to this group, send email to > >> > > > > [email protected]. > >> > > > > To unsubscribe from this group, send email to > >> > > > > [email protected]<google-appengine-java%2B > >> > > > > [email protected]><google-appengine-java%2B > >> [email protected]><google-appengine-java%2B > >> > > [email protected]> > >> > > > > . > >> > > > > For more options, visit this group at > >> > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > >> > > -- > >> > > You received this message because you are subscribed to the Google > >> Groups > >> > > "Google App Engine for Java" group. > >> > > To post to this group, send email to > >> > > [email protected]. > >> > > To unsubscribe from this group, send email to > >> > > [email protected]<google-appengine-java%2B > >> > > [email protected]><google-appengine-java%2B > >> [email protected]> > >> > > . > >> > > For more options, visit this group at > >> > >http://groups.google.com/group/google-appengine-java?hl=en. > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine for Java" group. > >> To post to this group, send email to > >> [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]<google-appengine-java%2B > >> [email protected]> > >> . > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine-java?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to > > [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine-java%2B > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
