Today I updated the mapreduce library. I see the same "only 1 shard" when I use the dev_server. The dev_server does not have the __scatter__ property of objects. The mapreduce library then falls back to a single shard.
And on the production it depends on how many objects have a __scatter__ property. If less then shard_count have __scatter__ you get less shards. GAE Team: What determines if an object gets a __scatter__ property? 2011/2/11 djidjadji <[email protected]>: > In your cron_mapreduce.py add these two lines > > shard_count=int(self.request.get("shard_count", > mr_control._DEFAULT_SHARD_COUNT)) > > mr_control.start_map( > self.request.get("name"), > self.request.get("reader_spec", "your_mapreduce.map"), > self.request.get("reader_parameters", > "mapreduce.input_readers.DatastoreInputReader"), > { "entity_kind": self.request.get("entity_kind", "models.YourModel"), > "processing_rate": int(self.request.get("processing_rate", 100)) }, > shard_count = shard_count, > mapreduce_parameters={"done_callback": self.request.get("done_callback", > None) } ) > > > 2011/2/10 Andrin von Rechenberg <[email protected]>: >> Hey there >> Today I created a library to run MapReduces as cron jobs in python. >> See >> here: http://devblog.miumeet.com/2011/02/schedule-mapreduce-daily-on-appengine.html >> However, I didn't figure out how to I'm able to set the >> shard_count programmatically. >> In mapreduce/control.py there is a function I call: >> >> def start_map(name, >> >> handler_spec, >> >> reader_spec, >> >> reader_parameters, >> >> shard_count=_DEFAULT_SHARD_COUNT, >> >> [...]) >> >> However, no matter what o I pass as the shard_count argument, it is ignored. >> >> Any ideas? >> >> Cheers, >> >> -Andrin > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
