This is one of the rare occasions, where I would disagree with Jeff here. For us namespaces are one of the best things in appengine. Obviously, this is just an abstraction and can be achieved manually. But having this implemented is a big timesaver and also is an important safety feature. We have implemented a CRM for small and medium businesses and, it helps to be able to isolate business data by namespace. The same way as DataStore by its design pushes developers to write scalable applications, namespaces help to enforce proper sandboxing of business data.
Obviously, it might not be the right tool for many other problems, but for us it works very well. I even wish, there was a namespace support for BigQuery and other services. Anyway, It's there, you can use it or leave. Having choice is great. But again, it would probably not be the right solution for most of the projects, so only use them if you understand the implications. As for migration, Jeff, thanks for gist. That was very useful On Sunday, November 29, 2015 at 11:29:47 PM UTC, Jeff Schnitzer wrote: > > First: I think the namespace feature is a horrible idea and should never > be used (and I say this as someone that runs a whitelabeled, multitenant > system). > > That said, doing migrations with namespaces shouldn't be all that > different from doing migrations without namespaces, you just have to add > one more layer of mapping (enqueue one rewrite task per namespace). Unless > you're doing something really exotic, data migration is really just > load/save entities and figuring out how to 'map' a task onto the task queue > for every entity in the system. > > Rather than the dataflow system (which seems far more complex than > necessary for this simple process), here's some code that may help: > > https://gist.github.com/stickfigure/1a345f9a8a56897da6a0 > > It just uses the scatter property to enqueue tasks per shard that > themselves enqueue tasks per entity. Just like the map/reduce framework. > The only twist is that you need to start with one task per namespace and > run the scatter query on each namespace separately. > > Jeff > > On Fri, Nov 27, 2015 at 1:02 AM, Rajesh Gupta < > [email protected] <javascript:>> wrote: > >> Hello Nick, >> Our data is organized by a tenant/company. Each tenant/company's data is >> in a separate namespace. >> So, we have many namespaces. >> >> The web app is continuously evolving, and hence we have to do many times >> schema migration of the datastore entities. >> >> Following Objectify link, tells more details on the kind of migrations we >> do. >> https://github.com/objectify/objectify/wiki/SchemaMigration >> >> Dataflow pipelines are created by specifying one namespace for a >> pipeline. The namespace issue is here >> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/83 >> >> >> >> On Fri, Nov 27, 2015 at 12:19 AM, Nick (Cloud Platform Support) < >> [email protected] <javascript:>> wrote: >> >>> Hey Rajesh, >>> >>> It seems that your question is a bit vague here. What kind of migration >>> or data transformation are you looking to perform, how are you using >>> namespaces, how would you like to use Dataflow, etc.? >>> >>> >>> On Wednesday, November 25, 2015 at 1:35:51 AM UTC-5, Rajesh Gupta wrote: >>>> >>>> Anyone using namespaces for real projects? >>>> >>>> Cloud Dataflow was my latest attempt to do the data changes for >>>> entities in large number of namespaces. The dataflow pipelines also does >>>> not help here >>>> >>>> Anyone has any framework developed to do migration/process data for >>>> large number of namespaces. >>>> -- >>>> Rajesh >>>> *www.VeersoftSolutions.com <http://www.VeersoftSolutions.com>* >>>> *www.AccountingGuru.net <http://www.AccountingGuru.net>* >>>> *Accounting/Inventory/Orders on Google Cloud Platform and Mobile* >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To post to this group, send email to [email protected] >>> <javascript:>. >>> Visit this group at http://groups.google.com/group/google-appengine. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/google-appengine/078f3c3d-b119-4f92-aa8b-15733afc8093%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/google-appengine/078f3c3d-b119-4f92-aa8b-15733afc8093%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Rajesh >> *www.VeersoftSolutions.com <http://www.VeersoftSolutions.com>* >> *www.AccountingGuru.net <http://www.AccountingGuru.net>* >> *Accounting/Inventory/Orders on Google Cloud Platform and Mobile* >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/google-appengine. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/CA%2BS7ijY0jg7SAHkVea%3DDazTe8_4cSGBH8Z68QCr_LSSfp0dTbg%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/google-appengine/CA%2BS7ijY0jg7SAHkVea%3DDazTe8_4cSGBH8Z68QCr_LSSfp0dTbg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/b6baf107-3498-41d0-bed9-c1a395b68f21%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
