Hi Hooman, For uploading data, if your dataset is relatively small, the bulkloader will do fine. You might want to also consider uploading as CSV to Cloud Storage, then running a mapreduce to copy from Cloud Storage to Datastore. This is the method we would currently recommend for larger datasets, please see https://developers.google.com/appengine/docs/python/googlestorage/#Using_Google_Storage
Also, if your issue is just that the bulkloader auto config is not picking up recent datastore changes, you could manually edit the configuration as described here: https://developers.google.com/appengine/docs/python/tools/uploadingdata#Editing_the_Configuration_File . Hope that helps! On Thursday, June 27, 2013 11:28:30 AM UTC-7, Hooman Korasani wrote: > > Hi Jesse, > > First off thank you so much for replying to this issue. We are a young > startup and are in process of migrating from AWS to GAE. > On day one (early this week) the bulkloader worked nicely even despite the > HRD. I understand I can't rely on it, but how else do we transfer our data > to google app engine? > > At least, if I were able to create one bulk_loader_config yaml, I would be > able to transfer our data via cvs files. Is there any other official way > for HDS? > > Many Thanks, > Hooman > > > On 27/06/13 18:58, Jesse Rohwer wrote: > > Hi Hooman, > > Bulkload operations on the HRD are eventually consistent, which means > you cannot use bulkloader with HRD and be guaranteed to get up-to-date data. > > You can read more in the bulkloader docs: > https://developers.google.com/appengine/docs/python/tools/uploadingdata > In particular, > *Note: *This document applies to apps that use the master/slave > datastore<https://developers.google.com/appengine/docs/python/datastore/hr>. > If your app uses the High Replication datastore, it is possible to copy > data from the app, but Google does not currently support this use case. If > you attempt to copy from a High Replication datastore, you'll see a > high_replication_warning error in the Admin Console, and the downloaded > data might not include recently saved entities. > > That said, it is true that in the past two days the HRD has had > longer-than-usual eventual consistency. This is returning to a more normal > range now, but again: for operations that are not strongly consistent, > including bulkloader, there are no guarantees. I hope this helps clarify > the situation. > > Thanks, > Jesse Rohwer on behalf of the App Engine team > > On Wednesday, June 26, 2013 2:41:40 AM UTC-7, Hooman Korasani wrote: >> >> Any one from google here? >> >> The statistics are failing since three days. Without up-to-date >> statistics there is no way to get the bulkloader working properly. >> >> >> http://stackoverflow.com/questions/17275202/how-to-force-create-bulkloader-config-to-pickup-latest-schema-data/17277052#comment25075099_17277052 >> >> >> On Tuesday, 25 June 2013 08:58:18 UTC+1, Hooman Korasani wrote: >>> >>> I have created a bulkloader.yaml like this: >>> >>> appcfg.py create_bulkloader_config --filename=tools/bulkloader.yaml -- >>> url=https://myid.appspot.com/_ah/remote_api >>> >>> >>> However it doesn't reflect my changes from a day ago. How long do I >>> have to wait until it picks it up? >>> >>> According to the data-store statistics, the last statistic update was *Last >>> updated: 1 day, 21:11:28 ago >>> *However it also says *"**Statistics are updated at least once per day" >>> >>> *Is there any other way to get the bulkloader consider my latest >>> changes?* >>> >>> *Many Thanks, >>> Hooman* >>> >>> * >>> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "Google App Engine" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/google-appengine/1zv8LNucS7g/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > [email protected] <javascript:>. > To post to this group, send email to [email protected]<javascript:> > . > Visit this group at http://groups.google.com/group/google-appengine. > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
