Before I go off to learn Python in order to backup the contents of my datastore, can I be certain the bulkloader will work with my Java app? I am not concerned with reusing the contents of the "live" datastore in a local environment, only that I can download a "backup" of the datastore and perform a complete restore if a worst case scenario occurred and the datastore became corrupted/unusable.
Ultimately, I'm not sure if I am missing something glaringly obvious but isn't a backup/restore feature fundamental to any service? You must have companies which rely on the app engine and the data they have within the datastore - are we expected to write our own tools to backup/restore our data? Does Google perform backups regularly we can access and restore? Are we just to cross our fingers and hope nothing goes awry? On Jul 1, 6:56 pm, "Ikai L (Google)" <[email protected]> wrote: > This bulkloader doesn't need any real Python knowledge to do basic > operations. If you know how to program, you can learn Python easily. This is > the book I used to get up to speed on Python. Took me about 30 minutes to > pick up 90% of what I needed: > > http://diveintopython.org/ > > That being said, we're always thinking about feature parity. I'm trying to > think of a good way to wrap the bulkloader work with Java. We've got to be > aware of tradeoffs - my sense is there may be issues with being able to > reuse locally enhanced classes (just a guess, since if I were to build this > out I would skip past trying to meet the requirement of being JDO/JPA aware > in a version 1). Scripting operations or operations where strict-typing is > more of a burden than a blessing are almost always easier in not-Java, which > is why this made sense to launch in Python first. > > If you need export capability or incremental export capability now, however, > don't wait on this since it won't be soon. My suggestion is to learn to use > the current bulkloader or build your own export mechanism. > > > > > > On Thu, Jul 1, 2010 at 8:16 AM, mscwd01 <[email protected]> wrote: > > I second this. > > > On Jul 1, 2:17 pm, xcdesz <[email protected]> wrote: > > > Ikai, > > > > I have read your post on using this bulkloader for java: > > > >http://ikaisays.com/2010/06/10/using-the-bulkloader-with-java-app-eng... > > > > However, without a Python background, this stuff is Latin to me.. Any > > > plans on making a java-friendly version of this Bulk Loader? > > > > I am one of the hundreds who starred issue 776 in the backlog, and > > > don't consider the current python-based bulkloader tool an adequate > > > solution to this problem. > > > > On Jun 30, 8:14 pm, "Ikai L (Google)" <[email protected]> wrote: > > > > > That was referring to the bulkloader, which lets you do those things. > > App > > > > Engine's datastore isn't a relational database. We can't do a dump of > > all > > > > your data without iterating through all of your indexes, then > > retrieving > > > > your Entities. > > > > > On Wed, Jun 30, 2010 at 4:02 PM, mscwd01 <[email protected]> wrote: > > > > > Thanks for replying Ikai. I assumed the "Datastore dump and restore > > > > > facility" on the roadmap was referring to an import/export tool - a > > > > > way for us to download the contents of the datastore for safekeeping > > > > > and restoring it if needs be? > > > > > Is this something different? > > > > > > Lastly, I would consider an import/export tool to be extremely useful > > > > > if not critical to most applications, surely there has been some > > plans > > > > > to offer one? > > > > > > On Jun 30, 7:57 pm, "Ikai L (Google)" <[email protected]> wrote: > > > > > > A web-based import/export tool is and was not ever on our roadmap: > > > > > > >http://code.google.com/appengine/docs/roadmap.html > > > > > > > The roadmap is not an all-inclusive list of features we plan to > > build, > > > > > but > > > > > > rather, a list of features we have high confidence will ship in the > > > > > upcoming > > > > > > 6 month timeframe. > > > > > > > You are absolutely welcome to build an open source web based > > > > > import/export > > > > > > tool for App Engine. Once you have done this, submit a request and > > we > > > > > will > > > > > > add it to the Open Source projects page: > > >http://groups.google.com/group/google-appengine/web/google-app-engine... > > > > > > > On Wed, Jun 30, 2010 at 8:51 AM, mscwd01 <[email protected]> > > wrote: > > > > > > > Can I attempt to get the attention of a Google employee to ask > > "When > > > > > > > can we expect an in-built web based import/export tool for the > > GAE?" > > > > > > > > Thanks :P > > > > > > > > On Jun 30, 11:04 am, François Masurel <[email protected]> wrote: > > > > > > > > I dont think the bulkloader is really convenient for doing > > daily > > > > > > > > datastore backups of several gigabytes, and I'm not talking > > about > > > > > > > > exploding quotas. > > > > > > > > > I would rather prefer some kind of snapshot mechanism on > > datastore > > > > > > > > side. > > > > > > > > > François > > > > > > > > > On 29 juin, 20:40, Matthew Blain <[email protected]> > > wrote: > > > > > > > > > > The bulk loader does indeed provide backup and restore > > > > > functionality > > > > > > > > > today. While the implementation is in Python it can talk to > > your > > > > > Java > > > > > > > > > server and the entities are shared between Java and Python. > > > > > > > > > > On Jun 28, 4:46 pm, mscwd01 <[email protected]> wrote: > > > > > > > > > > > Thanks for your replying guys, it would be great if we > > could get > > > > > an > > > > > > > > > > official Google response to update us on this as well > > though! > > > > > > > > > > > I'll definitely have a look at the Bulk Loader in the > > meantime... > > > > > > > > > > > On Jun 28, 1:52 pm, Mike Dillon <[email protected]> > > wrote: > > > > > > > > > > > > Mscwd01, > > > > > > > > > > > > I'm not a google employee, but I have had success backing > > up > > > > > our > > > > > > > > > > > datastore and then restoring our datastore with > > > > > > > > > > > the bulkloader utility. Currently the bulkloader is in > > the > > > > > python > > > > > > > sdk, > > > > > > > > > > > so all you need to do is download that and then > > > > > > > > > > > youll be able to do backups and restore from those > > backups. > > > > > Check > > > > > > > out > > > > > > > > > > > the literature on the app engine, and try > > > > > > > > > > > a couple searches to get your bearings. Nick Johnson has > > > > > commented > > > > > > > or > > > > > > > > > > > written every search result that comes > > > > > > > > > > > back when I was researching this a couple of weeks back, > > and > > > > > there > > > > > > > is > > > > > > > > > > > also an I/O video on the bulkloader. > > > > > > > > > > > > Make sure you read some of the fine print, i.e. one > > caveat is > > > > > that > > > > > > > you > > > > > > > > > > > must have had your datastore populated for > > > > > > > > > > > at least 12 hours which is the amount of time it takes > > for the > > > > > > > > > > > statistics info to be built. That what the bulkloader > > > > > > > > > > > works off of to do its datastore dump. > > > > > > > > > > > > - Mike > > > > > > > > > > > > On Jun 28, 7:16 am, mscwd01 <[email protected]> wrote: > > > > > > > > > > > > > Hey Google Employees, > > > > > > > > > > > > > There has been a backup/restore facility on the roadmap > > for a > > > > > > > while > > > > > > > > > > > > now. Can you let us know if and when we may get an > > in-built > > > > > tool > > > > > > > that > > > > > > > > > > > > will allow us to easily save the contents of the > > datastore to > > > > > our > > > > > > > > > > > > local machines and restore it if needs be? > > > > > > > > > > > > > In the meantime, what is the preferred solution to > > > > > backup/restore > > > > > > > our > > > > > > > > > > > > Java based apps? > > > > > > > > > > > > > Thanks > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the > > Google > > > > > Groups > > > > > > > "Google App Engine for Java" group. > > > > > > > To post to this group, send email to > > > > > > > [email protected]. > > > > > > > To unsubscribe from this group, send email to > > > > > > > [email protected]<google-appengine-java%2B > > > > > > > [email protected]><google-appengine-java%2B > > [email protected]><google-appengine-java%2B > > > > > [email protected]> > > > > > > > . > > > > > > > For more options, visit this group at > > > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > > > > -- > > > > > > Ikai Lan > > > > > > Developer Programs Engineer, Google App Engine > > > > > > Blog:http://googleappengine.blogspot.com > > > > > > Twitter:http://twitter.com/app_engine > > > > > > Reddit:http://www.reddit.com/r/appengine > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > Groups > > > > > "Google App Engine for Java" group. > > > > > To post to this group, send email to > > > > > [email protected]. > > > > > To unsubscribe from this group, send email to > > > > > [email protected]<google-appengine-java%2B > > > > > [email protected]><google-appengine-java%2B > > [email protected]> > > > > > . > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > > -- > > > > Ikai Lan > > > > Developer Programs Engineer, Google App Engine > > > > Blog:http://googleappengine.blogspot.com > > > > Twitter:http://twitter.com/app_engine > > > > Reddit:http://www.reddit.com/r/appengine > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to > > [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine-java%2B > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
