I'm definately going to start comparing the load times. I think you're right, reading a file seems to be a much faster idea.
There's one problem, however, since App Engine doesn't allow users to write to the disk, I won't be able to update the file when I do need to update it. So that means I have to do it manually when I do need to update it, right? That seems like a pretty tedious task. This got me thinking, instead of storing it in a file, what about just storing it in some huge blob entity. Won't that be essentially be doing the same thing, except with the benefit of being able to write and update to it when I need to? It would seem if I stored all the keywords in a json blob that could solve my problem as well. What are your thoughts on that? thanks for your help Nick On Dec 18, 10:23 am, "Nick Johnson (Google)" <[email protected]> wrote: > On Fri, Dec 18, 2009 at 6:12 PM, killer barney <[email protected]> wrote: > > You know I thought about that, is it really better to open a file and > > read the contents then just querying the database? > > If it'll fit in the file, and you can read it efficiently (eg, you're not > loading lots of data you don't need), absolutely! Files are stored on the > local filesystem, so access to them is much faster than datastore access. > > > > > Maybe for a table of my size, opening files would be faster, so > > definately a consideration. > > > So considering this solution, how would you load the keywords from a > > local file on a disk? If I am not wrong, you have to just download it > > via http because there is no direct local file load right? If so, then > > won't that also compound the loading times of a local file? > > No, you can store the data in a local file, and load it directly from disk > the same way you load templates. Simply store it in whatever format you > need, such as comma-delimited, and load it as you normally would. > > -Nick > > > > > > > > > On Dec 18, 2:34 am, "Nick Johnson (Google)" <[email protected]> > > wrote: > > > Hi, > > > > On Thu, Dec 17, 2009 at 10:48 PM, killer barney <[email protected]> > > wrote: > > > > sorry, hopefully I can explain in better detail :) > > > > > I have a list of keywords in a table that's the size of thousands. It > > > > changes, but not very often, so the list doesn't have to be updated > > > > all that often. > > > > > When I receive a body of text, I want to highlight the keywords in the > > > > body of text. So I have to constantly compare the body of text to my > > > > keywords. Because of that, I think querying would be too large a hit > > > > on the database. So I have a working version where I store in > > > > memcache keywords sharded by letter, for example storing all keywords > > > > starting with the letter 'A', and so forth. This is a temporary fix > > > > that will work for now but will cause problems later as my keywords > > > > exceed 1000 per letter. Not only that, when the memcache gets > > > > disposed, the user will have to query to populate that memcache, which > > > > could take a really long time for that unlucky person who has to > > > > populate the memcache for everybody else. > > > > > So looking for a better version of what I'm currently doing. > > > > Why not simply load the keywords from a local file on disk? > > > > -Nick Johnson > > > > > On Dec 16, 5:12 am, "Nick Johnson (Google)" <[email protected]> > > > > wrote: > > > > > Hi, > > > > > > It's difficult to comment usefully without more information. Where do > > you > > > > > get your list from? How is it calculated? How large is it? > > > > > > -Nick Johnson > > > > > > On Tue, Dec 15, 2009 at 11:11 PM, killer barney <[email protected]> > > > > wrote: > > > > > > I have a question that perhaps someone can give me some insight on. > > > > > > > I have a list of thousands of keywords in my website that I am > > going > > > > > > to constantly need to query and iterate through. So I thought > > rather > > > > > > than querying for every keyword in the datastore everytime, I > > should > > > > > > store it in the memcache. But even this doesn't seem like a very > > good > > > > > > solution as the memcache gets eliminated and I'm going to have to > > > > > > somehow query the whole list and store it back into memcache > > everytime > > > > > > it gets disposed of. > > > > > > > I thought about having the list stored as sharded arrays into > > > > > > memcache, so when it does get disposed of, then I only have to > > update > > > > > > the keywords that start with "N", for example, but this still > > doesn't > > > > > > seem like the best solution. > > > > > > > Is there a better way to do this? > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > > > "Google App Engine" group. > > > > > > To post to this group, send email to > > [email protected] > > > > . > > > > > > To unsubscribe from this group, send email to > > > > > > [email protected]<google-appengine%2Bunsubscrib > > > > > > [email protected]> > > <google-appengine%[email protected]<google-appengine%252Bunsub > > [email protected]> > > ><google-appengine%2Bunsubscrib > > > > [email protected]> > > > > > > . > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > > > > > Nick Johnson, Developer Programs Engineer, App Engine > > > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > > > Number: > > > > > 368047 > > > > > > -- > > > > > Nick Johnson, Developer Programs Engineer, App Engine > > > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > > > Number: > > > > > 368047 > > > > > -- > > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to [email protected] > > . > > > > To unsubscribe from this group, send email to > > > > [email protected]<google-appengine%2Bunsubscrib > > > > [email protected]> > > <google-appengine%[email protected]<google-appengine%252Bunsub > > [email protected]> > > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > > Nick Johnson, Developer Programs Engineer, App Engine > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > Number: > > > 368047 > > > -- > > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%2Bunsubscrib > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Nick Johnson, Developer Programs Engineer, App Engine > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
