Hi I'm quite new at GAE, so I'm probably saying something silly, but for what I read on GAE Java and GWT Application Development, I was under the impression that GAE would generate indexes automatically except for some variable types (blobs, text, etc) and for when we explicitly asked it not to. However, when I look at the datastore-indexes-auto.xml file, it is indeed empty. Through debugging I have been able to isolate that it's the bottleneck is pm.makeConsistent of each entity to insert. Wouldn't having indexes make the inserting even slower (since these indexes have to be updated/rebuilt)?
thanks C. On Mon, Mar 7, 2011 at 5:44 AM, Didier Durand <[email protected]>wrote: > Hi, > > The first issue that comes to mind in missing indexes: so huge scans > of all existing data when you upload an additional line. > > What are your indexes ? > > regards > > didier > > On Mar 6, 12:43 pm, Cláudio Coelho <[email protected]> wrote: > > Hi, > > My application needs a db of the world cities. I have a file that has > about > > 100k lines and I want to use it to populate the datastore. > > My first attempt failed because I didn't use tasks, so I was only able to > > upload a limited set of cities. It spent 2% of CPU total time on the app > > engine. > > I then changed the approach to use tasks. > > I basically read the file once to determine how many lines does it have > and > > then invoke tasks to read batches of 15k lines (supplying a start and end > > indexes). > > These invoked tasks (createCities(int start, int end)) read the file > again, > > get a list of 15k lines and then process it. > > The whole process takes about 15 seconds on my machine, however, when I > do > > this in the app engine, it takes 15 minutes (or more) and maxes out the > 6.5 > > hours of CPU time! I know there are plenty of optimizations I should do > to > > this process, but that doesn't seem to justify the 6.5 hours, so I must > be > > doing something seriously wrong. > > > > Does anybody have any clue of what I'm doing wrong? (Code attached) > > > > Thanks! > > > > private void createCities() { > > try > > { > > int count = countCitiesToLoad(); > > final int batchSize = 15000; > > for(int start=0;start<count;start+=batchSize) > > { > > int end=start+batchSize; > > Queue queue = QueueFactory.getQueue("dbQueue"); > > TaskOptions topt = TaskOptions.Builder.withUrl("/dbservlet"); > > topt.param("command", "createCities"); > > topt.param("start", ""+start); > > topt.param("end", ""+end); > > queue.add(topt); > > logInfo("Dispatched order to create cities "+start+" to "+end); > > Thread.sleep(500);} > > }catch(Exception e) > > > > { > > e.printStackTrace(); > > logError(e.getLocalizedMessage()); > > > > } > > } > > > > private void createCities(int start, int end) > > { > > try > > { > > logInfo("Reading cities "+start+" to "+end); > > BufferedReader br= new BufferedReader(new FileReader("data/cities.csv")); > > String line=br.readLine(); > > int counter=0; > > PersistenceManager pm = PMF.get().getPersistenceManager(); > > ArrayList<String> lines = new ArrayList<String>(); > > while(line!=null || counter<end) > > { > > if(counter>=start && counter <end) > > lines.add(line); > > counter++; > > line=br.readLine();} > > > > br.close(); > > logInfo("Adding cities "+start+" to "+end); > > createCities(lines); > > logInfo("Done cities "+start+" to "+end);} > > > > catch(Exception e) > > { > > e.printStackTrace(); > > logError(e.getLocalizedMessage()); > > > > } > > } > > > > private void createCities(ArrayList<String> lines) > > { > > if(lines==null) > > return; > > PersistenceManager pm = PMF.get().getPersistenceManager(); > > HashMap<String, Country> countryMap = loadCountryMap(); > > try{ > > for(String line : lines) > > if(line!=null) > > { > > String fields[]=line.split(","); > > if(fields.length<7) > > logError("length error in line:"+line); > > else > > { > > Location loc = new Location(); > > loc.setName(fields[2]); > > loc.setLatitude(Double.parseDouble(fields[3])); > > loc.setLongitude(Double.parseDouble(fields[4])); > > loc.setPopulation(Integer.parseInt(fields[6])); > > { > > Country c = countryMap.get(fields[5]); > > if(c==null) > > logError("Failed to get country for:"+line); > > else > > { > > loc.setCountryId(c.getCountryId()); > > loc.setCountry(c.getName()); > > pm.makePersistent(loc);} > > } > > } > > } > > } catch (Exception e) { > > > > e.printStackTrace(); > > > > } finally { > > pm.close(); > > } > > } > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- Cláudio Coelho "Great spirits have often encountered violent opposition from weak minds." A. Einstein -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
