Hi, Could you please disclose the source code of your objects stored in the ds ?
You maybe right or wrong depending on how they were defined. regards didier On Mar 7, 9:41 am, Cláudio Coelho <[email protected]> wrote: > Hi > I'm quite new at GAE, so I'm probably saying something silly, but for what I > read on GAE Java and GWT Application Development, I was under the impression > that GAE would generate indexes automatically except for some variable types > (blobs, text, etc) and for when we explicitly asked it not to. > However, when I look at the datastore-indexes-auto.xml file, it is indeed > empty. Through debugging I have been able to isolate that it's the > bottleneck is pm.makeConsistent of each entity to insert. Wouldn't having > indexes make the inserting even slower (since these indexes have to be > updated/rebuilt)? > > thanks > > C. > > On Mon, Mar 7, 2011 at 5:44 AM, Didier Durand <[email protected]>wrote: > > > > > Hi, > > > The first issue that comes to mind in missing indexes: so huge scans > > of all existing data when you upload an additional line. > > > What are your indexes ? > > > regards > > > didier > > > On Mar 6, 12:43 pm, Cláudio Coelho <[email protected]> wrote: > > > Hi, > > > My application needs a db of the world cities. I have a file that has > > about > > > 100k lines and I want to use it to populate the datastore. > > > My first attempt failed because I didn't use tasks, so I was only able to > > > upload a limited set of cities. It spent 2% of CPU total time on the app > > > engine. > > > I then changed the approach to use tasks. > > > I basically read the file once to determine how many lines does it have > > and > > > then invoke tasks to read batches of 15k lines (supplying a start and end > > > indexes). > > > These invoked tasks (createCities(int start, int end)) read the file > > again, > > > get a list of 15k lines and then process it. > > > The whole process takes about 15 seconds on my machine, however, when I > > do > > > this in the app engine, it takes 15 minutes (or more) and maxes out the > > 6.5 > > > hours of CPU time! I know there are plenty of optimizations I should do > > to > > > this process, but that doesn't seem to justify the 6.5 hours, so I must > > be > > > doing something seriously wrong. > > > > Does anybody have any clue of what I'm doing wrong? (Code attached) > > > > Thanks! > > > > private void createCities() { > > > try > > > { > > > int count = countCitiesToLoad(); > > > final int batchSize = 15000; > > > for(int start=0;start<count;start+=batchSize) > > > { > > > int end=start+batchSize; > > > Queue queue = QueueFactory.getQueue("dbQueue"); > > > TaskOptions topt = TaskOptions.Builder.withUrl("/dbservlet"); > > > topt.param("command", "createCities"); > > > topt.param("start", ""+start); > > > topt.param("end", ""+end); > > > queue.add(topt); > > > logInfo("Dispatched order to create cities "+start+" to "+end); > > > Thread.sleep(500);} > > > }catch(Exception e) > > > > { > > > e.printStackTrace(); > > > logError(e.getLocalizedMessage()); > > > > } > > > } > > > > private void createCities(int start, int end) > > > { > > > try > > > { > > > logInfo("Reading cities "+start+" to "+end); > > > BufferedReader br= new BufferedReader(new FileReader("data/cities.csv")); > > > String line=br.readLine(); > > > int counter=0; > > > PersistenceManager pm = PMF.get().getPersistenceManager(); > > > ArrayList<String> lines = new ArrayList<String>(); > > > while(line!=null || counter<end) > > > { > > > if(counter>=start && counter <end) > > > lines.add(line); > > > counter++; > > > line=br.readLine();} > > > > br.close(); > > > logInfo("Adding cities "+start+" to "+end); > > > createCities(lines); > > > logInfo("Done cities "+start+" to "+end);} > > > > catch(Exception e) > > > { > > > e.printStackTrace(); > > > logError(e.getLocalizedMessage()); > > > > } > > > } > > > > private void createCities(ArrayList<String> lines) > > > { > > > if(lines==null) > > > return; > > > PersistenceManager pm = PMF.get().getPersistenceManager(); > > > HashMap<String, Country> countryMap = loadCountryMap(); > > > try{ > > > for(String line : lines) > > > if(line!=null) > > > { > > > String fields[]=line.split(","); > > > if(fields.length<7) > > > logError("length error in line:"+line); > > > else > > > { > > > Location loc = new Location(); > > > loc.setName(fields[2]); > > > loc.setLatitude(Double.parseDouble(fields[3])); > > > loc.setLongitude(Double.parseDouble(fields[4])); > > > loc.setPopulation(Integer.parseInt(fields[6])); > > > { > > > Country c = countryMap.get(fields[5]); > > > if(c==null) > > > logError("Failed to get country for:"+line); > > > else > > > { > > > loc.setCountryId(c.getCountryId()); > > > loc.setCountry(c.getName()); > > > pm.makePersistent(loc);} > > > } > > > } > > > } > > > } catch (Exception e) { > > > > e.printStackTrace(); > > > > } finally { > > > pm.close(); > > > } > > > } > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to > > [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > Cláudio Coelho > > "Great spirits have often encountered violent opposition from weak minds." > A. Einstein -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
