Hi, The first issue that comes to mind in missing indexes: so huge scans of all existing data when you upload an additional line.
What are your indexes ? regards didier On Mar 6, 12:43 pm, Cláudio Coelho <[email protected]> wrote: > Hi, > My application needs a db of the world cities. I have a file that has about > 100k lines and I want to use it to populate the datastore. > My first attempt failed because I didn't use tasks, so I was only able to > upload a limited set of cities. It spent 2% of CPU total time on the app > engine. > I then changed the approach to use tasks. > I basically read the file once to determine how many lines does it have and > then invoke tasks to read batches of 15k lines (supplying a start and end > indexes). > These invoked tasks (createCities(int start, int end)) read the file again, > get a list of 15k lines and then process it. > The whole process takes about 15 seconds on my machine, however, when I do > this in the app engine, it takes 15 minutes (or more) and maxes out the 6.5 > hours of CPU time! I know there are plenty of optimizations I should do to > this process, but that doesn't seem to justify the 6.5 hours, so I must be > doing something seriously wrong. > > Does anybody have any clue of what I'm doing wrong? (Code attached) > > Thanks! > > private void createCities() { > try > { > int count = countCitiesToLoad(); > final int batchSize = 15000; > for(int start=0;start<count;start+=batchSize) > { > int end=start+batchSize; > Queue queue = QueueFactory.getQueue("dbQueue"); > TaskOptions topt = TaskOptions.Builder.withUrl("/dbservlet"); > topt.param("command", "createCities"); > topt.param("start", ""+start); > topt.param("end", ""+end); > queue.add(topt); > logInfo("Dispatched order to create cities "+start+" to "+end); > Thread.sleep(500);} > }catch(Exception e) > > { > e.printStackTrace(); > logError(e.getLocalizedMessage()); > > } > } > > private void createCities(int start, int end) > { > try > { > logInfo("Reading cities "+start+" to "+end); > BufferedReader br= new BufferedReader(new FileReader("data/cities.csv")); > String line=br.readLine(); > int counter=0; > PersistenceManager pm = PMF.get().getPersistenceManager(); > ArrayList<String> lines = new ArrayList<String>(); > while(line!=null || counter<end) > { > if(counter>=start && counter <end) > lines.add(line); > counter++; > line=br.readLine();} > > br.close(); > logInfo("Adding cities "+start+" to "+end); > createCities(lines); > logInfo("Done cities "+start+" to "+end);} > > catch(Exception e) > { > e.printStackTrace(); > logError(e.getLocalizedMessage()); > > } > } > > private void createCities(ArrayList<String> lines) > { > if(lines==null) > return; > PersistenceManager pm = PMF.get().getPersistenceManager(); > HashMap<String, Country> countryMap = loadCountryMap(); > try{ > for(String line : lines) > if(line!=null) > { > String fields[]=line.split(","); > if(fields.length<7) > logError("length error in line:"+line); > else > { > Location loc = new Location(); > loc.setName(fields[2]); > loc.setLatitude(Double.parseDouble(fields[3])); > loc.setLongitude(Double.parseDouble(fields[4])); > loc.setPopulation(Integer.parseInt(fields[6])); > { > Country c = countryMap.get(fields[5]); > if(c==null) > logError("Failed to get country for:"+line); > else > { > loc.setCountryId(c.getCountryId()); > loc.setCountry(c.getName()); > pm.makePersistent(loc);} > } > } > } > } catch (Exception e) { > > e.printStackTrace(); > > } finally { > pm.close(); > } > } > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
