Hi,
My application needs a db of the world cities. I have a file that has about
100k lines and I want to use it to populate the datastore.
My first attempt failed because I didn't use tasks, so I was only able to
upload a limited set of cities. It spent 2% of CPU total time on the app
engine.
I then changed the approach to use tasks.
I basically read the file once to determine how many lines does it have and
then invoke tasks to read batches of 15k lines (supplying a start and end
indexes).
These invoked tasks (createCities(int start, int end)) read the file again,
get a list of 15k lines and then process it.
The whole process takes about 15 seconds on my machine, however, when I do
this in the app engine, it takes 15 minutes (or more) and maxes out the 6.5
hours of CPU time! I know there are plenty of optimizations I should do to
this process, but that doesn't seem to justify the 6.5 hours, so I must be
doing something seriously wrong.
Does anybody have any clue of what I'm doing wrong? (Code attached)
Thanks!
private void createCities() {
try
{
int count = countCitiesToLoad();
final int batchSize = 15000;
for(int start=0;start<count;start+=batchSize)
{
int end=start+batchSize;
Queue queue = QueueFactory.getQueue("dbQueue");
TaskOptions topt = TaskOptions.Builder.withUrl("/dbservlet");
topt.param("command", "createCities");
topt.param("start", ""+start);
topt.param("end", ""+end);
queue.add(topt);
logInfo("Dispatched order to create cities "+start+" to "+end);
Thread.sleep(500);
}
}catch(Exception e)
{
e.printStackTrace();
logError(e.getLocalizedMessage());
}
}
private void createCities(int start, int end)
{
try
{
logInfo("Reading cities "+start+" to "+end);
BufferedReader br= new BufferedReader(new FileReader("data/cities.csv"));
String line=br.readLine();
int counter=0;
PersistenceManager pm = PMF.get().getPersistenceManager();
ArrayList<String> lines = new ArrayList<String>();
while(line!=null || counter<end)
{
if(counter>=start && counter <end)
lines.add(line);
counter++;
line=br.readLine();
}
br.close();
logInfo("Adding cities "+start+" to "+end);
createCities(lines);
logInfo("Done cities "+start+" to "+end);
}
catch(Exception e)
{
e.printStackTrace();
logError(e.getLocalizedMessage());
}
}
private void createCities(ArrayList<String> lines)
{
if(lines==null)
return;
PersistenceManager pm = PMF.get().getPersistenceManager();
HashMap<String, Country> countryMap = loadCountryMap();
try{
for(String line : lines)
if(line!=null)
{
String fields[]=line.split(",");
if(fields.length<7)
logError("length error in line:"+line);
else
{
Location loc = new Location();
loc.setName(fields[2]);
loc.setLatitude(Double.parseDouble(fields[3]));
loc.setLongitude(Double.parseDouble(fields[4]));
loc.setPopulation(Integer.parseInt(fields[6]));
{
Country c = countryMap.get(fields[5]);
if(c==null)
logError("Failed to get country for:"+line);
else
{
loc.setCountryId(c.getCountryId());
loc.setCountry(c.getName());
pm.makePersistent(loc);
}
}
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
pm.close();
}
}
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
