Since the bottleneck occurs when making objects of Location
persistent, I assume that is the only relevant class. If anything else
is relevant, tell me and I'll disclose.
@PersistenceCapable(identityType =
IdentityType.APPLICATION,detachable="true")
public class Location{
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
Long locationId;
@Persistent
long countryId;
@Persistent
String name;
@Persistent
String country=null;
@Persistent
double latitude,longitude;
@Persistent
int population;
@Persistent
long geoBoxL1,geoBoxL2,geoBoxL3,geoBoxL4,geoBoxL5;
public Location()
{
}
public Location(String name, long countryId,String country, double
latitude, double longitude)
{
this.name=name;
this.countryId=countryId;
this.latitude=latitude;
this.longitude=longitude;
this.country=country;
}
public Location(LocationDTO dto) {
this();
updateFromDTO(dto);
}
public void updateFromDTO(LocationDTO dto) {
setLocationId(dto.getLocationId());
setCountryId(dto.getCountryId());
setCountry(dto.getCountry());
setName(dto.getName());
setPopulation(dto.getPopulation());
setLatitude(dto.getLatitude());
setLongitude(dto.getLongitude());
setGeoBoxL1(dto.getGeoBoxL1());
setGeoBoxL2(dto.getGeoBoxL2());
setGeoBoxL3(dto.getGeoBoxL3());
setGeoBoxL4(dto.getGeoBoxL4());
setGeoBoxL5(dto.getGeoBoxL5());
}
public LocationDTO getDTO()
{
LocationDTO dto = new LocationDTO();
dto.setLocationId(getLocationId());
dto.setCountryId(getCountryId());
dto.setCountry(getCountry());
dto.setName(getName());
dto.setPopulation(getPopulation());
dto.setLatitude(getLatitude());
dto.setLongitude(getLongitude());
dto.setGeoBoxL1(getGeoBoxL1());
dto.setGeoBoxL2(getGeoBoxL2());
dto.setGeoBoxL3(getGeoBoxL3());
dto.setGeoBoxL4(getGeoBoxL4());
dto.setGeoBoxL5(getGeoBoxL5());
return dto;
}
public long getLocationId() {
return locationId;
}
public void setLocationId(long locationId) {
this.locationId = locationId;
}
public String getCountry() {
return country;
}
public int getPopulation() {
return population;
}
public void setPopulation(int population) {
this.population = population;
}
public void setCountry(String country) {
this.country = country;
}
public long getCountryId() {
return countryId;
}
public void setCountryId(long countryId) {
this.countryId = countryId;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public double getLatitude() {
return latitude;
}
public void setLatitude(double latitude) {
this.latitude = latitude;
}
public double getLongitude() {
return longitude;
}
public void setLongitude(double longitude) {
this.longitude = longitude;
}
public long getGeoBoxL1() {
return geoBoxL1;
}
public void setGeoBoxL1(long geoBoxL1) {
this.geoBoxL1 = geoBoxL1;
}
public long getGeoBoxL2() {
return geoBoxL2;
}
public void setGeoBoxL2(long geoBoxL2) {
this.geoBoxL2 = geoBoxL2;
}
public long getGeoBoxL3() {
return geoBoxL3;
}
public void setGeoBoxL3(long geoBoxL3) {
this.geoBoxL3 = geoBoxL3;
}
public long getGeoBoxL4() {
return geoBoxL4;
}
public void setGeoBoxL4(long geoBoxL4) {
this.geoBoxL4 = geoBoxL4;
}
public long getGeoBoxL5() {
return geoBoxL5;
}
public void setGeoBoxL5(long geoBoxL5) {
this.geoBoxL5 = geoBoxL5;
}
}
On Mar 7, 10:23 am, Didier Durand <[email protected]> wrote:
> Hi,
>
> Could you please disclose the source code of your objects stored in
> the ds ?
>
> You maybe right or wrong depending on how they were defined.
>
> regards
>
> didier
>
> On Mar 7, 9:41 am, Cláudio Coelho <[email protected]> wrote:
>
>
>
>
>
>
>
> > Hi
> > I'm quite new at GAE, so I'm probably saying something silly, but for what I
> > read on GAE Java and GWT Application Development, I was under the impression
> > that GAE would generate indexes automatically except for some variable types
> > (blobs, text, etc) and for when we explicitly asked it not to.
> > However, when I look at the datastore-indexes-auto.xml file, it is indeed
> > empty. Through debugging I have been able to isolate that it's the
> > bottleneck is pm.makeConsistent of each entity to insert. Wouldn't having
> > indexes make the inserting even slower (since these indexes have to be
> > updated/rebuilt)?
>
> > thanks
>
> > C.
>
> > On Mon, Mar 7, 2011 at 5:44 AM, Didier Durand
> > <[email protected]>wrote:
>
> > > Hi,
>
> > > The first issue that comes to mind in missing indexes: so huge scans
> > > of all existing data when you upload an additional line.
>
> > > What are your indexes ?
>
> > > regards
>
> > > didier
>
> > > On Mar 6, 12:43 pm, Cláudio Coelho <[email protected]> wrote:
> > > > Hi,
> > > > My application needs a db of the world cities. I have a file that has
> > > about
> > > > 100k lines and I want to use it to populate the datastore.
> > > > My first attempt failed because I didn't use tasks, so I was only able
> > > > to
> > > > upload a limited set of cities. It spent 2% of CPU total time on the app
> > > > engine.
> > > > I then changed the approach to use tasks.
> > > > I basically read the file once to determine how many lines does it have
> > > and
> > > > then invoke tasks to read batches of 15k lines (supplying a start and
> > > > end
> > > > indexes).
> > > > These invoked tasks (createCities(int start, int end)) read the file
> > > again,
> > > > get a list of 15k lines and then process it.
> > > > The whole process takes about 15 seconds on my machine, however, when I
> > > do
> > > > this in the app engine, it takes 15 minutes (or more) and maxes out the
> > > 6.5
> > > > hours of CPU time! I know there are plenty of optimizations I should do
> > > to
> > > > this process, but that doesn't seem to justify the 6.5 hours, so I must
> > > be
> > > > doing something seriously wrong.
>
> > > > Does anybody have any clue of what I'm doing wrong? (Code attached)
>
> > > > Thanks!
>
> > > > private void createCities() {
> > > > try
> > > > {
> > > > int count = countCitiesToLoad();
> > > > final int batchSize = 15000;
> > > > for(int start=0;start<count;start+=batchSize)
> > > > {
> > > > int end=start+batchSize;
> > > > Queue queue = QueueFactory.getQueue("dbQueue");
> > > > TaskOptions topt = TaskOptions.Builder.withUrl("/dbservlet");
> > > > topt.param("command", "createCities");
> > > > topt.param("start", ""+start);
> > > > topt.param("end", ""+end);
> > > > queue.add(topt);
> > > > logInfo("Dispatched order to create cities "+start+" to "+end);
> > > > Thread.sleep(500);}
> > > > }catch(Exception e)
>
> > > > {
> > > > e.printStackTrace();
> > > > logError(e.getLocalizedMessage());
>
> > > > }
> > > > }
>
> > > > private void createCities(int start, int end)
> > > > {
> > > > try
> > > > {
> > > > logInfo("Reading cities "+start+" to "+end);
> > > > BufferedReader br= new BufferedReader(new
> > > > FileReader("data/cities.csv"));
> > > > String line=br.readLine();
> > > > int counter=0;
> > > > PersistenceManager pm = PMF.get().getPersistenceManager();
> > > > ArrayList<String> lines = new ArrayList<String>();
> > > > while(line!=null || counter<end)
> > > > {
> > > > if(counter>=start && counter <end)
> > > > lines.add(line);
> > > > counter++;
> > > > line=br.readLine();}
>
> > > > br.close();
> > > > logInfo("Adding cities "+start+" to "+end);
> > > > createCities(lines);
> > > > logInfo("Done cities "+start+" to "+end);}
>
> > > > catch(Exception e)
> > > > {
> > > > e.printStackTrace();
> > > > logError(e.getLocalizedMessage());
>
> > > > }
> > > > }
>
> > > > private void createCities(ArrayList<String> lines)
> > > > {
> > > > if(lines==null)
> > > > return;
> > > > PersistenceManager pm = PMF.get().getPersistenceManager();
> > > > HashMap<String, Country> countryMap = loadCountryMap();
> > > > try{
> > > > for(String line : lines)
> > > > if(line!=null)
> > > > {
> > > > String fields[]=line.split(",");
> > > > if(fields.length<7)
> > > > logError("length error in line:"+line);
> > > > else
> > > > {
> > > > Location loc = new Location();
> > > > loc.setName(fields[2]);
> > > > loc.setLatitude(Double.parseDouble(fields[3]));
> > > > loc.setLongitude(Double.parseDouble(fields[4]));
> > > > loc.setPopulation(Integer.parseInt(fields[6]));
> > > > {
> > > > Country c = countryMap.get(fields[5]);
> > > > if(c==null)
> > > > logError("Failed to get country for:"+line);
> > > > else
> > > > {
> > > > loc.setCountryId(c.getCountryId());
> > > > loc.setCountry(c.getName());
> > > > pm.makePersistent(loc);}
> > > > }
> > > > }
> > > > }
> > > > } catch (Exception e) {
>
> > > > e.printStackTrace();
>
> > > > } finally {
> > > > pm.close();
> > > > }
> > > > }
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Google App Engine for Java" group.
> > > To post to this group, send email to
> > > [email protected].
> > > To unsubscribe from this group, send email to
> > > [email protected].
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine-java?hl=en.
>
> > --
> > Cláudio Coelho
>
> > "Great spirits have often encountered violent opposition from weak minds."
> > A. Einstein
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
