Hi, What captain says is true: you should remove your make persistent for each iteration and replace it with a makePersistentAll() every N iterations: that will batch the writes and make them more efficient. (but don't wait to have too many of have: I'd start with approx 100 and see if I can increase this number)
Beside that, I must admin that your code looks pretty standard... Question: do you use the regular DS or the new Hgih Replication DS ? HR DS is supposed to be slower for writes. Let's see the result of this change regards didier On Mar 8, 6:52 am, captain <[email protected]> wrote: > Hi Claudio > > Try makePersistentAll() > > It will certainly help > > Cheers > CB > > On Mar 7, 2:57 am, Cláudio Coelho <[email protected]> wrote: > > > Since the bottleneck occurs when making objects of Location > > persistent, I assume that is the only relevant class. If anything else > > is relevant, tell me and I'll disclose. > > > @PersistenceCapable(identityType = > > IdentityType.APPLICATION,detachable="true") > > public class Location{ > > @PrimaryKey > > @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) > > Long locationId; > > @Persistent > > long countryId; > > @Persistent > > String name; > > @Persistent > > String country=null; > > @Persistent > > double latitude,longitude; > > @Persistent > > int population; > > @Persistent > > long geoBoxL1,geoBoxL2,geoBoxL3,geoBoxL4,geoBoxL5; > > public Location() > > { > > > } > > public Location(String name, long countryId,String country, double > > latitude, double longitude) > > { > > this.name=name; > > this.countryId=countryId; > > this.latitude=latitude; > > this.longitude=longitude; > > this.country=country; > > } > > public Location(LocationDTO dto) { > > this(); > > updateFromDTO(dto); > > } > > > public void updateFromDTO(LocationDTO dto) { > > setLocationId(dto.getLocationId()); > > setCountryId(dto.getCountryId()); > > setCountry(dto.getCountry()); > > setName(dto.getName()); > > setPopulation(dto.getPopulation()); > > setLatitude(dto.getLatitude()); > > setLongitude(dto.getLongitude()); > > setGeoBoxL1(dto.getGeoBoxL1()); > > setGeoBoxL2(dto.getGeoBoxL2()); > > setGeoBoxL3(dto.getGeoBoxL3()); > > setGeoBoxL4(dto.getGeoBoxL4()); > > setGeoBoxL5(dto.getGeoBoxL5()); > > > } > > public LocationDTO getDTO() > > { > > LocationDTO dto = new LocationDTO(); > > dto.setLocationId(getLocationId()); > > dto.setCountryId(getCountryId()); > > dto.setCountry(getCountry()); > > dto.setName(getName()); > > dto.setPopulation(getPopulation()); > > dto.setLatitude(getLatitude()); > > dto.setLongitude(getLongitude()); > > dto.setGeoBoxL1(getGeoBoxL1()); > > dto.setGeoBoxL2(getGeoBoxL2()); > > dto.setGeoBoxL3(getGeoBoxL3()); > > dto.setGeoBoxL4(getGeoBoxL4()); > > dto.setGeoBoxL5(getGeoBoxL5()); > > > return dto; > > } > > > public long getLocationId() { > > return locationId; > > } > > public void setLocationId(long locationId) { > > this.locationId = locationId; > > } > > public String getCountry() { > > return country; > > } > > > public int getPopulation() { > > return population; > > } > > public void setPopulation(int population) { > > this.population = population; > > } > > public void setCountry(String country) { > > this.country = country; > > } > > > public long getCountryId() { > > return countryId; > > } > > public void setCountryId(long countryId) { > > this.countryId = countryId; > > } > > public String getName() { > > return name; > > } > > public void setName(String name) { > > this.name = name; > > } > > public double getLatitude() { > > return latitude; > > } > > public void setLatitude(double latitude) { > > this.latitude = latitude; > > } > > public double getLongitude() { > > return longitude; > > } > > public void setLongitude(double longitude) { > > this.longitude = longitude; > > } > > > public long getGeoBoxL1() { > > return geoBoxL1; > > } > > public void setGeoBoxL1(long geoBoxL1) { > > this.geoBoxL1 = geoBoxL1; > > } > > public long getGeoBoxL2() { > > return geoBoxL2; > > } > > public void setGeoBoxL2(long geoBoxL2) { > > this.geoBoxL2 = geoBoxL2; > > } > > public long getGeoBoxL3() { > > return geoBoxL3; > > } > > public void setGeoBoxL3(long geoBoxL3) { > > this.geoBoxL3 = geoBoxL3; > > } > > public long getGeoBoxL4() { > > return geoBoxL4; > > } > > public void setGeoBoxL4(long geoBoxL4) { > > this.geoBoxL4 = geoBoxL4; > > } > > public long getGeoBoxL5() { > > return geoBoxL5; > > } > > public void setGeoBoxL5(long geoBoxL5) { > > this.geoBoxL5 = geoBoxL5; > > } > > > } > > > On Mar 7, 10:23 am, Didier Durand <[email protected]> wrote:> Hi, > > > > Could you please disclose the source code of your objects stored in > > > the ds ? > > > > You maybe right or wrong depending on how they were defined. > > > > regards > > > > didier > > > > On Mar 7, 9:41 am, Cláudio Coelho <[email protected]> wrote: > > > > > Hi > > > > I'm quite new at GAE, so I'm probably saying something silly, but for > > > > what I > > > > read on GAE Java and GWT Application Development, I was under the > > > > impression > > > > that GAE would generate indexes automatically except for some variable > > > > types > > > > (blobs, text, etc) and for when we explicitly asked it not to. > > > > However, when I look at the datastore-indexes-auto.xml file, it is > > > > indeed > > > > empty. Through debugging I have been able to isolate that it's the > > > > bottleneck is pm.makeConsistent of each entity to insert. Wouldn't > > > > having > > > > indexes make the inserting even slower (since these indexes have to be > > > > updated/rebuilt)? > > > > > thanks > > > > > C. > > > > > On Mon, Mar 7, 2011 at 5:44 AM, Didier Durand > > > > <[email protected]>wrote: > > > > > > Hi, > > > > > > The first issue that comes to mind in missing indexes: so huge scans > > > > > of all existing data when you upload an additional line. > > > > > > What are your indexes ? > > > > > > regards > > > > > > didier > > > > > > On Mar 6, 12:43 pm, Cláudio Coelho <[email protected]> wrote: > > > > > > Hi, > > > > > > My application needs a db of the world cities. I have a file that > > > > > > has > > > > > about > > > > > > 100k lines and I want to use it to populate the datastore. > > > > > > My first attempt failed because I didn't use tasks, so I was only > > > > > > able to > > > > > > upload a limited set of cities. It spent 2% of CPU total time on > > > > > > the app > > > > > > engine. > > > > > > I then changed the approach to use tasks. > > > > > > I basically read the file once to determine how many lines does it > > > > > > have > > > > > and > > > > > > then invoke tasks to read batches of 15k lines (supplying a start > > > > > > and end > > > > > > indexes). > > > > > > These invoked tasks (createCities(int start, int end)) read the file > > > > > again, > > > > > > get a list of 15k lines and then process it. > > > > > > The whole process takes about 15 seconds on my machine, however, > > > > > > when I > > > > > do > > > > > > this in the app engine, it takes 15 minutes (or more) and maxes out > > > > > > the > > > > > 6.5 > > > > > > hours of CPU time! I know there are plenty of optimizations I > > > > > > should do > > > > > to > > > > > > this process, but that doesn't seem to justify the 6.5 hours, so I > > > > > > must > > > > > be > > > > > > doing something seriously wrong. > > > > > > > Does anybody have any clue of what I'm doing wrong? (Code attached) > > > > > > > Thanks! > > > > > > > private void createCities() { > > > > > > try > > > > > > { > > > > > > int count = countCitiesToLoad(); > > > > > > final int batchSize = 15000; > > > > > > for(int start=0;start<count;start+=batchSize) > > > > > > { > > > > > > int end=start+batchSize; > > > > > > Queue queue = QueueFactory.getQueue("dbQueue"); > > > > > > TaskOptions topt = TaskOptions.Builder.withUrl("/dbservlet"); > > > > > > topt.param("command", "createCities"); > > > > > > topt.param("start", ""+start); > > > > > > topt.param("end", ""+end); > > > > > > queue.add(topt); > > > > > > logInfo("Dispatched order to create cities "+start+" to "+end); > > > > > > Thread.sleep(500);} > > > > > > }catch(Exception e) > > > > > > > { > > > > > > e.printStackTrace(); > > > > > > logError(e.getLocalizedMessage()); > > > > > > > } > > > > > > } > > > > > > > private void createCities(int start, int end) > > > > > > { > > > > > > try > > > > > > { > > > > > > logInfo("Reading cities "+start+" to "+end); > > > > > > BufferedReader br= new BufferedReader(new > > > > > > FileReader("data/cities.csv")); > > > > > > String line=br.readLine(); > > > > > > int counter=0; > > > > > > PersistenceManager pm = > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
