Re: [MTT devel] More GDS questions
On Feb 12, 2010, at 11:36 AM, Andrew Senin wrote: > I would also like to add to Igor’s comment that CPU time shown by Google is a > sum of all CPUs of distributed system involved in update operation (and who > knows how many servers are involved?). Also they “define ‘CPU hour’ in terms > of a hypothetical 1.4 GHz processor, whereas the actual processors we use in > production vary but are generally faster than this” (see comment of > DonSchwarz: > http://groups.google.com/group/google-appengine-java/browse_thread/thread/aa9f18638b7bbea9?pli=1). > According to the same topic 6.5 CPU hours is about 2.3 minutes real time. I > think you may try to remove some of indexes which need to be updated on each > new file upload (see Datastore Indexes on Web admin console). Excellent information. Google seemed to agree that 2.3 mins of real time should be nowhere near 6.5 CPU hours quota and they claimed that they fixed at least one issue regarding bulk uploads. However, this thread does imply that trickling in data over time instead of doing bulk uploads is a good idea. Was the rationale of caching all MTTGDS info during the Submit phase and actually uploading it during Finalize just a measure to reduce submission latency? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [MTT devel] More GDS questions
Hello Jeff, I would also like to add to Igor's comment that CPU time shown by Google is a sum of all CPUs of distributed system involved in update operation (and who knows how many servers are involved?). Also they "define 'CPU hour' in terms of a hypothetical 1.4 GHz processor, whereas the actual processors we use in production vary but are generally faster than this" (see comment of DonSchwarz: http://groups.google.com/group/google-appengine-java/browse_thread/thread/aa 9f18638b7bbea9?pli=1). According to the same topic 6.5 CPU hours is about 2.3 minutes real time. I think you may try to remove some of indexes which need to be updated on each new file upload (see Datastore Indexes on Web admin console). Regards, Andrew Senin. From: Igor Ivanov [mailto:igor.iva...@argus-cv.com] Sent: Friday, February 12, 2010 6:02 PM To: Jeff Squyres Cc: Development list for the MPI Testing Tool; b...@argus-cv.com; Yiftah Shahar Subject: Re: More GDS questions Hi Jeff, You touched a sore point. App engine forums are in filled the questions as yours. I can not know clear answer now. Igor Jeff Squyres wrote: Igor et al. -- 1. I'm not sure you saw Ethan's and my posts from the past day or so about GDS on the mtt-devel list; it just occurred to me that I don't know if you're members of the list or not. We've posted a few questions and comments that you may not have received if you're not on the list: http://www.open-mpi.org/community/lists/mtt-devel/2010/02/index.php 2. I'm still looking into the perl syntax error that caused my Big Submit to GDS to fail. But looking at the Google logs, it looks like at least *some* of my test run results made it up to GDS. There was a BIG spike in CPU usage (3.2 hours of CPU time!) when it submitted -- see the attached CPU usage graph from the apps dashboard. Does anyone know why it takes so much CPU just to submit data to GDS? 3.2 CPU hours is a LOT! It makes me a bit concerned that only part of a single Cisco MTT run submit checked through almost half of our daily CPU quota (6.5 CPU hours/day). Is there any way to reduce the amount of CPU necessary just to submit data? _ __ Information from ESET NOD32 Antivirus, version of virus signature database 4861 (20100212) __ The message was checked by ESET NOD32 Antivirus. http://www.esetnod32.ru
Re: [MTT devel] More GDS questions
On Feb 12, 2010, at 10:02 AM, Igor Ivanov wrote: > You touched a sore point. App engine forums are in filled the questions as > yours. > I can not know clear answer now. Ok, bummer. :-) >> 2. I'm still looking into the perl syntax error that caused my Big Submit to >> GDS to fail. But looking at the Google logs, it looks like at least *some* >> of my test run results made it up to GDS. There was a BIG spike in CPU >> usage (3.2 hours of CPU time!) when it submitted -- see the attached CPU >> usage graph from the apps dashboard. >> >> Does anyone know why it takes so much CPU just to submit data to GDS? 3.2 >> CPU hours is a LOT! >> >> It makes me a bit concerned that only part of a single Cisco MTT run submit >> checked through almost half of our daily CPU quota (6.5 CPU hours/day). Is >> there any way to reduce the amount of CPU necessary just to submit data? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [MTT devel] More GDS questions
Hi Jeff, You touched a sore point. App engine forums are in filled the questions as yours. I can not know clear answer now. Igor Jeff Squyres wrote: Igor et al. -- 1. I'm not sure you saw Ethan's and my posts from the past day or so about GDS on the mtt-devel list; it just occurred to me that I don't know if you're members of the list or not. We've posted a few questions and comments that you may not have received if you're not on the list: http://www.open-mpi.org/community/lists/mtt-devel/2010/02/index.php 2. I'm still looking into the perl syntax error that caused my Big Submit to GDS to fail. But looking at the Google logs, it looks like at least *some* of my test run results made it up to GDS. There was a BIG spike in CPU usage (3.2 hours of CPU time!) when it submitted -- see the attached CPU usage graph from the apps dashboard. Does anyone know why it takes so much CPU just to submit data to GDS? 3.2 CPU hours is a LOT! It makes me a bit concerned that only part of a single Cisco MTT run submit checked through almost half of our daily CPU quota (6.5 CPU hours/day). Is there any way to reduce the amount of CPU necessary just to submit data? __ Information from ESET NOD32 Antivirus, version of virus signature database 4861 (20100212) __ The message was checked by ESET NOD32 Antivirus. http://www.esetnod32.ru
[MTT devel] More GDS questions
Igor et al. -- 1. I'm not sure you saw Ethan's and my posts from the past day or so about GDS on the mtt-devel list; it just occurred to me that I don't know if you're members of the list or not. We've posted a few questions and comments that you may not have received if you're not on the list: http://www.open-mpi.org/community/lists/mtt-devel/2010/02/index.php 2. I'm still looking into the perl syntax error that caused my Big Submit to GDS to fail. But looking at the Google logs, it looks like at least *some* of my test run results made it up to GDS. There was a BIG spike in CPU usage (3.2 hours of CPU time!) when it submitted -- see the attached CPU usage graph from the apps dashboard. Does anyone know why it takes so much CPU just to submit data to GDS? 3.2 CPU hours is a LOT! It makes me a bit concerned that only part of a single Cisco MTT run submit checked through almost half of our daily CPU quota (6.5 CPU hours/day). Is there any way to reduce the amount of CPU necessary just to submit data? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
