[MTT devel] MTT GDS -- one more...
Heh... even more questions... (BTW, Ethan and I have asked s many questions that if it helps, I can setup a webex and we can all discuss this in person rather than via 1,000,000 annoying emails from us. :-) Webex can call you; no one will need to pay for an international call) 1. It looks like the main benefits of using the Google App Engine -- specifically for MTT -- is that we can use the GDS and/or we can host an application on their web servers. Is that correct? 2. In reading through the Google Appengine docs, the GDS stuff looks like we mainly can access the data through GQL. I don't see any mention of doing map/reduce kinds of computations (Ethan and I were talking on the phone today about MTT Appengine possibilities). I'm new to all this stuff, so it's quite possible that a) I missed it, or b) I just don't understand what I'm seeing/reading yet. Or does GQL do map/reduce on the back end to do its magic? Is GQL the main/only way we have to access GDS? 3. Is there a reason that MTTGDS.pm doesn't use the python API to directly talk to GDS? I.e., what is the rationale for using a web app on appengine? Is the web app doing stuff that we can't do at the client? Ditto for bquery.pl and breport.pl. (these questions are partially fueled by my curiosity and concern about why we're using so much CPU at Google) -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[MTT devel] More GDS questions
Igor et al. -- 1. I'm not sure you saw Ethan's and my posts from the past day or so about GDS on the mtt-devel list; it just occurred to me that I don't know if you're members of the list or not. We've posted a few questions and comments that you may not have received if you're not on the list: http://www.open-mpi.org/community/lists/mtt-devel/2010/02/index.php 2. I'm still looking into the perl syntax error that caused my Big Submit to GDS to fail. But looking at the Google logs, it looks like at least *some* of my test run results made it up to GDS. There was a BIG spike in CPU usage (3.2 hours of CPU time!) when it submitted -- see the attached CPU usage graph from the apps dashboard. Does anyone know why it takes so much CPU just to submit data to GDS? 3.2 CPU hours is a LOT! It makes me a bit concerned that only part of a single Cisco MTT run submit checked through almost half of our daily CPU quota (6.5 CPU hours/day). Is there any way to reduce the amount of CPU necessary just to submit data? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[MTT devel] GDS errors
Looking in the appspot dashboard, I see a bunch of errors when Cisco tried to submit test run data. There's a few random errors, but a bunch that look like what I pasted below. How do I diagnose this further? Clearly, some field is too long -- how do I find out which one? - • 128.107.241.170 - - [11/Feb/2010:00:51:21 -0800] "POST /client HTTP/1.1" 500 1972 - "MPI Test MTTGDS Reporter,gzip(gfe)" "open-mpi-mtt.appspot.com" • E02-11 12:51AM 21.241 Property data_message_size is 667 bytes long; it must be 500 or less. Consider Text instead, which can store strings of any length. Traceback (most recent call last): File "/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 509, in __call__ handler.post(*groups) File "/base/data/home/apps/open-mpi-mtt/1.337140739868725607/main.py", line 961, in post status = self._submit(); File "/base/data/home/apps/open-mpi-mtt/1.337140739868725607/main.py", line 485, in _submit test_run_phase.put() File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 801, in put self._populate_internal_entity() File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 779, in _populate_internal_entity self._entity = self._populate_entity(_entity_class=_entity_class) File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 839, in _populate_entity self._to_entity(entity) File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1465, in _to_entity entity[key] = value File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 492, in __setitem__ datastore_types.ValidateProperty(name, value) File "/base/python_lib/versions/1/google/appengine/api/datastore_types.py", line 1290, in ValidateProperty prop_validator(name, v) File "/base/python_lib/versions/1/google/appengine/api/datastore_types.py", line 1181, in ValidatePropertyString ValidateStringLength(name, value, max_len=_MAX_STRING_LENGTH) File "/base/python_lib/versions/1/google/appengine/api/datastore_types.py", line 1171, in ValidateStringLength (name, len(value), max_len)) BadValueError: Property data_message_size is 667 bytes long; it must be 500 or less. Consider Text instead, which can store strings of any length. - -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [MTT devel] 500 Internal Server Error from open-mpi-mtt.appspot.com
After looking through the logs, Ethan and I *think* that this was just a query that was too large (i.e., it used too much CPU, and therefore it was killed). Can someone with a little more knowledge than us have a look at the logs and let us know if we're right? On Feb 11, 2010, at 2:05 PM, Ethan Mallove wrote: > Hi, > > I'm getting a 500 Internal Server Error using bquery.pl. I can --ping > successfully: > > $ client/bquery.pl --ping --server=http://open-mpi-mtt.appspot.com/ > --password=x --username=sun > Ping is successful. > > But an actual query gets an error: > > $ client/bquery.pl --server=http://open-mpi-mtt.appspot.com/ > --password=x --username=sun --query --gqls="select * from TestRunPhase > where status=1" --dir="bquery-test" > Error at http://open-mpi-mtt.appspot.com//client > 500 Internal Server Error > > -Ethan > ___ > mtt-devel mailing list > mtt-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel > -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[MTT devel] 500 Internal Server Error from open-mpi-mtt.appspot.com
Hi, I'm getting a 500 Internal Server Error using bquery.pl. I can --ping successfully: $ client/bquery.pl --ping --server=http://open-mpi-mtt.appspot.com/ --password=x --username=sun Ping is successful. But an actual query gets an error: $ client/bquery.pl --server=http://open-mpi-mtt.appspot.com/ --password=x --username=sun --query --gqls="select * from TestRunPhase where status=1" --dir="bquery-test" Error at http://open-mpi-mtt.appspot.com//client 500 Internal Server Error -Ethan
[MTT devel] MTTGDS issues
1. Can you guys describe what MTTGDS expects from the performance analyzer modules? I ran a bunch of netpipe results and MTTGDS performance analyzer failed to run -- did you guys change the specifications for the performance analyzer modules? *** WARNING: Could not run module MTT::Test::Analyze::Performance::NetPipe:PreReport: Undefined subroutine &MTT::Test::Analyze::Performance::NetPipe::PreReport called at (eval 335838) line 1. 2. I ran 24+ hours of MTT tests and the MTTGDS reporter failed to submit the results. :-( *** ERROR: Module aborted: MTT::Reporter::MTTGDS:Finalize: Nested quantifiers in regex; marked by <-- HERE in m/\s[\S/\\]*mpi2c++ <-- HERE _test.*/ at /home/jsquyres/svn/mtt/lib/MTT/Reporter/MTTGDS.pm line 498. Some of my INI section names have special characters in them (e.g., "mpi2c++"); it looks like this might be what tripped up some regexp. I'll have a look at this one now... Is there a way to re-submit my data to GDS? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
