John A Meinel has proposed merging lp:~jameinel/maas/manually-serialize-api
into lp:maas.
Commit message:
Change nodegroups/UUID/?op=node_hardware_details=XXX to directly serialize the
response, rather than having Piston do it.
It turns out that Piston's default stream is using encode_ascii=False, which
has a 10x performance overhead. Presumably it is because it is passing bad
regex's over the data to check for characters it doesn't like. This one change
essentially halves the time to rebuild a tag (from ~27s for 10,000 nodes down
to 14s).
This now properly swings the balance so that the MAAS appserver consumes less
CPU in a given request than the celeryd workers processing the content. (With 2
tags being updated concurrently, I get 2 celeryd at 60% cpu, and one maas at
32%cpu).
The other tweak is to change the number of hardware details we request in one
batch. We can return 100 nodes in < 100ms. That means that the connection and
request overheads start to be a large portion of the conversation. At a batch
size of 1000, the celeryd workers go up to 70-80% and the maas server drops to
15-30%. Rebuilding a single tag is down to 12s. Going up to a batch of 5000
still takes 12s to build a tag, though the CPU consumption does hit 100% for
the celery worker. However, the extra memory consumed in the content starts
showing up (24MB at 1k nodes, 120MB at 5k nodes). 1k nodes seems to be the
sweet spot.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~jameinel/maas/manually-serialize-api/+merge/129147
This gets rid of another major CPU overhead as part of tag rebuilding.
--
https://code.launchpad.net/~jameinel/maas/manually-serialize-api/+merge/129147
Your team Launchpad code reviewers is requested to review the proposed merge of
lp:~jameinel/maas/manually-serialize-api into lp:maas.
=== modified file 'src/maasserver/api.py'
--- src/maasserver/api.py 2012-10-10 13:46:34 +0000
+++ src/maasserver/api.py 2012-10-11 10:42:22 +0000
@@ -87,7 +87,7 @@
from functools import partial
import httplib
from inspect import getdoc
-import json
+import simplejson as json
import sys
from textwrap import dedent
@@ -1209,10 +1209,11 @@
nodegroup = get_object_or_404(NodeGroup, uuid=uuid)
if not request.user.is_superuser:
check_nodegroup_access(request, nodegroup)
- nodes = Node.objects.filter(
- system_id__in=system_ids, nodegroup=nodegroup)
- details = [(node.system_id, node.hardware_details) for node in nodes]
- return details
+ value_list = Node.objects.filter(
+ system_id__in=system_ids, nodegroup=nodegroup
+ ).values_list('system_id', 'hardware_details')
+ return HttpResponse(
+ json.dumps(list(value_list)), content_type='application/json')
DISPLAYED_NODEGROUP_FIELDS = (
=== modified file 'src/provisioningserver/tags.py'
--- src/provisioningserver/tags.py 2012-10-10 15:58:45 +0000
+++ src/provisioningserver/tags.py 2012-10-11 10:42:22 +0000
@@ -41,7 +41,7 @@
"""The MAAS URL or credentials are not yet set."""
-DEFAULT_BATCH_SIZE = 100
+DEFAULT_BATCH_SIZE = 1000
def get_cached_knowledge():
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-reviewers
Post to : [email protected]
Unsubscribe : https://launchpad.net/~launchpad-reviewers
More help : https://help.launchpad.net/ListHelp
