Thanks for clarifying. So we did find a solution - indexing the database
has speeded up the response by 5x.
1. using `format=json` to restrict the API from rendering did not change
performance, nor using header. in fact both methods had inconsistent
effects. Perhaps requests is already adding these parameters/headers? or if
requests addess accept:application/json, then the api is not rendered?
In [28]: %timeit requests.get(url=URL, auth=AUTH,
params=dict(PARAMS,format='json'), headers={'Accept': 'application/json'})
1 loop, best of 3: 611 ms per loop
In [29]: %timeit requests.get(url=URL, auth=AUTH,
params=dict(PARAMS,format='json'), headers={'Accept': 'application/json'})
1 loop, best of 3: 621 ms per loop
In [30]: %timeit requests.get(url=URL, auth=AUTH,
params=dict(PARAMS,format='json'), headers={'Accept': 'application/json'})
1 loop, best of 3: 595 ms per loop
In [31]: %timeit requests.get(url=URL, auth=AUTH,
params=dict(PARAMS,format='json'), headers={'Accept': 'application/json'})
1 loop, best of 3: 620 ms per loop
In [32]: %timeit requests.get(url=URL, auth=AUTH, params=PARAMS)
1 loop, best of 3: 608 ms per loop
In [33]: %timeit requests.get(url=URL, auth=AUTH, params=PARAMS)
1 loop, best of 3: 577 ms per loop
In [34]: %timeit requests.get(url=URL, auth=AUTH, params=PARAMS)
1 loop, best of 3: 569 ms per loop
In [35]: %timeit requests.get(url=URL, auth=AUTH, params=PARAMS,
headers={'Accept': 'application/json'})
1 loop, best of 3: 616 ms per loop
In [36]: %timeit requests.get(url=URL, auth=AUTH, params=PARAMS,
headers={'Accept': 'application/json'})
1 loop, best of 3: 615 ms per loop
In [37]: %timeit requests.get(url=URL, auth=AUTH, params=PARAMS,
headers={'Accept': 'application/json'})
1 loop, best of 3: 583 ms per loop
In [38]: %timeit requests.get(url=URL, auth=AUTH,
params=dict(PARAMS,format='json'))
1 loop, best of 3: 623 ms per loop
In [39]: %timeit requests.get(url=URL, auth=AUTH,
params=dict(PARAMS,format='json'))
1 loop, best of 3: 603 ms per loop
In [40]: %timeit requests.get(url=URL, auth=AUTH,
params=dict(PARAMS,format='json'))
1 loop, best of 3: 547 ms per loop
2. to index the database, we set db_index=True for each field in the
Weather model
https://docs.djangoproject.com/en/1.10/topics/db/optimization/
3. disabled pagination with custom viewset instead of ModelViewset
mixins.CreateModelMixin,
mixins.RetrieveModelMixin,
mixins.UpdateModelMixin,
mixins.DestroyModelMixin,
viewsets.GenericViewSet
now as you can see in #1 the response time is 0.6 seconds instead of 3 - 4
seconds.
thanks!
On Wednesday, January 11, 2017 at 11:27:03 AM UTC-8, Xavier Ordoquy wrote:
>
>
> Thanks for your answer. So the slow performance issue occurs whether the
> request is performed using JSON or from the browseable API.
>
>
> No, only this the browsable API.
>
>
> >>> import requests
> >>> PARAMS = {'latitude': 38.2, 'longitude': -122.1}
> >>> requests.get(url=URL, auth=AUTH, params=dict(PARAMS, limit=1000))
> # 1 loop, best of 3: 4.21 s per loop - using %timeit
>
>
> This isn’t a JSON request so it’ll trigger the browsable API.
>
> We do have django-filters with cripy-forms in installed apps, and it is
> listed as the default backend in the viewset:
>
> filter_backends = (filters.DjangoFilterBackend,)
>
> but would that be the cause of the issue? we will remove it, but it seems
> unlikely.
>
>
> Your mileage may vary but it is likely.
>
>
> Any ideas where we should debug or profile?
>
>
> Activate the logging for Django’s DB queries.
>
>
> when I mock the request and render the JSON manually, I get a response in
> 677[ms], this includes getting the queryset 757[microseconds] and
> serialization 57[microseconds], so this is what we expect. Also our
> previous API which is ASP.NET + MSSQL is about 600[ms] roundtrip.
>
> Thanks again!
>
> On Wednesday, January 11, 2017 at 1:00:56 AM UTC-8, Xavier Ordoquy wrote:
>>
>> Hi,
>>
>> It’s likely either the displayed filter or the create/update form from
>> the browsable API that spans some relation and gets the UI slow.
>> Try to perform the same request as JSON that should validate the previous
>> assomption.
>>
>> Regards,
>> Xavier Ordoquy,
>> Linovia.
>>
>>
>> We are experiencing very slow performance from one view in the API that
>> returns a queryset of a filter. We expected 0.5 - 1.0 sec, but the request
>> is taking up to 4 seconds. Turning off pagination decreases the wait to 2
>> sec.
>>
>> From the Django shell, the query is extremely fast, < 0.001 sec.
>>
>> >>> from django.db.models import F
>> >>> from app.models import Weather
>> >>> import numpy as np
>> >>> R_EARTH = 6371000.0 # [m] earth's radius
>> >>> search_radius = 160934
>> >>> latitude, longitude = 38.1, -122.1
>> >>> dtheta = np.rad2deg(search_radius / R_EARTH) # [degrees]
>> >>> queryset = Weather.objects.filter(
>> ... latitude__range=(latitude - dtheta, latitude + dtheta),
>> ... longitude__range=(longitude - dtheta, longitude + dtheta),
>> ... data_type__in=Weather.ALL_DATA_TYPES
>> ... ).annotate(distance_d=(
>> ... (F('latitude') - latitude) ** 2 + (F('longitude') - longitude) **
>> 2)
>> ... ).order_by('distance_d').exclude(distance_d__gt=(dtheta ** 2))
>> # 1000 loops, best of 3: 737 µs per loop - using %timeit
>>
>> I tested serializing the queryset and rendering the response, and It is
>> still very fast, < 1 sec
>>
>> # I know there is an easier/better way to do this, sorry :(
>> >>> from django.http import HttpRequest
>> >>> from rest_framework.request import Request
>> >>> from app.serializers import WeatherSearchSerializer
>> >>> request = Request(HttpRequest())
>> >>> request.META['SERVER_NAME'] = u'localhost'
>> >>> request.META['SERVER_PORT'] = 8000
>> >>> w = WeatherSearchSerializer(queryset, context={'request': request},
>> many=True)
>> # 10000 loops, best of 3: 56.5 µs per loop - using %timeit
>> >>> jsondata = JSONRenderer().render(w.data)
>> # 1 loops, best of 3: 677 ms per loop - using %timeit
>>
>> But if I use the browseable API or send a request, it takes 3-4 seconds.
>>
>> >>> import requests
>> >>> PARAMS = {'latitude': 38.2, 'longitude': -122.1}
>> >>> requests.get(url=URL, auth=AUTH, params=dict(PARAMS, limit=1000))
>> # 1 loop, best of 3: 4.21 s per loop - using %timeit
>>
>> requirements:
>>
>> Django>=1.8.5
>> djangorestframework>=3.3.3
>> psycopg2>=2.6.2
>> django-storages>=1.4
>> boto>=2.38.0
>> django-crispy-forms>=1.6.0
>> django-filter>=0.14.0
>>
>>
>> platform: AWS
>> EC2: c3.4xlarge instance based on Oracle Linux 7.2
>> RDS: Postgres on a db.m3.medium instance with 1TB of storage
>> S3
>> software: Apache httpd with mod_wsgi and python-2.7
>>
>> models and serializers: totally vanilla, nothing custom
>> * serializer is `serializers.HyperlinkedModelSerializer`
>>
>> views: totaly vanilla, nothing custom
>> * view uses `viewsets.ModelViewSet`
>>
>> Any ideas where the extra seconds are coming from? I'm not sure where to
>> profile. Any assistance would be greatly appreciated.
>>
>> Other relevant posts:
>> *
>> https://groups.google.com/d/msg/django-rest-framework/AkLKdfxCrcU/qyOB1k37pJAJ
>> * that's all I could find search terms: slow queryset
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Django REST framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
