Have you tried disabling the logging? Here's a good tip
http://stackoverflow.com/questions/1661275/disable-boto-logging-without-modifying-the-boto-files
On 09/12/2014 8:29 am, "Billy Reynolds" <[email protected]>
wrote:
> I looked at DDT, and it appears the that the queries aren't the issues.
> There only seems to be 10 db queries on each page in django-filebrowser.
>
> However, the DDT log files are huge. On a single page that where
> django-filebrowser is loading 10~ images and the directories where the
> audio is stored, the log file is 2000 lines. When I load the large
> directory of audio files on my local dev server, the log file is 290,000+
> lines and it slows my virtual machine to a crawl.
>
> Boto appears to be logging the following for each file in filebrowser:
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> path=/media/uploads/img_0057_3_-_reduced.jpg
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> auth_path=/bucket/media/uploads/img_0057_3_-_reduced.jpg
> path/to/boto/s3/connection.py:653\
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Method: HEAD
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Path: /media/uploads/img_0057_3_-_reduced.jpg
> path/to/boto/connection.py:898\
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Data: path/to/boto/connection.py:899\
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Headers: \{\}
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Host: bucket.s3.amazonaws.com
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Port: 443 path/to/boto/connection.py:902\
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Params: \{\} path/to/boto/connection.py:903\
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Token: None
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> StringToSign:\
>
> HEAD
>
> HEAD\
>
> \
>
> \
>
> Mon, 08 Dec 2014 20:54:11 GMT
>
> Mon
>
> 08 Dec 2014 20:54:11 GMT\
>
> /bucket/media/uploads/img_0057_3_-_reduced.jpg
>
> /bucket/media/uploads/img_0057_3_-_reduced.jpg
>
> path/to/boto/auth.py:144\
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Signature:\
>
> AWS key:key= path/to/boto/auth.py:148\
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Final headers: \{'Date': 'Mon
>
> 08 Dec 2014 20:54:11 GMT'
>
> DEBUG 03:54:11 12/08/2014
>
> boto
>
> Response headers: [('content-length'
>
> '132168')
>
> So I guess this is how my log files are so long in a directory with 3000
> audio files. Is this a standard for boto?
>
> At this point I'm at a loss as to how to solve this issue, and I feel like
> my only option moving forward is to bypass django-filebrowser for my audio
> files (which currently uses the mezzanine.FileField) and just using a
> standard django.FileField. I didn't want to do this as I like having the
> ability to change the audio file for each product from the admin without
> uploading a new one to the bucket anytime I want to change that particular
> product.
>
>
>
> On Wednesday, November 26, 2014 10:20:09 PM UTC-5, Mario Gudelj wrote:
>>
>> Is it possible that the app is making loads of DB queries? Have you had a
>> look at django debug toolbar? I have a Mezza site with a large number of
>> products and as this number got larger the admin slowed down drastically. I
>> looked as DDT and it was making 400 DB queries because of some django sites
>> query. So I disabled that and everything sped up. Perhaps something similar
>> is happening here.
>>
>> Cheers,
>>
>> Mario
>>
>>
>> P: +61 2 9281 3315
>> M: +61 415 193775
>> E: [email protected]
>> W: www.twoblokeswithapostie.com
>>
>>
>>
>>
>> On Thu, Nov 27, 2014 at 9:51 AM, Billy Reynolds <william.timo...@
>> gmail.com> wrote:
>>
>>> I'm having some issues with Django-Filebrowser and from what I can tell
>>> they seem to be related to Nginx. The two primary issues I'm having is that
>>> Django-Filebrowser fails to load directories with large amounts of Amazon
>>> S3 files in the mezzanine admin and I get an http error when trying to
>>> upload large files (500Mb) to Amazon S3 through filebrowser. I have
>>> multiple directories with 400+ large audio files (several hundred MB each)
>>> hosted on S3 that when I attempt to load in mezzanine admin/media-library,
>>> my server returns an nginx 500 (bad gateway) error. I didn't have any
>>> issues with this until the directories started getting bigger. I can also
>>> upload normal sized files (images, small audio files, etc.) without any
>>> issue. It's not until I try to upload large files that I get an error.
>>>
>>> It's probably worth noting a few things:
>>>
>>> 1. I only use Amazon S3 to serve the media files for the project,
>>> all static files are served locally through nginx.
>>> 2. All django-filebrowser functionality works correctly in
>>> directories that will actually load. (with the exception of large file
>>> uploads)
>>> 3. I created a test directory with 1000 small files and
>>> django-filebrowser loads the directory correctly.
>>> 4. In the nginx.conf settings listed below (proxy buffer size,
>>> proxy_connect_timeout, etc), I've tested multiple values, multiple times
>>> and I can never get the pages to consistently load. Now that the
>>> directories are larger, I'm can't even get them to load.
>>> 5. I've tried adding an additional location in my nginx conf for
>>> "admin/media-library/" with increased timeouts, and other settings I've
>>> tried... but nginx still did not load these large directories correctly.
>>>
>>> I believe my primary issues are an nginx or possible gunicorn issue as I
>>> have no trouble loading these directories or uploading large files in a
>>> local environment without nginx/gunicorn. My nginx error log throws the
>>> following error:
>>>
>>>> 2014/11/24 15:53:25 [error] 30816#0: *1 upstream prematurely closed
>>>> connection while reading response header from upstream, client:
>>>> xx.xxx.xxx.xxx, server: server, request: "GET /admin/media-library/browse/
>>>> HTTP/1.1", upstream: "http://127.0.0.1:8001/admin/media-library/browse/";,
>>>> host: "server name, referrer: "https://example/admin/";
>>>
>>>
>>>
>>> I've researched that error which led me to add these lines to my nginx
>>> conf file.
>>>
>>>> proxy_buffer_size 128k;
>>>> proxy_buffers 100 128k;
>>>> proxy_busy_buffers_size 256k;
>>>> proxy_connect_timeout 75s;
>>>> proxy_read_timeout 75s;
>>>> client_max_body_size 9999M;
>>>> keepalive_timeout 60s;
>>>
>>>
>>>
>>> Despite trying multiple nginx timeout configurations, I'm still stuck
>>> exactly where I started. My production server will not load large
>>> directories from Amazon S3 through django-filebrowser nor can I upload
>>> large files through django-filebrowser.
>>>
>>> Here are some other lines from settings/conf files that are relevant.
>>>
>>> Settings.py
>>>
>>>> DEFAULT_FILE_STORAGE = 's3utils.S3MediaStorage'
>>>> AWS_S3_SECURE_URLS = True # use http instead of https
>>>> AWS_QUERYSTRING_AUTH = False # don't add complex
>>>> authentication-related query parameters for requests
>>>> #AWS_PRELOAD_METADATA = True
>>>> AWS_S3_ACCESS_KEY_ID = 'key' # enter your access key id
>>>> AWS_S3_SECRET_ACCESS_KEY = 'secret key' # enter your secret access key
>>>> AWS_STORAGE_BUCKET_NAME = 'bucket'
>>>> AWS_S3_CUSTOM_DOMAIN = 's3.amazonaws.com/bucket'
>>>> S3_URL = 'https://s3.amazonaws.com/bucket/'
>>>> MEDIA_URL = S3_URL + 'media/'
>>>> MEDIA_ROOT = 'media/uploads/'
>>>> FILEBROWSER_DIRECTORY = 'uploads'
>>>
>>>
>>>
>>> /etc/nginx/sites-enabled/production.conf
>>>
>>> upstream name {
>>> server 127.0.0.1:8001;
>>> }
>>>
>>> server {
>>> listen 80;
>>> server_name www.example.com;
>>> rewrite ^(.*) http://example.com$1 permanent;
>>> }
>>>
>>> server {
>>>
>>> listen 80;
>>> listen 443 default ssl;
>>> server_name example.com;
>>> client_max_body_size 999M;
>>> keepalive_timeout 60;
>>>
>>> ssl on;
>>> ssl_certificate /etc/nginx/ssl/cert.crt;
>>> ssl_certificate_key /etc/nginx/ssl/key.key;
>>> ssl_session_cache shared:SSL:10m;
>>> ssl_session_timeout 10m;
>>> ssl_ciphers RC4:HIGH:!aNULL:!MD5;
>>> ssl_prefer_server_ciphers on;
>>>
>>> location / {
>>> proxy_redirect off;
>>> proxy_set_header Host $host;
>>> proxy_set_header X-Real-IP $remote_addr;
>>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>> proxy_set_header X-Forwarded-Protocol $scheme;
>>> proxy_pass http://example;
>>> add_header X-Frame-Options "SAMEORIGIN";
>>> proxy_buffer_size 128k;
>>> proxy_buffers 100 128k;
>>> proxy_busy_buffers_size 256k;
>>> proxy_connect_timeout 75s;
>>> proxy_read_timeout 75s;
>>> client_max_body_size 9999M;
>>> keepalive_timeout 60s;
>>> }
>>>
>>> location /static/ {
>>> root /path/to/static
>>> }
>>>
>>> location /robots.txt {
>>> root /path/to/robots;
>>> access_log off;
>>> log_not_found off;
>>> }
>>>
>>> location /favicon.ico {
>>> root /path/to/favicon;
>>> access_log off;
>>> log_not_found off;
>>> }
>>>
>>> }
>>>
>>> Is this even an nginx issue? If so, does anyone have any suggestions for
>>> resolving this error? If not, what am I missing that would cause timeouts
>>> only on these large directories/large file uploads?
>>>
>>> Is there a better way to approach this problem than my current setup?
>>>
>>> Any help would be greatly appreciated.
>>>
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Mezzanine Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Mezzanine Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
