Re: [mezzanine-users] Re: s3 and Bad Request 400 + thumbnails with s3
Ok cool, we might leave it for now since the new release is almost here - but thanks so much for documenting all your findings, no doubt it'll be useful. On Wed, Jul 1, 2015 at 1:01 PM, Lee H. wrote: > Regarding a fix. I guess there a few points: > > 1) This was all in Django 1.6 with the latest stable release of Mezzanine > (not the master dev branch). In Django 1.6, the suspicious operation gets > swallowed, but in later version of Django >1.6, a technical 500 response is > returned if the user has DEBUG True. The `django.core.handlers.base` code > was changed to be > > if settings.DEBUG: > return debug.technical_500_response(request, *sys.exc_info(), > status_code=400) > > so the user gets a debug screen with > > Exception Type: SuspiciousOperation > Exception Value: > > Attempted access to '/path/to/my/project/temp_media/uploads/' denied. > > > and full trace, which makes identifying the issue A LOT easier (I verified > this on the latest Mezzanine master branch and django 1.8). > > Hence there is an argument that nothing should be done, and that in future > version Mezzanine that bind to django >1.6 the user will get this nicer > debug screen anyway. > > 2) Whilst if I'd been using django >1.6 I wouldn't have had the headache. > I'd personally say there is still room to make this more transparent for an > end user who wants to use Mezzanine and s3. A user could quite easily > forget to set `MEDIA_ROOT= ''` perhaps, and then be greeted with the above > SuspiciousOperation and trace, which requires digging through a little bit > to realise it's down to the MEDIA_ROOT. Could there perhaps be a > misconfiguration warning coming in at the filebrowser-safe > S3BotoStorageMixin level if MEDIA_ROOT is not ' ' . Or the test using > self.location of S3BotoStorage and MEDIA_ROOT could be performed in the > mixin. Typically say, self.location = "media" for a custom S3Storage media > class, and MEDIA_ROOT might be naively set by the user as > '/path/to/project/temp_media`, so comparing these two could also raise an > error which asks the user to check the value of their MEDIA_ROOT setting if > it looks like the filepath (generated using MEDIA_ROOT) is going to be > outside of the location (in a very similar way to the check safe_join is > currently doing, but with an exception that is more specific to this > scenario, and more transparent: we can say it's not SuspciousOperation; > it's just the user is mixing a cloud location with local filesystem > MEDIA_ROOT). > > -- > You received this message because you are subscribed to the Google Groups > "Mezzanine Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to mezzanine-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Stephen McDonald http://jupo.org -- You received this message because you are subscribed to the Google Groups "Mezzanine Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [mezzanine-users] Re: s3 and Bad Request 400 + thumbnails with s3
Regarding a fix. I guess there a few points: 1) This was all in Django 1.6 with the latest stable release of Mezzanine (not the master dev branch). In Django 1.6, the suspicious operation gets swallowed, but in later version of Django >1.6, a technical 500 response is returned if the user has DEBUG True. The `django.core.handlers.base` code was changed to be if settings.DEBUG: return debug.technical_500_response(request, *sys.exc_info(), status_code=400) so the user gets a debug screen with Exception Type: SuspiciousOperation Exception Value: Attempted access to '/path/to/my/project/temp_media/uploads/' denied. and full trace, which makes identifying the issue A LOT easier (I verified this on the latest Mezzanine master branch and django 1.8). Hence there is an argument that nothing should be done, and that in future version Mezzanine that bind to django >1.6 the user will get this nicer debug screen anyway. 2) Whilst if I'd been using django >1.6 I wouldn't have had the headache. I'd personally say there is still room to make this more transparent for an end user who wants to use Mezzanine and s3. A user could quite easily forget to set `MEDIA_ROOT= ''` perhaps, and then be greeted with the above SuspiciousOperation and trace, which requires digging through a little bit to realise it's down to the MEDIA_ROOT. Could there perhaps be a misconfiguration warning coming in at the filebrowser-safe S3BotoStorageMixin level if MEDIA_ROOT is not ' ' . Or the test using self.location of S3BotoStorage and MEDIA_ROOT could be performed in the mixin. Typically say, self.location = "media" for a custom S3Storage media class, and MEDIA_ROOT might be naively set by the user as '/path/to/project/temp_media`, so comparing these two could also raise an error which asks the user to check the value of their MEDIA_ROOT setting if it looks like the filepath (generated using MEDIA_ROOT) is going to be outside of the location (in a very similar way to the check safe_join is currently doing, but with an exception that is more specific to this scenario, and more transparent: we can say it's not SuspciousOperation; it's just the user is mixing a cloud location with local filesystem MEDIA_ROOT). -- You received this message because you are subscribed to the Google Groups "Mezzanine Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [mezzanine-users] Re: s3 and Bad Request 400 + thumbnails with s3
Any thoughts on a fix?
On Wed, Jul 1, 2015 at 3:45 AM, Lee H. wrote:
> After killing myself to try and work out what throws this bad request
> (400) error when filebrowser-safe is installed and when MEDIA_ROOT is set
> to non ''. I finally (through blood sweat and tears), worked out that it
> comes from the "safe_join" function of the django-storages s3boto.py.
> Because I have a custom storage class with 'location="media"' for media
> files, and another with "location = "static"' for static files (to ensure
> static and media are in sep dirs in the bucket), the safe_join function
> uses "media" as the base dir. It finds that the path to the uploads dir is
> wherever I set it in 'MEDIA_ROOT' and it doesn't like it (it's trying to
> prevent users from uploading outside the MEDIA_ROOT of course). The
> exception is when MEDIA_ROOT = '' or '/'.
>
> if (not final_path.startswith(base_path) or
> final_path[base_path_len:base_path_len + 1] not in ('', '/')):
>
> raise ValueError('the joined path is located outside of the base
> path'
> ' component')
>
> This ValueError is caught by
>
> _normalize_name and traded for a "Supicious Operation".
> Finally this gets caught by
> `django.core.handlers.base.BaseHandler.get_response`, which if you have
> logging turned on logs the suspicious operation
> "As attempt to access dir denied" (I didnt). It then just sends this to a
> "Bad request (400)" so the user is left blind. It's almost the same
> as this https://code.djangoproject.com/ticket/21668#no2.
>
> It is filebrowser-safe's get_directory, that calls isdir (of the
> S3BotoStorageMixin) that calls isfile, that calls
> _normalize_name of s3Botostorage, that finally calls safe_join. Hence why I
> only saw the problem when filebrowser-safe was installed.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Mezzanine Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mezzanine-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
Stephen McDonald
http://jupo.org
--
You received this message because you are subscribed to the Google Groups
"Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mezzanine-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[mezzanine-users] Re: s3 and Bad Request 400 + thumbnails with s3
After killing myself to try and work out what throws this bad request (400)
error when filebrowser-safe is installed and when MEDIA_ROOT is set to non
''. I finally (through blood sweat and tears), worked out that it comes
from the "safe_join" function of the django-storages s3boto.py. Because I
have a custom storage class with 'location="media"' for media files, and
another with "location = "static"' for static files (to ensure static and
media are in sep dirs in the bucket), the safe_join function uses "media"
as the base dir. It finds that the path to the uploads dir is wherever I
set it in 'MEDIA_ROOT' and it doesn't like it (it's trying to prevent users
from uploading outside the MEDIA_ROOT of course). The exception is when
MEDIA_ROOT = '' or '/'.
if (not final_path.startswith(base_path) or
final_path[base_path_len:base_path_len + 1] not in ('', '/')):
raise ValueError('the joined path is located outside of the base
path'
' component')
This ValueError is caught by
_normalize_name and traded for a "Supicious Operation".
Finally this gets caught by
`django.core.handlers.base.BaseHandler.get_response`, which if you have logging
turned on logs the suspicious operation
"As attempt to access dir denied" (I didnt). It then just sends this to a "Bad
request (400)" so the user is left blind. It's almost the same
as this https://code.djangoproject.com/ticket/21668#no2.
It is filebrowser-safe's get_directory, that calls isdir (of the
S3BotoStorageMixin) that calls isfile, that calls
_normalize_name of s3Botostorage, that finally calls safe_join. Hence why I
only saw the problem when filebrowser-safe was installed.
--
You received this message because you are subscribed to the Google Groups
"Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mezzanine-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[mezzanine-users] Re: s3 and Bad Request 400 + thumbnails with s3
Just another note: if filebrowser-safe is uninstalled this bad request 400 does not occur, regardless if MEDIA_ROOT is set to some local fs, or empty, so I guess it must be a problem with filebrowser when MEDIA_ROOT is something other than ' ', and s3 storages being used. -- You received this message because you are subscribed to the Google Groups "Mezzanine Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[mezzanine-users] Re: s3 and Bad Request 400 + thumbnails with s3
There is one other simpler way to get this working MEDIA_ROOT = ''# stops django throwing bad request 400 in admin when looking at blogpost detail then thumbnails will attempt to temporarily write to the local filesystem in the dir 'uploads' relative the point of execuation, so if you get permissions errors, you need to find what this dir is. I added a `print os.getcwd()` and found out that the dir was `home/djangoUser`. I added the allowed permissions on this dir so uploads can be created and written too (my apache is running in daemon mode as 'djangoUser'). I guess normally it might be www-data, and /var/www or something. So this kinda of fixes this issue for me. I'd still really like to know why the bad request is thrown in admin when MEDIA_ROOT is anything other than '' or undefined however when using s3. -- You received this message because you are subscribed to the Google Groups "Mezzanine Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[mezzanine-users] Re: s3 and Bad Request 400 + thumbnails with s3
> > ### > ### STORAGE ## > ### > > # The custom storage just allows definition of different 'locations' so media goes in media/ static goes in /static # within the bucket. Otherwise same as S3 boto storage. > > # Bucket and creds > AWS_STORAGE_BUCKET_NAME = '..' > AWS_ACCESS_KEY_ID = '' > AWS_SECRET_ACCESS_KEY = '...' > > > AWS_PRELOAD_METADATA = True > AWS_QUERYSTRING_AUTH = False > AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME > > # Static storage > STATICFILES_LOCATION = 'static' > STATICFILES_STORAGE = 'custom_storages.StaticStorage' > STATIC_ROOT = os.path.join(PROJECT_ROOT, 'temp_static/') > STATIC_URL = "https://%s/%s/"; % (AWS_S3_CUSTOM_DOMAIN, > STATICFILES_LOCATION) > ADMIN_MEDIA_PREFIX = STATIC_URL + 'grappelli/' > > > # Media storage > MEDIAFILES_LOCATION = 'media' > MEDIA_URL = "https://%s/%s/"; % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION) > #MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'temp_media') # bad request 400 > in admin unless commented out > DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage' > > > -- You received this message because you are subscribed to the Google Groups "Mezzanine Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
