#986: Remove Nginx?
--------------------------+-------------------------------------------------
Reporter: anna | Owner:
Type: task | Status: new
Priority: major | Milestone: 4.6
Component: Architecture | Severity: New Ticket
Keywords: | Who will test this: And
--------------------------+-------------------------------------------------
Sam has suggested some ways to think of removing Nginx from the Plumi
stack, and rely upon Apache alone.
I've added all his notes here.
In general EM might want to consider simplifying their server
architecture. Plone with just Apache server in front of it, and a disk
cache configured in Apache, can serve busy sites very nicely ... this
one for example that we manage: http://www.ethicaljobs.com.au/ (see
somewhat dated visitor stats here:
http://www.ethicaljobs.com.au/news/ethicaljobs.com.au-
celebrates-115-000-visitors).
An additional advantage is that the buildout is simplified quite a bit
too - no nginx, varnish, ...
On 11/12/14 11:58 PM, Markos Gogoulos wrote:
> We had a really bad time with timeouts so after research we ended up in
nginx + nginx upload module (before that we were on Apache).
> Even if 2-3G file uploads are supported by apache/nginx (which I doubt),
the combination with uwsgi makes things much complex.
>
> In fact the nginx upload module uploads files in a /tmp dir, and when
the whole file is uploaded it handles it to zope, that moves it in
blobstorage etc.
>Nginx is used so that large files can be uploaded (we have tested with up
to 3G files), we are using an nginx's plugin for this. Without this
plugin, apache/nginx timeouts after some minutes of video uploading.
>
> On Thu, Dec 11, 2014 at 1:57 PM, Sam Stainsby
<[email protected]> wrote:
>
> On 11/12/14 19:41, Markos Gogoulos wrote:
>
> > Nginx is used so that large files can be uploaded (we have tested
with
> > up to 3G files), we are using an nginx's plugin for this. Without
this
> > plugin, apache/nginx timeouts after some minutes of video
uploading.
>
> Any recent mainstream web server (Apache, Nginx, ..) should handle
large
> (or long) file uploads just fine. Are you sure it's not a proxy
timeout
> you're seeing?
>
> Cheers,
> Sam.
>
I can think of at least three multimedia projects I've build that can do
this with plain Apache, so I suspect your timeout problems lies with
what runs behind your web server rather than the web server itself ...
or with bad proxy settings ... or just a bad network, in which case a
chunked, resumeable uploader Javascript library could be the solution.
In fact, the last system I built with file uploads wasn't designed with
large files in mind, but out of curiosity I'm testing it right now with
a 2.3GB upload. It's been running fine for 1.5 hours so far.
Unfortunately, at my ADSL1 network speed (current upload progress is at
7%!), it will probably take all day ... possibly more. As it's taking up
too much upstream bandwidth, I may have to stop it soon.
If interested, below are some technical details ...
The Apache virtual host is very plain and boring:
<VirtualHost 106.187.99.62:80>
ServerName admin.realbreakthroughs.net
CustomLog ${APACHE_LOG_DIR}/admin.realbreakthroughs.log combined
ErrorLog ${APACHE_LOG_DIR}/admin.realbreakthroughs.error.log
DocumentRoot /var/www/realbreakthroughs/admin_ui
<Directory />
Options FollowSymLinks
AllowOverride All
</Directory>
<Location />
AuthType Basic
AuthName "RB restricted area"
AuthUserFile /etc/apache2/realbreakthroughs_password
Require user rbadmin
</Location>
RewriteEngine On
RewriteRule ^/api/(.*) http://localhost:3122/$1 [P,L,E=proxy-
timeout:200s]
</VirtualHost>
- the front end uses angular-file-upload.js and the back end behind
Apache is a RESTful node.js API server. It streams uploaded files
directly into MongoDB GridFS using about 80 lines of code. Seriously,
this is the upload handing code in the node.js part:
exports.insert = function(db) {
var gridfs = GridFS(db, mongodb);
return function(req, res) {
var form = new multiparty.Form();
form.parse(req, function(err, fields, files) {
if (err) {
res.status(500);
return res.send(err);
}
try {
var metadata = JSON.parse(fields.metadata);
var fileKeys = Object.keys(files);
var nKeys = fileKeys.length;
if (nKeys < 1 || nKeys > 1) {
return res.send(422, 'wrong file count: ' + nKeys);
}
var file = files[fileKeys[0]][0];
var filename = file.originalFilename;
var ctype = file.headers && file.headers['content-type'] ||
'application/octet-stream';
var writeOption = {
filename: filename,
mode: 'w',
content_type: ctype,
metadata: metadata
};
var tmpPath = file.path;
var outstream = gridfs.createWriteStream(writeOption);
outstream.on('error', function(err) {
console.log('ERROR: FAILED storing file');
return res.send(500, err);
});
outstream.on('close', function(data) {
return res.send(201, data);
});
var instream = fs.createReadStream(tmpPath);
instream.pipe(outstream);
} catch(err) {
console.log('UPLOAD ERROR: ' + err);
res.status(500);
return res.send(err);
}
});
};
};
Cheers,
Sam.
--
Ticket URL: <http://plumi.org/ticket/986>
Plumi <http://plumi.org/>
Plumi - FOSS Video Sharing Platform
_______________________________________________
Discuss mailing list
[email protected]
http://lists.plumi.org/listinfo/discuss
