It's best to not put millions of keys in any javascript object. More than blow up the memory, it will blow up the GC. Hopefully this gets fixed in the newer V8 with node 0.8.x. In that case you need some external way to look up and store the hashes. But the general idea still works.
You could even use the disk itself as the database. Create a placeholder file using exclusive open when you register the route. Then on the http request, look for that file. If the resized image is not there create another lock file using exclusive open and start the resize, save the resized file when done. The logic is the same mostly, except all the lookups become async and can fail in races. The exclusive open flag will prevent this from being unsolvable, but you do need to check at every step. On Wed, May 16, 2012 at 8:36 AM, Alan Hoffmeister <[email protected] > wrote: > Thanks for the tip Anand. > > Tim, and if I have millions of images and more millions of dimensions, > your method wouldn't blow up the server's memory? > > -- > Att, > Alan Hoffmeister > > > 2012/5/16 Anand George <[email protected]>: > > Guess the issue was implementing a 'method on the view'. If async image > > processing and manipulation is of interest, > > https://github.com/hacksparrow/node-easyimage would help. > > > > > > On Wed, May 16, 2012 at 6:10 PM, yogesh agrawal <[email protected]> > > wrote: > >> > >> Not able to found any async image resize code in dropup > >> > >> > >> On Wed, May 16, 2012 at 6:03 PM, Anand George <[email protected]> > >> wrote: > >>> > >>> http://dropup.net/ seems to be a similar implementation. The code is > >>> available on Github https://github.com/daleharvey/dropup > >>> > >>> > >>> On Wed, May 16, 2012 at 5:57 PM, Tim Caswell <[email protected]> > wrote: > >>>> > >>>> > >>>> > >>>> On Wed, May 16, 2012 at 6:14 AM, Alan Hoffmeister > >>>> <[email protected]> wrote: > >>>>> > >>>>> Tim that's a very good idea! > >>>>> But how can I share the hash and the dimensions between the view > >>>>> helper and the resizing route? > >>>> > >>>> > >>>> Well that depends on the nature of your code. Is it a framework like > >>>> express, is it something else. If you're running the entire site in a > >>>> single process then a single variable in the outer closure would be > global > >>>> enough. Just like I wrote in the example, my lookup tables are > outside the > >>>> helper and the http route handler. As long as it's in a common place > they > >>>> can both see in their closures, they will have access. > >>>> > >>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Att, > >>>>> Alan Hoffmeister > >>>>> > >>>>> > >>>>> 2012/5/15 Tim Caswell <[email protected]>: > >>>>> > btw, this code isn't perfect. For example, there is no reason to > use > >>>>> > the > >>>>> > expensive md5 hash. It's not like the browser is sending you path, > >>>>> > width, > >>>>> > and height and you need to re-hash them. Any random unique string > >>>>> > would be > >>>>> > fine. Also the key variable would sufficient as well. Yes, the > url > >>>>> > contains the parameters, but they can't be modified to create a new > >>>>> > value. > >>>>> > Only existing keys would be served. > >>>>> > > >>>>> > The important part is to create some unique key to embed in the url > >>>>> > so that > >>>>> > later when the request comes in, you have a way to map it to the > >>>>> > original > >>>>> > resize command. The caching and batching pattern is described > >>>>> > here http://nodebits.org/distilled-patterns > >>>>> > > >>>>> > > >>>>> > On Tue, May 15, 2012 at 3:47 PM, Tim Caswell <[email protected]> > >>>>> > wrote: > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> On Tue, May 15, 2012 at 2:46 PM, Alan Hoffmeister > >>>>> >> <[email protected]> wrote: > >>>>> >>> > >>>>> >>> Hey guyz, of course I need to validate, I just don't want to > expose > >>>>> >>> the dimensions. > >>>>> >> > >>>>> >> > >>>>> >> The thing is you have to expose the dimensions if the request > coming > >>>>> >> from > >>>>> >> a browser includes the dimensions. > >>>>> >> > >>>>> >> If you want to emulate what php is doing, it's easy enough. > >>>>> >> Basically > >>>>> >> this would be a template helper that registers a new route when > >>>>> >> found. > >>>>> >> > >>>>> >> // load the createHash function from the crypto module > >>>>> >> var createHash = require('crypto').createHash; > >>>>> >> > >>>>> >> // Keep a history of registered routes > >>>>> >> var resizes = {}; > >>>>> >> // Store an index by hash for url routing > >>>>> >> var hashes = {}; > >>>>> >> // Register a new resize command, export this as a template > helper. > >>>>> >> function resize(path, width, height) { > >>>>> >> var key = path + width + "x" + height; // convert args to a > unique > >>>>> >> string > >>>>> >> if (resizes[key]) return resizes[key].url; // Check cache to see > >>>>> >> if it's > >>>>> >> already registered > >>>>> >> var hash = createHash("md5").update(key).digest("hex"); // > >>>>> >> calculate the > >>>>> >> md5 hash of the key to hide the params > >>>>> >> var entry = { > >>>>> >> url: "/images/" + hash + ".png", > >>>>> >> path: path, > >>>>> >> width: width, > >>>>> >> height: height > >>>>> >> }; > >>>>> >> // Store the object in the two tables. > >>>>> >> resizes[key] = entry; > >>>>> >> hashes[hash] = entry; > >>>>> >> // return the url. > >>>>> >> return entry.url; > >>>>> >> } > >>>>> >> > >>>>> >> > >>>>> >> Then later in your http request handler, add a route to handle > these > >>>>> >> requests. > >>>>> >> > >>>>> >> app.get("/images/:hash.png", function (req, res) { > >>>>> >> var entry = hashes[req.params.hash]; > >>>>> >> if (!entry) return doErrorHandling(); > >>>>> >> resizeImage(entry, function (err, resizedPath) { > >>>>> >> if (err) return doErrorHandling(); > >>>>> >> }); > >>>>> >> }); > >>>>> >> > >>>>> >> And here is some untested code that does caching and batching of > the > >>>>> >> async > >>>>> >> resize requests > >>>>> >> > >>>>> >> function resizeImage(entry, callback) { > >>>>> >> // Check if it's been resized already > >>>>> >> if (entry.resizedPath) { > >>>>> >> return callback(null, entry.resizedPath); > >>>>> >> } > >>>>> >> // Check if there is already a pending resize operation on this > >>>>> >> object > >>>>> >> if (entry.resizeQueue) { > >>>>> >> return entry.resizeQueue.push(callback); > >>>>> >> } > >>>>> >> entry.resizeQueue = [callback]; > >>>>> >> doRealResize(entry.path, entry.width, entry.height, function > (err, > >>>>> >> resizedPath) { > >>>>> >> if (err) return callback(err); > >>>>> >> var callbacks = entry.resizeQueue; > >>>>> >> delete entry.resizeQueue; > >>>>> >> entry.resizedPath = resizedPath; > >>>>> >> callbacks.forEach(function (callback) { > >>>>> >> callback(null, resizedPath); > >>>>> >> }); > >>>>> >> }); > >>>>> >> } > >>>>> >> > >>>>> >> function doRealResize(path, width, height, callback) { > >>>>> >> // Do actual resize and return the final filename's filepath > >>>>> >> } > >>>>> >> > >>>>> >>> -- > >>>>> >>> Att, > >>>>> >>> Alan Hoffmeister > >>>>> >>> > >>>>> >>> > >>>>> >>> 2012/5/15 Duncan Gmail <[email protected]>: > >>>>> >>> > You can POST the size as well as use GET. On the numbers, just > >>>>> >>> > validate the input - you should be doing this anyway for all > >>>>> >>> > inputted data, > >>>>> >>> > in all languages. > >>>>> >>> > > >>>>> >>> > - MRdNk > >>>>> >>> > > >>>>> >>> > On 15 May 2012, at 15:21, Alan Hoffmeister > >>>>> >>> > <[email protected]> > >>>>> >>> > wrote: > >>>>> >>> > > >>>>> >>> >> Marc, that would work, but I'm concerned about security... > What > >>>>> >>> >> if > >>>>> >>> >> someone access the url /images/100000/100000/avatar.jpg ? > >>>>> >>> >> > >>>>> >>> >> -- > >>>>> >>> >> Att, > >>>>> >>> >> Alan Hoffmeister > >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> 2012/5/15 Marc Deschamps <[email protected]>: > >>>>> >>> >>> I've done something like this using express: > >>>>> >>> >>> > >>>>> >>> >>> app.get('/images/:width/:height/:filename', > >>>>> >>> >>> routes.images.resize); > >>>>> >>> >>> > >>>>> >>> >>> Work great, in html i can do: > >>>>> >>> >>> > >>>>> >>> >>> <img src="/images/70/70/avatar.jpg"/> > >>>>> >>> >>> > >>>>> >>> >>> -- > >>>>> >>> >>> Job Board: http://jobs.nodejs.org/ > >>>>> >>> >>> Posting guidelines: > >>>>> >>> >>> > >>>>> >>> >>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >>>>> >>> >>> You received this message because you are subscribed to the > >>>>> >>> >>> Google > >>>>> >>> >>> Groups "nodejs" group. > >>>>> >>> >>> To post to this group, send email to [email protected] > >>>>> >>> >>> To unsubscribe from this group, send email to > >>>>> >>> >>> [email protected] > >>>>> >>> >>> For more options, visit this group at > >>>>> >>> >>> http://groups.google.com/group/nodejs?hl=en?hl=en > >>>>> >>> >> > >>>>> >>> >> -- > >>>>> >>> >> Job Board: http://jobs.nodejs.org/ > >>>>> >>> >> Posting guidelines: > >>>>> >>> >> > >>>>> >>> >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >>>>> >>> >> You received this message because you are subscribed to the > >>>>> >>> >> Google > >>>>> >>> >> Groups "nodejs" group. > >>>>> >>> >> To post to this group, send email to [email protected] > >>>>> >>> >> To unsubscribe from this group, send email to > >>>>> >>> >> [email protected] > >>>>> >>> >> For more options, visit this group at > >>>>> >>> >> http://groups.google.com/group/nodejs?hl=en?hl=en > >>>>> >>> > > >>>>> >>> > -- > >>>>> >>> > Job Board: http://jobs.nodejs.org/ > >>>>> >>> > Posting guidelines: > >>>>> >>> > > >>>>> >>> > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >>>>> >>> > You received this message because you are subscribed to the > >>>>> >>> > Google > >>>>> >>> > Groups "nodejs" group. > >>>>> >>> > To post to this group, send email to [email protected] > >>>>> >>> > To unsubscribe from this group, send email to > >>>>> >>> > [email protected] > >>>>> >>> > For more options, visit this group at > >>>>> >>> > http://groups.google.com/group/nodejs?hl=en?hl=en > >>>>> >>> > >>>>> >>> -- > >>>>> >>> Job Board: http://jobs.nodejs.org/ > >>>>> >>> Posting guidelines: > >>>>> >>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >>>>> >>> You received this message because you are subscribed to the > Google > >>>>> >>> Groups "nodejs" group. > >>>>> >>> To post to this group, send email to [email protected] > >>>>> >>> To unsubscribe from this group, send email to > >>>>> >>> [email protected] > >>>>> >>> For more options, visit this group at > >>>>> >>> http://groups.google.com/group/nodejs?hl=en?hl=en > >>>>> >> > >>>>> >> > >>>>> > > >>>>> > -- > >>>>> > Job Board: http://jobs.nodejs.org/ > >>>>> > Posting guidelines: > >>>>> > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >>>>> > You received this message because you are subscribed to the Google > >>>>> > Groups "nodejs" group. > >>>>> > To post to this group, send email to [email protected] > >>>>> > To unsubscribe from this group, send email to > >>>>> > [email protected] > >>>>> > For more options, visit this group at > >>>>> > http://groups.google.com/group/nodejs?hl=en?hl=en > >>>>> > >>>>> -- > >>>>> Job Board: http://jobs.nodejs.org/ > >>>>> Posting guidelines: > >>>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >>>>> You received this message because you are subscribed to the Google > >>>>> Groups "nodejs" group. > >>>>> To post to this group, send email to [email protected] > >>>>> To unsubscribe from this group, send email to > >>>>> [email protected] > >>>>> For more options, visit this group at > >>>>> http://groups.google.com/group/nodejs?hl=en?hl=en > >>>> > >>>> > >>>> -- > >>>> Job Board: http://jobs.nodejs.org/ > >>>> Posting guidelines: > >>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >>>> You received this message because you are subscribed to the Google > >>>> Groups "nodejs" group. > >>>> To post to this group, send email to [email protected] > >>>> To unsubscribe from this group, send email to > >>>> [email protected] > >>>> For more options, visit this group at > >>>> http://groups.google.com/group/nodejs?hl=en?hl=en > >>> > >>> > >>> -- > >>> Job Board: http://jobs.nodejs.org/ > >>> Posting guidelines: > >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >>> You received this message because you are subscribed to the Google > >>> Groups "nodejs" group. > >>> To post to this group, send email to [email protected] > >>> To unsubscribe from this group, send email to > >>> [email protected] > >>> For more options, visit this group at > >>> http://groups.google.com/group/nodejs?hl=en?hl=en > >> > >> > >> > >> > >> -- > >> Yogesh Agrawal > >> +91-9351507770 > >> > >> -- > >> Job Board: http://jobs.nodejs.org/ > >> Posting guidelines: > >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >> You received this message because you are subscribed to the Google > >> Groups "nodejs" group. > >> To post to this group, send email to [email protected] > >> To unsubscribe from this group, send email to > >> [email protected] > >> For more options, visit this group at > >> http://groups.google.com/group/nodejs?hl=en?hl=en > > > > > > -- > > Job Board: http://jobs.nodejs.org/ > > Posting guidelines: > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > You received this message because you are subscribed to the Google > > Groups "nodejs" group. > > To post to this group, send email to [email protected] > > To unsubscribe from this group, send email to > > [email protected] > > For more options, visit this group at > > http://groups.google.com/group/nodejs?hl=en?hl=en > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
