Aryeh Gregor wrote: > I'm not clear on why we don't just make the daemon synchronously > return a result the way ImageMagick effectively does. Given the level > of reuse of thumbnails, it seems unlikely that the latency is a > significant concern -- virtually no requests will ever actually wait > on it. > ( I basically outlined these issues on the soc page but here they are again with at bit more clarity )
I recommended that the image daemon run semi-synchronously since the changes needed to maintain multiple states and return non-cached place-holder images while managing updates and page purges for when the updated images are available within the wikimedia server architecture probably won't be completed in the summer of code time-line. But if the student is up for it the concept would be useful for other components like video transformation / transcoding, sequence flattening etc. But its not what I would recommend for the summer of code time-line. == per issues outlined in bug 4854 == I don't think its a good idea to invest a lot of energy into a separate python based image daemon. It won't avoid all problems listed in bug 4854 Shell-character-exploit issues should be checked against anyway (since not everyone is going to install the daemon) Other people using mediaWiki won't add a python or java based image resize and resolve dependency python or java component & libraries. It won't be easier to install than imagemagick or "php-gd" that are repository hosted applications and already present in shared hosting environments. Once you start integrating other libs like (java) Batik it becomes difficult to resolve dependencies (java, python etc) and to install you have to push out a "new program" that is not integrated into all the application repository manages for the various distributions. Potential to isolate CPU and memory usage should be considered in the core medaiWiki image resize support anyway . ie we don't want to crash other peoples servers who are using mediaWiki by not checking upper bounds of image transforms. Instead we should make the core image transform smarter maybe have a configuration var that /attempts/ to bind the upper memory for spawned processing and take that into account before issuing the shell command for a given large image transformation with a given sell application. == what would probably be better for the image resize efforts should focus on === (1) making the existing system "more robust" and (2) better taking advantage of multi-threaded servers. (1) right now the system chokes on large images we should deploy support for an in-place image resize maybe something like vips (?) (http://www.vips.ecs.soton.ac.uk/index.php?title=Speed_and_Memory_Use) The system should intelligently call vips to transform the image to a reasonable size at time of upload then use those derivative for just in time thumbs for articles. ( If vips is unavailable we don't transform and we don't crash the apache node.) (2) maybe spinning out the image transform process early on in the parsing of the page with a place-holder and callback so by the time all the templates and links have been looked up the image is ready for output. (maybe another function wfShellBackgroundExec($cmd, $callback_function) (maybe using |pcntl_fork then normal |wfShellExec then| ||pcntl_waitpid then callback function ... which sets some var in the parent process so that pageOutput knows its good to go) | If operationally the "daemon" should be on a separate server we should still more or less run synchronously ... as mentioned above ... if possible the daemon should be php based so we don't explode the dependencies for deploying robust image handling with mediaWiki. peace, --michael _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
