Re: [Wikitech-l] What's the best place to do post-upload processing on a file? Etc.
Thanks for the pointer Michael -- Timed Media Handler seems like a good example of job queuing use. The way TMH does job queuing seems like a feasible option for the post-upload processing I'm doing. Rather than enqueuing them upon the file's Upload Complete event firing, TMH seems to put video transcoding jobs into the queue the first time they're requested -- i.e. the first time a page containing the video is loaded. My initial impression was that it'd be faster from the user's perspective if the job were enqueued as soon as possible, which I assume would be onUploadComplete. Maybe there's a negligible difference here, or maybe I don't understand something -- any thoughts? On a separate note, I've found a way to speed up the post-upload processing needed for my extension. The ray-tracing can be divided among multiple CPU cores. (I've tried both ray-tracing libraries supported by the molecular visualization package I'm using, and they only support multicored, not multithreaded, distribution of the task.) The amount of time needed to do the post-upload processing seems to decrease proportional to the number of cores used. Given that, would it be possible to use multiple cores for this post-upload processing? If so, how many cores could be used for a given one of these ray-tracing tasks? If that distribution got the time needed to complete this processing down to something that was deemed reasonable for the user, would that make it unnecessary to enqueue jobs for this option? Thanks, Eric On Fri, May 4, 2012 at 7:19 PM, Michael Dale md...@wikimedia.org wrote: You will want to put into a jobQueue you can take a look at the Timed Media Handler extension for how post upload processor intensive transformations can be handled. --michael On 05/04/2012 04:58 AM, emw wrote: Hi all, For a MediaWiki extension I'm working on (see http://lists.wikimedia.org/**pipermail/wikitech-l/2012-** April/060254.htmlhttp://lists.wikimedia.org/pipermail/wikitech-l/2012-April/060254.html), an effectively plain-text file will need to be converted into a static image. I've got a set of scripts that does that, but it takes my medium-grade consumer laptop about 30 seconds to convert the plain-text file into a ray-traced static image. Since ray-tracing the images being created here substantially improves their visual quality, my impression is that it's worth a moderately expensive transformation operation like this, but only if the operation is done once. Given that, I assume it'd be best to do this transformation immediately after the plain-text file has completed uploading. Is that right? If not, what's a better time/way to do that processing? I've looked into MediaWiki's 'UploadComplete' event hook to accomplish this. That handler gives a way to access information about the upload and the local file. However, I haven't been able to find a way to get the uploaded file's path on the local file system, which I would need to do the transformation. Looking around related files I see references to $srcPath, which seems like what I would need. Am I just missing some getter method for file system path data in UploadBase.php or LocalFile.php? How can I get the information about an uploaded file's location on the file system while in an onUploadComplete-like object method in my extension? Thanks, Eric __**_ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/**mailman/listinfo/wikitech-lhttps://lists.wikimedia.org/mailman/listinfo/wikitech-l __**_ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/**mailman/listinfo/wikitech-lhttps://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] What's the best place to do post-upload processing on a file? Etc.
Hi all, For a MediaWiki extension I'm working on (see http://lists.wikimedia.org/pipermail/wikitech-l/2012-April/060254.html), an effectively plain-text file will need to be converted into a static image. I've got a set of scripts that does that, but it takes my medium-grade consumer laptop about 30 seconds to convert the plain-text file into a ray-traced static image. Since ray-tracing the images being created here substantially improves their visual quality, my impression is that it's worth a moderately expensive transformation operation like this, but only if the operation is done once. Given that, I assume it'd be best to do this transformation immediately after the plain-text file has completed uploading. Is that right? If not, what's a better time/way to do that processing? I've looked into MediaWiki's 'UploadComplete' event hook to accomplish this. That handler gives a way to access information about the upload and the local file. However, I haven't been able to find a way to get the uploaded file's path on the local file system, which I would need to do the transformation. Looking around related files I see references to $srcPath, which seems like what I would need. Am I just missing some getter method for file system path data in UploadBase.php or LocalFile.php? How can I get the information about an uploaded file's location on the file system while in an onUploadComplete-like object method in my extension? Thanks, Eric ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] What's the best place to do post-upload processing on a file? Etc.
I am just a random user lurking on the mail-list, nor a mw dev, but I wonder why you don't look on the SVG handling already on mediawiki. What you are doing is rendering, with one added dimension :D http://www.mediawiki.org/wiki/Manual:Configuration_settings#SVG This stuff has to do things in a stablished way, that you can just expand, or replicate the strategy. This is a super-casual comment. Wait for what the mw devs say. On 4 May 2012 13:58, emw emw.w...@gmail.com wrote: Hi all, For a MediaWiki extension I'm working on (see http://lists.wikimedia.org/pipermail/wikitech-l/2012-April/060254.html), an effectively plain-text file will need to be converted into a static image. I've got a set of scripts that does that, but it takes my medium-grade consumer laptop about 30 seconds to convert the plain-text file into a ray-traced static image. -- -- ℱin del ℳensaje. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] What's the best place to do post-upload processing on a file? Etc.
On Fri, May 4, 2012 at 7:58 AM, emw emw.w...@gmail.com wrote: However, I haven't been able to find a way to get the uploaded file's path on the local file system, which I would need to do the transformation. FileRepo::getHashPath() should be what you want. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] What's the best place to do post-upload processing on a file? Etc.
You will want to put into a jobQueue you can take a look at the Timed Media Handler extension for how post upload processor intensive transformations can be handled. --michael On 05/04/2012 04:58 AM, emw wrote: Hi all, For a MediaWiki extension I'm working on (see http://lists.wikimedia.org/pipermail/wikitech-l/2012-April/060254.html), an effectively plain-text file will need to be converted into a static image. I've got a set of scripts that does that, but it takes my medium-grade consumer laptop about 30 seconds to convert the plain-text file into a ray-traced static image. Since ray-tracing the images being created here substantially improves their visual quality, my impression is that it's worth a moderately expensive transformation operation like this, but only if the operation is done once. Given that, I assume it'd be best to do this transformation immediately after the plain-text file has completed uploading. Is that right? If not, what's a better time/way to do that processing? I've looked into MediaWiki's 'UploadComplete' event hook to accomplish this. That handler gives a way to access information about the upload and the local file. However, I haven't been able to find a way to get the uploaded file's path on the local file system, which I would need to do the transformation. Looking around related files I see references to $srcPath, which seems like what I would need. Am I just missing some getter method for file system path data in UploadBase.php or LocalFile.php? How can I get the information about an uploaded file's location on the file system while in an onUploadComplete-like object method in my extension? Thanks, Eric ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
