Re: [Mjpeg-users] B ... [WAS: y4mscaler: Upsampling to widescreen]
On Tue, Dec 20, 2005 at 11:15:43AM +1100, Mark Heath wrote: I'm not sure how this came about but I thought about training a Backpropagation neural net with sample images so that it may learn what the missing pixels looked like from the surrounding pixels. Just FYI, you might be interested in this work on Video Epitomes: http://www.psi.toronto.edu/~vincent/videoepitome.html -Dave Dodge --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637alloc_id=16865op=click ___ Mjpeg-users mailing list Mjpeg-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mjpeg-users
Re: [Mjpeg-users] B ... [WAS: y4mscaler: Upsampling to widescreen]
Andrew Stevens wrote: For the same x2 special-case asd your algorithm there is also some very interesting work based on ideas from some Sony researchers. Here you again choose your weights dynamically based on context in which your new pixel appears. However, the weight-selection function you use constructed by a 'learning' process. You 'train' your 'smart filter' on a huge set of training data (upscaling downsampled video and trying to match the original as closely as possible). The results are sometimes eerily good! The training approach (and the larger 3x3 'context' they use) allows them to avoid certain kinds of artefact I (suspect) your technique might have in common with median-like non-linear scaling filters. The usual complaint is a tendency to optical 'fattening' of fine features. Median filters also tend to be expensive (in HW) to get to work for fractional scale factors. Though I suspect yours is quite 'friendly' in that regard. I'm not sure how this came about but I thought about training a Backpropagation neural net with sample images so that it may learn what the missing pixels looked like from the surrounding pixels. At first the results looked surprisingly good. Later I discovered that the engine appears to be dependent on the sample images used to train it. IE if there are a lot of uphill diagonal lines in the sample images, all uphill lines in the upsampled picture look perfectly smooth, however the downhill lines look excessively jaggered. This also happens if the reverse is used to train the network. I thought that it may be that I was only using a 3x3 grid (effectively 6 pixels) so tried a 5x3 grid, which simply made things look worse. I then went to passing the RGB values through the network at the same time (back to a 3x3 grid), incase they might have an effect on each other, and it looked promising at first but as it was trained with more images, became progressively worse. The result appears no better than a box filter (as used by pamscale). I'm not sure if this is because I have not trained it enough, or if this is something that has no pattern and cannot be learnt, or that I'm using the wrong sort of images. I don't suppose anyone has any suggestions? this is how I am training the network, with a 3x3 grid: 1 2 3 4 5 6 7 8 9 take pixels 1,2,3,7,8 and 9 as the inputs to the neural net, and pixel 5 as the output. Mark --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637alloc_id=16865op=click ___ Mjpeg-users mailing list Mjpeg-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mjpeg-users
Re: [Mjpeg-users] B ... [WAS: y4mscaler: Upsampling to widescreen]
Hi Stefan, Nice Algorithm... it seems to fall into the class of 'median' style non-linear filters. These are very important in the x2 upscalers used in motion-adaptive deinterlacing in current mainstream Digital TV chips for flat panel displays. There are quite a few variations (every firm has to come up with their own to avoid the patents of other firms). The basic idea is just the one you came up with though: don't just pick a 'dumb' spatial average based on some signal theory that probably doesn't really apply too well to images. Instead try to come up with a pixel that is reasonably 'typical' (close to the mean / median) of a small 'context' around the sampling point. For the same x2 special-case asd your algorithm there is also some very interesting work based on ideas from some Sony researchers. Here you again choose your weights dynamically based on context in which your new pixel appears. However, the weight-selection function you use constructed by a 'learning' process. You 'train' your 'smart filter' on a huge set of training data (upscaling downsampled video and trying to match the original as closely as possible). The results are sometimes eerily good! The training approach (and the larger 3x3 'context' they use) allows them to avoid certain kinds of artefact I (suspect) your technique might have in common with median-like non-linear scaling filters. The usual complaint is a tendency to optical 'fattening' of fine features. Median filters also tend to be expensive (in HW) to get to work for fractional scale factors. Though I suspect yours is quite 'friendly' in that regard. Also, in fairness to the 'dumb' signal theory approach in the 'real world' you *never* just apply a reconstruction filter alone. You always run some kind of adaptive sharpening as well to steepen the otherwise unprettily 'flat' transitions a linear reconstruction filter alone would give you. The combined results can also be pretty good and the technique has the advantage of robustness and that it is clean and efficient to implement in 'streaming' 1-pixel-per-cycle VLSI hardware. cheers, Andrew --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637alloc_id=16865op=click ___ Mjpeg-users mailing list Mjpeg-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mjpeg-users
[Mjpeg-users] B ... [WAS: y4mscaler: Upsampling to widescreen]
Michael Hanke schrieb:
Ok. Now that you have said A I would ask that you should also say B: Would
it be possible to get a copy of your filter for experimenting, ideally with
adescription of the theory behind it since you said it's non-standard.
OK, here we go...
This is a much simplified and a lot faster variant of that scaler. It
took me some minutes to make it useful and understandable... (I hope)...
void
upscale (uint8_t * dst, uint8_t * src, int w, int h)
{
// triangulation scaler
// this scaler works by analysing the correlations between neighboring
pixels
// it is a rather simple super-resolution-approach but as TV is
lowpassfiltered
// in front of transmission we can't recover resolution with a
multi-image approach
// so if upscaling is needed at all, we try to do it visually correct...
//
// the results are quite similar to what modern plasma-screens do with
SD-signals
int x, y;
int dx, dy;
int m;
int a, b, c, d;
int ae, be, ce, de, me;
for (y = 0; y = h; y++)
for (x = 0; x = w; x++)
{
*(dst + (x * 2) + (y * 2) * (w * 2)) = *(src + x + y * w);
}
w *= 2;
h *= 2;
for (y = 1; y = h; y += 2)
for (x = 1; x = w; x += 2)
{
// fill in the four neighbor-pixels
a = *(dst + (x - 1) + (y - 1) * w);
b = *(dst + (x + 1) + (y - 1) * w);
c = *(dst + (x - 1) + (y + 1) * w);
d = *(dst + (x + 1) + (y + 1) * w);
// calculate the mean of the neighbors
m = (a + b + c + d) / 4;
// calculate the error for every neighbor-pixel
ae = (m - a) * (m - a);
be = (m - b) * (m - b);
ce = (m - c) * (m - c);
de = (m - d) * (m - d);
// find the maximum error-value
me = ae;
me = (me be) ? be : me;
me = (me ce) ? ce : me;
me = (me de) ? de : me;
// generate mixing coefficients
ae = me - ae;
be = me - be;
ce = me - ce;
de = me - de;
me = ae + be + ce + de;
if (me != 0)
m = (a * ae + b * be + c * ce + d * de) / me;
*(dst + x + y * w) = m;
}
for (y = 0; y = h; y += 2)
for (x = 1; x = w; x += 2)
{
// fill in the four neighbor-pixels
a = *(dst + (x - 1) + y * w);
b = *(dst + (x + 1) + y * w);
c = *(dst + x + (y - 1) * w);
d = *(dst + x + (y + 1) * w);
// calculate the mean of the neighbors
m = (a + b + c + d) / 4;
// calculate the error for every neighbor-pixel
ae = (m - a) * (m - a);
be = (m - b) * (m - b);
ce = (m - c) * (m - c);
de = (m - d) * (m - d);
// find the maximum error-value
me = ae;
me = (me be) ? be : me;
me = (me ce) ? ce : me;
me = (me de) ? de : me;
// generate mixing coefficients
ae = me - ae;
be = me - be;
ce = me - ce;
de = me - de;
me = ae + be + ce + de;
if (me != 0)
m = (a * ae + b * be + c * ce + d * de) / me;
*(dst + x + y * w) = m;
}
for (y = 1; y = h; y += 2)
for (x = 0; x = w; x += 2)
{
// fill in the four neighbor-pixels
a = *(dst + (x - 1) + y * w);
b = *(dst + (x + 1) + y * w);
c = *(dst + x + (y - 1) * w);
d = *(dst + x + (y + 1) * w);
// calculate the mean of the neighbors
m = (a + b + c + d) / 4;
// calculate the error for every neighbor-pixel
ae = (m - a) * (m - a);
be = (m - b) * (m - b);
ce = (m - c) * (m - c);
de = (m - d) * (m - d);
// find the maximum error-value
me = ae;
me = (me be) ? be : me;
me = (me ce) ? ce : me;
me = (me de) ? de : me;
// generate mixing coefficients
ae = me - ae;
be = me - be;
ce = me - ce;
de = me - de;
me = ae + be + ce + de;
if (me != 0)
m = (a * ae + b * be + c * ce + d * de) / me;
*(dst + x + y * w) = m;
}
// only very little lowpass-filtering (sometimes looks better
sometimes not...)...
#if 0
for (y = 0; y = h; y++)
for (x = 0; x = w; x++)
{
a = *(dst + (x - 1) + (y - 1) * w);
b = *(dst + (x + 1) + (y - 1) * w);
c = *(dst + (x - 1) + (y + 1) * w);
d = *(dst + (x + 1) + (y + 1) * w);
m = (a + b + c + d) / 4;
m += *(dst + x + y * w) * 3;
m /= 4;
*(dst + x + y * w) = m;
}
#endif
}
--
Gnomemeeting/Netmeeting: callto:ils.seconix.com/[EMAIL PROTECTED]
ICQ: 131490319
---
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637alloc_id=16865op=click
___
Mjpeg-users mailing list
Mjpeg-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mjpeg-users
