Re: [darktable-dev] denoise profile non local means: neighborhood parameter

[johannes hanika]

hi,

that doesn't sound like a bad idea at all. for what it's worth, in
practice the nlmeans doesn't let any grain at all through due to the
piecewise constant prior that it's based on. well, only in regions
where it finds enough other patches that is. in the current
implementation with a radius of 7 that is not always the case.

also, i usually use some blending to add the input buffer back on top
of the output. this essentially leaves the grain alone but tones it
down.

cheers,
 jo


On Thu, Jun 14, 2018 at 12:23 AM, Aurélien Pierre
 wrote:
> Hi,
>
> The problem of a 2-passes denoising method involving 2 differents
> algorithms, the later applied where the former failed, could be the grain
> structure (the shape of the noise) would be different along the picture,
> thus very unpleasing.
>
> I thought maybe we could instead create some sort of total variation
> threshold on other denoising modules :
>
> compute the total variation of each channel of each pixel as the divergence
> divided by the L1 norm of the gradient - we then obtain a "heatmap" of the
> gradients over the picture (contours and noise)
> let the user define a total variation threshold and form a mask where the
> weights above the threshold are the total variation and the weights below
> the threshold are zeros (sort of a highpass filter actually)
> apply the bilateral filter according to this mask.
>
> This way, if the user wants to stack several denoising modules, he could
> protect the already-cleaned areas from further denoising.
>
> What do you think ?
>
> Aurélien.
>
>
> Le 13/06/2018 à 03:16, rawfiner a écrit :
>
> Hi,
>
> I don't have the feeling that increasing K is the best way to improve noise
> reduction anymore.
> I will upload the raw next week (if I don't forget to), as I am not at home
> this week.
> My feeling is that doing non local means on raw data gives much bigger
> improvement than that.
> I still have to work on it yet.
> I am currently testing some raw downsizing ideas to allow a fast execution
> of the algorithm.
>
> Apart of that, I also think that to improve noise reduction such as the
> denoise profile in nlm mode and the denoise non local means, we could do a 2
> passes algorithm, with non local means applied first, and then a bilateral
> filter (or median filter or something else) applied only on pixels where non
> local means failed to find suitable patches (i.e. pixels where the sum of
> weights was close to 0).
> The user would have a slider to adjust this setting.
> I think that it would make easier to have a "uniform" output (i.e. an output
> where noise has been reduced quite uniformly)
> I have not tested this idea yet.
>
> Cheers,
> rawfiner
>
> Le lundi 11 juin 2018, johannes hanika  a écrit :
>>
>> hi,
>>
>> i was playing with noise reduction presets again and tried the large
>> neighbourhood search window. on my shots i could very rarely spot a
>> difference at all increasing K above 7, and even less so going above
>> 10. the image you posted earlier did show quite a substantial
>> improvement however. i was wondering whether you'd be able to share
>> the image so i can evaluate on it? maybe i just haven't found the
>> right test image yet, or maybe it's camera dependent?
>>
>> (and yes, automatic and adaptive would be better but if we can ship a
>> simple slider that can improve matters, maybe we should)
>>
>> cheers,
>>  jo
>>
>>
>>
>> On Mon, Jan 29, 2018 at 2:05 AM, rawfiner  wrote:
>> > Hi
>> >
>> > Yes, the patch size is set to 1 from the GUI, so it is not a bilateral
>> > filter, and I guess it corresponds to a patch window size of 3x3 in the
>> > code.
>> > The runtime difference is near the expected quadratic slowdown:
>> > 1,460 secs (8,379 CPU) for 7 and 12,794 secs (85,972 CPU) for 25, which
>> > means about 10.26x slowdown
>> >
>> > If you want to make your mind on it, I have pushed a branch here that
>> > integrates the K parameter in the GUI:
>> > https://github.com/rawfiner/darktable.git
>> > The branch is denoise-profile-GUI-K
>> >
>> > I think that it may be worth to see if an automated approach for the
>> > choice
>> > of K may work, in order not to integrate the parameter in the GUI.
>> > I may try to implement the approach of Kervann and Boulanger (the
>> > reference
>> > from the darktable blog post) to see how it performs.
>> >
>> > cheers,
>> > rawfiner
>> >
>> >
>> > 2018-01-27 13:50 GMT+01:00 johannes hanika :
>> >>
>> >> heya,
>> >>
>> >> thanks for the reference! interesting interpretation how the blotches
>> >> form. not sure i'm entirely convinced by that argument.
>> >> your image does look convincing though. let me get this right.. you
>> >> ran with radius 1 which means patch window size 3x3? not 1x1 which
>> >> would be a bilateral filter effectively?
>> >>
>> >> also what was the run time difference? is it near the expected
>> >> quadratic slowdown from 7 (i.e. 15x15) to 25 (51x51) so about 11.56x
>> >> slower with the large window size? (test with darktable -d 

Re: [darktable-dev] denoise profile non local means: neighborhood parameter

[Aurélien Pierre]

Hi,

The problem of a 2-passes denoising method involving 2 differents
algorithms, the later applied where the former failed, could be the
grain structure (the shape of the noise) would be different along the
picture, thus very unpleasing.

I thought maybe we could instead create some sort of total variation
threshold on other denoising modules :

 1. compute the total variation of each channel of each pixel as the
divergence divided by the L1 norm of the gradient - we then obtain a
"heatmap" of the gradients over the picture (contours and noise)
 2. let the user define a total variation threshold and form a mask
where the weights above the threshold are the total variation and
the weights below the threshold are zeros (sort of a highpass filter
actually)
 3. apply the bilateral filter according to this mask.

This way, if the user wants to stack several denoising modules, he could
protect the already-cleaned areas from further denoising.

What do you think ?

Aurélien.


Le 13/06/2018 à 03:16, rawfiner a écrit :
> Hi,
>
> I don't have the feeling that increasing K is the best way to improve
> noise reduction anymore.
> I will upload the raw next week (if I don't forget to), as I am not at
> home this week.
> My feeling is that doing non local means on raw data gives much bigger
> improvement than that.
> I still have to work on it yet.
> I am currently testing some raw downsizing ideas to allow a fast
> execution of the algorithm.
>
> Apart of that, I also think that to improve noise reduction such as
> the denoise profile in nlm mode and the denoise non local means, we
> could do a 2 passes algorithm, with non local means applied first, and
> then a bilateral filter (or median filter or something else) applied
> only on pixels where non local means failed to find suitable patches
> (i.e. pixels where the sum of weights was close to 0).
> The user would have a slider to adjust this setting.
> I think that it would make easier to have a "uniform" output (i.e. an
> output where noise has been reduced quite uniformly)
> I have not tested this idea yet.
>
> Cheers,
> rawfiner
>
> Le lundi 11 juin 2018, johannes hanika  > a écrit :
>
> hi,
>
> i was playing with noise reduction presets again and tried the large
> neighbourhood search window. on my shots i could very rarely spot a
> difference at all increasing K above 7, and even less so going above
> 10. the image you posted earlier did show quite a substantial
> improvement however. i was wondering whether you'd be able to share
> the image so i can evaluate on it? maybe i just haven't found the
> right test image yet, or maybe it's camera dependent?
>
> (and yes, automatic and adaptive would be better but if we can ship a
> simple slider that can improve matters, maybe we should)
>
> cheers,
>  jo
>
>
>
> On Mon, Jan 29, 2018 at 2:05 AM, rawfiner  > wrote:
> > Hi
> >
> > Yes, the patch size is set to 1 from the GUI, so it is not a
> bilateral
> > filter, and I guess it corresponds to a patch window size of 3x3
> in the
> > code.
> > The runtime difference is near the expected quadratic slowdown:
> > 1,460 secs (8,379 CPU) for 7 and 12,794 secs (85,972 CPU) for
> 25, which
> > means about 10.26x slowdown
> >
> > If you want to make your mind on it, I have pushed a branch here
> that
> > integrates the K parameter in the GUI:
> > https://github.com/rawfiner/darktable.git
> 
> > The branch is denoise-profile-GUI-K
> >
> > I think that it may be worth to see if an automated approach for
> the choice
> > of K may work, in order not to integrate the parameter in the GUI.
> > I may try to implement the approach of Kervann and Boulanger
> (the reference
> > from the darktable blog post) to see how it performs.
> >
> > cheers,
> > rawfiner
> >
> >
> > 2018-01-27 13:50 GMT+01:00 johannes hanika  >:
> >>
> >> heya,
> >>
> >> thanks for the reference! interesting interpretation how the
> blotches
> >> form. not sure i'm entirely convinced by that argument.
> >> your image does look convincing though. let me get this right.. you
> >> ran with radius 1 which means patch window size 3x3? not 1x1 which
> >> would be a bilateral filter effectively?
> >>
> >> also what was the run time difference? is it near the expected
> >> quadratic slowdown from 7 (i.e. 15x15) to 25 (51x51) so about
> 11.56x
> >> slower with the large window size? (test with darktable -d perf)
> >>
> >> since nlmeans isn't the fastest thing, even with this coalesced
> way of
> >> implementing it, we should certainly keep an eye on this.
> >>
> >> that being said if we can often times get much better 

Re: [darktable-dev] denoise profile non local means: neighborhood parameter

[rawfiner]

Hi

Yes, the patch size is set to 1 from the GUI, so it is not a bilateral
filter, and I guess it corresponds to a patch window size of 3x3 in the
code.
The runtime difference is near the expected quadratic slowdown:
1,460 secs (8,379 CPU) for 7 and 12,794 secs (85,972 CPU) for 25, which
means about 10.26x slowdown

If you want to make your mind on it, I have pushed a branch here that
integrates the K parameter in the GUI:
https://github.com/rawfiner/darktable.git
The branch is denoise-profile-GUI-K

I think that it may be worth to see if an automated approach for the choice
of K may work, in order not to integrate the parameter in the GUI.
I may try to implement the approach of Kervann and Boulanger (the reference
from the darktable blog post) to see how it performs.

cheers,
rawfiner


2018-01-27 13:50 GMT+01:00 johannes hanika :

> heya,
>
> thanks for the reference! interesting interpretation how the blotches
> form. not sure i'm entirely convinced by that argument.
> your image does look convincing though. let me get this right.. you
> ran with radius 1 which means patch window size 3x3? not 1x1 which
> would be a bilateral filter effectively?
>
> also what was the run time difference? is it near the expected
> quadratic slowdown from 7 (i.e. 15x15) to 25 (51x51) so about 11.56x
> slower with the large window size? (test with darktable -d perf)
>
> since nlmeans isn't the fastest thing, even with this coalesced way of
> implementing it, we should certainly keep an eye on this.
>
> that being said if we can often times get much better results we
> should totally expose this in the gui, maybe with a big warning that
> it really severely impacts speed.
>
> cheers,
>  jo
>
> On Sat, Jan 27, 2018 at 7:34 AM, rawfiner  wrote:
> > Thank you for your answer
> > I perfectly agree with the fact that the GUI should not become
> > overcomplicated.
> >
> > As far as I understand, the pixels within a small zone may suffer from
> > correlated noise, and there is a risk of noise to noise matching.
> > That's why this paper suggest not to take pixels that are too close to
> the
> > zone we are correcting, but to take them a little farther (see the
> caption
> > of Figure 2 for a quick explaination):
> >
> > https://pdfs.semanticscholar.org/c458/71830cf535ebe6c2b7656f6a205033
> 761fc0.pdf
> > (in case you ask, unfortunately there is a patent associated with this
> > approach, so we cannot implement it)
> >
> > Increasing the neighborhood parameter results in having proportionally
> less
> > problem of correlation between surrounding pixels, and decreases the
> size of
> > the visible spots.
> > See for example the two attached pictures: one with size 1, force 1, and
> K 7
> > and the other with size 1, force 1, and K 25.
> >
> > I think that the best would probably be to adapt K automatically, in
> order
> > not to affect the GUI, and as we may have different levels of noise in
> > different parts of an image.
> > In this post
> > (https://www.darktable.org/2012/12/profiling-sensor-and-photon-noise/),
> this
> > paper is cited:
> >
> > [4] charles kervrann and jerome boulanger: optimal spatial adaptation for
> > patch-based image denoising. ieee trans. image process. vol. 15, no. 10,
> > 2006
> >
> > As far as I understand, it gives a way to choose an adaptated window size
> > for each pixel, but I don't see in the code anything related to that
> >
> > Maybe is this paper related to the TODOs in the code ?
> >
> > Was it planned to implement such a variable window approach ?
> >
> > Or if it is already implemented, could you point me where ?
> >
> > Thank you
> >
> > rawfiner
> >
> >
> >
> >
> > 2018-01-26 9:05 GMT+01:00 johannes hanika :
> >>
> >> hi,
> >>
> >> if you want, absolutely do play around with K. in my tests it did not
> >> lead to any better denoising. to my surprise a larger K often led to
> >> worse results (for some reason often the relevance of discovered
> >> patches decreases with distance from the current point). that's why K
> >> is not exposed in the gui, no need for another irrelevant and cryptic
> >> parameter. if you find a compelling case where this indeed leads to
> >> better denoising we could rethink that.
> >>
> >> in general NLM is a 0-th order denoising scheme, meaning the prior is
> >> piecewise constant (you claim the pixels you find are trying to
> >> express /the same/ mean, so you average them). if you let that
> >> algorithm do what it would really like to, it'll create unpleasant
> >> blotches of constant areas. so for best results we need to tone it
> >> down one way or another.
> >>
> >> cheers,
> >>  jo
> >>
> >>
> >>
> >> On Fri, Jan 26, 2018 at 7:36 AM, rawfiner  wrote:
> >> > Hi
> >> >
> >> > I am surprised to see that we cannot control the neighborhood
> parameter
> >> > for
> >> > the NLM algorithm (neither for the denoise non local mean, nor for the
> >> > denoise profiled) from the GUI.
> >> > 

Re: [darktable-dev] denoise profile non local means: neighborhood parameter

[johannes hanika]

heya,

thanks for the reference! interesting interpretation how the blotches
form. not sure i'm entirely convinced by that argument.
your image does look convincing though. let me get this right.. you
ran with radius 1 which means patch window size 3x3? not 1x1 which
would be a bilateral filter effectively?

also what was the run time difference? is it near the expected
quadratic slowdown from 7 (i.e. 15x15) to 25 (51x51) so about 11.56x
slower with the large window size? (test with darktable -d perf)

since nlmeans isn't the fastest thing, even with this coalesced way of
implementing it, we should certainly keep an eye on this.

that being said if we can often times get much better results we
should totally expose this in the gui, maybe with a big warning that
it really severely impacts speed.

cheers,
 jo

On Sat, Jan 27, 2018 at 7:34 AM, rawfiner  wrote:
> Thank you for your answer
> I perfectly agree with the fact that the GUI should not become
> overcomplicated.
>
> As far as I understand, the pixels within a small zone may suffer from
> correlated noise, and there is a risk of noise to noise matching.
> That's why this paper suggest not to take pixels that are too close to the
> zone we are correcting, but to take them a little farther (see the caption
> of Figure 2 for a quick explaination):
>
> https://pdfs.semanticscholar.org/c458/71830cf535ebe6c2b7656f6a205033761fc0.pdf
> (in case you ask, unfortunately there is a patent associated with this
> approach, so we cannot implement it)
>
> Increasing the neighborhood parameter results in having proportionally less
> problem of correlation between surrounding pixels, and decreases the size of
> the visible spots.
> See for example the two attached pictures: one with size 1, force 1, and K 7
> and the other with size 1, force 1, and K 25.
>
> I think that the best would probably be to adapt K automatically, in order
> not to affect the GUI, and as we may have different levels of noise in
> different parts of an image.
> In this post
> (https://www.darktable.org/2012/12/profiling-sensor-and-photon-noise/), this
> paper is cited:
>
> [4] charles kervrann and jerome boulanger: optimal spatial adaptation for
> patch-based image denoising. ieee trans. image process. vol. 15, no. 10,
> 2006
>
> As far as I understand, it gives a way to choose an adaptated window size
> for each pixel, but I don't see in the code anything related to that
>
> Maybe is this paper related to the TODOs in the code ?
>
> Was it planned to implement such a variable window approach ?
>
> Or if it is already implemented, could you point me where ?
>
> Thank you
>
> rawfiner
>
>
>
>
> 2018-01-26 9:05 GMT+01:00 johannes hanika :
>>
>> hi,
>>
>> if you want, absolutely do play around with K. in my tests it did not
>> lead to any better denoising. to my surprise a larger K often led to
>> worse results (for some reason often the relevance of discovered
>> patches decreases with distance from the current point). that's why K
>> is not exposed in the gui, no need for another irrelevant and cryptic
>> parameter. if you find a compelling case where this indeed leads to
>> better denoising we could rethink that.
>>
>> in general NLM is a 0-th order denoising scheme, meaning the prior is
>> piecewise constant (you claim the pixels you find are trying to
>> express /the same/ mean, so you average them). if you let that
>> algorithm do what it would really like to, it'll create unpleasant
>> blotches of constant areas. so for best results we need to tone it
>> down one way or another.
>>
>> cheers,
>>  jo
>>
>>
>>
>> On Fri, Jan 26, 2018 at 7:36 AM, rawfiner  wrote:
>> > Hi
>> >
>> > I am surprised to see that we cannot control the neighborhood parameter
>> > for
>> > the NLM algorithm (neither for the denoise non local mean, nor for the
>> > denoise profiled) from the GUI.
>> > I see in the code (denoiseprofile.c) this TODO that I don't understand:
>> > "//
>> > TODO: fixed K to use adaptive size trading variance and bias!"
>> > And just some lines after that: "// TODO: adaptive K tests here!"
>> > (K is the neighborhood parameter of the NLM algorithm).
>> >
>> > In practice, I think that being able to change the neighborhood
>> > parameter
>> > allows to have a better noise reduction for one image.
>> > For  example, choosing a bigger K allows to reduce the spotted aspect
>> > that
>> > one can get on high ISO images.
>> >
>> > Of course, increasing K increase computational time, but I think we
>> > could
>> > find an acceptable range that would still be useful.
>> >
>> >
>> > Is there any reason for not letting the user control the neighborhood
>> > parameter in the GUI ?
>> > Also, do you understand the TODOs ?
>> > I feel that we would probably get better denoising by fixing these, but
>> > I
>> > don't understand them.
>> >
>> > I can spend some time on these TODOs, or to add the K parameter to the
>> > interface if you think it is 

Re: [darktable-dev] denoise profile non local means: neighborhood parameter

[Terry Duell]

On Sat, 27 Jan 2018 05:34:24 +1100, rawfiner  wrote:

Thank you for your answer I perfectly agree with the fact that the GUI  
should not become

overcomplicated.


...and neither should large attachments (9 MB) be sent directly to a  
mailing list.
Please use a link to large attached files, not everyone wants or needs to  
get it.


Cheers,
--
Regards,
Terry Duell
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] denoise profile non local means: neighborhood parameter

[johannes hanika]

hi,

if you want, absolutely do play around with K. in my tests it did not
lead to any better denoising. to my surprise a larger K often led to
worse results (for some reason often the relevance of discovered
patches decreases with distance from the current point). that's why K
is not exposed in the gui, no need for another irrelevant and cryptic
parameter. if you find a compelling case where this indeed leads to
better denoising we could rethink that.

in general NLM is a 0-th order denoising scheme, meaning the prior is
piecewise constant (you claim the pixels you find are trying to
express /the same/ mean, so you average them). if you let that
algorithm do what it would really like to, it'll create unpleasant
blotches of constant areas. so for best results we need to tone it
down one way or another.

cheers,
 jo



On Fri, Jan 26, 2018 at 7:36 AM, rawfiner  wrote:
> Hi
>
> I am surprised to see that we cannot control the neighborhood parameter for
> the NLM algorithm (neither for the denoise non local mean, nor for the
> denoise profiled) from the GUI.
> I see in the code (denoiseprofile.c) this TODO that I don't understand: "//
> TODO: fixed K to use adaptive size trading variance and bias!"
> And just some lines after that: "// TODO: adaptive K tests here!"
> (K is the neighborhood parameter of the NLM algorithm).
>
> In practice, I think that being able to change the neighborhood parameter
> allows to have a better noise reduction for one image.
> For  example, choosing a bigger K allows to reduce the spotted aspect that
> one can get on high ISO images.
>
> Of course, increasing K increase computational time, but I think we could
> find an acceptable range that would still be useful.
>
>
> Is there any reason for not letting the user control the neighborhood
> parameter in the GUI ?
> Also, do you understand the TODOs ?
> I feel that we would probably get better denoising by fixing these, but I
> don't understand them.
>
> I can spend some time on these TODOs, or to add the K parameter to the
> interface if you think it is worth it (I think so but it is only my personal
> opinion), but I have to understand what the TODOs mean before
>
> Thank you for your help
>
> rawfiner
>
> ___
> darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org