Re: [otb-users] Learning process: confusion matrix

[Jordi Inglada]

Hi Thierry,

I guess that the confusion matrix from TrainImagesClassifier is better than the 
one you get using your validation polygons. The training application samples 
pixels and therefore it will use pixels from the same polygon for training and 
for validation, which can yield better performance metrics than using a 
complete different set of polygons for training and for validation.

Maybe you can share the confusion matrices with us so we can give you a better 
guess of what is happening.

Thank you.

Jordi

On Mon 31-Jul-2017 at 09:05:01 +0200, Poughon Victor  
wrote: 
> Hi Thierry,
>
> In OTB version 5.10 and higher, TrainImagesClassifier uses the sample 
> procedure from SampleSelection, another OTB application:
>
> https://www.orfeo-toolbox.org/CookBook/Applications/app_SampleSelection.html
>
> You can set some parameters directly in TrainImagesClassifier, but if you 
> want greater control over the way the sampling is done, that’s one option.
>
> There’s a detailed tutorial in the cookbook here: 
> https://www.orfeo-toolbox.org/CookBook/recipes/pbclassif.html#samples-selection
>
> Concerning your issue, are the two confusion matrices you get really 
> different qualitatively, or is it just noise? Also, which OTB version are you 
> using.
>
> Thanks,
>
> Victor Poughon
>
> De : otb-users@googlegroups.com [mailto:otb-users@googlegroups.com] De la 
> part de Thierry Bélouard
> Envoyé : jeudi 27 juillet 2017 17:46
> À : otb-users
> Objet : [otb-users] Learning process: confusion matrix
>
> Hello,
>
> I would like to have some explanations about the calculus of the confusion 
> matrix (learning process) with OTB because I get totally different results 
> according to the way I proceed.
>
> On one side, I have split my reference data (polygons, 4 classes of 
> defoliation in forest) into 2 data files: a learning data set (50% of the 
> polygons) and a testing data set (others 50%). To do so, I have sampled 
> polygons regularly according to their size in each class of defoliation. An
> important point maybe is that I have sampled polygons and not points or 
> pixels. I calculate my random forest rule on my learning data set and then I 
> calculate my confusion matrix with my classification map and my testing data 
> set.
>
> On the other side, it seem to me than we can calculate simultaneously the 
> classifying rule and the matrix confusion with TrainImagesClassifier module 
> with the entire reference data set (learning and testing polygons all 
> together) and setting learning/validation ratio to 0.5. Isn’t? If yes, I
> don’t understand why I get completely different results. Can the reason be a 
> different sampling procedure of OTB as a systematic sampling of pixels for 
> example?
>
> Thank you for your answer.
>
> Thierry Bélouard
>
> -- 
> -- 
> Check the OTB FAQ at
> http://www.orfeo-toolbox.org/FAQ.html
>
> You received this message because you are subscribed to the Google
> Groups "otb-users" group.
> To post to this group, send email to otb-users@googlegroups.com
> To unsubscribe from this group, send email to
> otb-users+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/otb-users?hl=en
> --- 
> You received this message because you are subscribed to the Google Groups 
> "otb-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to otb-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> -- 

-- 
-- 
Check the OTB FAQ at
http://www.orfeo-toolbox.org/FAQ.html

You received this message because you are subscribed to the Google
Groups "otb-users" group.
To post to this group, send email to otb-users@googlegroups.com
To unsubscribe from this group, send email to
otb-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/otb-users?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"otb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to otb-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [otb-users] Learning process: confusion matrix

[Poughon Victor]

Hi Thierry,

In OTB version 5.10 and higher, TrainImagesClassifier uses the sample procedure 
from SampleSelection, another OTB application:
https://www.orfeo-toolbox.org/CookBook/Applications/app_SampleSelection.html

You can set some parameters directly in TrainImagesClassifier, but if you want 
greater control over the way the sampling is done, that’s one option.
There’s a detailed tutorial in the cookbook here: 
https://www.orfeo-toolbox.org/CookBook/recipes/pbclassif.html#samples-selection

Concerning your issue, are the two confusion matrices you get really different 
qualitatively, or is it just noise? Also, which OTB version are you using.

Thanks,

Victor Poughon

De : otb-users@googlegroups.com [mailto:otb-users@googlegroups.com] De la part 
de Thierry Bélouard
Envoyé : jeudi 27 juillet 2017 17:46
À : otb-users
Objet : [otb-users] Learning process: confusion matrix

Hello,

I would like to have some explanations about the calculus of the confusion 
matrix (learning process) with OTB because I get totally different results 
according to the way I proceed.

On one side, I have split my reference data (polygons, 4 classes of defoliation 
in forest) into 2 data files: a learning data set (50% of the polygons) and a 
testing data set (others 50%). To do so, I have sampled polygons regularly 
according to their size in each class of defoliation. An important point maybe 
is that I have sampled polygons and not points or pixels. I calculate my random 
forest rule on my learning data set and then I calculate my confusion matrix 
with my classification map and my testing data set.

On the other side, it seem to me than we can calculate simultaneously the 
classifying rule and the matrix confusion with TrainImagesClassifier module 
with the entire reference data set (learning and testing polygons all together) 
and setting learning/validation ratio to 0.5. Isn’t? If yes, I don’t understand 
why I get completely different results. Can the reason be a different sampling 
procedure of OTB as a systematic sampling of pixels for example?

Thank you for your answer.

Thierry Bélouard
--
--
Check the OTB FAQ at
http://www.orfeo-toolbox.org/FAQ.html

You received this message because you are subscribed to the Google
Groups "otb-users" group.
To post to this group, send email to 
otb-users@googlegroups.com
To unsubscribe from this group, send email to
otb-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/otb-users?hl=en
---
You received this message because you are subscribed to the Google Groups 
"otb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
otb-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- 
Check the OTB FAQ at
http://www.orfeo-toolbox.org/FAQ.html

You received this message because you are subscribed to the Google
Groups "otb-users" group.
To post to this group, send email to otb-users@googlegroups.com
To unsubscribe from this group, send email to
otb-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/otb-users?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"otb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to otb-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.