RE: AI-GEOSTATS: moving averages and trend
Sebastiano, I am struggling to understand why you are interested in doing trend + residual separation? There can be no unique decomposition of a data set into 'trend' and 'residual', it is a judgement about what model you feel is most appropriate given your prior beliefs and observations (evidence). The only thing you can say about the model is to validate it on out of sample data (even as a Bayesian I say this!). So in a sense there is no correct decomposition, and any decomposition is valid (so long as it is correctly implemented - maybe that is your question?). Are some decompositions better than others? Well yes they are likely to be, but this largely depends on your data (and the completeness of the overall model). In terms of your original question about the shape of the kernel there is no overall theory that I am aware of - different kernels will have different properties in terms of the function classes that they represent (e.g. differentiability, frequency response / characteristic length scales). Kernel families will have different null spaces which might or might not be important for your specific application and what you want to find out. I'm not sure if this is terribly helpful ... but I think it is the reality - everything depends on your data and your judgement (prior). Conditional on those you get a model and you need to validate this model carefully ... then you are OK. cheers Dan --- Dr Dan Cornford Senior Lecturer, Computer Science and NCRG Aston University, Birmingham B4 7ET www: http://wiki.aston.ac.uk/DanCornford/ tel: +44 (0)121 204 3451 mob: 07766344953 --- From: owner-ai-geost...@jrc.ec.europa.eu [mailto:owner-ai-geost...@jrc.ec.europa.eu] On Behalf Of seba Sent: 02 February 2010 08:39 To: Pierre Goovaerts Cc: ai-geostats@jrc.it Subject: Re: AI-GEOSTATS: moving averages and trend Hi Pierre I think that for my task factorial kriging is a little bit too much sophisticated (nevertheless, is there any open source or free implementation of it ??? I remember that it is implemented in Isatis.). I have an exhaustive and regularly spaced data set (i.e. a grid) and I need to calculate locally the spatial variability of the residual surface or better I would like to calculate the spatial variability of the high frequency component. Here I'm lucky because I know exactly what I want to see and what I need to filter out. In theory, using (overlapping) moving window averages (but here it seems better to use some more complex kernel) one should be able to filter out the short range variability (characterized by an eventual variogram range within the window size???). Seeing the problem from another perspective, in the case of a perfect sine wave behavior, I should be able to filter out spatial variability components with wave lengths up to the window size. But maybe there is something flawed in my reasoningso feedback is appreciated! Bye Sebastiano At 16.27 01/02/2010, you wrote: well Factorial Kriging Analysis allows you to tailor the filtering weights to the spatial patterns in your data. You can use the same filter size but different kriging weights depending on whether you want to estimate the local or regional scales of variability. Pierre 2010/2/1 seba sebastiano.trevis...@libero.itmailto:sebastiano.trevis...@libero.it Hi José Thank you for the interesting references. I'm going to give a look! Bye Sebastiano At 15.46 01/02/2010, José M. Blanco Moreno wrote: Hello again, I am not a mathematician, so I never worried too much on the theoretical reasons. You may be able to find some discussion on this subject in Eubank, R.L. 1999. Nonparametric Regression and Spline Smoothing, 2a ed. M. Dekker, New York. You may be also interested on searching information in and related to (perhaps citing) this work: Altman, N. 1990. Kernel smoothing of data with correlated errors. Journal of the American Statistical Association, 85: 749-759. En/na seba ha escrit: Hi José Thank you for your reply. Effectively I'm trying to figure out the theoretical reasons for their use. Bye Sebas -- Pierre Goovaerts Chief Scientist at BioMedware Inc. 3526 W Liberty, Suite 100 Ann Arbor, MI 48103 Voice: (734) 913-1098 (ext. 202) Fax: (734) 913-2201 Courtesy Associate Professor, University of Florida Associate Editor, Mathematical Geosciences Geostatistician, Computer Sciences Corporation President, PGeostat LLC 710 Ridgemont Lane Ann Arbor, MI 48103 Voice: (734) 668-9900 Fax: (734) 668-7788 http://goovaerts.pierre.googlepages.com/
Re: AI-GEOSTATS: moving averages and trend
Cornford, Dan wrote:
Sebastiano,
I am struggling to understand why you are interested in doing trend +
residual separation? There can be no unique decomposition of a data
set into ‘trend’ and ‘residual’, it is a judgement about what model
you feel is most appropriate given your prior beliefs and observations
(evidence). The only thing you can say about the model is to validate
it on out of sample data (even as a Bayesian I say this!). So in a
sense there is no correct decomposition, and any decomposition is
valid (so long as it is correctly implemented – maybe that is your
question?). Are some decompositions better than others? Well yes they
are likely to be, but this largely depends on your data (and the
completeness of the overall model).
An article by Tomislav Hengl goes into separately estimating the trend
and then interpolating the residuals [1]. He calls it regression
kriging. Might be interesting to have a look at it in light of this
discussion.
cheers,
Paul
[1] http://dx.doi.org/10.1016/j.cageo.2007.05.001
@ARTICLE{Hengl2007,
author = {Hengl, T. and Heuvelink, G.B.M. and Rossiter, D.G.},
title = {About regression-kriging: From equations to case studies},
journal = {Computers \ Geosciences},
year = {2007},
volume = {33},
pages = {1301--1315},
number = {10},
}
In terms of your original question about the shape of the kernel there
is no overall theory that I am aware of – different kernels will have
different properties in terms of the function classes that they
represent (e.g. differentiability, frequency response / characteristic
length scales). Kernel families will have different null spaces which
might or might not be important for your specific application and what
you want to find out.
I’m not sure if this is terribly helpful … but I think it is the
reality – everything depends on your data and your judgement (prior).
Conditional on those you get a model and you need to validate this
model carefully … then you are OK.
cheers
Dan
---
Dr Dan Cornford
Senior Lecturer, Computer Science and NCRG
Aston University, Birmingham B4 7ET
www: http://wiki.aston.ac.uk/DanCornford/
tel: +44 (0)121 204 3451
mob: 07766344953
---
*From:* owner-ai-geost...@jrc.ec.europa.eu
[mailto:owner-ai-geost...@jrc.ec.europa.eu] *On Behalf Of *seba
*Sent:* 02 February 2010 08:39
*To:* Pierre Goovaerts
*Cc:* ai-geostats@jrc.it
*Subject:* Re: AI-GEOSTATS: moving averages and trend
Hi Pierre
I think that for my task factorial kriging is a little bit
too much sophisticated (nevertheless, is there any open source or
free implementation of it ??? I remember that it is implemented in
Isatis.).
I have an exhaustive and regularly spaced data set (i.e. a grid) and I
need
to calculate locally the spatial variability of the residual surface
or better
I would like to calculate the spatial variability of the high
frequency component.
Here I'm lucky because I know exactly what I want to see and what I
need to filter out.
In theory, using (overlapping) moving window averages (but here it
seems better to use some more complex kernel)
one should be able to filter out the short range variability
(characterized by an eventual variogram range within the window size???).
Seeing the problem from another perspective, in the case of a perfect
sine wave behavior, I should be able to filter out spatial
variability components with wave lengths up to the window size.
But maybe there is something flawed in my reasoningso feedback is
appreciated!
Bye
Sebastiano
At 16.27 01/02/2010, you wrote:
well Factorial Kriging Analysis allows you to tailor the filtering weights
to the spatial patterns in your data. You can use the same filter size but
different kriging weights depending on whether you want to estimate
the local or regional scales of variability.
Pierre
2010/2/1 seba sebastiano.trevis...@libero.it
mailto:sebastiano.trevis...@libero.it
Hi José
Thank you for the interesting references. I'm going to give a look!
Bye
Sebastiano
At 15.46 01/02/2010, José M. Blanco Moreno wrote:
Hello again,
I am not a mathematician, so I never worried too much on the
theoretical reasons. You may be able to find some discussion on this
subject in Eubank, R.L. 1999. Nonparametric Regression and Spline
Smoothing, 2a ed. M. Dekker, New York.
You may be also interested on searching information in and related to
(perhaps citing) this work: Altman, N. 1990. Kernel smoothing of data
with correlated errors. Journal of the American Statistical
Association, 85: 749-759.
En/na seba ha escrit:
Hi José
Thank you for your reply.
Effectively I'm trying to figure out the theoretical reasons for their
use.
Bye
Sebas
--
Pierre Goovaerts
Chief Scientist at BioMedware Inc.
3526 W Liberty, Suite 100
Ann Arbor, MI 48103
Voice: (734)
Re: AI-GEOSTATS: moving averages and trend
Hi Paul
Well, in some way regression kriging is not too
much different to kriging with external drift.
My question is more directed to kernel smoothing
techniques than to classical geostatistical tools.
I should have read the mentioned paper some time
ago...I'll go to give another look.
Sebastiano
At 13.56 02/02/2010, you wrote:
Cornford, Dan wrote:
Sebastiano,
I am struggling to understand why you are
interested in doing trend + residual
separation? There can be no unique
decomposition of a data set into trend and
residual, it is a judgement about what model
you feel is most appropriate given your prior
beliefs and observations (evidence). The only
thing you can say about the model is to
validate it on out of sample data (even as a
Bayesian I say this!). So in a sense there is
no correct decomposition, and any decomposition
is valid (so long as it is correctly
implemented maybe that is your question?).
Are some decompositions better than others?
Well yes they are likely to be, but this
largely depends on your data (and the completeness of the overall model).
An article by Tomislav Hengl goes into
separately estimating the trend and then
interpolating the residuals [1]. He calls it
regression kriging. Might be interesting to have
a look at it in light of this discussion.
cheers,
Paul
[1] http://dx.doi.org/10.1016/j.cageo.2007.05.001
@ARTICLE{Hengl2007,
author = {Hengl, T. and Heuvelink, G.B.M. and Rossiter, D.G.},
title = {About regression-kriging: From equations to case studies},
journal = {Computers \ Geosciences},
year = {2007},
volume = {33},
pages = {1301--1315},
number = {10},
}
In terms of your original question about the
shape of the kernel there is no overall theory
that I am aware of different kernels will
have different properties in terms of the
function classes that they represent (e.g.
differentiability, frequency response /
characteristic length scales). Kernel families
will have different null spaces which might or
might not be important for your specific
application and what you want to find out.
Im not sure if this is terribly helpful
but
I think it is the reality everything depends
on your data and your judgement (prior).
Conditional on those you get a model and you
need to validate this model carefully
then you are OK.
cheers
Dan
---
Dr Dan Cornford
Senior Lecturer, Computer Science and NCRG
Aston University, Birmingham B4 7ET
www: http://wiki.aston.ac.uk/DanCornford/
tel: +44 (0)121 204 3451
mob: 07766344953
---
*From:* owner-ai-geost...@jrc.ec.europa.eu
[mailto:owner-ai-geost...@jrc.ec.europa.eu] *On Behalf Of *seba
*Sent:* 02 February 2010 08:39
*To:* Pierre Goovaerts
*Cc:* ai-geostats@jrc.it
*Subject:* Re: AI-GEOSTATS: moving averages and trend
Hi Pierre
I think that for my task factorial kriging is a little bit
too much sophisticated (nevertheless, is there any open source or
free implementation of it ??? I remember that
it is implemented in Isatis.).
I have an exhaustive and regularly spaced data set (i.e. a grid) and I need
to calculate locally the spatial variability of
the residual surface or better
I would like to calculate the spatial
variability of the high frequency component.
Here I'm lucky because I know exactly what I
want to see and what I need to filter out.
In theory, using (overlapping) moving window
averages (but here it seems better to use some more complex kernel)
one should be able to filter out the short
range variability (characterized by an eventual
variogram range within the window size???).
Seeing the problem from another perspective, in the case of a perfect
sine wave behavior, I should be able to filter out spatial
variability components with wave lengths up to the window size.
But maybe there is something flawed in my
reasoningso feedback is appreciated!
Bye
Sebastiano
At 16.27 01/02/2010, you wrote:
well Factorial Kriging Analysis allows you to tailor the filtering weights
to the spatial patterns in your data. You can use the same filter size but
different kriging weights depending on whether you want to estimate
the local or regional scales of variability.
Pierre
2010/2/1 seba sebastiano.trevis...@libero.it
mailto:sebastiano.trevis...@libero.it
Hi José
Thank you for the interesting references. I'm going to give a look!
Bye
Sebastiano
At 15.46 01/02/2010, José M. Blanco Moreno wrote:
Hello again,
I am not a mathematician, so I never worried
too much on the theoretical reasons. You may be
able to find some discussion on this subject in
Eubank, R.L. 1999. Nonparametric Regression and
Spline Smoothing, 2a ed. M. Dekker, New York.
You may be also interested on searching
information in and related to (perhaps citing)
this work: Altman, N. 1990. Kernel smoothing of
data with
Re: AI-GEOSTATS: moving averages and trend
Hello, Factorial kriging is not very sophisticated, it's just a slight variant of kriging that requires the modification of just a few lines of codes. Anyways, I just posted a program to perform factorial kriging analysis in the download section of my website. I hope your grid is not too big, The program filter.exe (FORTRAN source code filter.f) is a modified version of the Gslib program kt3d.f that allows performing a kriging analysis. Based on the number of nested structures of the variogram model specified in the parameter file filter.par, the program will estimate the values of the noise and noise-filtered signal (1 structure), or the values of the noise, local and regional components (2 structures). Following Goovaerts (1997, page 167), the regional component includes both the long-range component and the trend component in order to attenuate the impact of the search window on the estimation of these long-range spatial components. The zipped folder includes the executable, the source code, as well as a sample parameter file for the Jura dataset. In another paper concerned with noise-filtering of imagery, I run the program for a single pixel to get the kernel weights and then apply the same kernel everywhere (since the data geometry does not change except at the edges of the image). Goovaerts, P., Jacquez, G.M., and W.A. Marcus. 2005. Geostatistical and local cluster analysis of high resolution hyperspectral imagery for detection of anomalies. *Remote Sensing of the Environment*, 95, 351-367.http://home.comcast.net/%7Epgoovaerts/RSE-2005.pdf Cheers, Pierre On Tue, Feb 2, 2010 at 3:39 AM, seba sebastiano.trevis...@libero.it wrote: Hi Pierre I think that for my task factorial kriging is a little bit too much sophisticated (nevertheless, is there any open source or free implementation of it ??? I remember that it is implemented in Isatis.). I have an exhaustive and regularly spaced data set (i.e. a grid) and I need to calculate locally the spatial variability of the residual surface or better I would like to calculate the spatial variability of the high frequency component. Here I'm lucky because I know exactly what I want to see and what I need to filter out. In theory, using (overlapping) moving window averages (but here it seems better to use some more complex kernel) one should be able to filter out the short range variability (characterized by an eventual variogram range within the window size???). Seeing the problem from another perspective, in the case of a perfect sine wave behavior, I should be able to filter out spatial variability components with wave lengths up to the window size. But maybe there is something flawed in my reasoningso feedback is appreciated! Bye Sebastiano At 16.27 01/02/2010, you wrote: well Factorial Kriging Analysis allows you to tailor the filtering weights to the spatial patterns in your data. You can use the same filter size but different kriging weights depending on whether you want to estimate the local or regional scales of variability. Pierre 2010/2/1 seba sebastiano.trevis...@libero.it Hi José Thank you for the interesting references. I'm going to give a look! Bye Sebastiano At 15.46 01/02/2010, José M. Blanco Moreno wrote: Hello again, I am not a mathematician, so I never worried too much on the theoretical reasons. You may be able to find some discussion on this subject in Eubank, R.L. 1999. Nonparametric Regression and Spline Smoothing, 2a ed. M. Dekker, New York. You may be also interested on searching information in and related to (perhaps citing) this work: Altman, N. 1990. Kernel smoothing of data with correlated errors. Journal of the American Statistical Association, 85: 749-759. En/na seba ha escrit: Hi José Thank you for your reply. Effectively I'm trying to figure out the theoretical reasons for their use. Bye Sebas -- Pierre Goovaerts Chief Scientist at BioMedware Inc. 3526 W Liberty, Suite 100 Ann Arbor, MI 48103 Voice: (734) 913-1098 (ext. 202) Fax: (734) 913-2201 Courtesy Associate Professor, University of Florida Associate Editor, Mathematical Geosciences Geostatistician, Computer Sciences Corporation President, PGeostat LLC 710 Ridgemont Lane Ann Arbor, MI 48103 Voice: (734) 668-9900 Fax: (734) 668-7788 http://goovaerts.pierre.googlepages.com/ -- Pierre Goovaerts Chief Scientist at BioMedware Inc. 3526 W Liberty, Suite 100 Ann Arbor, MI 48103 Voice: (734) 913-1098 (ext. 202) Fax: (734) 913-2201 Courtesy Associate Professor, University of Florida Associate Editor, Mathematical Geosciences Geostatistician, Computer Sciences Corporation President, PGeostat LLC 710 Ridgemont Lane Ann Arbor, MI 48103 Voice: (734) 668-9900 Fax: (734) 668-7788 http://goovaerts.pierre.googlepages.com/
