On 28 Feb 2003, Patrick Noffke wrote:
> I have a regression application where I fit data to polynomials of
> up to 3 inputs (call them x1, x2, and x3).
If I understand you correctly, you have three predictors (since you call
them x1, x2, x3; if they were response variables I would expect you to
have called them y1, y2, y3). Do you in fact have more than one
response variable, y, say? If what you're trying to fit is a model of
the general form
y = f(x1, x2, x3)
where f is a polynomial function in three variables, this is only
multiple (univariate) regression, not multivariate regression. But I
may have misunderstood you.
> The problem I'm having is selecting terms for the regression since
> there is a lot of inter-dependence (e.g. x1^4 looks a lot like
> x1^6). If I add one more term, the coefficients of other terms
> change greatly, and it becomes difficult to evaluate the
> significance of each term and the overall quality of the fit.
Yes. This is characteristic of variables obtained by simply taking
successive powers, especially if the original variable (x1, say) only
took on positive values. This is the reason textbooks on multiple
regression (see, e.g., Draper & Smith, Applied regression analysis)
often recommend using orthogonal polynomials.
> With one variable, the answer is to use orthogonal polynomials.
> Then the lower order coefficients don't change by adding higher
> order terms. Does this concept extend to multivariate regression?
Yes; especially if by "multivariate" you really mean "multiple".
It can be somewhat more complicated, though.
> Are there multivariate orthogonal polynomials?
You can specify orthogonal polynomials for each predictor separately.
The complicated part arises when you want to control also for
correlations between x1 (and its polynomials) and x2 (and its
polynomials). [Aside: polynomials may be used as surrogates for other
kinds of nonlinear functions -- log(x), log(y), e^x, e^y spring to mind
-- and in such cases it is nearly always *much* better to fit the proper
function rather than a polynomial approximation. If you want more
commentary on this point, ask.]
> My first guess would be there are (maybe with terms like cos^2(u),
> cos(u)cos(v), and cos^2(v)?).
Umm... these don't look like polynomials to me; and I do not see what
(if any) relation you are visualizing between the original variables x1,
x2, x3 and these variables u, v.
> Does the idea that addition of higher order terms won't affect your
> lower order coefficients also extend to the multivariate case?
The short answer is "Yes", reading "multiple predictors" for
"multivariate". (I think it may also be "Yes" for truly multivariate
multiple regression, where there are several y's whose multivariate
vector is to be fitted by polynomials in (x1, x2, x3); but I have not
personally dealt with such problems, and am not sure my intuition is
trustworthy.)
A longer answer: if x1 and x2 (for openers) are correlated, you will
have the same kind of problem disntangling their joint effects from
their separate effects that you have with x1^4 and x1^6 above. You can
address the problem in a logically equivalent way, however. In the case
of x1 and its quadratic, cubic, ... components, one sort of assigns a
logical priority to x1, the original variable, and chooses a quadratic
function of x1 that is uncorrelated with x1 as the orthogonal polynomial
of order (or do I mean degree?) two; and then a cubic function of x1
that is uncorrelated both with x1 and with the quadratic function; and
so on. Similarly, one can decide that x1 is logically prior to x2, and
then substitute for x2 a function of (x2,x1) that is uncorrelated with
x1. This function one might call the part of x2 that is orthogonal to
x1. (One then has another priority decision: should this part of x2
also be orthogonal to the quadratic part of x1, or should we define the
quadratic part of x1 in this context to be orthogonal to x2? You can do
it either way. Similar questions must be addressed in defining the
quadratic part of x2 (presumably to be orthogonal to x2, x1, probably
the quadratic part of x1, ...).
Now: if you also want to model interaction between x1 and x2 (etc.),
you will want a function of (x1*x2) [where "*" means "multiply"] that is
orthogonal to both x1 and x2. And you'll need to decide how far out to
carry the polynomial interactions: (x1:x2^2, x1^2:x2, x1^2:x2^2, ...).
> Could someone please enlighten me or point me to a reference
> (preferably one with practical & useful examples in addition to the
> theory)?
There's a White Paper of mine on the Minitab web page (www.minitab.com):
"On modelling and interpreting interactions in multiple regression".
The URL is http://www.minitab.com/resources/whitepapers/burril2.htm
Doesn't deal with polynomials, but does show how to construct
interaction terms that are orthogonal to all lower-order interactions
(and, a fortiori, the main effects). The extension to polynomials
should be straightforward, especially if you consult Draper & Smith's
section on orthogonalizing the X matrix (in Chapter 5, I think) as well.
> Thank you kindly in advance for your help,
> Pat
I hope this will have been helpful. -- Don.
-----------------------------------------------------------------------
Donald F. Burrill [EMAIL PROTECTED]
56 Sebbins Pond Drive, Bedford, NH 03110 (603) 626-0816
.
.
=================================================================
Instructions for joining and leaving this list, remarks about the
problem of INAPPROPRIATE MESSAGES, and archives are available at:
. http://jse.stat.ncsu.edu/ .
=================================================================
