Bryan Caplan wrote:
> Chris Auld wrote:

> > outcome.  Would it be accepted by any economics journal?  I don't
> > think so -- so why should I think that if we put a couple of dozen
> > of these together, we arrive at something compelling?
> 
> I think so.  Anyone can get one marginally convincing result.  But
> getting hundreds shows something.  

Not if all of them are faulty!  You ignored the first part of the
paragraph: taken alone, are any of M/H's results up to the minimum
standards of quantitative research in economics?  You also ignored
the question: how would M/H's results discern whether we live in a
world where ability has no direct effect, but ability is correlated
with any of the many relevant excluded regressors?


> There is even a formal test using
> this intuition - the p-lambda test.   Eight results (out of 10)
> significant the 20% level, for example, could almost never arise from
> pure chance. 

Suppose I were to take price/quantity clouds from many markets.  In each
market, I regress price on quantity and announce I've measured the supply
curve.  If I do this two dozen times rather than once, have I found
compelling evidence on the slope of supply curves?  Your argument is
indeed valid to show that it's very unlikely that AFQT is actually
an irrelevant regressor in M/H's specifications.  I'm not arguing that
the true coefficient in their spefications is zero -- I'm arguing that
they consistently misinterpret their findings.  


> > in each regression, rather then the ad hoc index.
> 
> There's as much or as little reason to use an index in TBC as anywhere. 
> Should they have put in each test sub-scale separately, too?

I don't understand this, Bryan.  By this reasoning, why do we ever bother
with more than one or two regressors?  Why not always just create ad hoc
indexes of everything we think we ought to be controlling for?  We
_should_ put in each sub-scale if we have reason to believe that, say,
math scores affect outcomes much differently from verbal scores and we are
interested in how these effects differ.  We should not if our question is
whether some generalized cognitive ability score affects outcomes.  Also
notice there is a natural way of aggregating test scores, whereas M/H's
SES index is completely ad hoc.

 
> I don't think it's possible for them to show there's "no way" to do it. 
> They could certainly point out data limitations, and offer their
> judgment that these are insuperable.  But I wouldn't call that judgment
> a "finding."

What they show is that time differences in returns to education and
returns to ability are nonparametrically unidentified by data based on
a single cohort.  That's a theoretical finding.  They then show that
the structure typically imposed to identify these effects, linear
seperability, is rejected by the NLS-Y data.  Of course, M/H never 
even get in the ballpark of realizing that all this is even an issue.

 
> If there's "no way" to do it for the whole population, how did they
> manage? :-)

They imposed the identification assumptions that everyone else does.
This is at least useful for showing the sensitivity of this type of
result under the usual assumptions, even if we also know we are 
imposing unwarranted structure.  Notice also the non-identification 
paper postdates this one.


>  Seriously, I'd expect that you could re-do almost anyone's
> results this way.  In each case, you would learn more, but unless the
> whole-sample results drastically reversed I don't see why it's so
> interesting.

Because it directly contradicts M/H's interpretation of their findings?
Here, for instance, is the abstract from Cawley et al 1997:

 This paper presents new evidence from the NLSY on the importance of
 meritocracy in American society. In it, we find that general
 intelligence, or g -- a measure of cognitive ability--is dominant in 
 explaining test score variance.  The weights assigned to tests by g are
 similar for all major demographic groups. These results support
 Spearman's theory of g. We also find that g and other measures of ability
 are not rewarded equally across race and gender, evidence against the
 view that the labor market is organized on meritocratic principles.
 Additional factors beyond g are required to explain wages and
 occupational choice. However, both blue collar and white collar wages are
 poorly predicted by g or even multiple measures of ability. Observed
 cognitive ability is only a minor predictor of social performance. White
 collar wages are more g loaded than blue collar wages. Many noncognitive
 factors determine blue collar wages. 
 

> > Many textbooks recommend dropping a regressor, then interpreting the
> > coefficients on the remaining regressors _as if_ the dropped regressor
> > was still in the equation?  That's clearly not true.  What are these
> > "many textbooks," out of curiosity (I can't recall ever reading anyone
> > recommending that one "solve" colinearity problems by just tossing
> > out regressors)?  And surely they note that such an exclusion affects how
> > one should interpret the coefficients on the remaining regressors, no?
> 
> I'm sure they put it more circumspectly, but that's what it amounts to.
> If you did a regression with both "what you said your education is" and
> "what your brother said your education is" and the SEs on both exploded,
> what would any econometrics prof tell you?  That "there's no way to
> decide" which matters, and we have to be agnostic?  That's the purist
> answer, but I think most practioners would tell you to drop the second
> measure and call the results from the new specification the "return to
> education."

Let's try a different example: I regress income on years of education,
with no other explanatory variables, and call the coefficient the causal
effect of education on income.  When someone, say, Bryan, comes along and
tells me that I've mis-interpreted the coefficient because I've failed to
control for variables correlated with education, such as ability, family
background, race, and gender, I announce that any econometrics prof would
recommend dropping such colinear regressors and ignoring the exclusions,
so my take is valid.  I would be, of course, mistaken.

What Bryan has actually said is that any econometrics prof would recommend
imposing exclusion restrictions that are strongly supported by theory in
order to improve efficiency, not that any econometrics prof would 
recommend dropping regressors willy nilly to "solve" colinearity problems,
then ignoring the effect the excluded variables might have had when 
interpreting the coefficients.

We would, of course, drop "what your brother said your education is" 
because it's a more noisy measure of the same underlying variable of
interest, and we have no reason to believe that "what your brother said
your education is" would have any explanatory power beyond your true
education level.  Of course, if we drop that variable, we can't test
that assumption, and we would be off base if we made implicit or explicit
claims that we'd proved "what your brother..." has no independent effect.
We do have reason to believe that education and income have affects beyond
cognitive ability in M/H's regressions, so dropping them and then
pretending we haven't is bad methodology.

Notice further that, in Bryan's counterexample, we may want to use the
extra information in "what my brother said your education is" if we can.
We might, for instance, estimate a system of equations, with one equation
representing true education level using information in both measures, and
the other governing labor market success.  Something like this is actually
what Bishop, AER, 1989 does in the context of incomes, IQs and education.


Chris Auld                          (403)220-4098
Economics, University of Calgary    <mailto:[EMAIL PROTECTED]>
Calgary, Alberta, Canada            <URL:http://jerry.ss.ucalgary.ca/>


Reply via email to