Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder
Spencer, I tried the mixed effects approach you suggest using the random effects module of AD Model Builder: (http://www.otter-rsch.ca/admbre/admbre.html). What are 94 unbounded parameters in Schnute et al (1998), now become realizations of a Gaussian random variable, with the corresponding standard deviation being estimated as a parameter. The approach works, but the computation time is increased substantially. This is however understandable as the computational problem is a very different one. The likelihood function now involves an integral in dimension 94, which I believe cannot be broken into a product of lower dimensional integrals as is usual for clustered data (the reason being the recursive nature of the population dynamics). hans ___ Spencer Graves wrote: Have you considered nonlinear mixed effects models for the types of problems considered in the comparison paper you cite? Those benchmark trials consider T years of data ... for A age classes and the total number of parameters is m = T+A+5. Without knowing more about the problem, I suspect that the T year parameters and the A age class parameters might be better modeled as random effects. If this were done, the optimization problem would then involve 7 parameters, the 5 fixed-effect parameters suggested by the computation of m plus two variance parameters, one for the random year effects and another for the random age class effect. This would replace the problem of maximizing, e.g., a likelihood over T+A+5 parameters with one of maximizing a marginal likelihood over 2+5 parameters after integrating out the T and A random effects. These integrations may not be easy, and I might stick with the fixed-effects solution if I couldn't get answers in the available time using a model I thought would be theoretically more appropriate. Also, I might use the fixed-effects solution to get starting values for an attempt to maximize a more appropriate marginal likelihood. For the latter, I might first try 'nlmle'. If that failed, I might explore Markov Chain Monte Carlo (MCMC). I have not done MCMC myself, but the MCMCpack R package looks like it might make it feasible for the types of problems considered in this comparison. The CRAN summary of that package led me to an Adobe Acrobat version of a PPT slide presentation that seemed to consider just this type of problem (e.g., http://mcmcpack.wustl.edu/files/MartinQuinnMCMCpackslides.pdf). Have you considered that? Hope this helps. Spencer Graves [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder
Douglas Bates wrote: snip Don't you find it somewhat disingenuous that you publish a comparison between the AD Model Builder software that you sell and R - a comparison that shows a tremendous advantage for your software - and then you write I am not proficient in R? I think there is a misunderstanding here. I did not pick this example. As I said it was undertaken by Schnute and his colleagues. I had nothing to do with it except of course to sell them my software. However as I stated at that time Splus could not run the model without crashing after a time so that no comparison was possible. I was aware of those results and decided to use R to complete the comparison. I don't see why Schnute would want to unfairly promote my software. I believe he was simply looking for the best tool for the job he had in mind. Had you been proficient in R you might have known about the symbolic differentiation capabilities, specifically the deriv function, that have been part of the S language since the late 1980s. I believe that the 'AD' in AD model builder stands for automatic differentiation, which is actually something that John Chambers and I discussed at length when we were developing nonlinear modeling methods for S. In the end we went with symbolic differentiation rather than automatic differentiation because we felt that symbolic was more flexible. Yes I am aware of the symbolic differentiation capabilities. I have checked out the deriv function and it does not seem to be capable of calculating derivatives for a model of this complexity in an efficient manner. Of course I could be wrong. There is a paper by Andreas Griewank (whose title I have forgotten but perhaps some list member recalls) around 1990 where he compares symbolic and automatic differentiation for a simple model of an oil reservoir. He demonstrates quite decisively that symbolic differentiation is not the way to go. This is not to say that automatic differentiation isn't a perfectly legitimate technique. However, my recollection is that it would have required extensive revisions to the arithmetic expression evaluator, which is already very tricky code because of the recycling rule and the desire to shield users from knowledge of the internal representations and such details as whether you are using logical or integer or double precision operands or a combination. If you want to see these details you can, of course, examine the source code. I don't believe we would have the opportunity to examine how you implemented automatic differentiation. I must also agree with Spencer Graves that when I start reading a description of a nonlinear model with over 100 parameters, the example that you chose, I immediately start thinking of nonlinear mixed effects models. In my experience the only way in which a nonlinear model ends up with that number of parameters is through applying an underlying model with a low number of parameters to various groups within the data. Table 2 in the Schnute et al. paper to which you make reference states that the number of parameters in the model is T + A + 5 where T is the number of years of data and A is the number of age classes. To me that looks a lot like a nonlinear mixed effects model. I agree that this makes a good nonlinear random effects example. Of course 10 years ago AD Model Builder did not have that capability. It now does and my colleague Hans Skaug has modified the code to incorporate random effects. I believe the model converges in a few minutes. He will report the results and hopefully they can be compared to nlme or any other software in R which can carry out the calculations. Also, your choice of subject heading for your message seems deliberatively provocative. You seem to be implying that you are discussing a comparisons of AD Model Builder and R on all aspects of nonlinear statistical modeling but you are only discussing one comparison on simulated data using a model from the applications area for which you wrote AD Model Builder. Then you follow up by saying I am not proficient in R and your results for R are from applying code that someone else gave you. It seems that ADMB had a bit of a home-field advantage in this particular comparison. I don't quite get your point. Of course I am only going to present examples where I believe ADMB is (far) superior to R. Otherwise I would just be wasting everyones time. ADMB is much more narrowly focused than R. I think that people can examine the arguments and make up their own minds. I view nonlinear statistical modeling differently. I have had a bit of experience in the area and I find that I want to examine the data carefully, usually through plots, before I embark on fitting complicated models. I like to have some assurance that the model makes sense in the context of the data. (In your example you don't need to worry about appropriateness of the model because the data were
Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder
On 11/24/06, dave fournier [EMAIL PROTECTED] wrote: Dave Did you try supplying gradient information to nlminb? (I note that nlminb is used for the optimization, but I don't see any gradient information supplied to it.) I would suspect that supplying gradient information would greatly speed up the computation (as you note in comments at http://otter-rsch.ca/tresults.htm.) Actually you should probably ask Norm Olsen these questions. I am not proficient in R and am just using his code. Don't you find it somewhat disingenuous that you publish a comparison between the AD Model Builder software that you sell and R - a comparison that shows a tremendous advantage for your software - and then you write I am not proficient in R? Had you been proficient in R you might have known about the symbolic differentiation capabilities, specifically the deriv function, that have been part of the S language since the late 1980s. I believe that the 'AD' in AD model builder stands for automatic differentiation, which is actually something that John Chambers and I discussed at length when we were developing nonlinear modeling methods for S. In the end we went with symbolic differentiation rather than automatic differentiation because we felt that symbolic was more flexible. This is not to say that automatic differentiation isn't a perfectly legitimate technique. However, my recollection is that it would have required extensive revisions to the arithmetic expression evaluator, which is already very tricky code because of the recycling rule and the desire to shield users from knowledge of the internal representations and such details as whether you are using logical or integer or double precision operands or a combination. If you want to see these details you can, of course, examine the source code. I don't believe we would have the opportunity to examine how you implemented automatic differentiation. I must also agree with Spencer Graves that when I start reading a description of a nonlinear model with over 100 parameters, the example that you chose, I immediately start thinking of nonlinear mixed effects models. In my experience the only way in which a nonlinear model ends up with that number of parameters is through applying an underlying model with a low number of parameters to various groups within the data. Table 2 in the Schnute et al. paper to which you make reference states that the number of parameters in the model is T + A + 5 where T is the number of years of data and A is the number of age classes. To me that looks a lot like a nonlinear mixed effects model. Also, your choice of subject heading for your message seems deliberatively provocative. You seem to be implying that you are discussing a comparisons of AD Model Builder and R on all aspects of nonlinear statistical modeling but you are only discussing one comparison on simulated data using a model from the applications area for which you wrote AD Model Builder. Then you follow up by saying I am not proficient in R and your results for R are from applying code that someone else gave you. It seems that ADMB had a bit of a home-field advantage in this particular comparison. I view nonlinear statistical modeling differently. I have had a bit of experience in the area and I find that I want to examine the data carefully, usually through plots, before I embark on fitting complicated models. I like to have some assurance that the model makes sense in the context of the data. (In your example you don't need to worry about appropriateness of the model because the data were simulated.) I would never try to fit a nonlinear model with 100 parameters to data without carefully examining the data, and especially selected subsets of the data, first. For this the flexibility of the S language and tools like lattice graphics that were developed in this language are invaluable to me. The flexibility of data manipulation and graphics for interactive exploration of data is what attracted me to S in the first place. I realize that for many people the area of nonlinear statistical modeling is reduced to Fit this model to these data and don't ask any questions. Just give me parameter estimates and p-values. If that is your situation then it would make sense to use software that gets you those estimates as quickly as possible with a minimum of effort. I'm just happy that I get to turn down people who ask me to do that. I like that fact that I can spend my time asking questions about the data and of the data. However I can say that providing derivatives for such a model is a highly nontrivial exercise. As I said in my posting, the R script and data are available to anyone who feels that the exercise was not carried out properly and would like to improve on it. Also one does not need to provide derivatives to the AD Model Builder program. Finally suppose that you are very good at calculating derivatives and manage to get them right.
[R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder
There has recently been some discussion on the list about AD Model builder and the suitability of R for constructing the types of models used in fisheries management. https://stat.ethz.ch/pipermail/r-help/2006-January/086841.html https://stat.ethz.ch/pipermail/r-help/2006-January/086858.html I think that many R users understimate the numerical challenges that some of the typical nonlinear statistical model used in different fields present. R may not be a suitable platform for development for such models. Around 10 years ago John Schnute, Laura Richards, and Norm Olsen with Canadian federal fisheries undertook an investigation comparing various statistical modeling packages for a simple age-structured statistical model of the type commonly used in fisheries. They compared AD Mdel Builder, Gauss, Matlab, and Splus. Unfortunately a working model could not be produced with Splus so its times could not be included in the comparison. It is possible to produce a working model with the present day version of R so that R can now be directly compared with AD Model Builder for this type of model. I have put the results of the test together with the original Schnute and Richards paper and the working R and AD Model Builder codes on Otter's web site http://otter-rsch.ca/tresults.htm The results are that AD Model builder is roughly 1000 times faster than R for this problem. ADMB takes about 2 seconds to converge while R takes over 90 minutes. This is a simple toy example. Real fisheries models are often hundred of times more computationally intensive as this one. Cheers, Dave ~ -- David A. Fournier P.O. Box 2040, Sidney, B.C. V8l 3S3 Canada Phone/FAX 250-655-3364 http://otter-rsch.com __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder
dave fournier [EMAIL PROTECTED] wrote: I think that many R users understimate the numerical challenges that some of the typical nonlinear statistical model used in different fields present. R may not be a suitable platform for development for such models. Around 10 years ago John Schnute, Laura Richards, and Norm Olsen with Canadian federal fisheries undertook an investigation comparing various statistical modeling packages for a simple age-structured statistical model of the type commonly used in fisheries. [...] It is possible to produce a working model with the present day version of R so that R can now be directly compared with AD Model Builder for this type of model. The results are that AD Model builder is roughly 1000 times faster than R for this problem. ADMB takes about 2 seconds to converge while R takes over 90 minutes. Our group's experiences reflect, at least qualitatively, what Dave says above. We use R for analyzing results from models written in his AD Model Builder, and a couple of years ago, we started programming one of our models directly in R. We quickly abandoned that idea because of lengthy execution time under R. That is not a judgement of either piece of software. R and ADMB are designed for different types of task, and it seems to me that they complement each other well. That experience was in part the genesis of our X2R software (now at CRAN -- pardon the plug), which saves results from ADMB models into a format that R can read as a list. We feel that now we have the best of both worlds -- fast execution with ADMB, followed by the programming ease and excellent graphics of R for analysis of results and projections under numerous scenarios. -- Mike Prager, NOAA, Beaufort, NC * Opinions expressed are personal and not represented otherwise. * Any use of tradenames does not constitute a NOAA endorsement. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder
Did you try supplying gradient information to nlminb? (I note that nlminb is used for the optimization, but I don't see any gradient information supplied to it.) I would suspect that supplying gradient information would greatly speed up the computation (as you note in comments at http://otter-rsch.ca/tresults.htm.) I'm curious -- when you say R may not be a suitable platform for development for such models, what aspect of R do you feel is lacking? Is it the specific optimization routines available, or is it some other more general aspect? Also, another optimization algorithm available in R is the L-BFGS-B method for optim() in the MASS package. I've had extremely good experiences with using this code in S-PLUS. It can take box constraints, and can use gradient information. It is my first choice for most optimization problems, and I believe it is very widely used. Did you try using that optimization routine with this problem? -- Tony Plate dave fournier wrote: There has recently been some discussion on the list about AD Model builder and the suitability of R for constructing the types of models used in fisheries management. https://stat.ethz.ch/pipermail/r-help/2006-January/086841.html https://stat.ethz.ch/pipermail/r-help/2006-January/086858.html I think that many R users understimate the numerical challenges that some of the typical nonlinear statistical model used in different fields present. R may not be a suitable platform for development for such models. Around 10 years ago John Schnute, Laura Richards, and Norm Olsen with Canadian federal fisheries undertook an investigation comparing various statistical modeling packages for a simple age-structured statistical model of the type commonly used in fisheries. They compared AD Mdel Builder, Gauss, Matlab, and Splus. Unfortunately a working model could not be produced with Splus so its times could not be included in the comparison. It is possible to produce a working model with the present day version of R so that R can now be directly compared with AD Model Builder for this type of model. I have put the results of the test together with the original Schnute and Richards paper and the working R and AD Model Builder codes on Otter's web site http://otter-rsch.ca/tresults.htm The results are that AD Model builder is roughly 1000 times faster than R for this problem. ADMB takes about 2 seconds to converge while R takes over 90 minutes. This is a simple toy example. Real fisheries models are often hundred of times more computationally intensive as this one. Cheers, Dave ~ __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder
Dave Did you try supplying gradient information to nlminb? (I note that nlminb is used for the optimization, but I don't see any gradient information supplied to it.) I would suspect that supplying gradient information would greatly speed up the computation (as you note in comments at http://otter-rsch.ca/tresults.htm.) Actually you should probably ask Norm Olsen these questions. I am not proficient in R and am just using his code. However I can say that providing derivatives for such a model is a highly nontrivial exercise. As I said in my posting, the R script and data are available to anyone who feels that the exercise was not carried out properly and would like to improve on it. Also one does not need to provide derivatives to the AD Model Builder program. Finally suppose that you are very good at calculating derivatives and manage to get them right. Then someone else comes along who wants to modify the model. Unless they are also very good at calculating derivatives there will be trouble. I'm curious -- when you say R may not be a suitable platform for development for such models, what aspect of R do you feel is lacking? Is it the specific optimization routines available, or is it some other more general aspect? 2 seconds vs 90 minutes. For a real problem of tihs type the timings would probably be something like 10 minutes vs more than 2,700 minutes. Also, another optimization algorithm available in R is the L-BFGS-B method for optim() in the MASS package. I've had extremely good experiences with using this code in S-PLUS. It can take box constraints, and can use gradient information. It is my first choice for most optimization problems, and I believe it is very widely used. Did you try using that optimization routine with this problem? -- Tony Plate dave fournier wrote: There has recently been some discussion on the list about AD Model builder and the suitability of R for constructing the types of models used in fisheries management. https://stat.ethz.ch/pipermail/r-help/2006-January/086841.html https://stat.ethz.ch/pipermail/r-help/2006-January/086858.html I think that many R users understimate the numerical challenges that some of the typical nonlinear statistical model used in different fields present. R may not be a suitable platform for development for such models. Around 10 years ago John Schnute, Laura Richards, and Norm Olsen with Canadian federal fisheries undertook an investigation comparing various statistical modeling packages for a simple age-structured statistical model of the type commonly used in fisheries. They compared AD Mdel Builder, Gauss, Matlab, and Splus. Unfortunately a working model could not be produced with Splus so its times could not be included in the comparison. It is possible to produce a working model with the present day version of R so that R can now be directly compared with AD Model Builder for this type of model. I have put the results of the test together with the original Schnute and Richards paper and the working R and AD Model Builder codes on Otter's web site http://otter-rsch.ca/tresults.htm The results are that AD Model builder is roughly 1000 times faster than R for this problem. ADMB takes about 2 seconds to converge while R takes over 90 minutes. This is a simple toy example. Real fisheries models are often hundred of times more computationally intensive as this one. Cheers, Dave ~ -- David A. Fournier P.O. Box 2040, Sidney, B.C. V8l 3S3 Canada Phone/FAX 250-655-3364 http://otter-rsch.com -- David A. Fournier P.O. Box 2040, Sidney, B.C. V8l 3S3 Canada Phone/FAX 250-655-3364 http://otter-rsch.com __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder
Hi, Mike Dave: Have you considered nonlinear mixed effects models for the types of problems considered in the comparison paper you cite? Those benchmark trials consider T years of data ... for A age classes and the total number of parameters is m = T+A+5. Without knowing more about the problem, I suspect that the T year parameters and the A age class parameters might be better modeled as random effects. If this were done, the optimization problem would then involve 7 parameters, the 5 fixed-effect parameters suggested by the computation of m plus two variance parameters, one for the random year effects and another for the random age class effect. This would replace the problem of maximizing, e.g., a likelihood over T+A+5 parameters with one of maximizing a marginal likelihood over 2+5 parameters after integrating out the T and A random effects. These integrations may not be easy, and I might stick with the fixed-effects solution if I couldn't get answers in the available time using a model I thought would be theoretically more appropriate. Also, I might use the fixed-effects solution to get starting values for an attempt to maximize a more appropriate marginal likelihood. For the latter, I might first try 'nlmle'. If that failed, I might explore Markov Chain Monte Carlo (MCMC). I have not done MCMC myself, but the MCMCpack R package looks like it might make it feasible for the types of problems considered in this comparison. The CRAN summary of that package led me to an Adobe Acrobat version of a PPT slide presentation that seemed to consider just this type of problem (e.g., http://mcmcpack.wustl.edu/files/MartinQuinnMCMCpackslides.pdf). Have you considered that? Hope this helps. Spencer Graves Mike Prager wrote: dave fournier [EMAIL PROTECTED] wrote: I think that many R users understimate the numerical challenges that some of the typical nonlinear statistical model used in different fields present. R may not be a suitable platform for development for such models. Around 10 years ago John Schnute, Laura Richards, and Norm Olsen with Canadian federal fisheries undertook an investigation comparing various statistical modeling packages for a simple age-structured statistical model of the type commonly used in fisheries. [...] It is possible to produce a working model with the present day version of R so that R can now be directly compared with AD Model Builder for this type of model. The results are that AD Model builder is roughly 1000 times faster than R for this problem. ADMB takes about 2 seconds to converge while R takes over 90 minutes. Our group's experiences reflect, at least qualitatively, what Dave says above. We use R for analyzing results from models written in his AD Model Builder, and a couple of years ago, we started programming one of our models directly in R. We quickly abandoned that idea because of lengthy execution time under R. That is not a judgement of either piece of software. R and ADMB are designed for different types of task, and it seems to me that they complement each other well. That experience was in part the genesis of our X2R software (now at CRAN -- pardon the plug), which saves results from ADMB models into a format that R can read as a list. We feel that now we have the best of both worlds -- fast execution with ADMB, followed by the programming ease and excellent graphics of R for analysis of results and projections under numerous scenarios. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
