Dear Crispin,
As we’ve discussed (off the list), I think you and I agree that if remaining char is actually discarded in practice, then the test metric should reflect that. If char is sometimes discarded and sometimes used in practice, then two different test metrics could reflect the two cases. Following is my response to your message (copied below) regarding our recent article published in the journal, Environmental Science and Technology, available at the web site: http://pubs.acs.org/doi/abs/10.1021/es301693f Crispin: “The spreadsheet does not have a place to enter the amount of raw fuel needed to accomplish a cooking task.” Jim: The spreadsheet does have a place to enter the amount of raw fuel. The spreadsheet is available at: http://www.pciaonline.org/testing Equivalent dry fuel is calculated (in the spreadsheet) to provide a more fair comparison between stoves tested with fuels with different moisture content. It would be relatively easy to use the data already entered in the spreadsheet to calculate fuel use assuming char is discarded. Crispin: “In the paper it is labeled 'overall thermal efficiency', but is it actually a proxy for the heat transfer efficiency…” Jim: Overall thermal efficiency is not a proxy for heat transfer efficiency. Please see a discussion of this in the Supporting Information (available on the journal web site), Section 10, Page S19. Crispin: “The statistical significance (or rather, the confidence we can have in the result) is undermined by the use of an averaging technique that is not accepted by statisticians as valid.” Jim: We performed a minimum of three replications for each stove/fuel combination we tested, and we reported results as the average and standard deviation of replicates. The standard deviation indicates the variation between replicates, but does not indicate statistically significant differences between stove/fuel results. We used the Student’s t-test (assuming unequal variance) to evaluate statistically significant differences for some of the results discussed in the article, but we did not do this evaluation for the entire data set. Crispin: “A peer reviewer might have noticed, for example, that the heat value of rice hull char was credited with a heat content of 29.5 MJ/kg. This is far from the actual heat content which is closer to 12-14 MJ (an error of ?100%).” Jim: The heat value of rice hull char was not credited with a heat content of 29.5 MJ/kg. As we described in the article and supporting information, heat of combustion for fuels and remaining char was measured using ASTM Standard Method D4442-07 (bomb calorimeter). Measured heat of combustion values are shown in Figure S6 in the supporting information. The heat of combustion for rice hull char from the Mayon Turbo stove seemed high, but we had a second sample analyzed with nearly the same result. Crispin: “If the Journal provides us with the spreadsheets from the actual tests as supplementary material for the published article (normally required for publication), we will be able to reproduce the work mathematically and see what the effect of making the necessary corrections would be. I expect there would be a significant change to the ratings given to all char-making stoves…” Jim: In the supporting information, we provided detailed information on the stoves, fuels (including moisture content and heat of combustion), cooking pots, operation (including fuel burning rates – see Figures S14-S16), test protocol, and equipment used. I believe the information is sufficient for others to reproduce the experiments. We can calculate fuel use with the assumption of discarded char, if there is a need for these data. There would certainly be a significant increase in fuel consumption for char-making stoves if we assume the char is discarded. Crispin: “What we have never seen in a Journal is a review of the procedures and mathematical methods conducted by an independent lab to put some confidence intervals on these WBT results. Dr Penn Taylor's Masters and PhD theses have not been published. They address the issues and give a precision of about 50% for the WBT 3.1 (which was under review at the time).” Jim: Robert Pendleton Taylor’s thesis is available at the web site: http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1534 <http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1534&context=etd> &context=etd >From Page 65 of the thesis: “Taking these errors together, the existing UCBWBT >should be considered to have a minimum method error of between five and ten >percent. Depending on the fuel used, the method error may be as high as twenty >percent, as in the case of a fuel with a high ash content. If the test is >altered to properly account for ash, the minimum method error drops to about >five percent. The UCBWBT currently reports the uncertainty of fuel use and >other results as the intra-test variation between three trials. It is not >uncommon to see uncertainties of one percent or less reported for factors such >as fuel use and energy efficiency. The sources of error identified in this >thesis indicate that even if the repeatability of the test can be brought to >such a fine level, the actual deviation of the test results from true values >is much more likely to be on the order of ten percent. If results of the >current UCBWBT are being used to compare two stove designs, the relative error >in thermal efficiency, specific fuel use, firepower, turn-down ratio, and any >emissions factors expressed on a per-energy or per-mass-of-fuel-consumed basis >should be assumed to be ten percent, regardless of that cited as the >intra-test error.” Some of these sources of error have been addressed since the thesis was published in 2009, but many of us are working to continue to improve existing methods and to develop new methods (e.g., methods for plancha stoves and charcoal stoves). USEPA QA personnel (independent from our project) are working on an uncertainty analysis including error propagation for the equipment and methods we are currently using. The methodology may be useful for other labs using different equipment and methods. Crispin: “Until the process of correcting the WBT is formalised the sorts of problems described above and in previous communications will continue to dog the stove community. In the meantime it is perfectly reasonable for stove projects to develop and use alternative test methods that are scientifically sound, properly documented and independently validated.” Jim: With support and leadership from the Global Alliance for Clean Cookstoves, I expect we will soon be engaged in a more formalized process for developing and improving test protocols and standards. We have challenging work to do, and I think this is an exciting time to be working together! Thank you for constructive comments. Respectfully, Jim ------------------------------ Date: Sun, 25 Nov 2012 17:24:04 -0500 From: "Crispin Pemberton-Pigott" <[email protected]> To: "'Discussion of biomass cooking stoves'" <[email protected]> Subject: Re: [Stoves] ETHOS 2013: Where is the New Data on Stove Performance in the Field? Message-ID: <[email protected]> Content-Type: text/plain; charset="koi8-r" Dear Dean The tests were performed using the WBT 4: "In this study, twenty-two cookstoves burning six fuel types (wood, charcoal, pellets, corn cobs, rice hulls, and plant oil) at two fuel moisture levels were examined under laboratory-controlled operating conditions as outlined in the Water Boiling Test (WBT) protocol, Version 4." You said: "You can see that fuel use is measured, not assumed, etc." I beg to disagree. The fuel use is not measured, it is calculated from the energy available from the mass of fuel burned, compensated for unburned char. I have just had confirmation of this from Jim yesterday morning. In other words, he followed the experimental procedure laid down in the WBT protocol exactly. The spreadsheet does not have a place to enter the amount of raw fuel needed to accomplish a cooking task. That is why the results of the Quad 2 stove test (636 dry g) are so different from the actual fuel needed to perform a 5-litre WBT (1550 dry g). Rocket stoves, which also produce quite a bit of char, are credited with a fuel consumption number that is not reflective of their actual performance. That will probably interest quite a number of organisations. It may explain a lot of the difference between WBT, CCT and KPT results. Maybe not - there are several other problems with the protocol and calculations. You have many times referred testers to the CCT for field confirmation of actual fuel use and warned people not to rely on the WBT results. The CCT makes the same type of calculation for 'dry fuel consumed' and does not determine the amount of fuel the stove actually draws from the forest for each cooking event. I checked the formulas. Like the WBT, it too finds out the approximate amount of heat that could be generated from the fuel and divides it by the dry heat content of the fuel type used for that test. The calculation method assumes that charcoal produced is actually raw fuel not consumed which leads to a significant misrepresentation of the performance with certain stove types, generally crediting them with using much less fuel than they do and over-rating the overall thermal efficiency (but still giving a proxy for the heat transfer efficiency). 'Overall thermal efficiency' is not the same as the heat transfer efficiency. The WBT 4.1.2 tries to measure the HTE (see Fig 2 in the paper) but the paper calls it the OTE. I understand the 'overall efficiency' as being the amount of raw fuel needed to feed the stove to accomplish a heating type task (like heating water). There are many industrial examples of this. In the paper it is labelled 'overall thermal efficiency', but is it actually a proxy for the heat transfer efficiency with several losses uncounted (as can be seen by the way the number is generated). >And, as you can imagine, in this EPA peer reviewed study the statistical significance is good. Again, I beg to disagree. The statistical significance (or rather, the confidence we can have in the result) is undermined by the use of an averaging technique that is not accepted by statisticians as valid. How can a test method (and then the derived test results obtained by using that method) be 'statistically significant' if the comparison of individual test results is only made after grouping them first? This demonstrates that the peer review is only as good as the reviewers. If the reviewers do not themselves review the test method, or do not ask that the author provide the results of an independent review of the method, you 'get what you pay for'. Even then, an independent review might be challenged if it contains its own mistakes. These things happen all the time. A peer reviewer might have noticed, for example, that the heat value of rice hull char was credited with a heat content of 29.5 MJ/kg. This is far from the actual heat content which is closer to 12-14 MJ (an error of ?100%). Because the fuel use number is calculated from the net heat value of fuel burned (according to both me and Jim) and because the rice hull char remaining has a far lower heat content than the number applied (according to any reference you care to find) the stove was credited with a significantly higher thermal efficiency than it actually has. This resulted (because of the erroneous manner in which the fuel consumption is determined) in a much better fuel consumption rating than it actually has for two reasons instead of the usual one: 1) the error in the heat value for the char remaining, and 2) the error of not measuring the amount of raw fuel the stoves consumes to perform each test. If the Journal provides us with the spreadsheets from the actual tests as supplementary material for the published article (normally required for publication), we will be able to reproduce the work mathematically and see what the effect of making the necessary corrections would be. I expect there would be a significant change to the ratings given to all char-making stoves and all those burning rice hull. How much? We can't tell until we see the supporting documentation for the article. My guess? 50-100% increase in fuel consumed per WBT and a halving of the number for the overall thermal efficiency. That supporting documentation will also include the version of the WBT (might be v.4 or 4.1 or 4.1.1 or 4.1.2) so we can reproduce it exactly. The published article system is where the real progress comes in science and there is no problem with challenging the content of peer-reviewed, published material. Published articles are a form of discourse. If something is published without the supporting material required to replicate the result, including calculations, it is not normally considered to have entered the published domain and remains 'grey literature'. There are scores of grey literature articles, monographs and pamphlets claiming that the WBT, CCT and KPT are good tools for developing and determining the performance of stoves. What we have never seen in a Journal is a review of the procedures and mathematical methods conducted by an independent lab to put some confidence intervals on these WBT results. Dr Penn Taylor's Masters and PhD theses have not been published. They address the issues and give a precision of about 50% for the WBT 3.1 (which was under review at the time). The accuracy was not determined. Berkeley's review of hundreds of old WBT 3.1's may be perfectly valid as a method, but it's results rest on the platform of the validity of those tests for which there is yet no independent confirmation and very real suspicion that the results are not representative of true stove performance from an efficiency or fuel consumption point of view. The frequent warning by Aprovecho and others not to rely on WBT results (it is even written into the IWA is it not?) is itself enough to view with suspicion anything based on a review of hundreds of them made using an old, uncorrected version of the UCB test, not so? That is what Tom Miles was asking. What exactly were they trying to show? Regarding confirmation of more recent tests, there is a bit of a problem with the WBT 4.1.2 because it appears to be getting some undocumented changes made anonymously. I wrote to several people asking who is actually controlling this (signing off on suggested changes). I have no name yet. The reason I asked is because the current 19 Jun 2012 version has at least two changes that were not present earlier in the month but the version number is the same. One change I noticed is that the order of the test sheets reads Test 2, Test 1, Test3 instead of Test 1, Test 2, Test 3. The other is the introduction of an error in the formula in cell i30 of the Results tab that materially affects the CoV for the burn rate which may lead to a set of tests being considered valid when they are not (as far as some labs are concerned - not all labs use the CoV). Jim Jetter mentioned another one so that is three just on that version. Until the process of correcting the WBT is formalised the sorts of problems described above and in previous communications will continue to dog the stove community. In the meantime it is perfectly reasonable for stove projects to develop and use alternative test methods that are scientifically sound, properly documented and independently validated. Regards Crispin
_______________________________________________ Stoves mailing list to Send a Message to the list, use the email address [email protected] to UNSUBSCRIBE or Change your List Settings use the web page http://lists.bioenergylists.org/mailman/listinfo/stoves_lists.bioenergylists.org for more Biomass Cooking Stoves, News and Information see our web site: http://www.bioenergylists.org/
