Dear Tom and Dean
Tom asks: >Do the tests mean anything? The tests give calculated results. Numbers. The interpretation of those numbers is what matters as much as the numbers themselves. But first the numbers have to be valid. Each metric in a test has a precision attached to it. For example you cannot get a 0.1% precision from a gas cell that reads to 2.5% precision by measuring 25 times. That is basic science. You can nail down where the centre of the error bar is, but all readings are still ±2.5%. When all the actions and readings in a test are rated in this manner, you get an overall precision. According to Penn Taylor the WBT precision is about ±50%. I have found a number of systematic errors additional to the ones he pointed out in his Thesis which increase it to about ±80% for most stoves, ±200% for TLUD gasifiers. So before we go creating averages of final outputs, we need to be clear that a complicated test with imprecise steps cannot give precise results. This still says nothing about accuracy. Accuracy asks, Is the result is close to the real result? The real fuel consumption, for example, is determined by running a series of tests and taking the one in the middle where the old fuel and the char and the ash are dealt with on a daily basis and the amount of new fuel required to keep the stove going each cooking session is assessed. This straightforward approach removes all sorts of systematic errors. Measured carefully, such a method can give a result within perhaps 2% of the truth. Now compare the result of a standard WBT result with that result. There is always a large difference. That difference shows the accuracy or lack of it in the WBT. Tom wrote> We should be at the point where we have large numbers of tests for each condition to arrive at performance numbers rather than having to rely on averages of averages of data generated by using outdated versions of the testing protocols. Averages: So you noticed that too? Good. Taking three tests and creating an average of them gives another number. The error bar on those numbers is not reduced in size by the averaging process. Comparing one set of averaged numbers with another set of averaged numbers is an invalid method of demonstrating the precision or accuracy of the answers given. This is basic statistics. I was surprised to see Berkeley doing this in the report. They should get a red X and be sent back to do it properly. Tom wrote: >If there are still flaws in the testing protocol does the whole stack of cards fall down? Of course it does. It is well known that the results from the UCB-WBT 3.1 are not a valid reporting of fuel consumption or efficiency of stoves. That is why the exercise to correct the method was started by ETHOS, and it got started when Berkeley realised it. I say well known because it was discussed here 4 years ago in detail and hundreds or thousands of people in the stove community are aware that the WBT 4.1.2 was created to try to correct the most significant errors in the 3.1 version. How then can any general statement be based on hundreds of 3.1 version tests? Have the result been corrected even to the 4.1.2 stage? Take for example the calculation of the fuel used to complete a 5 lite WBT which is near the bottom of the Results page. It reports a mass of dry fuel which it indicates is the consumption of fuel needed to boil then simmer 5 litres of water for 45 minutes. (To be clear, the EPA lab does not use this metric). The figure is a concatenation of several conceptual errors and one invalid mathematical step. The final number is often used to compare stove performance against Aprovechos 850 g benchmark for an improved stove. There are many examples published on line. But the number is completely invalid as a measure of fuel consumption. Just look at the formulas. Any small change in the mass of char remaining after simmering creates a large change in the final number well in excess of any possible change in the fuel consumed to create that change in char mass. That is a hint there is something terribly wrong. These days we are seeing stove test results that are patently impossible. Tom wrote: >What is the statistical probability of changing the outcome if we find that we have to change one element the protocol (e.g. assumption about charcoal) to make it more accurate? The call is I guess to fix it. A lot of us have had a go at fixing it by pointing out what the conceptual and methodological and statistical problems are. Who is listening and who is not? Dean, I want to run this one by you: The WBT says that the quantity of water boiled at the end of phase 1 and 2 is the water remaining in the pot, and after the simmering phase it is the mass of water remaining in the pot. You were always very clear on this approach and always rejected the alternative. The alternative is that when you put in 5 litres of water and bring it to a boil, you have boiled 5 litres. If there is 100 cc of water missing, it has been boiled away otherwise it would not be missing (obviously). So the fuel consumed should be divided by which figure? The 5 litres that were boiled or the 4.9 that remains? You and I have disagreed about this. The error created in the WBT 5 liter benchmark number is mostly caused by choosing to use the final mass instead of the initial mass, three times in a row. That is what I mean by a concatenation of errors. In order to correct this basic error (Specific fuel consumption to boil) will require giving up the concept that the missing 100 g of water was not boiled by the fuel that was burned and that the volume boiled is always 5 litres. Is there a willingness to accept this? If the correct formula is applied the 5 litre Benchmark number doubles for some stoves and does not change at all for others. This shows it is a systematic error that cannot be mapped to some other performance assessment method because it applies to each stove to a different degree. A doubling is a pretty major correction if you were to apply it to everything in the Berkeley analysis document. Tom asks: >What is the impact on our understanding of stove performance and what stoves we need to improve, or abandon? The impact is massive! Tom asks: >Another important question: do failures in stove programs have anything to do with stove technology or are they due to problems with the implementing organizations? Most stove programmes start off assuming the solutions already exist and the thing to do is to find out which ones are best then roll them out. Best has often turned out to be a lab evaluation based on some or other WBT containing all or most of the errors in the UCB-WBT. The fact is that these WB Tests cant meaningfully tell the difference between stoves. The CCT is a simplified field version that as I pointed out earlier today does not measure the amount of fuel consumed. It calculates the dry fuel equivalent of the energy used in the thermal efficiency calculation. So the field test cant tell the difference between stoves either. Good grief! Yet we still have proclaimed documents lauding both as valid methods to be used for determining what to promote. Tom asks: > Are there things we can do to strengthen our stove organizations? For a start they should start asking for due diligence checks on their methods! Do you think Health organisations can spend a couple of million $ on a survey using a method that has not been checked for suitability and precision, accuracy and appropriate statistical method? Why is it different with stoves? >This is going to take a lot of beers. . . In the circumstances a tempting prospect even to an abstainer Regards Crispin Tom From: Stoves [mailto:[email protected]] On Behalf Of Dean Still Sent: Friday, November 23, 2012 9:21 AM To: Discussion of biomass cooking stoves Cc: [email protected] Subject: Re: [Stoves] ETHOS 2013: Where is the New Data on Stove Performance in the Field? Hi Tom, Great suggestions for ETHOS! A Stove Performance Inventory was released this week with both lab and field data funded by the Global Alliance. "This Stove Performance Inventory, developed by the Berkeley Air Monitoring Group in partnership with the Alliance and with funding from Environment Canada, contains data from over 600 sets of cookstove performance tests. A detailed report <http://www.cleancookstoves.org/resources_files/stove-performance-inventory- pdf.pdf> is available in the Resources section of the Alliances website." Best, Dean On Fri, Nov 23, 2012 at 8:36 AM, Tom Miles <[email protected]> wrote: Mark, You have listed a number of standards and protocol topics. Now that GACC exists have we made any gains on the science side? Where is the data? We should be generating good field data by now. I dont want to just hear reports about what people have decided in political committees that nobody seems to be able to attend or vote in. I would like to see a critical review of the testing protocols and methods with regard to the key metrics and emissions. For example, the protocols to date have made assumptions about heating values of fuels and residues ash and charcoal that I have always felt were just placeholders until someone with larger research budgets could validate them. Determining the amount of energy left in the ash or charcoal is a good example. I often get asked what value to use for the remaining ash/charcoal. I havent seen a test where the remaining charcoal/ash has been directly tested for ash content and heating value. Testing the benchmarks. Do the benchmarks that were derived several years ago make sense now that we have improved tools for measuring stove performance or do we get the same numbers because were using the same tools? Do the benchmarks tell us anything about stove performance in the field? Do stoves preform in the field in the same relative way they are shown to perform in the lab or are some stoves much better than others (or much worse) when they are used in the field? Are the stoves designed to the test (e.g. WBT) or to the use in the field? Do some stoves perform best when they are tested in the lab and fail in the field? Or, are we even testing for this? Field applications. Are our tools and metrics of any use for improving stove performance in actual use? If so then how are projects in developing countries using these tools to improve their fuel use and health? How do local, nation, or regional stove projects use these tools to improve their stoves, or do they just ignore them? QA/QC. When a program buys container loads of stoves how do we know that they perform within the expectations created by the test results? Do any of the manufactured stove suppliers test the quality and performance of their stoves on a regular basis? Is there any monitoring? How has testing been used for different fuels? I was inspired this week by a photo from Mexican which a construction worker was using an LPG burner in an eCocina stove (Stove Team International) because it substantially reduced his LG use. Can we compare fuel consumption for different fuels? How good are our fuel consumption metrics? Health. We still do not have proven direct correlation between stove emissions and heath. Most of the data seems to be recycled. Are there new health studies? Has GACC and the many supporting organizations funded any of the fundamental health studies that every year Jay Smith tells us are lacking? These are some of the questions that I would like to see addressed at ETHOS. Looking forward to another productive ETHOS. Thanks Tom T R Miles Technical Consultants, Inc. [email protected] www.trmiles.com www.stove.bioenergylists.org From: [email protected] [mailto:[email protected]] On Behalf Of Bryden, Kenneth [M E] Sent: Monday, October 22, 2012 2:05 PM To: Discussion of biomass cooking stoves; ETHOS - Listserve Subject: [Ethos] ETHOS 2013 All, It's that time of year again! We're starting to get the activities together together for the ETHOS conference. The conference will be January 25 - 27, 2013 in Seattle. Proposed discussion topics include - Update on the Global Alliance and their activities - Standardizing Reporting on IWA Indicators - Stove Performance Inventory, Sharing Public Data, and Establishing Common Data Formats - Update on ISO Process - Updates on Protocol Developments (including possibly charcoal, plancha, batch-fed, durability, finalizing WBT from public comment period) Let us know what else you would like to talk about. A lot is happening and I'm sure it will be as exciting as ever. I'd appreciate having your ideas on topics and panels by November 15. Abstracts for papers and talks are due January 1. For more details, to register, and to submit your abstract the conference web site is http://www.vrac.iastate.edu/ethos/conference.php. Please send your ideas directly to Dean and I. Best regards Mark
_______________________________________________ Stoves mailing list to Send a Message to the list, use the email address [email protected] to UNSUBSCRIBE or Change your List Settings use the web page http://lists.bioenergylists.org/mailman/listinfo/stoves_lists.bioenergylists.org for more Biomass Cooking Stoves, News and Information see our web site: http://www.bioenergylists.org/
