Re: [Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Peter, Thank you for your in depth, forthright and professional response. You have answered my questions. And you have cleared up a lot of confusion I had in my mind. (and I read ditto as i dont know) I appreciate the extra information. :) This is a top post and there is nothing inline so i snipped it. cheers, steve On 03/10/2012 06:03, Peter Vessenes wrote: Oops, I accidentally didn't reply-all to Steve. I am ccing my detailed response to steve to the list since I think people are wondering about how the Foundation fits in; trash the e-mail if you don't really care. :) On Tue, Oct 2, 2012 at 10:01 PM, Peter Vessenes pe...@coinlab.com wrote: -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQbGKXAAoJEFvEB9dQFvtQ5QcIAJR0RlkjUb5Xm09eo7wjV2QV IYXbyOBx6bZWox0wxmbygpPL23grEKxlehavf18Q1S6VjdtFs75K5GV83FZb9KPk YeB0hz5ht48Ig7uQ3zu7MBCaNerTst+6fQ/k5Uu6l2WKCVwk18WykGrnXMnSTbpM qONQCc4HQhOm7sgEsJD9KNp73eGZt/BG0hQAa8zLh2rHA0to8TER9nRUUx+mH0uf Cj58EXKVz5WI5+G31M/4v4UgRRv3Z7efaVbczrvhvv6DZfVkCpuD6t0kt2ZlcFCn oO1sKD05/VtC7DZxQvPEvvkikOp8onmue0xCJBvDpG269Xr55qNqGM039Yjy80c= =1NTS -END PGP SIGNATURE- -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/10/2012 17:52, Peter Vessenes wrote: I'm a big proponent of a testing project. I am very happy to hear this, however, your actual words are slightly evasive. I do not expect you to be up to speed on this. Gavin started a project called 'the bitcoin testing project' This project solicited donations, about 80 coins last time I checked. However these 80 odd coins were donated to 'the bitcoin testing project' This would seem to be an official bitcoin (both protocol and client) testing project. I signed up to work on this, and organise as much as i could of this. for various reasons I did not manage to do the testing I wanted to on 0.7 i over committed myself. Are the donations solicited for the 'bitcoin testing project) funds going to be given to 'the foundation'? Does the foundation support 'the bitcoin testing project'? does the foundation have any involvement with 'the bitcoin testing project'? I think if one could self organize that Gavin and team wanted to bless we could put up some BTC as bounties or funding. We won't have our heads around the foundation budget for a few more weeks, but self-organization is often slower than budgeting. :) Im ready to go, more or less. Please check out the links in my previous emails. I have over 400 testcases (8 platforms * 50 release tests) - Also I am not sure what you mean by bless, I take it that is a euphemism for pay? I have tried my hardest to get bettermeans to work, but it doesnt. It does show quite a lot of work that I have done though. If you were to say to me, 'steve, by monday we need end to end, requirements based testing' It would be done. (I have already spent over 4 months on this) Leaderless leadership is something I am having a hard time with, bettermeans is excellent at this. But I have found very little in regards to voting and polling that integrates with the project in an effortless way like bettermeans. I understand that the budget from the foundation is something that needs to be worked on and organised. I offer my services in this area (qa only). I would be happy to submit my cv and refs for this, if required. I am now feeling frustrated and useless. has my last 4 months of work been for nothing? it feels like it. I know I bang on about processes but they are sorted, you can only attract talent like Arklan if he has a process to follow. i feel like a broken record. tl;dr version 1 - Will donations to the 'bitcoin testing project' as started by gavin going to be given to the foundation? 2 - Is the work bill hees and myself going to be binned? 3 - I feel like I have the knowledge and drive to push this, but I cant do it on my own. 4 - Is bill or I entitled to any of the cash raised for 'the bitcoin testing project' 5 - Do I have to join the foundation to have a say in how the project (testing) is done? 6 - sorry for being so mercenary, but am I going to receive any coin for work I have done? 7 - It really probably is the time for a bitcoin-test list to appear. Is there anything I can do to make this happen? This is just my opinion, but I would like very, very much to move the current specification into unit tests so that anyone could validate their alternate bitcoin implementation. This is a lot of work, some of which has been done, much of which hasn't. have a look at the stuff in bettermeans. I personally think we can go a step further and publish guidelines (similar to RFC's and all the tests that we would do against a ref client) But I dont want to waste any more time on stuff that is going to be ignored, life is short. So, my two cents, plus an offer to bring it up at our next budgeting meeting. I accept that offer. and I really appreciate it. I have some more questions I would like you to ask in regards to QA. (Gavin and I skyped about this a while ago and we didnt really come to a resolution, weworked out the problems though ;) ) I have an exceptionally detailed qa process (based off the game certification process) - but I have gone on about this at length in previous messages. I thank you for your email and your involvement with this, but do you think we are closer to getting stuff tested? call my bluff... Not one person has asked for login details to my proposals - and i even have a bugzilla version now. I need to sleep. sorry if i rambled. nite nite, steve Peter On Mon, Oct 1, 2012 at 7:28 AM, steve st...@mistfpga.net wrote: On 01/10/2012 14:52, Arklan Uth Oslin wrote: Hi guys. So, as I mentioned on the bitcointalk.org forums thread about the foundation, I want to get involved in the QA side of bitcoin development. I've done functional testing in the video game industry for years. Nice one, I worked in games for quite a few years. (before getting into finance then pentesting) there are about 6 keen testers now. maybe we should get a bitcoin-test mailing list, where we can discuss stuff without disturbing the
Re: [Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/10/2012 03:02, Gregory Maxwell wrote: On Tue, Oct 2, 2012 at 9:15 PM, steve st...@mistfpga.net wrote: Im ready to go, more or less. Please check out the links in my previous emails. I have over 400 testcases (8 platforms * 50 release tests) - Also I am not sure what you mean by bless, I take it that is a euphemism for pay? Perhaps a bit bluntly here— but since you seem to be rather boldly insisting on getting paid: that is not what i was getting at, nor what i meant at all.I am sorry it came across this way. Im a bit drunk. (yeah that old excuse) I was only trying to make an example of the quandary that exists, in what happens to the current donations. There are far more deserving people who have a claim on that coin rather than I. I have stated on more than on occasion (written in public and verbal with gavin) that I would be willing to work for free. But I would love to be able to ditch the day job. (however i see that more as services than mining/testing) rereading what i wrote yeah, fair play i can see how you read my mail like that. it wasnt meant to :/ i feel a bit of a turnip. the tone is not what i meant. it does read pretty mercenary. With all of this testing where can I find the issues you've uncovered? Searching on your name/email in the issue tracker reports nothing, likewise I can't find anything in my email (beyond abstract discussion of testing). This is a different issue. But I see why you have raised it and I would like to address it. I personally believe you should earn bitcoins for writing testcases, executing testcases, passing or failing test cases. in that order (failing a testcase normally generates a new one therefore encouraging indepth and recorded testing.). QA should not be judged based solely on bug reports - this is unfair and will result in race to report bugs. I have worked on a few projects that have tried it. This is one reason we need workflow and testcase software, so we can measure and compensate people for their work. As I have stated in previous emails I should have, but for various excuses did not manage to run a single testcase for 0.7. but I did setup a frame work, and I know some people did do some good work on 0.7 I believe these people should be compensated from the (limited) funds the project received... something needs to happen with those coins. I am sorry you find my emails abstract (and therefore meaningless?) this is a really confusing time. i dont know if i am doing right from wrong. I am trying to lead, without leading... i have always preferred benevolent dictator. I know a number of bitcoin businesses that are really keen to work with the project, but I have no authority, noone seems to. Someone should be able to say, nice one, thanks for the vm's. maybe use $200 worth of coin to get a technet licence, that gives 10 installs of each ms OS. slipstreaming in the service packs means you can have whatever era pc you want (and the vps providers will allow various configuration of vm's...) then even if you do it by hand you can still do 6 or 7 at once... as for bug reports, give me 1 week from now (168hrs), lets see what i can do. (priority, protocol, daemon, qt ref?) but still process is more important... I still think that writing and doing testcases should get more coin than bug reports. but from what I have read, the big bounties will be paid by the foundation. im not sure if that is true or where i read it. (protocol level bugs) it feels like I am wasting peoples time,and I should get back in my box. so I will. contact me off list if you want to have a look at the various workflow stuff i have done. cheers, steve -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQa6pAAAoJEFvEB9dQFvtQ0H8H/jUjvzmsp61w1bmDnHR+KmF4 LNu7WwVLrTvrT8AHSNh3mClWvMM3muJJA7NMb2WthAgVe3jtrGimfreAlstDsObL XNEcGvU6WN1YosH0MkN7hyDl8jnrDFoiH1P5qsMecuZIxwq7Z0vCOHEJ9DPmZilW R+G8OmoGcpaeWs9VqXR6zR7Uyz69KaDAQpMRE1GTu3zQP9HSSolciy3ESeJRR9Sd yO7EcCGdQot80rOG/VIZ0wkOmzGGm1thzYzayD6Zn2eW4Hw+ME1en9ksIbXJFpSv IdgThEm7p5UuBo0jFkbX4Awrf9hfusZSEGWfhZqdASqqkSBnYqLmWF1sLprDRF8= =Iv87 -END PGP SIGNATURE- -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
And, finally, when I say Ditto to above I mean I have no idea, not nope. Double oops. -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
Hi guys. So, as I mentioned on the bitcointalk.org forums thread about the foundation, I want to get involved in the QA side of bitcoin development. I've done functional testing in the video game industry for years. I've read all the messages in this thread, but I'm left unclear how I can most effectively and quickly being helping out. Could I get a bit of a directional nudge? Arklan -- As long as there is light, the darkness holds no fear. And yet, even in the deepest black, there is life. - Arklan Uth Oslin I want to leave this world the same way I came into it: backwards and on fire. - Arklan Uth Oslin On Sat, Sep 29, 2012 at 12:26 PM, steve st...@mistfpga.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Gavin, Sorry for the delayed response, I wanted to take a couple of days to reflect on your email. On 26/09/2012 19:09, Gavin Andresen wrote: And their are other methods too. The GUI::Test package for perl will allow this to be greatly automated. (I have done this before on the localisation of photoshop.) this why we need detailed testscripts and plans. so we know what has and hasnt been done. The more boring the task the more work that needs to go into testcase development. This is the area I see as my greatest failing last time. I have a large number of virtual machines and should have at least this work. But we need very detailed testcases. with decent testplans just downloading the software, syncing the block chain, syncing an existing wallet, rescanning the blockchain and verifying the balance would cover a large number of tests. The idea behind having lots of very specific testcases is you get to see what tests have not been run. I understand your concern, however I have taken a couple of days to reflect on this and I still strongly feel that in order to make sure that this sticks, and is still useful in 1 years time we need to lay proper foundations. Those foundations are not word documents, spreadsheets, etc. they are selecting the right tools for the job. We can gain so much benefit from using 3rd party software. (bettermeans would rock if it wasnt rotting) I am sure you could do your coding work just using vi, but an sdk makes it much easier and allows you to work in a more productive manner. I have had a couple of off list emails with some testers and they also feel that it is very important to make sure we have a sound foundation (mantis is so much more than just a bug reporting tool, I see the bug reporting functionality as secondary to the main test run functionality - but it doesnt have to be mantis based, we do need workflow and testcase software though - and proper software for this is much better than just a massive google doc.) however I am checking out some other software that has been recommended. It will be very hard to change 'the process' once we have something we are used too (just look at the current resistance) I promise nothing will change for the dev team. But test does need other tools, and processes. If you feel that strongly that I am going about this the wrong way, I am happy to step back and let someone else sort it out (I will still do all the testing I possibly can). I would feel that this would be a real shame and we have the chance to setup requirements to functionality to tests all with traceability. why not do it right from the start? I will open up my vps' somepoint over the next few days and you can see what I mean. I will setup a fake git project, and sort out the interactions. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQZz1pAAoJEFvEB9dQFvtQRLkIAJtPCkW1R9vmMPY9u4o+ET1t w4pV/+W2PXo2p86HnljCIPLV/cua/1I/EJp7XR7s145Nj4KZUbzHGhvUUmwDOHW2 TGvJs+HO1bjsJfh4pWEb6PXcW3TguZxZSt5/rBAAI/5BeomSuRcZOdoV87D1xnK8 TSlgaseWrJcpKLO30/FQA3QnH/bjJ4OBmtHp8WaOtSnfww9Zbb5VYca37O15c2U4 2d0RUunDg1w2kRbkKjztxr3YasSOX+07Uvj4d5Lw7zgA0U93krNWVT1Ypo94dNJ7 6SyKi30UuqDdJ9XxZrMB/LBVNGOLlIBNWL++ocu5GFnOn9pnw57ZMBZM5g6YDpo= =ekQ/ -END PGP SIGNATURE- -- How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y?
Re: [Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/10/2012 14:52, Arklan Uth Oslin wrote: Hi guys. So, as I mentioned on the bitcointalk.org forums thread about the foundation, I want to get involved in the QA side of bitcoin development. I've done functional testing in the video game industry for years. Nice one, I worked in games for quite a few years. (before getting into finance then pentesting) there are about 6 keen testers now. maybe we should get a bitcoin-test mailing list, where we can discuss stuff without disturbing the dev team. I've read all the messages in this thread, but I'm left unclear how I can most effectively and quickly being helping out. Could I get a bit of a directional nudge? Great question... for me I feel structure is the most important thing to sort out first. However we desperately need detailed testcases for the release of a new version. - Not too much on the change log stuff, more on the noddy stuff (as gavin points out below), downloading and making sure it works on a non dev machine, make sure the wallet isnt overwritten, etc.) doing games qa I imagine this would be an ideal place for you to start. I have a MSDN and TechNet licence so if you need some reference ms virtual machines I can help you out. However we need some testcase software. Please check out what was done on bettermeans for the stuff I was planning out... It details everything from recompense and testcases. bettermeans kinda died a death though... check out: Bitcoin over all- https://secure.bettermeans.com/projects/4180/wiki/Page_index discussion https://secure.bettermeans.com/projects/4180/boards 0.7 https://secure.bettermeans.com/projects/4256/boards and https://secure.bettermeans.com/projects/4256/wiki I still have the testcases, but until we get some proper testcase software I am loathed to publish them in a half arsed format. (they worked well on bettermeans, then just vanished one day...) what testcase software are you familiar with? apart from that, what do you feel you can do for the project? how long have you been involved in bitcoin? It may well be worth reading up all the dev stuff on the wiki so you can get you head around how the bitcoin protocol is different from the daemon and qt client. What do you think you can and will enjoy doing? What is your skill set in regard to networking, crypto and operating systems. (not that you need any, in any we still want and need you. :) there really is room for you to do whatever role you want, and as little or as much as you want - however funding is now a very tricky issue. so much so that I am not sure I want anything to do with it(distribution of coin based on work.). - I just paid for some logo spec work out of my own pocket (for example). I have some testers i know irl who are willing to work for coin. NOTE: This response has nothing to do with the bitcoin foundation. I am not a member of the foundation. I do not speak for them or even probably with them. I am still trying to work out how much qa the foundation should be responsible for, and/how it is supposed to work. I think the games cert process would be ideal for this. This however this a discussion that probably wont have my involvement. (personally I believe that the foundation should publish requirements with example code and testcases for each aspect of the reference client. (on reference platforms - I do not expect many to agree with this though) As a side note, what happens to the donations to the bitcoin testing project? do they get moved over to the foundation? this question is bigger than this email. as far as I know they are all on an address Gavin holds. Actually I would like to be involved in any discussions that would impact QA, does this mean I need to join the foundation or just go lone wolf? tbh I dont really understand foundations. I always thought they were just a tax dodge. Sorry for the long message. :) Arklan -- As long as there is light, the darkness holds no fear. And yet, even in the deepest black, there is life. - Arklan Uth Oslin I want to leave this world the same way I came into it: backwards and on fire. - Arklan Uth Oslin On Sat, Sep 29, 2012 at 12:26 PM, steve st...@mistfpga.net wrote: Hi Gavin, Sorry for the delayed response, I wanted to take a couple of days to reflect on your email. On 26/09/2012 19:09, Gavin Andresen wrote: And their are other methods too. The GUI::Test package for perl will allow this to be greatly automated. (I have done this before on the localisation of photoshop.) this why we need detailed testscripts and plans. so we know what has and hasnt been done. The more boring the task the more work that needs to go into testcase development. This is the area I see as my greatest failing last time. I have a large number of virtual machines and should have at least this work. But we need very detailed testcases. with decent testplans just
Re: [Bitcoin-development] Bitcoin Testing Project
I'm a big proponent of a testing project. I think if one could self organize that Gavin and team wanted to bless we could put up some BTC as bounties or funding. We won't have our heads around the foundation budget for a few more weeks, but self-organization is often slower than budgeting. :) This is just my opinion, but I would like very, very much to move the current specification into unit tests so that anyone could validate their alternate bitcoin implementation. This is a lot of work, some of which has been done, much of which hasn't. So, my two cents, plus an offer to bring it up at our next budgeting meeting. Peter On Mon, Oct 1, 2012 at 7:28 AM, steve st...@mistfpga.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/10/2012 14:52, Arklan Uth Oslin wrote: Hi guys. So, as I mentioned on the bitcointalk.org forums thread about the foundation, I want to get involved in the QA side of bitcoin development. I've done functional testing in the video game industry for years. Nice one, I worked in games for quite a few years. (before getting into finance then pentesting) there are about 6 keen testers now. maybe we should get a bitcoin-test mailing list, where we can discuss stuff without disturbing the dev team. I've read all the messages in this thread, but I'm left unclear how I can most effectively and quickly being helping out. Could I get a bit of a directional nudge? Great question... for me I feel structure is the most important thing to sort out first. However we desperately need detailed testcases for the release of a new version. - Not too much on the change log stuff, more on the noddy stuff (as gavin points out below), downloading and making sure it works on a non dev machine, make sure the wallet isnt overwritten, etc.) doing games qa I imagine this would be an ideal place for you to start. I have a MSDN and TechNet licence so if you need some reference ms virtual machines I can help you out. However we need some testcase software. Please check out what was done on bettermeans for the stuff I was planning out... It details everything from recompense and testcases. bettermeans kinda died a death though... check out: Bitcoin over all- https://secure.bettermeans.com/projects/4180/wiki/Page_index discussion https://secure.bettermeans.com/projects/4180/boards 0.7 https://secure.bettermeans.com/projects/4256/boards and https://secure.bettermeans.com/projects/4256/wiki I still have the testcases, but until we get some proper testcase software I am loathed to publish them in a half arsed format. (they worked well on bettermeans, then just vanished one day...) what testcase software are you familiar with? apart from that, what do you feel you can do for the project? how long have you been involved in bitcoin? It may well be worth reading up all the dev stuff on the wiki so you can get you head around how the bitcoin protocol is different from the daemon and qt client. What do you think you can and will enjoy doing? What is your skill set in regard to networking, crypto and operating systems. (not that you need any, in any we still want and need you. :) there really is room for you to do whatever role you want, and as little or as much as you want - however funding is now a very tricky issue. so much so that I am not sure I want anything to do with it(distribution of coin based on work.). - I just paid for some logo spec work out of my own pocket (for example). I have some testers i know irl who are willing to work for coin. NOTE: This response has nothing to do with the bitcoin foundation. I am not a member of the foundation. I do not speak for them or even probably with them. I am still trying to work out how much qa the foundation should be responsible for, and/how it is supposed to work. I think the games cert process would be ideal for this. This however this a discussion that probably wont have my involvement. (personally I believe that the foundation should publish requirements with example code and testcases for each aspect of the reference client. (on reference platforms - I do not expect many to agree with this though) As a side note, what happens to the donations to the bitcoin testing project? do they get moved over to the foundation? this question is bigger than this email. as far as I know they are all on an address Gavin holds. Actually I would like to be involved in any discussions that would impact QA, does this mean I need to join the foundation or just go lone wolf? tbh I dont really understand foundations. I always thought they were just a tax dodge. Sorry for the long message. :) Arklan -- As long as there is light, the darkness holds no fear. And yet, even in the deepest black, there is life. - Arklan Uth Oslin I want to leave this world the same way I came into it: backwards and on fire. - Arklan Uth Oslin On Sat, Sep 29,
Re: [Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Matt, Glad to have another ninja onboard :) On 25/09/2012 21:41, Matt Corallo wrote: Although Jenkins may not be the best system, we already have jenkins and pull-tester (which is a dumb python script I wrote to test all incoming pull requests from github). I have never heard of jenkins before. I need to do some more digging. is this the right thing? https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin Mantis on the other hand, I know exceptionally well. I hate duplication of work/data unless absolutely necessary. I will check jenkins out (just out of interest what is it actually meant to do? the website implies framework, but not what its for) So, currently there are 4 potential places for bugs to be reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 - forum (bitcointalk...) 5? - is there still the ability to add bugs via sourceforge? Adding to this doesnt make sense. Each one of these reporting methods is for a different thing. I am not seeking to replace these (or even unify them) I am looking for software that will take testcases and bug reports against them [and allow for test campaigns]. Mantis is so flexible and industry standard and if the jenkins plugin works... then we can keep things as they are until they fit into better places. The reason I am so behind mantis as the backbone is it works with more or less anything, and can easily modded to work with whatever people are most comfortable with - however it is exceptionally powerful and has had a constant stream of workflow improvements over the past few years. They both run the same set of scripts, namely those at https://github.com/TheBlueMatt/test-scripts (its pretty basic right now, but since it is on github, I was hoping someone would find the inspiration to add to it). I will check it out. I wrote a very basic script that wikified the changelog, and linked to the changes and created wiki pages for the testcases. have you seen the stuff I put on bettermeans? bits keep vanishing then re appearing. This is the outline of the testing that I setup for 0.7 https://secure.bettermeans.com/projects/4256/wiki I dont really care if we keep using jenkins, but I figure we might as well keep all the tests in one place? Yes, I would love to unify all build testing and testcases into one place. I am still on the fence as to including unit tests into this. However I do see 3 distinct type of testcases 1 - requirements based testcases (requirements based off the current block chain rules - these are edge cases and known interoperability issues) 2 - Acceptance based testcases - these are testcases that should be run for every build. Check out the General Acceptance Tests in the wiki link for examples and testcases 3 - Testcases for reference implementations of things (like multisig - i see these working like the /test folder when you install a new perl module) These three things alone are a massive task. and they still wont cover everything. I would like to get the workflow so that people can sponsor or donate to a specific campaign (eg a new feature is implemented, people want it tested so can donate just for that campaign [developing testcases, structure, requirements, etc]) Once this is done, I will get to do some exciting stuff (like writing fuzzers, automation, etc) unfortunately I do not know python, only perl. Anyway, I'm all for more testing (I'm always complaining about how we need more tests for stuff...). Nice, I love testing. I think we will get on :) And I would rather go for interoperability between testing rather than rewriting it all. Cheers, steve Matt On Tue, 2012-09-25 at 19:32 +0100, steve wrote: Hi All, After the failure to get any real testing done for the 0.7 release (all of which is my fault) I have decided to rejig things. I am heavily into test driven development, and I have a strong background in requirements management, and automation. I want to leave bettermeans behind, maybe we might be able to keep the voting aspect of it, and link it into mantis. So, what I have been doing over the past few weeks is developing a rudimentary requirements set, basic requirement tracking, tests to prove/stress the requirements. The next most important thing is to get release/acceptance tests done - these primarily focus on new stuff doesnt break old (ie lose a wallet, etc) and needs no special requirements. To this end I have installed various opensource applications (mantis, salomeTMF, bugzilla, etc) and am currently evaluating the best workflow process. This was a much longer post, but decided against it. :) So, what I want to know is who wants to be a part helping me out with all this? I am finalising the workflow flow diagrams and they should be ready for inspection soon. Anyone interested in helping out/reviewing processes? even if it is just some encouragement, it is all
Re: [Bitcoin-development] Bitcoin Testing Project
Steve, So, currently there are 4 potential places for bugs to be reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 - forum (bitcointalk...) 5? - is there still the ability to add bugs via sourceforge? Currently github is the authoritative place to report issues. When someone reports a bug on the mailing list, IRC or forum, they are generally asked to make a github issue (or, someone else makes the issue for them). Failed tests are generally also reported on github, by the pull tester. We currently have 232 issues, mostly classified into categories such as Bug, Improvement, GUI, Wallet, and so on. Also it's easy to refer to github issues in commits with #123, with automatic linking. I'm not sure it is worth the effort to move to another system (especially if you need a another login etc...). But I'm probably misunderstanding what you're trying to do. Wladimir -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 26/09/2012 13:49, Wladimir wrote: Steve, Hi Wladimir, So, currently there are 4 potential places for bugs to be reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 - forum (bitcointalk...) 5? - is there still the ability to add bugs via sourceforge? Currently github is the authoritative place to report issues. When someone reports a bug on the mailing list, IRC or forum, they are generally asked to make a github issue (or, someone else makes the issue for them). Failed tests are generally also reported on github, by the pull tester. excellent, that makes things much easier. We currently have 232 issues, mostly classified into categories such as Bug, Improvement, GUI, Wallet, and so on. Also it's easy to refer to github issues in commits with #123, with automatic linking. I'm not sure it is worth the effort to move to another system (especially if you need a another login etc...). But I'm probably misunderstanding what you're trying to do. I think you might be misunderstanding a little. I am not trying to replace the current system, I need to make sure that what I do will be compatible with it (seamlessly so for the developer). I do not want this to generate extra work for the development team. However testing is a lot more than just bug reporting, dont get me wrong bug reports are important, but so is running a testcase and that testcase passing, especially if that testcase is linked to the proof of a requirement. I am trying to develop a qa environment that is conducive to testing and will allow the testers to shine in all their glory :) and we need different tools and methodologies. Git is too developer centric to be useful for organising testing. - however there is a large amount of software that is compatible with git, so the core development team only ever need to work with git. The linking between a bug, the requirement, the fix, the retest, and updating of regression testplan is vital. So is the ability to organise testing campaigns and assigning tests, work units and test relevant docs/scripts/ideas, etc. I hope this clears things up a bit? Cheers, steve Wladimir - -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYwESAAoJEFvEB9dQFvtQ/GUH/jv2c5L0OcL/kHkX/z0Yqbl/ 2IntPLdjXNKLuz0A7BMz7XfUyVmWlZrw44qxmi+Vyk5PKNBjYIidm763xHnTeJLN ULQBckYexMvan9hAyYZUOt85IpesdNgqTIsqh8f49y4roHOy8GT4M/2fhzXpnsGg G9d2m8jWGpj/kxl9qE7/WjVQC4APwBi/NiJsCrcHswgweN+zENc/Pot9YBLxAZu/ ACBUX/xFymRdaZN8P2LWBXuKx6E2WEcBdPCCWArX07wPiBlrashx9Gz6tiNzIiNq x2c4ltLzRa45AmiDtQhwqyTprz/DbyeAYO1sIsfpUxDeu9e3xTb/Zd96jfKIWI0= =iHI1 - -END PGP SIGNATURE- -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYwGsAAoJEFvEB9dQFvtQxOsIAKgBBOKHNFtoV2cN+GVqzlip yy0qiMvMTZKrraOhEw8QNNuOlB3LRchi+RDR/PvQkVfuwi/jHB2gUBzlapLoECBv EH8pgT/MO281pXzARgRSVkRYqkb3ljhQz3mEQg9RhR9h5t9g2mL3Tvppt7249Bg8 oGXPj6xmMcrbClF5qDbwQUUDGJfOo4eti0jSVD3qp2NE7QpPVQwuN5buchpoKt3P 9aJnjeZdLmuAk2RPoDaLXUFc9unT8AcnW96juD0zoVA9wKvAa6/8IZQf0mzV4iZP yiWGNOQtBZ+jyu2ixiEnvHqqG2ZmjtUVqWtjHkxYgrCyuuK2jOcTMNEWfn7SfKc= =yP7N -END PGP SIGNATURE- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
Running a concurrent Mantis tracker would be confusing and fragment the development pathway. We have an issue tracker; it's on github. What's being talked about here are two separate things. Jenkins is a continuous integration system. It can be configured to run the suite of unit tests, regression tests, and any kind of automated functional tests for every commit on github and every pull request. Github is our issue tracker. Github, and only github, is where new issues should be reported (unless it's security related, in which case an email should be sent to the core devs directly). Certainly developers should be responsible for making sure that regression tests for bugs they fix make it either into the unit tests or Matt's functional test repository. QA should hold them accountable for that (re-opening tickets for bugs that have been fixed but without regression tests). The other thing we're talking about is coordinated release testing--getting release candidates in the hands of actual users and making sure that issues are reported. This is something that can't be automated as the point really is to pick up on things that the testing suite missed. You sound more qualified than me for coming up with a process, but in the end discovered issues should be reported to github, the final repository of issues that hold up Gavin from doing a release. Just my 0.002BTC Mark On Wed, Sep 26, 2012 at 6:22 AM, steve st...@mistfpga.net wrote: I think you might be misunderstanding a little. I am not trying to replace the current system, I need to make sure that what I do will be compatible with it (seamlessly so for the developer). I do not want this to generate extra work for the development team. However testing is a lot more than just bug reporting, dont get me wrong bug reports are important, but so is running a testcase and that testcase passing, especially if that testcase is linked to the proof of a requirement. I am trying to develop a qa environment that is conducive to testing and will allow the testers to shine in all their glory :) and we need different tools and methodologies. Git is too developer centric to be useful for organising testing. - however there is a large amount of software that is compatible with git, so the core development team only ever need to work with git. The linking between a bug, the requirement, the fix, the retest, and updating of regression testplan is vital. So is the ability to organise testing campaigns and assigning tests, work units and test relevant docs/scripts/ideas, etc. I hope this clears things up a bit? Cheers, steve -- How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
On Wed, Sep 26, 2012 at 12:06 PM, Mark Friedenbach m...@monetize.io wrote: Certainly developers should be responsible for making sure that regression tests for bugs they fix make it either into the unit tests or Matt's functional test repository. QA should hold them accountable for that (re-opening tickets for bugs that have been fixed but without regression tests). As a goal or general principle, this is agreeable. But slavish attention to this will only get ignored. There is finite developer resources, and regression tests for certain types of bugs, like prickly P2P network interaction bugs or RPC API bugs, could potentially involve many days or weeks of coding, to sufficiently simulate the environment. The ability to easily, automatically and programmatically reproduce certain classes of bugs is simply out of reach right now, and nobody is going to shut down development to fix that problem. We should move towards this direction, yes, but bitcoin test cases are not always going to be as easy as writing (say) a compiler testcase. We can always use the help of a few good QA coders: simulating a P2P environment and checking the RPC API are two examples of very complicated problems that -can- be automated for testing... with a lot of work. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
There are test cases that can be automated. That's Jenkins, and those will be run automagically. Then there are tests that cannot be automated; things like Does the GUI look OK on all of the platforms that we support (Windows XP/2000/Vista/7/8, Ubuntu/Debian blah with window managers foo and bar, OSX 10.5/6/7/8). Thanks to Matt, we're doing great with automated functional test cases (can always do better, of course). We're failing on simple, boring stuff like making sure we actually run on all of the platforms that we say we run on BEFORE final release. That is where I think a QA team can add a lot of value. Steve: I'm worried you're over-designing The Process. A release acceptance test plan could be nothing more than a step-by-step checklist on a wiki page, Google Doc, or Drobox shared folder... -- -- Gavin Andresen -- How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi All, After the failure to get any real testing done for the 0.7 release (all of which is my fault) I have decided to rejig things. I am heavily into test driven development, and I have a strong background in requirements management, and automation. I want to leave bettermeans behind, maybe we might be able to keep the voting aspect of it, and link it into mantis. So, what I have been doing over the past few weeks is developing a rudimentary requirements set, basic requirement tracking, tests to prove/stress the requirements. The next most important thing is to get release/acceptance tests done - these primarily focus on new stuff doesnt break old (ie lose a wallet, etc) and needs no special requirements. To this end I have installed various opensource applications (mantis, salomeTMF, bugzilla, etc) and am currently evaluating the best workflow process. This was a much longer post, but decided against it. :) So, what I want to know is who wants to be a part helping me out with all this? I am finalising the workflow flow diagrams and they should be ready for inspection soon. Anyone interested in helping out/reviewing processes? even if it is just some encouragement, it is all greatly appreciated. Drop me an email if you want access to the current setup and help me review the different software for the bitcoin workflow process. cheers, steve - -- my PGP public key is at pgp.mit.edu id: 0x5016FB50 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYfjMAAoJEFvEB9dQFvtQSmsH/R/FEdOQRB7ncTnHhaP8woLu nIiGX2DgLOWLOF9launSuTCrtVm2G56B9Dgl/BqScFxeuJGbzje7+kp7LgjtA3uy kS9DUZ1zhUfhslGP0UpVJJGX6Yfk8GbQ4nUcuL1VTv6nSZXWP2EvLMDPpRgKwyi5 z1FiyBg2A3Kg3Er+VmHPmpI0zZAGB5ytaenUp4xXGhL7Nk66i5X0twVr51xlEm0L zKCDXHzWTvNNlT7TzMjIxShJ/EcgCI1r6tVD3T+2e9QeVm0QNw3xeNUkMxKn+ul8 d1v1OxJbHD1CsNqW+XgVvFE2SJReizaHNOFwrqcpVCp7bABnWAB5eyTzB9B9IX8= =di5x -END PGP SIGNATURE- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
Although Jenkins may not be the best system, we already have jenkins and pull-tester (which is a dumb python script I wrote to test all incoming pull requests from github). They both run the same set of scripts, namely those at https://github.com/TheBlueMatt/test-scripts (its pretty basic right now, but since it is on github, I was hoping someone would find the inspiration to add to it). I dont really care if we keep using jenkins, but I figure we might as well keep all the tests in one place? Anyway, I'm all for more testing (I'm always complaining about how we need more tests for stuff...). Matt On Tue, 2012-09-25 at 19:32 +0100, steve wrote: Hi All, After the failure to get any real testing done for the 0.7 release (all of which is my fault) I have decided to rejig things. I am heavily into test driven development, and I have a strong background in requirements management, and automation. I want to leave bettermeans behind, maybe we might be able to keep the voting aspect of it, and link it into mantis. So, what I have been doing over the past few weeks is developing a rudimentary requirements set, basic requirement tracking, tests to prove/stress the requirements. The next most important thing is to get release/acceptance tests done - these primarily focus on new stuff doesnt break old (ie lose a wallet, etc) and needs no special requirements. To this end I have installed various opensource applications (mantis, salomeTMF, bugzilla, etc) and am currently evaluating the best workflow process. This was a much longer post, but decided against it. :) So, what I want to know is who wants to be a part helping me out with all this? I am finalising the workflow flow diagrams and they should be ready for inspection soon. Anyone interested in helping out/reviewing processes? even if it is just some encouragement, it is all greatly appreciated. Drop me an email if you want access to the current setup and help me review the different software for the bitcoin workflow process. cheers, steve -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
on 09/25/2012 02:32 PM steve said the following: Anyone interested in helping out/reviewing processes? even if it is just some encouragement, it is all greatly appreciated. not enough time in the day for me to seriously help out, but since you asked, here's some encouragement. :) more testing == good. signature.asc Description: OpenPGP digital signature -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
This is definitely worth doing and I wish you every encouragement. For my part I'm working on a different area of the Bitcoin ecosystem and that is taking up all my time so I can only cheer you on from the sidelines. On 25 September 2012 21:49, Daniel F nanot...@gmail.com wrote: on 09/25/2012 02:32 PM steve said the following: Anyone interested in helping out/reviewing processes? even if it is just some encouragement, it is all greatly appreciated. not enough time in the day for me to seriously help out, but since you asked, here's some encouragement. :) more testing == good. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
I think it's a great initiative Steve, On Tue, Sep 25, 2012 at 10:41 PM, Matt Corallo bitcoin-l...@bluematt.me wrote: They both run the same set of scripts, namely those at https://github.com/TheBlueMatt/test-scripts (its pretty basic right now, but since it is on github, I was hoping someone would find the inspiration to add to it). The pulltester and Jenkins are really useful, I think it's best to keep these and improve them if needed. I dont really care if we keep using jenkins, but I figure we might as well keep all the tests in one place? Which brings me to another issue: we desperately need a page that links all the developer resources for bitcoin dev: - link to my doxygen docs (https://dev.visucore.com/bitcoin/doxygen/) - bluematt's pulltester and jenkins - any testing dashboards Steve is going to add But also: - how to check out the source code, how to find the build instructions for your platform - contribution guidelines - where to ask w/ development problems (this mailing list, #bitcoin-dev) - where to begin, which bugs to solve first ... Maybe on bitcoin.org, but at least easy to find for people that want to contribute, not just some deeply nested wiki page. Ie, something like https://www.mozilla.org/en-US/contribute/ http://www.ogre3d.org/developers ... I'm willing to write this. But I know these kinds of proposals always end in a big discussion about what should be and what should not be on bitcoin.org, however we should be a bit pragmatic here. Wladimir -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
