Re: [Bitcoin-development] On-going data spam

[steve]

On 09/04/2013 15:39, Caleb James DeLisle wrote:
> Agreed on the legality aspect but another case which is worth considering is
> what anti-virus software might do when certain streams of bytes are sent 
> across
> the tcp socket or persisted to disk.

Do you mean firewalls or something like snort or other deep packet
inspection for the tcp sockets statement? I dont see much of an issue
with either.

set up your own private testnet and have a play with this

http://www.eicar.org/83-0-Anti-Malware-Testfile.html

The eicar test virus.

> Perhaps worth contacting an AV company and
> asking what is the smallest data they have a signature on.

I have tried a few ways of getting the eicar string into the blockchain
(on a private testnet) and getting it flagged by AV, however it is a bit
tricky (the getting it flagged bit). and tbh you would exclude the
bitcoin directory and runtime from antivirus scans so i stopped bothering.

I am making vague assumptions about using windows with antivirus. (and
linux for deep packet inspection, but the idea is the same whatever.)

I found no greater attack surface area (in the blockchain) than
cookies... thinking about it a bit more, there is a bit more potential
as a bounce pad/egg drop location but not much - no heap spraying as
such, or d/c tors, or heap header structs, etc. Im sure someone is sure
to come up with something very clever tho. just not me.

cheers,

steve

> 
> Thanks,
> Caleb
> 
> 
> On 04/09/2013 06:42 AM, Mike Hearn wrote:
>> OK, as the start of that conversation is now on the list, I might as well 
>> post the other thoughts we had. Or at least that I had :)
>>
>> It's tempting to see this kind of abuse through the lens of fees, because we 
>> only have a few hammers and so everything looks like a kind of nail. The 
>> problem is the moment you try to define "abuse" economically you end up 
>> excluding legitimate and beneficial uses as well. Maybe Peters patch for 
>> uneconomical outputs is different because of how it works. But mostly it's 
>> true. In this case, fees would never work - Peter said the guy who uploaded 
>> Wikileaks paid something like $500 to do it. I guess
>> by now it's more like $600-$700. It's hard for regular end users to compete 
>> with that kind of wild-eyed dedication to "the cause".
>>
>> The root problem here is people believe the block chain is a data structure 
>> that will live forever and be served by everyone for free, in perpetuity, 
>> and is thus the perfect place for "uncensorable" stuff. That's a reasonable 
>> assumption given how Bitcoin works today. But there's no reason it will be 
>> true in the long run (I know this can be an unpopular viewpoint).
>>
>> Firstly, legal issues - I think it's very unlikely any sane court would care 
>> about illegal stuff in the block chain given you need special tools to 
>> extract it (mens rea). Besides, I guess most end users will end up on SPV 
>> clients as they mature. So these users already don't have a copy of the 
>> entire block chain. I don't worry too much about this.
>>
>> Secondly, the need to host blocks forever. In future, many (most?) full 
>> nodes will be pruning, and won't actually store old blocks at all. They'll 
>> just have the utxo database, some undo blocks and some number of old blocks 
>> for serving, probably whatever fits in the amount of disk space the user is 
>> willing to allocate. But very old blocks will have been deleted. 
>>
>> This leads to the question of what incentives people have to not prune. The 
>> obvious incentive is money - charge for access to older parts of the chain. 
>> The fewer people that host it, the more you can charge. In the worst case 
>> scenario where, you know, only 10 different organizations store a copy of 
>> the chain, it might mean that bootstrapping a new node in a trust-less 
>> manner is expensive. But I really doubt it'd ever get so few. Serving large 
>> static datasets just isn't that expensive. Also, you
>> don't actually need to replay from the genesis block to bring up a new code, 
>> you can copy the UTXO database from somewhere else. By comparing the 
>> databases of lots of different nodes together, the chances of you being in a 
>> matrix-like sybil world can be reduced to "beyond reasonable doubt". Maybe 
>> nodes would charge for copies of their database too, but ideally there are 
>> lots of nodes and so the charge for that should be so close to zero as makes 
>> no odds - you can trivially undercut someone by
>> buying access to the dataset and then reselling it for a bit

Re: [Bitcoin-development] 0.7.1 release candidate 1 ready for testing

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


Hi,

On 11/10/2012 16:46, Gavin Andresen wrote:
> Any progress on a release candidate QA sanity testing plan?

essentially yes - in all seriousness, I have come around to your way
of thinking, it is not worth arguing about. we end up in the same place.

:) nothing is final though. I will be opening it up to Arklan in a bit
(I have a pool match tonight so after that depending on how drunk i
get.) and to another couple of people who have emailed me off list who
would like to be involved in testing.

I have not created any new tests for the new stuff.  this is still the
noddy stuff.

It covers everything that needs to be covered for a newbe or someone
upgrading clients.

and it is in many formats the moment (from wiki to test runner) none
particularly useful (about to import into mantis, just fiddling with
the fields so it does testcases)

I have been trying to separate the tests into what can be automated,
recorded and checked by watching the video and have sanity checks
along the way (published sha and sha256 of the binaries match) -
(obviously the test is flagged to be run manually if there is any doubt)

I did take myself outside and had a little sit down and talk to
myself.  Lets not run before we can walk. I will still keep the grand
ideas, but they are on the back burner. (I got called a blue skies
thinker today, wtf is one of them? I said smiled and said thanks)

the main things we make sure we get tested right are all the GAT tests
in bettermeans.They are more protocol based and should be run against
all releases.  Lets nail that process, peer review it, retest it, then
use that as a basis for The Process. I will temper my zealousness. I
realise that I can come across as bullish or even aggressive in my
emails.  I never mean too.(unless I say I am. That is not an excuse
for my poor english language skills.)

But I still think that the people running the tests should have the
greatest say in what software is used. So once you have access to the
server and the cases feel free to install your favourite software and
see what other people think.

but we have a vague structure and workflow so it will be interesting
to see how it works out.

the git integration will be trivial.

please email me if you would like access.  I am off out in 15 mins but
I promise to have all access sorted by tomorrow.  I am only using 3
vps to show example testing and test platforms.  I would like it if we
could get a technet license with the donations so far, I use mine (and
my msdn) for work.

also check out bettermeans for the formats that I will be using.

bollocks. bollocks. bollocks

https://secure.bettermeans.com/projects/4180

bye bye bettermeans. how we never knew ye.

Gavin, can we get a bitcoin-test mailing list please.  it would be
used for discussion of testing, tools, ideas, chatter, etc.  it would
not be a place reporting bugs... This list seems very focused and I
always feel like I am disrupting things with my emails about test.  I
dont mind admining the list. I would rather be vocal and inclusive, I
dont think the dev list is the place for that.

> 
> 
> Bitcoin version 0.7.1 release candidate 1 is now available from: 
> http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.7.1/test/
>
>  This is a bug-fix minor release.
> 
> New features 
> 
> * Added a boolean argument to the RPC 'stop' command, if true sets 
> -detachdb to create standalone database .dat files before shutting
> down.
> 
> * -salvagewallet command-line option, which moves any existing
> wallet.dat to wallet.{timestamp}.dat and then attempts to salvage
> public/private keys and master encryption keys (if the wallet is
> encrypted) into a new wallet.dat. This should only be used if your
> wallet becomes corrupted, and is not intended to replace regular
> wallet backups.
> 
> * Import $DataDir/bootstrap.dat automatically, if it exists.
> 
> Dependency changes --
> 
> * Qt 4.8.2 for Windows builds
> 
> * openssl 1.0.1c
> 
> Bug fixes -
> 
> * When running -testnet, use RPC port 18332 by default.
> 
> * Better detection and handling of corrupt wallet.dat and
> blkindex.dat files. Previous versions would crash with a
> DB_RUNRECOVERY exception, this version detects most problems and
> tells you how to recover if it cannot recover itself.
> 
> * Fixed an uninitialized variable bug that could cause transactions
> to be reported out of order.
> 
> * Fixed a bug that could cause occasional crashes on exit.
> 
> * Warn the user that they need to create fresh wallet backups after
> they encrypt their wallet.
> 
> 

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQdwWxAAoJEFvEB9dQFvtQiuMIAJpdu23biq56apMO/KfqmSB0
kQPflDD2XMxTknfqzRs35/EFgL0mJ/cYfY3qTW9JOWL/cit4ieq2EA4P3uQeFFDC
NCxIuDxOzOIaZ+SzRZENXdpVWKRNP1RgUCANm3YfrJBlavr9a/om36nP1IK0W4MB
QcPXrrZvipt1xhx1G/V6NvYbZA3lTAJBFz

[Bitcoin-development] Fwd: Re: Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


I think he had a typo in the CC.  here is a forward of the email.
You will have to work out the indentations yourselves :)

-  Original Message 
Subject: Re: [Bitcoin-development] Bitcoin Testing Project
Date: Tue, 2 Oct 2012 22:01:19 -0700
From: Peter Vessenes 
To: steve 

On Tue, Oct 2, 2012 at 6:15 PM, steve  wrote:

> On 01/10/2012 17:52, Peter Vessenes wrote:
>> I'm a big proponent of a testing project.
> 
> I am very happy to hear this, however, your actual words are
> slightly evasive. I do not expect you to be up to speed on this.
> Gavin started a project called 'the bitcoin testing project' This
> project solicited donations, about 80 coins last time I checked.
> However these 80 odd coins were donated to 'the bitcoin testing
> project' This would seem to be an official bitcoin (both protocol
> and client) testing project.  I signed up to work on this, and
> organise as much as i could of this. for various reasons I did not
> manage to do the testing I wanted to on 0.7 i over committed
> myself.
> 
> Are the donations solicited for the 'bitcoin testing project)
> funds going to be given to 'the foundation'?
> 
> 
> Not as far as I know; sounds like they should go toward testing.
> 
> 
> Does the foundation support 'the bitcoin testing project'? does
> the foundation have any involvement with 'the bitcoin testing
> project'?
> 
> I personally support the idea of a testing project. I would like
> the Foundation to fund it if it can't crowdsource funding from the
> forums; sounds like so far the support hasn't been enough to get
> all the work done. The Foundation has no formal role with the
> bitcoin testing project that I'm aware of.
> 
> 
> 
> 
>> I think if one could self organize that Gavin and team wanted to 
>> bless we could put up some BTC as bounties or funding. We won't 
>> have our heads around the foundation budget for a few more
>> weeks, but self-organization is often slower than budgeting. :)
> 
> Im ready to go, more or less.  Please check out the links in my 
> previous emails. I have over 400 testcases (8 platforms * 50
> release tests) - Also I am not sure what you mean by bless, I take
> it that is a euphemism for pay?
> 
> Wow, that's awesome! I use bless to mean "Gavin saying that it
> sounds good."
> 
> 
> I have tried my hardest to get bettermeans to work, but it doesnt.
> It does show quite a lot of work that I have done though. If you
> were to say to me, 'steve, by monday we need end to end,
> requirements based testing' It would be done. (I have already spent
> over 4 months on this)  Leaderless leadership is something I am
> having a hard time with, bettermeans is excellent at this.  But I
> have found very little in regards to voting and polling that
> integrates with the project in an effortless way like bettermeans.
> 
> I understand that the budget from the foundation is something that 
> needs to be worked on and organised.  I offer my services in this
> area (qa only).  I would be happy to submit my cv and refs for
> this, if required.
> 
> I am now feeling frustrated and useless.  has my last 4 months of
> work been for nothing? it feels like it.  I know I bang on about
> processes but they are sorted, you can only attract talent like
> Arklan if he has a process to follow. i feel like a broken record.
> 
> I'm a little late to this conversation, so I don't know what to say
> in response. I will answer your questions below, though.
> 
> 
> tl;dr version 1 - Will donations to the 'bitcoin testing project'
> as started by gavin going to be given to the foundation?
> 
> 
> I don't expect so, although we'd take them if whoever is in charge
> of the testing project wants to do so. I'd expect that if the
> testing project is good and community approved and supported by the
> dev team the funding flow would go the other way, but we'll need to
> wait for budgets to get finished.
> 
> 
> 2 - Is the work bill hees and myself going to be binned?
> 
> 
> I have no idea whatsoever, I would guess that's up to you and bill
> hees and the dev team.
> 
> 
> 3 - I feel like I have the knowledge and drive to push this, but I 
> cant do it on my own.
> 
> 
> Totally understand the feeling!
> 
> 
> 4 - Is bill or I entitled to any of the cash raised for 'the
> bitcoin testing project'
> 
> 
> I have no idea what the bitcoin testing project finance situation
> is.
> 
> 
> 5 - Do I have to join the foundation to have a say in how the
&g

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Peter,

Thank you for your in depth, forthright and professional response. You
have answered my questions.  And you have cleared up a lot of
confusion I had in my mind. (and I read ditto as i dont know) I
appreciate the extra information. :)

This is a top post and there is nothing inline so i snipped it.

cheers,

steve

On 03/10/2012 06:03, Peter Vessenes wrote:
> Oops, I accidentally didn't reply-all to Steve. I am ccing my
> detailed response to steve to the list since I think people are
> wondering about how the Foundation fits in; trash the e-mail if you
> don't really care. :)
> 
> On Tue, Oct 2, 2012 at 10:01 PM, Peter Vessenes 
> wrote:
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQbGKXAAoJEFvEB9dQFvtQ5QcIAJR0RlkjUb5Xm09eo7wjV2QV
IYXbyOBx6bZWox0wxmbygpPL23grEKxlehavf18Q1S6VjdtFs75K5GV83FZb9KPk
YeB0hz5ht48Ig7uQ3zu7MBCaNerTst+6fQ/k5Uu6l2WKCVwk18WykGrnXMnSTbpM
qONQCc4HQhOm7sgEsJD9KNp73eGZt/BG0hQAa8zLh2rHA0to8TER9nRUUx+mH0uf
Cj58EXKVz5WI5+G31M/4v4UgRRv3Z7efaVbczrvhvv6DZfVkCpuD6t0kt2ZlcFCn
oO1sKD05/VtC7DZxQvPEvvkikOp8onmue0xCJBvDpG269Xr55qNqGM039Yjy80c=
=1NTS
-END PGP SIGNATURE-

--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 03/10/2012 03:02, Gregory Maxwell wrote:
> On Tue, Oct 2, 2012 at 9:15 PM, steve  wrote:
>> Im ready to go, more or less.  Please check out the links in my 
>> previous emails. I have over 400 testcases (8 platforms * 50
>> release tests) - Also I am not sure what you mean by bless, I
>> take it that is a euphemism for pay?
> 
> Perhaps a bit bluntly here— but since you seem to be rather boldly 
> insisting on getting paid:

that is not what i was getting at, nor what i meant at all.I am sorry
it came across this way. Im a bit drunk. (yeah that old excuse) I was
only trying to make an example of the quandary that exists, in what
happens to the current donations. There are far more deserving people
who have a claim on that coin rather than I.

I have stated on more than on occasion (written in public and verbal
with gavin) that I would be willing to work for free. But I would love
to be able to ditch the day job. (however i see that more as services
than mining/testing)

rereading what i wrote yeah, fair play i can see how you read my mail
like that. it wasnt meant to :/ i feel a bit of a turnip. the tone is
not what i meant. it does read pretty mercenary.

> 
> With all of this testing where can I find the issues you've
> uncovered? Searching on your name/email in the issue tracker
> reports nothing, likewise I can't find anything in my email (beyond
> abstract discussion of testing).
> 

This is a different issue. But I see why you have raised it and I
would like to address it. I personally believe you should earn
bitcoins for writing testcases, executing testcases, passing or
failing test cases. in that order (failing a testcase normally
generates a new one therefore encouraging indepth and recorded
testing.). QA should not be judged based solely on bug reports - this
is unfair and will result in race to report bugs.  I have worked on a
few projects that have tried it.

This is one reason we need workflow and testcase software, so we can
measure and compensate people for their work.

As I have stated in previous emails I should have, but for various
excuses did not manage to run a single testcase for 0.7. but I did
setup a frame work, and I know some people did do some good work on
0.7 I believe these people should be compensated from the (limited)
funds the project received... something needs to happen with those coins.

I am sorry you find my emails abstract (and therefore meaningless?)

this is a really confusing time. i dont know if i am doing right from
wrong. I am trying to lead, without leading... i have always preferred
benevolent dictator. I know a number of bitcoin businesses that are
really keen to work with the project, but I have no authority, noone
seems to. Someone should be able to say, nice one, thanks for the
vm's. maybe use $200 worth of coin to get a technet licence, that
gives 10 installs of each ms OS. slipstreaming in the service packs
means you can have whatever era pc you want (and the vps providers
will allow various configuration of vm's...) then even if you do it by
hand you can still do 6 or 7 at once...

as for bug reports, give me 1 week from now (168hrs), lets see what i
can do. (priority, protocol, daemon, qt ref?) but still process is
more important...

I still think that writing and doing testcases should get more coin
than bug reports. but from what I have read, the big bounties will be
paid by the foundation. im not sure if that is true or where i read
it. (protocol level bugs)

it feels like I am wasting peoples time,and I should get back in my
box. so I will.

contact me off list if you want to have a look at the various workflow
stuff i have done.

cheers,

steve
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQa6pAAAoJEFvEB9dQFvtQ0H8H/jUjvzmsp61w1bmDnHR+KmF4
LNu7WwVLrTvrT8AHSNh3mClWvMM3muJJA7NMb2WthAgVe3jtrGimfreAlstDsObL
XNEcGvU6WN1YosH0MkN7hyDl8jnrDFoiH1P5qsMecuZIxwq7Z0vCOHEJ9DPmZilW
R+G8OmoGcpaeWs9VqXR6zR7Uyz69KaDAQpMRE1GTu3zQP9HSSolciy3ESeJRR9Sd
yO7EcCGdQot80rOG/VIZ0wkOmzGGm1thzYzayD6Zn2eW4Hw+ME1en9ksIbXJFpSv
IdgThEm7p5UuBo0jFkbX4Awrf9hfusZSEGWfhZqdASqqkSBnYqLmWF1sLprDRF8=
=Iv87
-END PGP SIGNATURE-

--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 01/10/2012 17:52, Peter Vessenes wrote:
> I'm a big proponent of a testing project.

I am very happy to hear this, however, your actual words are slightly
evasive. I do not expect you to be up to speed on this. Gavin started
a project called 'the bitcoin testing project' This project solicited
donations, about 80 coins last time I checked.  However these 80 odd
coins were donated to 'the bitcoin testing project' This would seem to
be an official bitcoin (both protocol and client) testing project.  I
signed up to work on this, and organise as much as i could of this.
for various reasons I did not manage to do the testing I wanted to on
0.7 i over committed myself.

Are the donations solicited for the 'bitcoin testing project) funds
going to be given to 'the foundation'?

Does the foundation support 'the bitcoin testing project'? does the
foundation have any involvement with 'the bitcoin testing project'?

> 
> I think if one could self organize that Gavin and team wanted to
> bless we could put up some BTC as bounties or funding. We won't
> have our heads around the foundation budget for a few more weeks,
> but self-organization is often slower than budgeting. :)

Im ready to go, more or less.  Please check out the links in my
previous emails. I have over 400 testcases (8 platforms * 50 release
tests) - Also I am not sure what you mean by bless, I take it that is
a euphemism for pay?

I have tried my hardest to get bettermeans to work, but it doesnt.  It
does show quite a lot of work that I have done though. If you were to
say to me, 'steve, by monday we need end to end, requirements based
testing' It would be done. (I have already spent over 4 months on
this)  Leaderless leadership is something I am having a hard time
with, bettermeans is excellent at this.  But I have found very little
in regards to voting and polling that integrates with the project in
an effortless way like bettermeans.

I understand that the budget from the foundation is something that
needs to be worked on and organised.  I offer my services in this area
(qa only).  I would be happy to submit my cv and refs for this, if
required.

I am now feeling frustrated and useless.  has my last 4 months of work
been for nothing? it feels like it.  I know I bang on about processes
but they are sorted, you can only attract talent like Arklan if he has
a process to follow. i feel like a broken record.

tl;dr version
1 - Will donations to the 'bitcoin testing project' as started by
gavin going to be given to the foundation?
2 - Is the work bill hees and myself going to be binned?
3 - I feel like I have the knowledge and drive to push this, but I
cant do it on my own.
4 - Is bill or I entitled to any of the cash raised for 'the bitcoin
testing project'
5 - Do I have to join the foundation to have a say in how the project
(testing) is done?
6 - sorry for being so mercenary, but am I going to receive any coin
for work I have done?
7 - It really probably is the time for a bitcoin-test list to appear.
 Is there anything I can do to make this happen?

> 
> This is just my opinion, but I would like very, very much to move
> the current specification into unit tests so that anyone could
> validate their alternate bitcoin implementation. This is a lot of
> work, some of which has been done, much of which hasn't.

have a look at the stuff in bettermeans.  I personally think we can go
a step further and publish guidelines (similar to RFC's and all the
tests that we would do against a ref client)

But I dont want to waste any more time on stuff that is going to be
ignored, life is short.

> 
> So, my two cents, plus an offer to bring it up at our next
> budgeting meeting.

I accept that offer. and I really appreciate it.  I have some more
questions I would like you to ask in regards to QA. (Gavin and I
skyped about this a while ago and we didnt really come to a
resolution, weworked out the problems though ;) )

I have an exceptionally detailed qa process (based off the game
certification process) - but I have gone on about this at length in
previous messages.

I thank you for your email and your involvement with this, but do you
think we are closer to getting stuff tested? call my bluff... Not one
person has asked for login details to my proposals - and i even have a
bugzilla version now.

I need to sleep.  sorry if i rambled.

nite nite,

steve

> 
> Peter
> 
> 
> On Mon, Oct 1, 2012 at 7:28 AM, steve  wrote:
> 
> On 01/10/2012 14:52, Arklan Uth Oslin wrote:
>>>> Hi guys.
>>>> 
>>>> So, as I mentioned on the bitcointalk.org forums thread about
>>>> the foundation, I want to get involved in the QA side of
>>>> bitcoin development. I've done functional testing in the
>>

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 01/10/2012 14:52, Arklan Uth Oslin wrote:
> Hi guys.
> 
> So, as I mentioned on the bitcointalk.org forums thread about the 
> foundation, I want to get involved in the QA side of bitcoin 
> development. I've done functional testing in the video game
> industry for years.

Nice one, I worked in games for quite a few years. (before getting
into finance then pentesting) there are about 6 keen testers now.
maybe we should get a bitcoin-test mailing list, where we can discuss
stuff without disturbing the dev team.

> I've read all the messages in this thread, but I'm left unclear
> how I can most effectively and quickly being helping out. Could I
> get a bit of a directional nudge?

Great question... for me I feel structure is the most important thing
to sort out first.  However we desperately need detailed testcases for
the release of a new version. - Not too much on the change log stuff,
more on the noddy stuff (as gavin points out below), downloading and
making sure it works on a non dev machine, make sure the wallet isnt
overwritten, etc.) doing games qa I imagine this would be an ideal
place for you to start.  I have a MSDN and TechNet licence so if you
need some reference ms virtual machines I can help you out.

However we need some testcase software.  Please check out what was
done on bettermeans for the stuff I was planning out...

It details everything from recompense and testcases. bettermeans kinda
died a death though...

check out:

Bitcoin over all-
https://secure.bettermeans.com/projects/4180/wiki/Page_index
discussion
https://secure.bettermeans.com/projects/4180/boards

0.7
https://secure.bettermeans.com/projects/4256/boards
and
https://secure.bettermeans.com/projects/4256/wiki

I still have the testcases, but until we get some proper testcase
software I am loathed to publish them in a half arsed format. (they
worked well on bettermeans, then just vanished one day...) what
testcase software are you familiar with?

apart from that, what do you feel you can do for the project? how long
have you been involved in bitcoin?  It may well be worth reading up
all the dev stuff on the wiki so you can get you head around how the
bitcoin protocol is different from the daemon and qt client. What do
you think you can and will enjoy doing? What is your skill set in
regard to networking, crypto and operating systems. (not that you need
any, in any we still want and need you. :)

there really is room for you to do whatever role you want, and as
little or as much as you want - however funding is now a very tricky
issue. so much so that I am not sure I want anything to do with
it(distribution of coin based on work.). - I just paid for some logo
spec work out of my own pocket (for example).  I have some testers i
know irl who are willing to work for coin.

NOTE: This response has nothing to do with the bitcoin foundation.  I
am not a member of the foundation. I do not speak for them or even
probably with them. I am still trying to work out how much qa the
foundation should be responsible for, and/how it is supposed to work.
I think the games cert process would be ideal for this.  This however
this a discussion that probably wont have my involvement.  (personally
I believe that the foundation should publish requirements with example
code and testcases for each aspect of the reference client. (on
reference platforms - I do not expect many to agree with this though)

As a side note, what happens to the donations to the bitcoin testing
project? do they get moved over to the foundation? this question is
bigger than this email. as far as I know they are all on an address
Gavin holds. Actually I would like to be involved in any discussions
that would impact QA, does this mean I need to join the foundation or
just go lone wolf?

tbh I dont really understand foundations.  I always thought they were
just a tax dodge.

Sorry for the long message. :)

> 
> Arklan
> 
> -- As long as there is light, the darkness holds no fear.
> And yet, even in the deepest black, there is life. - Arklan Uth
> Oslin
> 
> I want to leave this world the same way I came into it: backwards
> and on fire. - Arklan Uth Oslin
> 
> 
> 
> On Sat, Sep 29, 2012 at 12:26 PM, steve 
> wrote:
> 
> Hi Gavin,
> 
> Sorry for the delayed response, I wanted to take a couple of days
> to reflect on your email.
> 
> On 26/09/2012 19:09, Gavin Andresen wrote:
> 
> 
> And their are other methods too.
> 
> 
> 
> The GUI::Test package for perl will allow this to be greatly 
> automated. (I have done this before on the localisation of
> photoshop.)
> 
> 
> 
> this why we need detailed testscripts and plans.  so we know what
> has and hasnt been done. The more boring the task the more work
> that needs to go into testcase development.  This is the area I see
> 

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Gavin,

Sorry for the delayed response, I wanted to take a couple of days to
reflect on your email.

On 26/09/2012 19:09, Gavin Andresen wrote:


And their are other methods too.



The GUI::Test package for perl will allow this to be greatly
automated. (I have done this before on the localisation of photoshop.)



this why we need detailed testscripts and plans.  so we know what has
and hasnt been done. The more boring the task the more work that needs
to go into testcase development.  This is the area I see as my
greatest failing last time.  I have a large number of virtual machines
and should have at least this work.  But we need very detailed
testcases.  with decent testplans just downloading the software,
syncing the block chain, syncing an existing wallet, rescanning the
blockchain and verifying the balance would cover a large number of
tests.  The idea behind having lots of very specific testcases is you
get to see what tests have not been run.



I understand your concern, however I have taken a couple of days to
reflect on this and I still strongly feel that in order to make sure
that this sticks, and is still useful in 1 years time we need to lay
proper foundations. Those foundations are not word documents,
spreadsheets, etc.  they are selecting the right tools for the job.

We can gain so much benefit from using 3rd party software.
(bettermeans would rock if it wasnt rotting)

I am sure you could do your coding work just using vi, but an sdk
makes it much easier and allows you to work in a more productive manner.

I have had a couple of off list emails with some testers and they also
feel that it is very important to make sure we have a sound foundation
(mantis is so much more than just a bug reporting tool, I see the bug
reporting functionality as secondary to the main test run
functionality - but it doesnt have to be mantis based, we do need
workflow and testcase software though - and proper software for this
is much better than just a massive google doc.) however I am checking
out some other software that has been recommended.  It will be very
hard to change 'the process' once we have something we are used too
(just look at the current resistance) I promise nothing will change
for the dev team.  But test does need other tools, and processes.

If you feel that strongly that I am going about this the wrong way, I
am happy to step back and let someone else sort it out (I will still
do all the testing I possibly can). I would feel that this would be a
real shame and we have the chance to setup requirements to
functionality to tests all with traceability. why not do it right from
the start?

I will open up my vps' somepoint over the next few days and you can
see what I mean. I will setup a fake git project, and sort out the
interactions.


-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQZz1pAAoJEFvEB9dQFvtQRLkIAJtPCkW1R9vmMPY9u4o+ET1t
w4pV/+W2PXo2p86HnljCIPLV/cua/1I/EJp7XR7s145Nj4KZUbzHGhvUUmwDOHW2
TGvJs+HO1bjsJfh4pWEb6PXcW3TguZxZSt5/rBAAI/5BeomSuRcZOdoV87D1xnK8
TSlgaseWrJcpKLO30/FQA3QnH/bjJ4OBmtHp8WaOtSnfww9Zbb5VYca37O15c2U4
2d0RUunDg1w2kRbkKjztxr3YasSOX+07Uvj4d5Lw7zgA0U93krNWVT1Ypo94dNJ7
6SyKi30UuqDdJ9XxZrMB/LBVNGOLlIBNWL++ocu5GFnOn9pnw57ZMBZM5g6YDpo=
=ekQ/
-END PGP SIGNATURE-

--
How fast is your code?
3 out of 4 devs don\\\'t know how their code performs in production.
Find out how slow your code is with AppDynamics Lite.
http://ad.doubleclick.net/clk;262219672;13503038;z?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 26/09/2012 17:06, Mark Friedenbach wrote:
> Running a concurrent Mantis tracker would be confusing and fragment
> the development pathway. We have an issue tracker; it's on github.

I think you misunderstand what I am proposing.

QA needs more than just an issue tracker. i have yet to find any
opensource software that integrates testcases, nor any method for
generating testplans, nor any method for linking testcases and plans
to requirements, that work with git.

We will need software for this (as well as workflow software) and it
is much easier to integrate this into mantis/bugzilla. They are both
so much more functional than Git.

both mantis and bugzilla have full two way functionality with git.

> 
> What's being talked about here are two separate things. Jenkins is
> a continuous integration system. It can be configured to run the
> suite of unit tests, regression tests, and any kind of automated
> functional tests for every commit on github and every pull
> request.

well 3 but okay.  Jenkins integrates both with mantis (and therefore a
testsuite, etc) and with git. I do not see why anything should be any
different.  again I am not trying to change any current process, just
develop some new ones.

> 
> Github is our issue tracker. Github, and only github, is where new
> issues should be reported (unless it's security related, in which
> case an email should be sent to the core devs directly).

You will only ever receive bug reports via git. How they are entered
should not be of concern.

There will be no space in mantis/zilla for bugs that are not related
to testcases.

> 
> Certainly developers should be responsible for making sure that
> regression tests for bugs they fix make it either into the unit
> tests or Matt's functional test repository. QA should hold them
> accountable for that (re-opening tickets for bugs that have been
> fixed but without regression tests).

I feel very strongly that developers should not do regression testing
or any signoff testing on their own code. QA should do the testing. I
am 50/50 if they should write the testcases. the QA process should
make things easier for the dev team, not generate more work for them.

> 
> The other thing we're talking about is coordinated release
> testing--getting release candidates in the hands of actual users
> and making sure that issues are reported. This is something that
> can't be automated as the point really is to pick up on things that
> the testing suite missed. You sound more qualified than me for
> coming up with a process, but in the end discovered issues should
> be reported to github, the final repository of issues that hold up
> Gavin from doing a release.

All the core development team will still use git. the extra software
is needed by test.

(And the third point was coming up with a suite of tests for 3rd party
developers to test their interoperability - this will having nothing
to do with git, or mantis. But the solution should be compatible with
mantis/zilla)

> 
> Just my 0.002BTC Mark
> 
> On Wed, Sep 26, 2012 at 6:22 AM, steve  wrote:
>> 
>> I think you might be misunderstanding a little. I am not trying
>> to replace the current system, I need to make sure that what I do
>> will be compatible with it (seamlessly so for the developer). I
>> do not want this to generate extra work for the development
>> team.
>> 
>> However testing is a lot more than just bug reporting, dont get
>> me wrong bug reports are important, but so is running a testcase
>> and that testcase passing, especially if that testcase is linked
>> to the proof of a requirement. I am trying to develop a qa
>> environment that is conducive to testing and will allow the
>> testers to shine in all their glory :) and we need different
>> tools and methodologies.
>> 
>> Git is too developer centric to be useful for organising testing.
>> - however there is a large amount of software that is compatible
>> with git, so the core development team only ever need to work
>> with git.
>> 
>> The linking between a bug, the requirement, the fix, the retest,
>> and updating of regression testplan is vital. So is the ability
>> to organise testing campaigns and assigning tests, work units and
>> test relevant docs/scripts/ideas, etc.
>> 
>> I hope this clears things up a bit?
>> 
>> Cheers,
>> 
>> steve
>> 
> 

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYz8CAAoJEFvEB9dQFvtQjkMH/Apa95IRh21mfNIuyK8kOSdt
55tLoT9a6DFyF1IPTgjHQlPN/A0JCPy/p2rIEEL7X

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

- -BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 26/09/2012 13:49, Wladimir wrote:
> Steve,

Hi Wladimir,

> 
>> So, currently there are 4 potential places for bugs to be 
>> reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 
>> - forum (bitcointalk...) 5? - is there still the ability to add 
>> bugs via sourceforge?
> 
> Currently github is the authoritative place to report issues. When
>  someone reports a bug on the mailing list, IRC or forum, they are
>  generally asked to make a github issue (or, someone else makes the
>  issue for them). Failed tests are generally also reported on 
> github, by the pull tester.

excellent, that makes things much easier.

> 
> We currently have 232 issues, mostly classified into categories 
> such as "Bug", "Improvement", "GUI", "Wallet", and so on.
> 
> Also it's easy to refer to github issues in commits with #123, with
> automatic linking.
> 
> I'm not sure it is worth the effort to move to another system 
> (especially if you need a another login etc...). But I'm probably 
> misunderstanding what you're trying to do.

I think you might be misunderstanding a little. I am not trying to
replace the current system, I need to make sure that what I do will be
compatible with it (seamlessly so for the developer). I do not want this
to generate extra work for the development team.

However testing is a lot more than just bug reporting, dont get me
wrong bug reports are important, but so is running a testcase and that
testcase passing, especially if that testcase is linked to the proof
of a requirement. I am trying to develop a qa environment that is
conducive to testing and will allow the testers to shine in all their
glory :) and we need different tools and methodologies.

Git is too developer centric to be useful for organising testing. -
however there is a large amount of software that is compatible with
git, so the core development team only ever need to work with git.

The linking between a bug, the requirement, the fix, the retest, and
updating of regression testplan is vital. So is the ability to
organise testing campaigns and assigning tests, work units and test
relevant docs/scripts/ideas, etc.

I hope this clears things up a bit?

Cheers,

steve

> 
> Wladimir
> 

- -BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYwESAAoJEFvEB9dQFvtQ/GUH/jv2c5L0OcL/kHkX/z0Yqbl/
2IntPLdjXNKLuz0A7BMz7XfUyVmWlZrw44qxmi+Vyk5PKNBjYIidm763xHnTeJLN
ULQBckYexMvan9hAyYZUOt85IpesdNgqTIsqh8f49y4roHOy8GT4M/2fhzXpnsGg
G9d2m8jWGpj/kxl9qE7/WjVQC4APwBi/NiJsCrcHswgweN+zENc/Pot9YBLxAZu/
ACBUX/xFymRdaZN8P2LWBXuKx6E2WEcBdPCCWArX07wPiBlrashx9Gz6tiNzIiNq
x2c4ltLzRa45AmiDtQhwqyTprz/DbyeAYO1sIsfpUxDeu9e3xTb/Zd96jfKIWI0=
=iHI1
- -END PGP SIGNATURE-
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYwGsAAoJEFvEB9dQFvtQxOsIAKgBBOKHNFtoV2cN+GVqzlip
yy0qiMvMTZKrraOhEw8QNNuOlB3LRchi+RDR/PvQkVfuwi/jHB2gUBzlapLoECBv
EH8pgT/MO281pXzARgRSVkRYqkb3ljhQz3mEQg9RhR9h5t9g2mL3Tvppt7249Bg8
oGXPj6xmMcrbClF5qDbwQUUDGJfOo4eti0jSVD3qp2NE7QpPVQwuN5buchpoKt3P
9aJnjeZdLmuAk2RPoDaLXUFc9unT8AcnW96juD0zoVA9wKvAa6/8IZQf0mzV4iZP
yiWGNOQtBZ+jyu2ixiEnvHqqG2ZmjtUVqWtjHkxYgrCyuuK2jOcTMNEWfn7SfKc=
=yP7N
-END PGP SIGNATURE-

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Matt,

Glad to have another ninja onboard :)

On 25/09/2012 21:41, Matt Corallo wrote:
> Although Jenkins may not be the best system, we already have
> jenkins and pull-tester (which is a dumb python script I wrote to
> test all incoming pull requests from github).

I have never heard of jenkins before.  I need to do some more digging.
is this the right thing?

https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin

Mantis on the other hand, I know exceptionally well.  I hate
duplication of work/data unless absolutely necessary.  I will check
jenkins out (just out of interest what is it actually meant to do? the
website implies framework, but not what its for)

So, currently there are 4 potential places for bugs to be reported
1 - jenkins (and unit tests)
2 - git
3 - mailing list
4 - forum (bitcointalk...)
5? - is there still the ability to add bugs via sourceforge?

Adding to this doesnt make sense.  Each one of these reporting methods
is for a different thing.  I am not seeking to replace these (or even
unify them) I am looking for software that will take testcases and bug
reports against them [and allow for test campaigns].  Mantis is so
flexible and industry standard and if the jenkins plugin works... then
we can keep things as they are until they fit into better places.

The reason I am so behind mantis as the backbone is it works with more
or less anything, and can easily modded to work with whatever people
are most comfortable with - however it is exceptionally powerful and
has had a constant stream of workflow improvements over the past few
years.

> 
> They both run the same set of scripts, namely those at 
> https://github.com/TheBlueMatt/test-scripts (its pretty basic right
> now, but since it is on github, I was hoping someone would find
> the inspiration to add to it).

I will check it out. I wrote a very basic script that wikified the
changelog, and linked to the changes and created wiki pages for the
testcases.  have you seen the stuff I put on bettermeans? bits keep
vanishing then re appearing.

This is the outline of the testing that I setup for 0.7

https://secure.bettermeans.com/projects/4256/wiki

> 
> I dont really care if we keep using jenkins, but I figure we might
> as well keep all the tests in one place?

Yes, I would love to unify all build testing and testcases into one
place.  I am still on the fence as to including unit tests into this.
However I do see 3 distinct type of testcases
1 - requirements based testcases (requirements based off the current
block chain rules - these are edge cases and known interoperability
issues)

2 - Acceptance based testcases - these are testcases that should be
run for every build.  Check out the General Acceptance Tests in the
wiki link for examples and testcases

3 - Testcases for reference implementations of things (like multisig -
i see these working like the /test folder when you install a new perl
module)

These three things alone are a massive task. and they still wont cover
everything.  I would like to get the workflow so that people can
sponsor or donate to a specific campaign (eg a new feature is
implemented, people want it tested so can donate just for that
campaign [developing testcases, structure, requirements, etc])

Once this is done, I will get to do some exciting stuff (like writing
fuzzers, automation, etc) unfortunately I do not know python, only perl.

> 
> Anyway, I'm all for more testing (I'm always complaining about how
> we need more tests for stuff...).

Nice, I love testing.  I think we will get on :)

And I would rather go for interoperability between testing rather than
rewriting it all.

Cheers,

steve

> 
> Matt
> 
> On Tue, 2012-09-25 at 19:32 +0100, steve wrote:
>> Hi All,
>> 
>> After the failure to get any real testing done for the 0.7
>> release (all of which is my fault) I have decided to rejig
>> things.
>> 
>> I am heavily into test driven development, and I have a strong 
>> background in requirements management, and automation.
>> 
>> I want to leave bettermeans behind, maybe we might be able to
>> keep the voting aspect of it, and link it into mantis.
>> 
>> So, what I have been doing over the past few weeks is developing
>> a rudimentary requirements set, basic requirement tracking, tests
>> to prove/stress the requirements.
>> 
>> The next most important thing is to get release/acceptance tests
>> done - these primarily focus on new stuff doesnt break old (ie
>> lose a wallet, etc) and needs no special requirements.
>> 
>> To this end I have installed various opensource applications
>> (mantis, salomeTMF, bugzilla, etc) and am currently evaluating
>> the best workflow process.
>> 
>> This was a much longer post, but decided against it

[Bitcoin-development] Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi All,

After the failure to get any real testing done for the 0.7 release (all
of which is my fault) I have decided to rejig things.

I am heavily into test driven development, and I have a strong
background in requirements management, and automation.

I want to leave bettermeans behind, maybe we might be able to keep the
voting aspect of it, and link it into mantis.

So, what I have been doing over the past few weeks is developing a
rudimentary requirements set, basic requirement tracking, tests to
prove/stress the requirements.

The next most important thing is to get release/acceptance tests done -
these primarily focus on new stuff doesnt break old (ie lose a wallet,
etc) and needs no special requirements.

To this end I have installed various opensource applications (mantis,
salomeTMF, bugzilla, etc) and am currently evaluating the best workflow
process.

This was a much longer post, but decided against it. :)

So, what I want to know is who wants to be a part helping me out with
all this? I am finalising the workflow flow diagrams and they should be
ready for inspection soon.

Anyone interested in helping out/reviewing processes? even if it is just
some encouragement, it is all greatly appreciated.

Drop me an email if you want access to the current setup and help me
review the different software for the bitcoin workflow process.

cheers,

steve

- -- 
my PGP public key is at pgp.mit.edu id: 0x5016FB50
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYfjMAAoJEFvEB9dQFvtQSmsH/R/FEdOQRB7ncTnHhaP8woLu
nIiGX2DgLOWLOF9launSuTCrtVm2G56B9Dgl/BqScFxeuJGbzje7+kp7LgjtA3uy
kS9DUZ1zhUfhslGP0UpVJJGX6Yfk8GbQ4nUcuL1VTv6nSZXWP2EvLMDPpRgKwyi5
z1FiyBg2A3Kg3Er+VmHPmpI0zZAGB5ytaenUp4xXGhL7Nk66i5X0twVr51xlEm0L
zKCDXHzWTvNNlT7TzMjIxShJ/EcgCI1r6tVD3T+2e9QeVm0QNw3xeNUkMxKn+ul8
d1v1OxJbHD1CsNqW+XgVvFE2SJReizaHNOFwrqcpVCp7bABnWAB5eyTzB9B9IX8=
=di5x
-END PGP SIGNATURE-

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] The Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 03/08/2012 01:07, Gary Rowe wrote:
> Hi Steve,
> 
> This looks like a good idea to me. The test suites could act
> similarly to the 100% Pure Java approach that successfully fended
> off a lot of corrupting influences to Java over the years.
> 
> Maybe it's worth putting together a small starter suite of tests
> and showing them to the community then providing a suitable
> process, perhaps through BIPs, to allow tests to be created,
> reviewed and updated before getting incorporated into a reference.
> I imagine a BIP would cover an aspect of the blockchain rather than
> a single test or test suite since having that many BIPs would get
> onerous fast.
> 
> Kind regards,
> 
> Gary
> 

Hi Gary,

Thanks for the response. :)

I have started all this in bettermeans, but lost a lot of work (which
I am working on how to redo - but I am on holiday at the moment, and
have restricted access to my test setups)

Here is the discussion thread I had with gavin about acceptance tests.

https://secure.bettermeans.com/boards/4316/topics/7261

Here is the work I have currently done (note, it was losing all the
General Acceptance Tests and getting no response from bettermeans that
has lead me to not want to use it)

also note that the terminology I have been using is a little wrong, I
refer to release 0.7 as testnet release 0.7 - I will tidy it up. - I
did this before and it looks like it got reverted somehow.

here is the main wiki space that I have been using.

https://secure.bettermeans.com/projects/4256/wiki

For the General Accceptance Tests, check

https://secure.bettermeans.com/projects/4256/wiki/Dev_general_acceptance_tests

These are the basic acceptance tests based off the changelog

https://secure.bettermeans.com/projects/4256/wiki/Dev_acceptance_tests

However, notice no tests are in there yet.

There is plenty more stuff on bettermeans so please go have a poke
around.  I will try to get at least a wiki setup on a vps I have
control of (in germany, and provided by CINFU and paid for with
bitcoins) and get the stuff moved over and put the tests back up.  I
can do a limited amount of testing if the release is to happen in the
next week or two, So I will focus my efforts on the installation tests
and wallet tests. Ideally I would like to get all the GAT's done.

Hopefully this fleshes things out a bit more.

Please feel free to add/edit but remember your stuff might magically
disappear, it might be better to wait for the wiki move.  I intend on
doing the wiki move today. but that might not happen.

(I have a feeling that it was me adding stuff from an account that is
not on the BTP that blew things up, just a feeling though)

I would like to keep tests informal if possible, but that kinda goes
against the purpose of the tests. this is a bit tricky to explain if
you have not had a look at bettermeans.  bettermeans kinda has its own
BIP and voting mechanism, which is quite nice and it is what I want to
emulate - this should reduce the need for vetting of tests... have a
look at what is there and see what you think.

:)

cheers,

steve
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQGxkFAAoJEFvEB9dQFvtQeD0IAJe9BJz/mv+kZjhk7LH1d7HH
c46D7s2Y8a+2Yobve4KtRGMoQZQiqqXGIdZ2nHVO77s0zICixqdtcKlRvBZHybw9
pB8hFYmeBdXvMHj7TR4kMbMKqTJ2z/B6m1qEKFfCRIXQXnyD5qNYhFocyQMwz53A
dkwhpoiWNVqcgnz51XEnphyohu0TPsPbOOyCrT7ORdyAgLJAs5Ig1sKbTAdSxOux
flEYKOVk0gse2b8lO2ly+eLwcQgI7jrzy+qkSKmNajRKFdvHUODXo4RraR08qiaJ
SUpmN/43uQZ4atMdOCZxD5DWKjBO96sj6mkB/po5lzIEEtkhzyp/wmKdHtlvZ/Q=
=Fonn
-END PGP SIGNATURE-

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] The Bitcoin Testing Project

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi all,

I know most of you have more important things to do, and not enough
time as it is, but I would really like your feedback on the testing
project.

Bettermeans seemed ideal, it allowed for leaderless leadership with
people being able to dip in and out of what they wanted to do.
however it seems that bettermeans is in no way a finished product and
is rotting fast.

I would like to move away from bettermeans - whilst retaining the
voting and sub workstream style of working.  I would like to set
something up around MantisBT (Mantis Bug Tracker) - this is an
exceptionally versatile bit of software with plugins and interaction
with other testing products (everything from test setups and plans to
test runs and results) I would go as far as saying it is industry
standard (yeah, i know about bugzilla, etc. mantis beats them all
hands down.) obviously we would need a wiki and other software - this
is not a problem.

I am happpy to pay for the VPS' to host this stuff and set it all up.

I have quite a bit of experience with mantis and other opensource
testing stuff.

I see the testing of bitcoin to be very similar to the testing process
in the games industry.  for those that do not know how this works it
is like this:-

A company comes up with a device (xbox/ps),
They then publish a set of dos and donts for this device (TRC for sony
and TCRs for microsoft.  - I wrote quite a few of the MS TCR's for the
original xbox.)
They (ms/sony) then test your game against these rules and depending
on how many you pass/fail your game can be released or not.

I see this as mapping to bitcoin very well, the device is the
blockchain, and the TRC/TCR _tests_ are published so third party
developers can see how they fair [look at gavins recent blockchain
edge cases for an example] (ms/sony do not publish their testcases
only the requirements)

I believe that this will allow the bitcoin testing project to be able
to cope with the stable builds, bleeding edge builds and 3rd party
implementations all at the same time.

It doesnt matter what the app is, it is its interaction with the
blockchain, the safe guarding of the blockchain and compatibility with
the previous/future versions that are tested.

[for the bitcoin dev list]
A little about me:-
The below is more or less a cut and paste of some of the stuff I sent
gavin in my initial email about wanting to be in on the project.

I can back all of this up with references. I can go into more detail if
needed.

I was heavily involved with setting up the microsoft xbox european cert
department
I set up qa department for europes largest independant games developer
(although they are no longer)

worked for microsoft secure science designing security automation tools

setup the internal pentest for thales e-security (now TITS [Thales
Information Technology Solutions (or Trotters Independant Traders ;),
on thier datacryptors (fpga, crypto and product) - have done full test
cycles on hsm 8000, payshield 9000, dc2k and thier latest line of
military spec comms equipment.

Setup and pentested Thales and nCiphers credit control software (to
fips level 4 standard - This shit bitcoin exchanges need!! I know bank
is a dirty word in the bitcoin world, but we should be at least as
secure as them)

I currently find exploits in stuff like office, quicktime, ie, ff, etc
and sell them to companies like ZDI (3com) iDefense (verisgn) and some
pentest companies that require zero day exploits.

however I would like to ditch this and get my bitcoin related stuff
off the ground - with the BTP being top of the list.

my spelling is rubbish, and sometimes i forget to spellcheck before i
hit send. sorry about that.

any feedback would be really appreciated. please! I feel that this has
stagnated enough and I want to get my work out there and I want it to
be useful. (I lost 60 or so testcases because Kev left himself logged
into bettermeans and I added them under his account... none appeared
on the wiki, none appeared anywhere...)

There is already lots of stuff on the wiki that outlines how i see
things holding together - but you cant see who posted what because of
a bug that has appeared recently [meta info is just displayed as its
meta tag, {name} on {date} wrote.]

as apposed to Mistfpga on 30/8/2012 wrote...

so the tl;dr
1 - I want to use something other than bettermeans
2 - I can admin opensource software to do the same
3 - I want to take the voting/hiearachy style from bettermeans and
apply it to mantis and test workflow.
4 - I want to get some testing done asap.
5 - I have a full msdn and technet licence
6 - I have a vast array of machines [nearly 100 cores] that I can use
to automate testing and to test different setups.
7 - been mining for 18 months or so.

cheers,

steve

I do not belive this to be related, but I am not ashamed and feel no
stigma. I have posted this on a public forum. I suffer from fast
cycling (withing a day) type two bipola

Re: [Bitcoin-development] Scalability issues

[steve]

On 25/07/2012 10:45, Michael Grønager wrote:
> Hi Steve,
> 
> I see dramatic differences in performance on virtual machines vs 
> running directly on the iron. I am not an expert in virtual
> machines,

They can be, it depends on how they are set up.  For reference, these
VM's used to test network stacks and file format bugs.  They do this
via debug tracing, trace into, not over.  They then dump this data in
files and it can keep up with a core2 duo laptop for file io. however
I moved to using an in ram database (4gb chunks, that then get dumped
over the network port to a db on a seperate machine)

I am not sure I have the skills to instrament this into bitcoind

> 
> I would like to do a test keeping database log files in memory. It 
> should not matter for durability of the wallet, as it flushes at
> each write anyway. As for the blockindex, it will remain
> consistent, but might be lagging some blocks behind at startup,
> which shouldn't really matter (except that the same block could end
> up appearing twice in the block00X files, inelegant, but not really
> a problem).
> 
> Otherwise the system you describe (raid0 over 6 disks) should
> perform like crazy wrt disk i/o, at least on par with SSD. It is
> your virtualization I am worried about.

I will test that when I get back :)

> 
> Have a safe trip to down under!

will do, thanks for the response. :)

> 
> /M
> 
> On 24/07/2012, at 21:56, steve wrote:
> 
>> Hi Michael,
>> 
>> from what I have noticed, bitcoin blockchain
>> download/verfication all happens in 1 thread.  (so multicores
>> doesnt really help)
>> 
>> That said, I have never tried on an ssd.
>> 
>> What I do have is 6 SATA 6gbs configed as RAID0 Drives. 32gb of 
>> ram. ubuntu 64 (yeah I know), this runs upto 16 VM's (I have 4
>> of these)
>> 
>> However I have not tried to download the blockchain on the
>> master os, just in virtulisation.  However, the dedicated
>> machines that I have been using for benchmarking the VM's against
>> is a q6600 8gb ram sata2 hdd - Win 7 (seems faster than
>> slackware...) to me it has always felt like network bandwidth was
>> the issue.  I might instrument the bitcoin-qt exe to only pick
>> low ping nodes (has someone already done this?)
>> 
>> I guess it is time to start some benchmarking (like the gpu 
>> comparison page)
>> 
>> hte verification for the 5 past 5 days was negliglable. I am off
>> on a flight to australia tomorrrow, so I will set some
>> breakpoints and do some timings in a debugger.
>> 
>> This will all happen on an e-450 (wonderful machine!)
>> 
>> Thanks very much for your response. it would seem that I am
>> 'doing it wrong' :/
>> 
>> cheers mate,
>> 
>> steve
>> 
>> (this message isnt signed because I have forgotten my password.)
>> 
>> On 24/07/2012 09:25, Michael Grønager wrote:
>>> Hi Steve,
>>> 
>>> 45-90 minutes - note that its numbers from March/April, so a
>>> bit longer today, but far, far away from the 12 hours.
>>> 
>>> I am using libcoin and the bitcoind build based on this.
>>> Libcoin is based on the Satoshi client, but refactured to use
>>> an async concurrency model. I also did a minor tweeks to the
>>> db parameters. It has earlier been tested up against Satoshi
>>> bitcoin where on some OS'es it performs similarly (at least on
>>> some linuxes) and on some faster (e.g. mac).
>>> 
>>> What is your CPU load during a block download ? (both 
>>> initially/up to the point where verification sets in and
>>> after). The initial download is typically disk I/O bound, the 
>>> verification stage CPU bound, though I lean to believe that
>>> even there it is disk I/O bound (at least on my system ~50% CPU
>>> load). What should be better in libcoin is the concurrency
>>> model. The Satoshi client uses a pure reentrant mutexes model,
>>> that is not generally believed to motivate the best coding
>>> practice nor performance, you might end up without the
>>> concurrency you initially strived for *). As mentioned earlier
>>> libcoin uses a pure async concurrency model (and so does
>>> libbitcoin btw).
>>> 
>>> I would like to stress again that these numbers will depend 
>>> largely on the system running the test - I would call my laptop
>>> a bit over the average today (MB Pro, 2.66Ghz i7 dual core,
>>> 8GBRAM, 512GB SSD). But again 12 hours - I only reach such
>>> numbers on some of my VPS'es (linode 1024) that are known 

Re: [Bitcoin-development] Scalability issues

[steve]

Hi Michael,

from what I have noticed, bitcoin blockchain download/verfication all
happens in 1 thread.  (so multicores doesnt really help)

That said, I have never tried on an ssd.

What I do have is 6 SATA 6gbs configed as RAID0 Drives.
32gb of ram. ubuntu 64 (yeah I know), this runs upto 16 VM's
(I have 4 of these)

However I have not tried to download the blockchain on the master os,
just in virtulisation.  However, the dedicated machines that I have been
using for benchmarking the VM's against is a q6600 8gb ram sata2 hdd -
Win 7 (seems faster than slackware...) to me it has always felt like
network bandwidth was the issue.  I might instrument the bitcoin-qt exe
to only pick low ping nodes (has someone already done this?)

I guess it is time to start some benchmarking (like the gpu comparison page)

hte verification for the 5 past 5 days was negliglable. I am off on a
flight to australia tomorrrow, so I will set some breakpoints and do
some timings in a debugger.

This will all happen on an e-450 (wonderful machine!)

Thanks very much for your response. it would seem that I am 'doing it
wrong' :/

cheers mate,

steve

(this message isnt signed because I have forgotten my password.)

On 24/07/2012 09:25, Michael Grønager wrote:
> Hi Steve,
> 
> 45-90 minutes - note that its numbers from March/April, so a bit
> longer today, but far, far away from the 12 hours.
> 
> I am using libcoin and the bitcoind build based on this. Libcoin is
> based on the Satoshi client, but refactured to use an async
> concurrency model. I also did a minor tweeks to the db parameters. It
> has earlier been tested up against Satoshi bitcoin where on some
> OS'es it performs similarly (at least on some linuxes) and on some
> faster (e.g. mac).
> 
> What is your CPU load during a block download ? (both initially/up to
> the point where verification sets in and after). The initial download
> is typically disk I/O bound, the verification stage CPU bound, though
> I lean to believe that even there it is disk I/O bound (at least on
> my system ~50% CPU load). What should be better in libcoin is the
> concurrency model. The Satoshi client uses a pure reentrant mutexes
> model, that is not generally believed to motivate the best coding
> practice nor performance, you might end up without the concurrency
> you initially strived for *). As mentioned earlier libcoin uses a
> pure async concurrency model (and so does libbitcoin btw).
> 
> I would like to stress again that these numbers will depend largely
> on the system running the test - I would call my laptop a bit over
> the average today (MB Pro, 2.66Ghz i7 dual core, 8GBRAM, 512GB SSD).
> But again 12 hours - I only reach such numbers on some of my VPS'es
> (linode 1024) that are known for notoriously slow disk I/O. (here I
> have a few % CPU load during the verification indicating indeed that
> the disk i/o is the culprit).
> 
> Cheers,
> 
> Michael
> 
> 
> *) I like this Dave Butenhof quote: "The biggest of all the big
> problems with recursive mutexes is that they encourage you to
> completely lose track of your locking scheme and scope. This is
> deadly. Evil. It's the "thread eater". You hold locks for the
> absolutely shortest possible time. Period. Always. If you're calling
> something with a lock held simply because you don't know it's held,
> or because you don't know whether the callee needs the mutex, then
> you're holding it too long. You're aiming a shotgun at your
> application and pulling the trigger. You presumably started using
> threads to get concurrency; but you've just PREVENTED concurrency."
> 
> 
> 
> 
> On 23/07/2012, at 17:54, steve wrote:
> 
> Hi Michael,
> 
> On 23/07/2012 10:00, Michael Grønager wrote:
>>>> I get a full blockchain from scratch in 45 minutes on my
>>>> laptop, /M
>>>> 
> Hang on a sec, in 45 minutes you can download the entire chain from 
> the genesis block?
> 
> I have been doing extensive testing in this area and would love to 
> know what is special about your setup (I have never had the entire 
> chain in under 12 hours, infact it is normally closerto 24.) I have
> an extensive setup of test machines, everything from e4300 to
> phenom2x6 to i5's.
> 
> as an example on an amd e-450 with 4gb ram, and approx 3gb/s
> internet connection it took 2 hours to sync the last 5 days.
> 
> Maybe i am missing something important...
> 
> Any additional information that you could provide to help me with 
> testing would be really appreciated.
> 
> cheers,
> 
> steve
> 
>> 
>> --
>>
>> 
Live S

Re: [Bitcoin-development] Scalability issues

[steve]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Michael,

On 23/07/2012 10:00, Michael Grønager wrote:
> I get a full blockchain from scratch in 45 minutes on my laptop,
> /M
> 
Hang on a sec, in 45 minutes you can download the entire chain from
the genesis block?

I have been doing extensive testing in this area and would love to
know what is special about your setup (I have never had the entire
chain in under 12 hours, infact it is normally closerto 24.) I have an
extensive setup of test machines, everything from e4300 to phenom2x6
to i5's.

as an example on an amd e-450 with 4gb ram, and approx 3gb/s internet
connection it took 2 hours to sync the last 5 days.

Maybe i am missing something important...

Any additional information that you could provide to help me with
testing would be really appreciated.

cheers,

steve

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQDXO4AAoJEFvEB9dQFvtQxdcH/ieqQkyDCg8mKeOa6CqsWaS6
fhoeny3Ke2b/CsvhYmsThCvntN9volIqR2CTn5tkHiVwG9OmlxyHZcNpN0ZTHhK5
lsfLap/Y0QpiysXpV4Bu7Z4Hwp9jnhOP74TshT305r2pX6EGXPQ0CrlHqlIry/X/
vNcunUclliou+KjL7EHcY50GH5wDpqJAjlNyF97Lj9YiPrAC9vahGwWdxkbCYtG+
KUuWGBKMMdHuMAgcQh7nI9q0WT3k/gzRQtuC2kf+v0wvQhaGlTVkku4uanhpuw4p
99blRF3/SfWimGuQgsm6wT3Y7dk+z8MFHLb6XGPxmgV9+gF+TWNczfU3GRzfcXw=
=CQkI
-END PGP SIGNATURE-

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Alert System

[Steve Coughlan]

Who knows, it might be the only way we'll ever hear from Satoshi again.
On Sep 9, 2011 1:21 AM, "Matt Corallo"  wrote:
> On Thu, 2011-09-08 at 07:42 -0700, David Perry wrote:
>> There has been some discussion on the new Bitcoin StackExchange site
>> lately about the alert protocol. A few have suggested that it might
>> carry the potential for abuse (spam/DoS) and others have argued that
>> it's merely deprecated. In any case, enough have voiced concerns that
>> I've forked bitcoin/bitcoin, removed the snippet of code from main.cpp
>> that makes the questionable call and submitted a pull request. On that
>> pull request it was noted by Gavin Andresen that it merited discussion
>> here and some kind of consensus should be reached before acting on
>> that pull request. It was also mentioned that he thought the feature
>> was still more useful than dangerous and that he would argue against.
>>
>>
>> So I pose the question to you fine fellows: Is the alert system
>> valuable, an unnecessary risk or merely a snippet of deprecated code?
>> Should it be removed?
>
> The alert system requires a signature verification when it receives an
> alert, but so do blocks and transactions so it really isn't a DoS target
> (remember that the alert system requires alerts to be signed by a key
> that only gavin and satoshi have).
>
> The alert system could prove very, very valuable. In much software it
> carries the risk for abuse or simply seems wrong that the developers can
> send a message to everyone's computer to notify them of something, but
> keep in mind that Bitcoin is financial software. If there is an urgent
> problem (like the overflow bug) there must be a way to notify people to
> upgrade immediately, which is exactly what alerts provide. Since alerts
> no longer carry the ability to put Bitcoin into RPC safe-mode, they are
> literally just a message and I see no reason why they should be removed.
>
>
>
--
> Doing More with Less: The Next Generation Virtual Desktop
> What are the key obstacles that have prevented many mid-market businesses
> from deploying virtual desktops? How do next-generation virtual desktops
> provide companies an easier-to-deploy, easier-to-manage and more
affordable
> virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
--
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Alert System

[Steve]

I think there's a significant risk to not having it at this stage.  
There's many reasons why an urgent update may been to rapidly propagated 
in this stage of the network's lifecycle.  Perhaps if there's a 
perceived threat of abuse the protocol could be altered slightly so it 
can't carry content.  Only a notification of the fact that there is an 
alert.  Then it would be up to individual clients whether they react to 
it or not.  The main clients would probably check a central trusted 
server for actual alert content.  This would give a lot more flexibility 
in how to deal with the alert.  Alert content servers could for example 
implement a json api to provide alert content with meta data like target 
client version, priority etc.


I think it should be removed in the future but not for a good while yet.

On 09/09/11 00:42, David Perry wrote:
There has been some discussion on the new Bitcoin StackExchange 
 site lately about the alert 
protocol. A few have suggested that it might carry the potential for 
abuse (spam/DoS) and others have argued that it's merely deprecated. 
In any case, enough have voiced concerns that I've forked 
bitcoin/bitcoin, removed the snippet of code from main.cpp that makes 
the questionable call and submitted a pull request 
. On that pull request it 
was noted by Gavin Andresen that it merited discussion here and some 
kind of consensus should be reached before acting on that pull 
request. It was also mentioned that he thought the feature was still 
more useful than dangerous and that he would argue against.


So I pose the question to you fine fellows: Is the alert system 
valuable, an unnecessary risk or merely a snippet of deprecated code? 
Should it be removed?


Sources:
http://bitcoin.stackexchange.com/questions/583/what-is-the-alert-system-in-the-bitcoin-protocol-how-does-it-work/590
http://bitcoin.stackexchange.com/questions/636/is-the-alert-system-still-in-the-main-clients-code-will-it-be-removed/711


--
Doing More with Less: The Next Generation Virtual Desktop
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/


___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development
--
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] bitcoind multiplexing proxy - request/response routing problem

[Steve]

> It's probably best to keep this discussion on just one mailing list. 
> It's confusing to have duplicate threads in different places. People 
> will end up making the same points.
>

Fair enough I'll take it to the bitcoinj list.  I wanted to post here in 
case I got any nibbles from c developers about option 2.  If anyone 
want's the follow this discussion on the other list it's here:
https://groups.google.com/forum/#!topic/bitcoinj/kqiBq9VxL-k



--
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] bitcoind multiplexing proxy - request/response routing problem

[Steve]

4a/ Serialize all request/response exchanges.  i.e. request comes in 
from remote node, proxy aquires lock on the proxy-localdaemon channel 
and sends request.  Channel remains locked until response is received or 
timeout (in which case remote node gets no response).  Unlock channel 
after response received and send to client.

Possibly messages that don't expect a response (e.g. relaying a tx 
broadcast from remote node) can be pushed down a locked channel to 
improve performance as they won't interfere with sequencing.  Locked 
channels may also receive other unsolicited messages from local daemon 
before the expected response message which would be dealt with the same 
as if they came from an unlocked channel.

Disadvantages:  Idle time for channel while waiting for response.  As 
per option 2 this allows the proxy to stay dumb/thin but loses 
opportunity for de-duplicating/caching unless option 1 is layered on top.

4b/ As per 4a but use all 125 available bitcoind connections in a 
channel pool.  Acquiring a lock on a channel consists of checking for an 
unlocked channel first then waiting in a queue for one to become available.

--
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] bitcoind multiplexing proxy - request/response routing problem

[Steve diy-eco-life]

This a reworking of a post I made on the bitcoinj list under a
different topic but it's something I'd like to throw out there for
input.

I'm going to build a proof of concept this weekend of the multiplexing
proxy that Gavin and Mike were talking about here:
http://sourceforge.net/mailarchive/message.php?msg_id=28049519

Initially just a dumb as doornails proxy between one local bitcoind
and one remote node.  Once I've got that far the next step is to work
out how deal with request-response exchanges from multiple remote
nodes.  I discussed this tatcm on IRC last night.  The problem is
after relaying several requests from different remote nodes to the
local daemon you expect multiple responses to come back.  How identify
which response matches which client's request. Bitcoind can implicitly
identify the recipient based on which connection made the request.  By
piling all the requests onto one channel we lose this identifier.  I
can think of 3 approaches to dealing with this:

1/ Generate a unique key from the request and can also be generated
from the response.  e.g. getheaders key could be "headers" +
hash_start.  We locally store a mapping to client (or clients) that
requested it and pass it to bitcoind.  When we get a headers packet
back unique key = "headers" + hash_of_first_header, so we can lookup
the clients who requested it and send it back.

The unique key should have the following properties:
 - can be reliably generated from both the request and the response.
 - identical requests from different clients should generate the same
unique key (this allows us to recycle responses)

Problems:

This is dependent on each pair of request/response messages being
guaranteed to contain information needed to create an identical unique
key.  I haven't looked in detail at every request/response pair yet to
confirm this.  If it is the case then this is an onerous obligation to
place on the protocol to fulfil this condition for all future protocol
changes.

To obtain guaranteed uniqueness may require large keys.  Is
'almost-unique' an option?  e.g. generating a key off a getblocks
request using the first n bytes of each block_locator_hash would be
much smaller/faster and very likely to be unique.  Are the risks of
sending back the wrong response to a request acceptable?


2/ Modify bitcoind to accept sequence numbers for request/response
type messages, similar to 'id' field in json-rpc.  This is more
reliable but potentially quite invasive to the bitcoin protocol.  It
also loses the inherent de-duplication of requests that we get with
the previous solution.  If it were to be implemented I'd suggest
something like a separate sequence number message.  i.e. proxy sends
seq message containing a unique ID.  The contract is that the seq
message refers to the next message that comes over the wire.  When the
response is ready the bitcoind sends a seq message with matching id
then sends the response message.  A separate seq message at least
leaves the existing protocol untouched as the handling will remain
unchanged unless a request type message is preceded by a seq message.

This approach allows the proxy remain a lot thinner and dumber but we
lose a lot of the de-duplicating efficiencies from the first approach.
 If we want to add this capability as well we essentially need to
implement option 1 as well although in this case we have a reliable
fallback.  i.e. If we can't generate a unique key then we just use
single request/response matching via the seq id.

3/ Make the proxy intelligent enough to handle these requests itself.
Using getheaders as an example again.  Proxy maintains it's own local
cache of headers.  when a getheaders message comes in the proxy checks
if it has all requested.  If not it requests the missing ones from the
local daemon, adds to it's cache and builds a headers response itself.
 In this case the proxy definately has to be protocol version aware...

Advantages:
 - This probably achieves the best combination of request/response
matching reliability and de-duplication of work.

Disadvantages:
- Complexity.  The proxy needs to be far more protocol aware which
creates a maintenance dependency for future protocol changes.

Having spent the last couple of days studying the protocol I'm
inclined toward the first approach as an initial easy implementation
with a view to moving to the 3rd approach.  It appears that most
response type messages could be relatively easily constructed from a
local cache.  Before I looked at the protocol I would have said no way
to the 3rd but the depth of protocol awareness for 1 or 3 is not
really much different.  Option 2 allows for a much dumber and thinner
proxy but loses a lot of potential efficiencies and if those were to
be regained it would require the same level of protocol awareness
inherent in 1 and 3 anyway.  It would also require someone on the
bitcoind side to put their hand up to add the seq message
functionality as I don't have any c skills to speak of.

U

Re: [Bitcoin-development] Building a node crawler to map network

[Steve]

Thanks for the overview Mike.  I just bailed up Gavin on IRC and between 
that convo and what you've just written I'm starting to picture a plan 
in my head... This sounds right up my alley, I wish I didn't have to go 
to bed right now as I've got a ton of ideas buzzing around I'd like to 
get started on right now.  But I'll be onto it as soon as I've got a 
free moment...


On 07/09/11 00:52, Mike Hearn wrote:
On Tue, Sep 6, 2011 at 4:17 PM, Steve <mailto:shadders@gmail.com>> wrote:


I'm not really understanding the use case though.  I believe most
bitcoind's have a default max connections of 8.  Is the goal to
increase this without fundamentally altering the bitcoind
concurrency model?


bitcoind already uses asynchronous IO. That's not the problem.

The issue came up in a conversation about scalability. If Bitcoins 
popularity continues to grow, users are very likely to migrate away 
from running full verifying nodes to lightweight clients, either a 
different mode of the Satoshi client or different implementations like 
the Android Wallet or MultiBit.


Lightweight clients cannot verify thus should not relay. And they'll 
be run by users who just want to send/receive coins from time to time, 
so don't leave the programs running 24/7. The result could be running 
out of sockets (like we have had problems with recently). It's 
especially true because lightweight clients cannot check transactions 
for themselves. If they want to show transactions appearing 
immediately (and they do), they have to use "heard from lots of nodes" 
as a proxy for validity. So lightweight clients are likely to be 
socket intensive.


We could solve this by just hoping that lots of people run full nodes. 
The problem is that a full node is quite an intensive thing already, 
it uses lots of CPU and disk seeks, and will just get more expensive 
in future. And as transaction traffic increases, that leaves less CPU 
time available to service thousands of connected clients. The ROI of 
bringing up a new node decreases at the same time as the userbase 
increases.


One traditional approach to solving this is frontend proxies. 
Jabber.com/org used this technique many years ago, and Google has also 
used it to scale up the lockservice 
<http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf> (see 
section 3.1). It's effective because often maintaining connections to 
thousands of clients doesn't involve much brainwork, just shifting 
bytes around. This is especially true of Bitcoin. So if somebody is 
running a full node already they could increase their client capacity 
by just bringing up a frontend proxy and having it handle things like 
outbound tx broadcasts/deduping inbound broadcasts, connection setup, 
relaying recently found blocks etc. A well written proxy could 
probably support tens of thousands of simultaneous clients which frees 
up the bitcoinds time for verification and wallet manipulation.
--
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Many-output transactions in the main chain

[Steve]

Is this the ArtForz solidcoin 'attack'?

On 07/09/11 01:21, Gavin Andresen wrote:
> Somebody has been inserting transactions with lots of outputs into the
> main bitcoin block chain:
>
> http://blockexplorer.com/block/0305f98ffbe1db8445ce847fb9a924551945b465386c828f136f
>
> Their next step will be creating transactions with thousands of inputs
> from those transactions. The result will be lots of excessive disk
> space usage.
>
> The fix is this patch:
>https://github.com/bitcoin/bitcoin/pull/491
>
> Suggestions on the best way to let merchants, miners, and pools know
> about the potential problem?
> I hate to take time away from the 0.4 release to re-spin 0.3.24 with
> the patch, but we may have to.
>

--
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Building a node crawler to map network

[Steve]

Hi Mike,

I expect I'll be submitting patches for bitcoinj sometime in the future 
but I'm not really across it yet to the point where I'd be confident 
submitting patches right now...


This proxy sound like a good match for what I've been up to lately 
though so long as it wouldn't involve direct changes to bitcoind on my 
part.  My c/c++ skills are non-existent.


However I have been building a pool protocol using protobufs and netty 
for non-blocking IO and I'd imagine the kind of multiplexing proxy 
you're talking about could be easily implemented using netty.


I'm not really understanding the use case though.  I believe most 
bitcoind's have a default max connections of 8.  Is the goal to increase 
this without fundamentally altering the bitcoind concurrency model?  Or 
is it to provide capactity for a more hub/client oriented network?  If 
the latter then presumably this is functionality that should ideally be 
native to the client in the long term in the form of NIO?


On 06/09/11 23:31, Mike Hearn wrote:


I've looked but can't find a post like you're talking about.  Can
you point me to it?

https://groups.google.com/forum/?pli=1#!topic/bitcoinj/LSlZdUWcCdk 



If so then bollocks... I'm looking for something useful to do atm.
 PoolServerJ is in a holding pattern atm as I've stabilisied all
the bugs I know about and am waiting for several pools to finish
testing and move into production so I'm twiddling thumbs trying to
figure out how to spend my time.


Patches to BitCoinJ are always welcome :-)

If you'd rather do your own thing, you could experiment with writing a 
proxy that sits in front of bitcoind and multiplexes connections. 
Gavin is concerned about socket exhaustion as users move to 
lightweight clients. Multiplexing proxies are a battle-tested 
technique for reducing the strain of this type of thing. BitCoinJ uses 
thread-per-connection so wouldn't do a good job of that right now, but 
allowing it to use a mix of async io and multi-threading would be a 
nice improvement. It'd need some changes to bitcoind as well for a 
really good effort, to allow for IPs to be forwarded. I'm happy to 
discuss it more with you over on the bitcoinj list if wanted.
--
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Building a node crawler to map network

[Steve]

Hi Mike,

I've looked but can't find a post like you're talking about.  Can you 
point me to it?

If so then bollocks... I'm looking for something useful to do atm.  
PoolServerJ is in a holding pattern atm as I've stabilisied all the bugs 
I know about and am waiting for several pools to finish testing and move 
into production so I'm twiddling thumbs trying to figure out how to 
spend my time.

On 06/09/11 22:49, Mike Hearn wrote:
> Actually Steve, take a look at the bitcoinj mailing list today. 
> Somebody has already built this and has it running. It's accumulating 
> data at the moment, they'll announce it more widely soon. But I think 
> there's no need to duplicate work.

--
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Building a node crawler to map network

[Steve]

While I'm asking questions I'll add one more regarding the getaddr 
message.


Talking to myself here.  I just sent this message then found this 
brilliant set of articles in the Dev & Tech forum which answers the 
question very nicely: *https://bitcointalk.org/index.php?topic=41722.0 


*
Anyway just as an FYI I've been running v0.0.0.0.0.0.0.0.1 for about an 
hour.  It's only running 10 concurrent connections due to girlfriend 
complaining she couldn't watch youtube but here's some early results.


New nodes: 19319 // node address discovered but no contact attempt made yet
Contacted nodes: 754
Uncontactable nodes: 3253
Limbo nodes: 9 //not as exciting as it sounds, just nodes with connect 
in progress
Total nodes: 23335 // about 5000 from initial IRC discover, the rest are 
from getaddr


Versions: {
300=1,
31900=7,
31902=1,
32000=2,
32001=7,
32002=22,
32100=100,
32200=24,
32300=277,
32400=317,
32500=2}

Fails: {
ConnectException: Connection refused=377,
IOException: Socket is disconnected=87,
SocketException: Network is unreachable=2,
ProtocolException: Error deserializing message =1,
NoRouteToHostException: No route to host=115,
SocketException: Connection reset=149,
SocketTimeoutException: connect timed out=2521}


--
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] Building a node crawler to map network

[Steve]

Hi All,

I started messing around today with building a node crawler to try and 
map out the bitcoin network and hopefully provide some useful 
statistics.  It's very basic so far using a mutilated bitcoinj to 
connect (due me being java developer and not having a clue with c/c++). 
  If it's worthwhile I'll hack bitcoinj some more to run on top Netty to 
take advantage of it's NIO architecture (netty's been shown to handle 
1/2 million concurrent connections so would be ideal for the purpose).

Hoping to a get a bit of input into what would be useful as well as 
strategy for getting max possible connections without distorted data.  I 
seem to recall Gavin talking about the need for some kind of network 
health monitoring so I assume there's a need for something like this...

Firstly at the moment basically I'm just storing version message and the 
results of getaddr for each node that I can connect to.  Is there any 
other useful info that can be extracted from a node that's worth collecting?

Second and main issue is how to connect.  From my first very basic 
probing it seems the very vast majority of nodes don't accept incoming 
connections no doubt due to lack of upnp.  So it seems the active crawl 
approach is not really ideal for the purpose.  Even if it was used the 
resultant data would be hopelessly distorted.

A honeypot approach would probably be better if there was some way to 
make a node 'attractive' to other nodes to connect to.  That way it 
could capture non-listening nodes as well.  If there is some way to 
influence other nodes to connect to the crawler node that solves the 
problem.  If there isn't which I suspect is the case then perhaps 
another approach is to build an easy to deploy crawler node that many 
volunteers could run and that could then upload collected data to a 
central repository.

While I'm asking questions I'll add one more regarding the getaddr 
message.  It seems most nodes return about 1000 addresses in response to 
this message.  Obviously most of these nodes haven't actually talked to 
all 1000 on the list so where does this list come from?  Is it mixture 
of addresses obtained from other nodes somehow sorted by timestamp? 
Does it include some nodes discovered by IRC/DNS? Or are those only used 
to find the first nodes to connect to?

Thanks for any input... Hopefully I can build something that's useful 
for the network...

--
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development