Re: [Bitcoin-development] Endianness (was: Linux packaging letter)
On Tue, Jul 23, 2013 at 9:07 PM, Gregory Maxwell wrote: > order to and from the wire, which is part of why the protocol is LE > everywhere, *before someone corrects me, it's not LE everywhere (I meant "manywhere" :P)— there is just enough BE to keep you on your toes. :P -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Endianness (was: Linux packaging letter)
On Tue, Jul 23, 2013 at 8:54 PM, Wendell wrote: > Forking for curiosity's sake: > Is there a substantial barrier to endian independence in the Bitcoin codebase? Not really. The software was originally written to write out memory order to and from the wire, which is part of why the protocol is LE everywhere, so fixing that much is pretty typical endianness fixes. There is an extra kink in that almost everything Bitcoin sends and receives is an authenticated data structure— the stuff gets hashed for authentication. So that simply swizzling the byte order on immediately on input isn't enough because sometimes you'll go on to hash that data and it can't be in memory order for that. Luke gave an initial crack at it a long time ago: http://gitorious.org/~Luke-Jr/bitcoin/luke-jr-bitcoin/commits/endian But it wasn't enough yet. Seems like its just enough of an undertaking that absent a really good reason to care about it no real progress in fixing it is happening. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Endianness (was: Linux packaging letter)
On Wednesday, July 24, 2013 3:54:25 AM Wendell wrote:
> Is there a substantial barrier to endian independence in the Bitcoin
> codebase?
I got the obvious stuff ('endian' branch in my repo), but it still didn't work
when I moved on. I haven't had time to try to figure out why not yet.
Luke
--
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Endianness (was: Linux packaging letter)
Forking for curiosity's sake: Is there a substantial barrier to endian independence in the Bitcoin codebase? -wendell grabhive.com | twitter.com/grabhive On Jul 24, 2013, at 3:45 AM, Douglas Huff wrote: > The fact that you're even trying to package and/or at some point have > packaged and shipped big endian binaries is straight up *NEGLIGENT.* -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 7:35 PM, zooko wrote: > I think some > package maintainers might perceive this version of the letter as high-handed > -- > telling someone else how to do their job -- and they might not notice the > actual facts included in the letter explaining why Bitcoin really *is* > different than a lot of software. Bummer, because this was a explicit consideration while writing it and a concern several people had with the initial draft Mike did. We're very much aware that upstreams frequently cry (wolf) at the mutilation of their unique and precious snowflake. The intention was that second paragraph acknowledging the many good motivations for the existing norms and the third paragraph talking about consensus systems would address these concerns— showing that we aren't totally clueless, and pointing out that we have an actually unusual situation. In intermediate drafts they were longer and more elaborate, but we were struggling against length and trying to avoid delving into a highly technical discussion which would lose anyone who wasn't already very interested. We also compromised on an initial approach of "please don't package this at all" to "please understand first", in part at the protest of our gentoo package (which also bundles leveldb but hard locks it to an exact version in the package system with exact build flags, which is a sophisticated compromise which might not generalize to other distributors) maintainer (uh, Luke-Jr, not exactly the most representative sample). As a first step it's at least important to know that there is a concern here shared by a bunch of people. Helping talk people through understanding it is part of the job here. I certainly didn't expect the discussion to stop with the letter but getting it out there is a way to start the discussion and make it more likely that we have it again with the next packager who comes around. I guess the first priority though is avoiding gratuitously offending people. Can anyone point out any specific tweaks that would reduce initial bristling? On Tue, Jul 23, 2013 at 6:45 PM, Douglas Huff wrote: > Honestly, until I read the quoted part of your response, Oh be nice. If any of this were easy it would all be _done_ already. :) There is naturally some tension when people with different priorities and backgrounds interact, ... I've seen a lot of upstreams run into disagreements with packagers the result is usually better for everyone. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
Folks: With all due respect, I think the letter as I see it at https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi should be changed before being shown to package maintainers. I think some package maintainers might perceive this version of the letter as high-handed -- telling someone else how to do their job -- and they might not notice the actual facts included in the letter explaining why Bitcoin really *is* different than a lot of software. You should understand that without a careful read, this letter sounds much like a cry that packagers have heard from hundreds of other authors who say things to the effect that "my software is different and more important and packager maintainers have to do things my way". Why not solicit the cooperation of a few package maintainers and write a joint letter with them signing on? Instead of it being a one-sided lecture from Bitcoin devs to packagers, it would be a shared statement *and* packagers, and it would be phrased in language that would make it instantly clear to other packagers that this isn't just another whine from ignorant devs. If you're interested in that, there are lots of packagers who would be happy to help. Greg Troxel (pkgsrc) is one, who has already posted to this thread. I'd be happy to invite the ones that I've worked with to package the software that I am a dev on -- Tahoe-LAFS. What I'm proposing is that we contact some packagers and say "Here's this rough draft, and we'd like you to suggest edits that would make it into the kind of letter that you'd sign your name to.". At the very least, we'd learn something from the ensuing conversation. Regards, Zooko -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 6:26 PM, Luke-Jr wrote: > This means a lot of additional work for the > maintainers of the library packages, and the security team; for example, the > security team must understand that they *cannot* deploy a critical security > bugfix to LevelDB until someone competent has reviewed that it is > behaviourally (including bug behaviours!) compatible with the versions in use > everywhere else/previously. I think it is likely all this additional > work/delays will be considered unacceptable to your library/security teams, > thus using the bundled/embedded LevelDB is probably the best solution. The above is a key point, lukejr addressed it well "I think it is likely all this additional work/delays will be considered unacceptable to your library/security teams, thus using the bundled/embedded LevelDB is probably the best solution." >> MIPS has been failing recently, but no one has looked into it yet. >> Perhaps it's not worth developers efforts yet, but at some point the >> technology should reach a point it can be redistributed. > > MIPS (and any other big endian architecture) has *always* failed on the > Satoshi codebase, and will not be supported until someone has time to dedicate > to fixing the numerous little-endian assumptions in the code. I have an > incomplete port, but it isn't very high on my priority list to figure out what > else it's missing. To be clear, bitcoind/bitcoin-qt is only built on little endian machines https://buildd.debian.org/status/package.php?p=bitcoin > Debian could probably get away with packaging Bitcoin-Qt and bitcoind as-is > with no modifications, and not have any problems. It's only when you begin > making modifications that it becomes a problem. There are also some concerns > that it puts a much larger price on compromising Debian's build servers and/or > repositories (suddenly the attacker can steal a LOT of money). The only current modification is to use system leveldb instead of the packaged leveldb. (There is also a patch porting libmemenv.a to several other architectures, but that is only used in test suites - so it shouldn't pose a risk to users). > > The official binaries are not simply built by upstream developers: using > Gitian, *anyone* can produce bit-for-bit identical binaries. Official releases > are only published after 3 or more people have done an independent compile and > signed the output. It would be excellent if the whole of Debian could work > toward achieving this level of security eventually, which would make > distributing Bitcoin node software much safer as well. Ironically, debian (in general) doesn't trust upstream security maintenance of third part libraries - that's why they typically get dropped in favor of use system libraries. In this case, upstream doesn't trust (rightfully) that some future debian security team bug fix to a stable library won't be tested properly against bitcoin, causing problems for users (since bitcoin might expect buggy library behavior). I'm not the original packager or maintainer - I just came across the package in really bad shape and helped bring it to something reasonable and have done the most recent uploads (since 0.8, I believe). Since updated libraries could pose a security risk because bitcoin may expect buggy behavior, I think that is a good argument for debian to use the included library. However, I'm just a recent helper - I still want to hear what people who have been doing this for longer think. ~Scott -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 9:45 PM, Douglas Huff wrote: > Honestly, until I read the quoted part of your response, I actually wasn't in > favor of this whole thing since in general the types of issues being > mentioned are, in large part, the types of issues that maintainers deal with > all the time. > > On Jul 23, 2013, at 3:02 PM, Scott Howard wrote: > >> Response: Is there a way to "certify" the Debian libraries? Debian >> bitcoind/bitcoin-qt runs the compile test during all architectures. >> MIPS has been failing recently, but no one has looked into it yet. >> Perhaps it's not worth developers efforts yet, but at some point the >> technology should reach a point it can be redistributed. > > > The fact that you're even trying to package and/or at some point have > packaged and shipped big endian binaries is straight up *NEGLIGENT.* > > Stop that. Now. It won't work. > > Thanks for showing that this *is* necessary, I guess. before people get too upset, I'm talking about little-endian MIPS (debian-mipsel) http://www.debian.org/ports/mips/ -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
Honestly, until I read the quoted part of your response, I actually wasn't in favor of this whole thing since in general the types of issues being mentioned are, in large part, the types of issues that maintainers deal with all the time. On Jul 23, 2013, at 3:02 PM, Scott Howard wrote: > Response: Is there a way to "certify" the Debian libraries? Debian > bitcoind/bitcoin-qt runs the compile test during all architectures. > MIPS has been failing recently, but no one has looked into it yet. > Perhaps it's not worth developers efforts yet, but at some point the > technology should reach a point it can be redistributed. The fact that you're even trying to package and/or at some point have packaged and shipped big endian binaries is straight up *NEGLIGENT.* Stop that. Now. It won't work. Thanks for showing that this *is* necessary, I guess. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 4:23 PM, Greg Troxel wrote: > Is the repeatable build infrastructure portable (to any reasonable > mostly-POSIX-compliant system, with gcc or clang)? I have the vague It's "portable" to anything that can run the relevant VMs. Uh provided you don't mind cross compiling everything from an unbuntu VM. It certainly would be nice if the trusted-computing-base for gitian were a bit smaller, thats an area for long term improvement for sure. It may need some massaging. The tor project is beginning to use the same infrastructure, so this could be usefully conserved work. Likewise expanding the supported output targets would be great— though in the case of Bitcoin this is bounded by resources to adequately QA builds on alternative targets. > Requiring precise library depdendencies is quite awkward. Certainly > requiring new enough to avoid known bugs is understandable, but that > should be caught at configure time and fail. In some cases packages solving bugs is problematic for Bitcoin. This is something that it seems to take a whiteboard to explain, so I apologize for the opacity of simple email here. From a technical perspective Bitcoin's whole purpose is getting a whole bunch of computers world wide to reach a bit identical agreement on the content of a database, subject to a whole pile of rules, in the face of potentially malicious input, without any trusted parties at all (even the guy you got the software from, assuming you have the resources to audit it). I'll walk through a simple example: Say Bitcoin used a backing database which had an unknown a bug where any item with a key that begins with 0xDEADBEEF returns not found when queried, even if its in the DB. Once discovered, any database library would want to fix that quickly and they'd fix it in a point release without reservation. They might not even release note that particular fix it if went along with some others, it could even be fixed accidentally. Now say that we have a state where half the Bitcoin network is running the old buggy version, and half is running the fixed version. Someone creates a transaction with ID 0xDEADBEEF... and then subsequently spends the output of that transaction. This could be by pure chance or it could be a malicious act. To half the network that spending transaction looks like someone spending coin from nowhere, a violation of the rules. The consensus would then fork, effectively partitioning the network. On each fork any coin could be spent twice, and the fork will only be resolvable by one side or the other abandoning their state (generally the more permissive side would need to be abandoned because the permissive one is tolerant of the restrictive one's behavior) by manually downgrading or patching software. As a result of this parties who believed some of their transactions were safely settled would find them reversed by people who exploited the inconsistent consensus. To deploy such a fix in Bitcoin without creating a risk for participants we need to make a staged revision of the network protocol rules: There would be a protocol update that fixed the database bug _and_ explicitly rejected 0xDEADBEEF transactions until either some far out future date or until triggered by quorum sensing (or both). The users of Bitcoin would all be advised that they had to apply fixes/workaround by the switchover point or be left out of service or vulnerable. At designated time / quorum nodes would simultaneously switch to the new behavior. (Or in some cases, we'd just move the 'bug' into the Bitcoin code so that it can be fixed in the database, and we'd then just keep it forever, depending on how harmful it was to Bitcoin, a one if 4 billion chance of having to rewrite a transaction wouldn't be a big deal) We've done these organized updates to solve problems (as various flaws in Bitcoin itself can present similar consensus risks) several times with great success, typical time horizons spanning for months to years. But it cannot work if the behavior is changed out from under the software. Fortunately, if the number of users running with an uncontrolled consensus relevant inconsistent behavior is small the danger is only to themselves (and, perhaps, their customers). I'm not happy to see anyone get harmed, but it's better if its few people than many. This is part of the reason that it's a "linux packaging letter", since for Bitcoin the combination of uncoordinated patching and non-trivial userbases appears to be currently unique to GNU/Linux systems. Though indeed, the concerns do apply more broadly. > multiple packages is difficult, and runs into A wants only n of C, while > B wants only m. My understanding is that gentoo is actually able to handle this (and does, for Bitcoin)— and really I presume just about everything else could with enough effort. I certainly wouldn't ask anyone else to do that. If you're really getting into the rathole of building separate libraries just for Bit
Re: [Bitcoin-development] Linux packaging letter
On Tuesday, July 23, 2013 11:23:27 PM Greg Troxel wrote: > I find it interesting that this is a "linux packaging letter". How much > of this applies to pkgsrc, FreeBSD ports, OpenBSD ports, and other > non-Linux packaging systems (pkgsrc supports Linux as on of 20 operating > systems, but is not a "Linux packaging system")? It was written with bitcoind/Bitcoin-Qt in mind, which don't work on BSD. :p > Is the repeatable build infrastructure portable (to any reasonable > mostly-POSIX-compliant system, with gcc or clang)? I have the vague > impression it's Ubuntu only, but I am very unclear on this point. How > does this repeatableness interact with building for multiple operating > systems and cpu types (say 20 OS, with typically 3 versions of the OS > for each, with 1-20 cpu types per OS, for a cross-product of perhaps 200 > combinations)? It should be portable to other systems, though hasn't been done yet. Would be nice if the concepts it uses could be integrated into the package- building systems. > Requiring precise library depdendencies is quite awkward. Certainly > requiring new enough to avoid known bugs is understandable, but that > should be caught at configure time and fail. The problem is that we require bugs. That is, if your library has those bugs fixed, you have introduced a security vulnerability. > It seems like a bug that the package will build on BE systems and then > fail tests. If it's known not to be ok, it seems that absent some > configure-time flag the build should fail. There is no configure-time for this node software yet. The autoconf-based one in the works *does* make this check, however. > Asking people not to patch should mean willingnesss to make accomodation > in the master sources for build issues for multiple packaging systems. > I haven't gotten around to packaging this for pkgsrc - so far I only > have the energy to lurk (due to too many things on the todo list). But > I often find that some changes are needed. If you're willing (in > theory) to add in configure flags to control build behavior (in a way > that you can audit and decide is safe), that's great, and of course we > can discuss an actual situation when one gets figured out. The review process is very slow and strict, but that is because of the same reasons it isn't safe to distribute patched versions in general. Merging your patches to mainline is not only a good idea, but it helps to ensure they get the necessary testing needed to be safe. Luke signature.asc Description: This is a digitally signed message part. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
I find it interesting that this is a "linux packaging letter". How much of this applies to pkgsrc, FreeBSD ports, OpenBSD ports, and other non-Linux packaging systems (pkgsrc supports Linux as on of 20 operating systems, but is not a "Linux packaging system")? Is the repeatable build infrastructure portable (to any reasonable mostly-POSIX-compliant system, with gcc or clang)? I have the vague impression it's Ubuntu only, but I am very unclear on this point. How does this repeatableness interact with building for multiple operating systems and cpu types (say 20 OS, with typically 3 versions of the OS for each, with 1-20 cpu types per OS, for a cross-product of perhaps 200 combinations)? Requiring precise library depdendencies is quite awkward. Certainly requiring new enough to avoid known bugs is understandable, but that should be caught at configure time and fail. Synchronous updates of multiple packages is difficult, and runs into A wants only n of C, while B wants only m. So if you are talking about running regression tests with the set of versions of a dependency that are considered reasonable, and there's therefore a solution to the multiple-package constraint problem, that seems ok. It seems like a bug that the package will build on BE systems and then fail tests. If it's known not to be ok, it seems that absent some configure-time flag the build should fail. Asking people not to patch should mean willingnesss to make accomodation in the master sources for build issues for multiple packaging systems. I haven't gotten around to packaging this for pkgsrc - so far I only have the energy to lurk (due to too many things on the todo list). But I often find that some changes are needed. If you're willing (in theory) to add in configure flags to control build behavior (in a way that you can audit and decide is safe), that's great, and of course we can discuss an actual situation when one gets figured out. Greg -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 10:01:55PM +0200, Mike Hearn wrote: > The trigger for this is the discovery that Debian bitcoind's got split out > of the consensus some time in April, for reasons that nobody yet figured > out but is presumably related to a patch (eg it uses system leveldb). Just to make sure there are no misunderstandings, as far as I know there is no reason to assume the reported problem (comment on #2726) is: 1) a fork (it's an indeterministic and avoidable database corruption, it seems) 2) related to leveldb 3) reproducible by more than one person 4) debian's fault. That said, I think reaching out to packagers to educate them about the risks is a good idea - but let's not blame people before we understand our own problems. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tuesday, July 23, 2013 10:02:28 PM Scott Howard wrote: > 1) It appears that the consensus of upstream developers is that any > distributed binary should only be linked against libraries that the > bitcoin developers have tested and audited since any compatibility bug > is a risk to both the user and the network. > > Response: Is there a way to "certify" the Debian libraries? Debian > bitcoind/bitcoin-qt runs the compile test during all architectures. It doesn't need to be audited by any given person or team, just someone who understands the issues and can dedicate the time to doing a competent audit. Testing bitcoind/bitcoin-qt is not sufficient: you must guarantee that your libraries (especially LevelDB) are bug-for-bug compatible with the ones used by everyone else. And not only the current versions, but every future version to ever hit the repository. This means a lot of additional work for the maintainers of the library packages, and the security team; for example, the security team must understand that they *cannot* deploy a critical security bugfix to LevelDB until someone competent has reviewed that it is behaviourally (including bug behaviours!) compatible with the versions in use everywhere else/previously. I think it is likely all this additional work/delays will be considered unacceptable to your library/security teams, thus using the bundled/embedded LevelDB is probably the best solution. > MIPS has been failing recently, but no one has looked into it yet. > Perhaps it's not worth developers efforts yet, but at some point the > technology should reach a point it can be redistributed. MIPS (and any other big endian architecture) has *always* failed on the Satoshi codebase, and will not be supported until someone has time to dedicate to fixing the numerous little-endian assumptions in the code. I have an incomplete port, but it isn't very high on my priority list to figure out what else it's missing. > 2) Bitcoin is new technology, so any patches have the ability of > harming both the network and user. > > Response: I, and I'm sure everyone else working on it, totally agrees. > All patches are public [1], you can see that the patches are only to > the build system (except for one adding a debug message). Is there a > specific patch or bug that is problematic? This seems to be > reiterating (1) above: don't patch the build system to use libraries > that were not audited by the developers. > > The two solutions are: (1) no one besides the upstream developers > compiles and distributes binaries, ever, or (2) upstream comes up with > a system where someone besides them can compile working binaries for > distribution. Most likely the best solution is to do (1) until > upstream sets up a system to allow (2). Debian could probably get away with packaging Bitcoin-Qt and bitcoind as-is with no modifications, and not have any problems. It's only when you begin making modifications that it becomes a problem. There are also some concerns that it puts a much larger price on compromising Debian's build servers and/or repositories (suddenly the attacker can steal a LOT of money). The official binaries are not simply built by upstream developers: using Gitian, *anyone* can produce bit-for-bit identical binaries. Official releases are only published after 3 or more people have done an independent compile and signed the output. It would be excellent if the whole of Debian could work toward achieving this level of security eventually, which would make distributing Bitcoin node software much safer as well. Luke signature.asc Description: This is a digitally signed message part. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 4:01 PM, Mike Hearn wrote: > Hi, > > Some of us have put together an open letter to the Linux packaging > community, outlining why Bitcoin is different to other programs and asking > them to not patch or modify the upstream sources. > > Please consider signing it if you agree (I think the wording by now is fine, > so don't edit the contents - use the comment feature if you want to though). > > https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi > > The trigger for this is the discovery that Debian bitcoind's got split out > of the consensus some time in April, for reasons that nobody yet figured out > but is presumably related to a patch (eg it uses system leveldb). Hi Mike, Debian's bitcoin is maintained on an open and archived mailing list and git repo: Debian Bitcoin Packaging Team If there is a problem or question, getting an answer should be really easy. It would be good to include them in the discussion there (I CC:ed the list). If the upstream developers have a consensus that distribution packaging is not in the best interest of the project, then I personally would defer to their judgement and request removal. I'm leaving this open for other opinions from the Debian side. That said, let me summarize the arguments and see if we can figure out a permanent solution: 1) It appears that the consensus of upstream developers is that any distributed binary should only be linked against libraries that the bitcoin developers have tested and audited since any compatibility bug is a risk to both the user and the network. Response: Is there a way to "certify" the Debian libraries? Debian bitcoind/bitcoin-qt runs the compile test during all architectures. MIPS has been failing recently, but no one has looked into it yet. Perhaps it's not worth developers efforts yet, but at some point the technology should reach a point it can be redistributed. 2) Bitcoin is new technology, so any patches have the ability of harming both the network and user. Response: I, and I'm sure everyone else working on it, totally agrees. All patches are public [1], you can see that the patches are only to the build system (except for one adding a debug message). Is there a specific patch or bug that is problematic? This seems to be reiterating (1) above: don't patch the build system to use libraries that were not audited by the developers. The two solutions are: (1) no one besides the upstream developers compiles and distributes binaries, ever, or (2) upstream comes up with a system where someone besides them can compile working binaries for distribution. Most likely the best solution is to do (1) until upstream sets up a system to allow (2). I'm curious as to other's opinions. Thanks, Scott [1] http://anonscm.debian.org/gitweb/?p=collab-maint/bitcoin.git;a=tree;f=debian/patches;h=ba576f9f3ddad47a2f85dcbfb7a0b3482834f19f;hb=HEAD -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 1:32 PM, Mike Hearn wrote: > Yes. Someone decided to actually delete the people who had signed so far and Some people/person went and actually started making substantive edits to the text. The text it's rolled back to is missing the last copyedits from last night too. The text that had been ACKed last night was a3e52973, available at http://luke.dashjr.org/tmp/code/20130723-linux-distribution-packaging-and-bitcoin.md As far as the PGP goes— I think using the PGP is good: it's making use of the right tools, avoids issues like we just had where people go changing the content after names had been affixed, shows solidarity with people building security infrastructure that our ecosystem depends on. If you only use it occasionally then its easy for someone to strip it when it _is_ needed and disguise that just as regular non-use. It's my general view that for people working in our domain basic competence and use of these tools, even when they kinda stink, is a kind of civic hygiene. At the same time it's not urgent. It's poorly used by people and will be ignored by most but packagers are the most frequent users of it that I've encountered. Fortunately, it's harmless in any case. If people are interested in offering PGP signatures of it: wget http://luke.dashjr.org/tmp/code/20130723-linux-distribution-packaging-and-bitcoin.md gpg --clearsign 20130723-linux-distribution-packaging-and-bitcoin.md and post the little signature asc. The result composes nicely: http://luke.dashjr.org/tmp/code/20130723-linux-distribution-packaging-and-bitcoin.md.asc -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
Yes. Someone decided to actually delete the people who had signed so far and replace it with a request for PGP signing - no. Not everyone even uses PGP, which is overkill for this anyway. I'm going to roll the document back and lock it. Sorry, I had hoped people would respect my request to not fiddle with the content, which they did not do. If you'd like to have your name on it, let me know or post here and I'll add it. On Tue, Jul 23, 2013 at 10:14 PM, Gregory Maxwell wrote: > On Tue, Jul 23, 2013 at 1:01 PM, Mike Hearn wrote: > > Hi, > > > > Some of us have put together an open letter to the Linux packaging > > community, outlining why Bitcoin is different to other programs and > asking > > them to not patch or modify the upstream sources. > > > > Please consider signing it if you agree (I think the wording by now is > fine, > > so don't edit the contents - use the comment feature if you want to > though). > > > > > https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi > > Ah, this is not entirely in sync with the last (mostly minor) > copyedits that had been signed off by Gavin, Pieter, Jgarzik, and I... > but that appears to be a smaller issue than the fact that the whole > thing is now being rewritten by "anonymous beaver" and friends. > -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 1:01 PM, Mike Hearn wrote: > Hi, > > Some of us have put together an open letter to the Linux packaging > community, outlining why Bitcoin is different to other programs and asking > them to not patch or modify the upstream sources. > > Please consider signing it if you agree (I think the wording by now is fine, > so don't edit the contents - use the comment feature if you want to though). > > https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi Ah, this is not entirely in sync with the last (mostly minor) copyedits that had been signed off by Gavin, Pieter, Jgarzik, and I... but that appears to be a smaller issue than the fact that the whole thing is now being rewritten by "anonymous beaver" and friends. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 7/23/13 3:29 AM, Andreas Schildbach wrote: > > Yes, I understand that. For this reason, I would vote for adding the > usual HTTP authentication/SSL stuff to the REST API. That way, SPV users > can decide to run their own instance of the API (providing the needed > resources themselves). > > Or, a trusted party can set up a server. For example, I would be willing > to set it up for users of Bitcoin Wallet. I don't expect shitloads of > paper wallets sweeps for the forseeable future. > > Anyone who wants HTTP authentication or TLS can wrap it with nginx, or something similar. In the process they could put appropriate restrictions in place on incoming requests, and the onus would be on them, not us to keep it secure. Mark -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJR7ttWAAoJEAdzVfsmodw4UmIP/36lK2TDc7mLTT8rbflJhl3v TL4CFKhXj6OuzG7tyino3Djs4EQnyk+CbpfOmJ8kYr29GPaZttuDJhYXtJqQBQCi DPq79ktudHnVMLPirEs7dUrLo+TAqhYX+8Sj+eTlW+p6YZg3JbkOAIPJG7597OK4 zzU8Oxr0XKJFfGscKfkPThxJboNqzJYGl3otHUMXM4HsbIRYmrx4QSr8y7dsVgTd YZnD4bJO+eY4ZPzCcFdkPD/8bXQyKC5nPOH8/79lARNLESwB4OW79uf9q86EuH2O jZQ1qwpRNHblrNWS1/U2E4+7hEidvgZBwQhj+HbWgKiPWh4Df1lEXq6bLQQwdn6/ b+jfiwg7xpb7eB2M4gPZ0uF/1TIcGJN3+LWEULFNTT/vsjyD/UU63ahZ1kVv7X0m W1NrbKjXxDbip+x3N7HLIu3zqAAaa0ele7OysyFCL6ZlwwafwJiEZZgHn2Iw7I1L S7lYBbFoLfXlOMVXNaKHPEV5gQEveMROJVBtnWkqShPQM0N/+Z+TXZes37up0GVo d7ptPfNbUNDTFc8Jj3+5rIyy3dUvSyMJlHZhsLmtCUnbQ867ZOgeUS52a8XQ+nJY 8IsShLfLk6fRWmHrwo9lzZQ/TbbUNyoUje0Ns6iL7G3IZwDqJH3kAGb/bkj/piDu tPNcN8bkYeNobTFIH+o4 =jV80 -END PGP SIGNATURE- -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Linux packaging letter
Hi, Some of us have put together an open letter to the Linux packaging community, outlining why Bitcoin is different to other programs and asking them to not patch or modify the upstream sources. Please consider signing it if you agree (I think the wording by now is fine, so don't edit the contents - use the comment feature if you want to though). https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi The trigger for this is the discovery that Debian bitcoind's got split out of the consensus some time in April, for reasons that nobody yet figured out but is presumably related to a patch (eg it uses system leveldb). -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 4:36 AM, Pieter Wuille wrote: > Apart from that, exposing this HTTP-based interface publicly has its > own problems, like security risks and potential DoS risks. If > anything, we should be reducing the attack surface rather than > increase it. IMHO, the only thing that should be exposed in the P2P > protocol, which is inevitable, and already has some DoS protections. > > I like this HTTP interface, but it should really only be used for > trusted local applications and debugging. > We already have a good private HTTP interface. Most benefits of this REST interface come from exposing it publicly. As always, the challenge is balancing costs and benefits. I'm not confident that either can be known with certainty until a well-written prototype is running in the wild. As some nodes expose this interface, we'll gain concrete experience from which node operators can make informed security trade offs. -- Michael -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tuesday 23 July 2013 11:17:28 Peter Todd wrote: > On Tue, Jul 23, 2013 at 11:00:24AM +0100, Andy Parkins wrote: > > > Increasing the resource usage by SPV clients on full nodes is > > > undesirable; > > > > I don't think that's thinking big enough. What I imagine is that making > > it easier and easier to store a partial blockchain would result in lower > > demand on full nodes. > > Read my proposal for "Partial UTXO" mode: > http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg02 > 511.html Very interesting. I love the idea of the UTXO set being tied to a block. > > I might run a client that has only fetched blocks that contain > > transactions needed to verify my balances, right back to the genesis > > block. That will be some small subset of the block chain and will take > > me very little resource to maintain. I join the network and am my > > client is willing to verify based on information I have, or supply (by > > REST or bitcoin protocol) blocks. Imagine then that everyone with a > > wallet were doing this. The blockchain would be distributed massively. > > Obviously the miners would still be keeping the entire chain, but we'd > > have a lot more nodes in the network, each contributing a little bit and > > so reducing the load on the full nodes. > > Actually the really scary thing about partial UTXO mode is miners can > get away without keeping the entire chain provided they don't (often) > try to mine transactions spending UTXO's that they haven't verified > themselves. You're right. That is scary. > > > In any case UTXO data currently requires you to have full trust in > > > whomever is providing you with it, and that situation will continue > > > until UTXO commitments are implemented - if they are implemented. > > > > Almost; because you can go and ask someone else the same question, it's > > pretty > > How do you know they actually are someone else? You don't. You can't invalidate the lie if all you have access to is lies. But if you have access to just one honest node; that will reveal the liars. I'm not claiming that headers-only nodes can ever be made as secure as a full node. Just _more_ secure than they are now; and potentially able to act as one of those honest nodes. > > easy to check if you're being lied to. Also, it's far easier to maintain > > a headers-only block chain. When you fetch your relevant block subset, > > you can easily see that they are real blocks in your headers-only > > blockchain; and so it's pretty much impossible to lie to "give me the > > block containing transaction X". > > Do you think you have SPV or full security in that situation? > Do you know the difference? There is absolutely no need to get condescendingly shirty. I thought this was a friendly list; and we were having a discussion. If you don't want to respond to posts -- don't. I also didn't realise I had to pass an exam before I was allowed to speak. Yes: I know the difference between SPV and full security. SPV is headers only and so has no way to verify that the transaction outputs references as inputs to any new as-yet-unverified transaction are valid. Instead it relies on having some way of proving it's in the chain; and then looking for the number of blocks built on top of it as "verification". "Full security" (which is itself a very poor name), is obviously just checking that every output referenced in the inputs is unspent; that necessarily requires full blocks. The difference in security being that in SPV there is no way to know if the referenced Unspent TransaXtion Output really is unspent -- it might have been spent elsewhere then referenced again in this new transaction. My suggestion was that we want to be able to fetch a block by transaction; and that simple nodes can all, in aggregate offer contribution to the network rather than just being parasitical on the full nodes. When I ask for a block that contains a transaction, and I do that repeatedly, I have part of the block chain. If lots of simple nodes are doing that, then the whole chain should be available if there are enough of them. They would then gain the ability to do transaction-forwarding in some cases. This is only possible if a few extra facilities are added to the protocol. One of which is the new feature I suggested: block-given-transaction. It's not enough on its own, but if you also add in the ability for a node to tell another about the output transactions (basically, what block spends it), _then_ the simple nodes are able to become much more secure -- not 100% of course, they're still not full nodes, because they have no way of knowing if they are being lied to when they are told (this transaction is unspent), but all it takes is one honest node to point them at the truth, and the lie is then exposed. That facility is just a drain on full nodes for the most part; except if you start encouraging it whole-sale. The simple node wou
[Bitcoin-development] Distributing low POW headers
I was thinking about a change to the rules for distinguishing between forks
and maybe a BIP..
*Summary*
- Low POW headers should be broadcast by the network
If a header has more than 1/64 of the POW of a block, it should be
broadcast. This provides information on which fork is getting most of the
hashing power.
- Miners should use the header information to decide on longest chain
The fork selection rule for miners should be biased towards staying on the
fork that has the most hashing power.
This means that they might keep hashing on a fork that is 1-2 blocks
shorter.
If most miners follow the rule, then it is the best strategy for other
miners to also follow this rule.
- Advantages
This lowers the probability of natural and malicious reversals.
*Distributing low POW headers*
First block header messages that have more than 1/64 of the standard POW
requirements would be forwarded.
This means the client needs to maintain a short term view of the entire
header tree.
if (header extends header tree) {
if (header meets full POW) {
add to header tree;
forward to peers;
check if any blocks in storage now extend the header tree
} else {
if (header meets POW / 64) {
forward to peers;
}
} else {
if (header meets POW) {
add to orphan header storage
}
}
The storage could be limited and headers could be discarded after a while.
This has the extra advantage that it informs clients of forks that are
receiving hashing power.
This could be linked to a protocol version to prevent disconnects due to
invalid header messages.
*Determining the longest chain*
Each link would get extra credit for headers received.
Assume there are 2 forks starting at block A as the fork point.
A(63) <- B(72) <- C(37) <- D(58)
and
A(63) <- B'(6) <- C'(9) <- D'(4) <- E(7) <- F(6)
The numbers in brackets are the number of low POW headers received that
have those blocks as parent.
The new rule is that the POW for a block is equal to
POW * (1 + (headers / 16))
Only headers within of the end of the (shorter) chain
count. However, in most cases, that doesn't matter since the fork point
will act as the start point. As long as miners keep headers for 30-40
blocks, they will likely have all headers back to any reasonable fork point.
This means that the top fork is considered longer, since it has much more
headers, even though it has 2 less blocks.
If 75% of the miners follow this rule, then the top fork will eventually
catch up and win, so it is in the interests of the other 25% to follow the
rule too.
Even if there isn't complete agreement on headers received, the fork that
is getting the most hashing will naturally gain most of the headers, so
ties will be broken quickly.
--
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 12:29 PM, Andreas Schildbach wrote: > Yes, I understand that. For this reason, I would vote for adding the > usual HTTP authentication/SSL stuff to the REST API. That way, SPV users > can decide to run their own instance of the API (providing the needed > resources themselves). > > Or, a trusted party can set up a server. For example, I would be willing > to set it up for users of Bitcoin Wallet. I don't expect shitloads of > paper wallets sweeps for the forseeable future. I don't object to using a trusted server for this if people want that, but I don't think the reference client should encourage this. Apart from that, exposing this HTTP-based interface publicly has its own problems, like security risks and potential DoS risks. If anything, we should be reducing the attack surface rather than increase it. IMHO, the only thing that should be exposed in the P2P protocol, which is inevitable, and already has some DoS protections. I like this HTTP interface, but it should really only be used for trusted local applications and debugging. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On 07/23/2013 11:47 AM, Peter Todd wrote: >> Is it planned to expose the UXTO set of a given address? That would be >> useful for SPV wallets to be able to swipe a previously unknown private >> key (e.g. paper wallet). > > The REST API has nothing to do with SPV clients; it's similar to the RPC > interface and won't be exposed to the network as a whole. > > Increasing the resource usage by SPV clients on full nodes is undesirable; we > have a lot of work to do regarding DoS attacks. Yes, I understand that. For this reason, I would vote for adding the usual HTTP authentication/SSL stuff to the REST API. That way, SPV users can decide to run their own instance of the API (providing the needed resources themselves). Or, a trusted party can set up a server. For example, I would be willing to set it up for users of Bitcoin Wallet. I don't expect shitloads of paper wallets sweeps for the forseeable future. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 12:17 PM, Andreas Schildbach wrote: > Sweeping paper wallets is a common feature request. People switch to > centralized services just for getting that. That means they value convenience more than the trust-freeness of a decentralized solution. The only way to avoid that is by making sure the decentralized one is convenient enough. But relying on unauthenticated data itself is equally bad - it means you lose whatever benefit the decentralization had. > It is my understanding that for the usecase, an address-indexed UXTO is > enough. So you probably don't need to worry about script-indexed for now. The difference between script-indexed and address-indexed is absolutely trivial compared to the effort needed to implement and maintain such authenticated trees by all full nodes. Restricting things at the network level (which doesn't even know about a thing like an address) to address-based indexes is ridiculous IMHO. > Security issues could be mitigated by applying trust to the REST server, > e.g. because its your own or the one of your apps vendor. Of course, > link-level security would be needed for this (e.g. SSL). Sure, once you introduce trust, a lot can be done. But it's not really Bitcoin anymore in that case - it's relying on a third party to do the heavy indexing for you. And if that is the best-scaling solution, sure - but I don't think we should encourage that. Or at least, we should first search for alternatives. And encourage infrastructure that doesn't require it. > Paper wallets that include the necessary additional information is > something I have been thinking about. I see some issues: > > - Paper wallets are already quite widespread. You still won't be able to > sweep those. > - Some people like to "top up" a paper wallet or even just sweep a > portion of it. That would not be possible, and in some cases even lead > to loss of coins because of the "involuntary fee" you described. Yeah, those are inherent problems with how there are used today. But today there is also little problem - the UTXO set is tiny. > - Does the necessary info fit into a QR code? Absolutely, though a slightly bigger one. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 12:00 PM, Andy Parkins wrote: >> The REST API has nothing to do with SPV clients; it's similar to the RPC >> interface and won't be exposed to the network as a whole. > > Yes; I know that. I'm saying that it would make it easier for SPV (and other > lightweight clients) for that matter. In what way? >> Increasing the resource usage by SPV clients on full nodes is undesirable; > > I don't think that's thinking big enough. What I imagine is that making it > easier and easier to store a partial blockchain would result in lower demand > on full nodes. A partial blockchain is quite useless, as you can't build an UTXO set from it. If you're talking simply about the storage requirements for maintaining history, perhaps, but why rely on SPV nodes for this? Right now, those don't store any blocks at all, and there is no reason why they should. The only requirement is that old blocks remain available for new full nodes to be able to bootstrap. It's certainly not required that everyone keeps every block ever created. That's how the software currently works, but as soon as we get to a few protocol changes that would allow new full nodes to find specific peers with the data they need, we can have fully-verifying yet (partially) pruning nodes. I think that's a much better idea than conflating "maintaining a wallet" with "maintaining a subset of historical block data". > Obviously the miners would still be keeping the entire > chain, but we'd have a lot more nodes in the network, each contributing a > little bit and so reducing the load on the full nodes. I disagree strongly here. The rules of the network are enforced by full nodes, not by miners - they are just trying to satisfy the rules demaned by the network. And as far as I know, there is no way to do some "partial validation" using just the blocks you care about (and remember, SPV nodes currently have none at all). One interesting possibility here is fraud proofs, where the network can relay proofs that certain blocks or transactions are violating certain network rules. In this case, even SPV nodes can just communicate with eachother and get some "herd immunity". But storing some blocks doesn't matter here - it is all about whether you can maintain the UTXO set or not. > >> In any case UTXO data currently requires you to have full trust in >> whomever is providing you with it, and that situation will continue >> until UTXO commitments are implemented - if they are implemented. > > Almost; because you can go and ask someone else the same question, it's pretty > easy to check if you're being lied to. Also, it's far easier to maintain a > headers-only block chain. When you fetch your relevant block subset, you can > easily see that they are real blocks in your headers-only blockchain; and so > it's pretty much impossible to lie to "give me the block containing > transaction X". That's assuming there is no powerful enough attacker that can benefit from doing a sybil attack on you. For SPV nodes currently, that risk is limited to an attacker that can spend enough on faking a chain with valid proof-of-work, to make you accept a transaction that will be reversed. If you go let SPV nodes rely on unauthenticated UTXO set data, there is no such limitation, and they can let you believe *anything*. There are some safeguards, like combining it with merkle paths for all outputs that credit you (which again requires a more expensive index on the other side), but you can't ever guarantee that a particular outputs isn't spent yet. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On 07/23/2013 11:37 AM, Pieter Wuille wrote: >> Is it planned to expose the UXTO set of a given address? That would be >> useful for SPV wallets to be able to swipe a previously unknown private >> key (e.g. paper wallet). > > Depends what you mean by expose. > > Maintaining an address/script-indexed UTXO is generally useful, in > particular for things like sweeping addresses. I certainly have > less problems with 'exposing' this than exposing a fully-indexed > block chain history. > > However, and I expect that's what your question is about, this isn't > really useful for SPV (or less) nodes, as there is no way to > authenticate this data. If you can fake a UTXO entry, you can make > a peer believe anything about their balance, potentially resulting > in creating a valid transaction that sends change it didn't know > was there as fee to miners. Other than for normal block chain data, > there is no way to detect this without at least partial validation. > > The only way to do this safely at an SPV security assumption, is by > having an address-indexed committed merkle UTXO-set tree, like the > one proposed by Alan Reiner, and being implemented by Mark > Friedenback. I know Michael Gronager has something similar implemented, > but I don't know whether it is script-indexed. To be actually useful, > it likely needs to be enforced by miners - putting a significant > burden on validation nodes. Still, if it can be done efficiently, > I think this would be worth it, but more research is needed first in > any case. > > Regarding sweeping keys in the first place - I think using those, > and relying on address-indexed UTXO sets or blockchains to import > them, is an idea that doesn't scale very well in the first place. > If it is for things like scratch card or physical coins, with a > pre-set value, the obvious solution IMHO is storing the crediting > transaction with its merkle path together with the key. If that's > not possible, just the txid:vout of the credit output can suffice. > Yes, that's more data than is necessary now, but it's so much more > trivial to use. I certainly don't want to push ideas that won't work for whatever reason. So I fully respect whatever you decide regarding that feature. Personally I have never felt any need for being able to sweep paper wallets, I am more or less just relaying the need of users. Let me just say this: Sweeping paper wallets is a common feature request. People switch to centralized services just for getting that. It is my understanding that for the usecase, an address-indexed UXTO is enough. So you probably don't need to worry about script-indexed for now. Security issues could be mitigated by applying trust to the REST server, e.g. because its your own or the one of your apps vendor. Of course, link-level security would be needed for this (e.g. SSL). Paper wallets that include the necessary additional information is something I have been thinking about. I see some issues: - Paper wallets are already quite widespread. You still won't be able to sweep those. - Some people like to "top up" a paper wallet or even just sweep a portion of it. That would not be possible, and in some cases even lead to loss of coins because of the "involuntary fee" you described. - Does the necessary info fit into a QR code? -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 11:00:24AM +0100, Andy Parkins wrote: > > Increasing the resource usage by SPV clients on full nodes is undesirable; > > I don't think that's thinking big enough. What I imagine is that making it > easier and easier to store a partial blockchain would result in lower demand > on full nodes. Read my proposal for "Partial UTXO" mode: http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg02511.html > I might run a client that has only fetched blocks that contain transactions > needed to verify my balances, right back to the genesis block. That will be > some small subset of the block chain and will take me very little resource to > maintain. I join the network and am my client is willing to verify based on > information I have, or supply (by REST or bitcoin protocol) blocks. Imagine > then that everyone with a wallet were doing this. The blockchain would be > distributed massively. Obviously the miners would still be keeping the > entire > chain, but we'd have a lot more nodes in the network, each contributing a > little bit and so reducing the load on the full nodes. Actually the really scary thing about partial UTXO mode is miners can get away without keeping the entire chain provided they don't (often) try to mine transactions spending UTXO's that they haven't verified themselves. They can get away with accepting blocks without checking that the UTXO's exist, at least until enough miners do so that someone creates an invalid block and the majority of hashing power never notices. Remember that only with a complete UTXO set can you know that a UTXO *doesn't* exist. We're going to have to force miners to prove they possess the full UTXO set in the future or the security of Bitcoin will be seriously threatened. > > In any case UTXO data currently requires you to have full trust in > > whomever is providing you with it, and that situation will continue > > until UTXO commitments are implemented - if they are implemented. > > Almost; because you can go and ask someone else the same question, it's > pretty How do you know they actually are someone else? > easy to check if you're being lied to. Also, it's far easier to maintain a > headers-only block chain. When you fetch your relevant block subset, you can > easily see that they are real blocks in your headers-only blockchain; and so > it's pretty much impossible to lie to "give me the block containing > transaction X". Do you think you have SPV or full security in that situation? Do you know the difference? -- 'peter'[:-1]@petertodd.org 0070f3d118303a611e1f44ea6482a3b59a16056e69af088b1ffa signature.asc Description: Digital signature -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 12:02 PM, Andy Parkins wrote: > On Tuesday 23 July 2013 10:56:02 Pieter Wuille wrote: > >> The block chain is not involved at all to verify transactions, it's >> just a historical >> record to serve to other nodes, and to do wallet rescans with. > > It must be involved to some extent. Certainly during a temporary fork, there > are two branches growing, and you have to be able, when verifying a new > transaction, to say which branch it's one... which branch of the blockchain. No, not really. The UTXO set is the state you need to validate blocks and transactions. You can see blocks as authenticated patches to the UTXO set (consumes some outputs, produces others). During validation, we store "undo data", basically (non-authenticated) reverse patches to the UTXO set, so we can walk back in case of a reorganization. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tuesday 23 July 2013 10:56:02 Pieter Wuille wrote: > The block chain is not involved at all to verify transactions, it's > just a historical > record to serve to other nodes, and to do wallet rescans with. It must be involved to some extent. Certainly during a temporary fork, there are two branches growing, and you have to be able, when verifying a new transaction, to say which branch it's one... which branch of the blockchain. Andy -- Dr Andy Parkins andypark...@gmail.com -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tuesday 23 July 2013 10:47:03 Peter Todd wrote: > On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote: > > One additional URL makes this pretty much perfect: > > GET /rest/block-with-tx/TX-HASH > > > > Construction of the transaction-hash-to-block database is something the > > full client's have to do anyway, so this query is no harder than the > > others for them to supply; but suddenly makes it possible for an SPV > > client to trace the providence of any transaction without needing to > > maintain the entire chain. > The REST API has nothing to do with SPV clients; it's similar to the RPC > interface and won't be exposed to the network as a whole. Yes; I know that. I'm saying that it would make it easier for SPV (and other lightweight clients) for that matter. > Increasing the resource usage by SPV clients on full nodes is undesirable; I don't think that's thinking big enough. What I imagine is that making it easier and easier to store a partial blockchain would result in lower demand on full nodes. I might run a client that has only fetched blocks that contain transactions needed to verify my balances, right back to the genesis block. That will be some small subset of the block chain and will take me very little resource to maintain. I join the network and am my client is willing to verify based on information I have, or supply (by REST or bitcoin protocol) blocks. Imagine then that everyone with a wallet were doing this. The blockchain would be distributed massively. Obviously the miners would still be keeping the entire chain, but we'd have a lot more nodes in the network, each contributing a little bit and so reducing the load on the full nodes. > In any case UTXO data currently requires you to have full trust in > whomever is providing you with it, and that situation will continue > until UTXO commitments are implemented - if they are implemented. Almost; because you can go and ask someone else the same question, it's pretty easy to check if you're being lied to. Also, it's far easier to maintain a headers-only block chain. When you fetch your relevant block subset, you can easily see that they are real blocks in your headers-only blockchain; and so it's pretty much impossible to lie to "give me the block containing transaction X". Andy -- Dr Andy Parkins andypark...@gmail.com -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 11:52 AM, Andy Parkins wrote: >> There is actually no such index being maintained by default, and doing so >> is an unnecessary burden IMHO (you need to enable -txindex since 0.8 to >> get this). Of course, if enabled, it can be exposed. > > Wow. I'm surprised at that. How does a newly received transaction have its > inputs verified then? Multiple linear brute force searches of the block chain > for every new transaction? Or is it that transactions are only recorded if > they were in a block, and just their presence indicates they're valid? The block chain is not involved at all to verify transactions, it's just a historical record to serve to other nodes, and to do wallet rescans with. For validation, a separate database with just unspent transaction outputs is used (around 230 MB now). -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
> > The only way to do this safely at an SPV security assumption, is by > having an address-indexed committed merkle UTXO-set tree, like the > one proposed by Alan Reiner, and being implemented by Mark > Friedenback. I know Michael Gronager has something similar implemented, > but I don't know whether it is script-indexed. The MerkleTrie I have in libcoin is indexed on UTXOs only. However, adding an extra index for scripts would be pretty easy (half day of coding), or even having the two merged into one index. The burden imposed on validating nodes for keeping such an index is really minimal. When using the UTXO MerkleTrie I switch off the sqlite index of these and vise versa, so there are hardly any measurable timing difference. However, the MerkleTrie index is currently re-build on startup (which takes ~30 sec on my laptop), keeping it synced with disk would be optimal and in the long run necessary as even the UTXO set will grow over time. > To be actually useful, > it likely needs to be enforced by miners - putting a significant > burden on validation nodes. Still, if it can be done efficiently, > I think this would be worth it, but more research is needed first in > any case. > -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tuesday 23 July 2013 10:42:05 Pieter Wuille wrote: > On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote: > > One additional URL makes this pretty much perfect: > > GET /rest/block-with-tx/TX-HASH > > > > Construction of the transaction-hash-to-block database is something the > > full client's have to do anyway, so this query is no harder than the > > others for them to supply; but suddenly makes it possible for an SPV > > client to trace the providence of any transaction without needing to > > maintain the entire chain. > > There is actually no such index being maintained by default, and doing so > is an unnecessary burden IMHO (you need to enable -txindex since 0.8 to > get this). Of course, if enabled, it can be exposed. Wow. I'm surprised at that. How does a newly received transaction have its inputs verified then? Multiple linear brute force searches of the block chain for every new transaction? Or is it that transactions are only recorded if they were in a block, and just their presence indicates they're valid? Andy -- Dr Andy Parkins andypark...@gmail.com -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote: > One additional URL makes this pretty much perfect: > > GET /rest/block-with-tx/TX-HASH > > Construction of the transaction-hash-to-block database is something the full > client's have to do anyway, so this query is no harder than the others for > them to supply; but suddenly makes it possible for an SPV client to trace the > providence of any transaction without needing to maintain the entire chain. On Tue, Jul 23, 2013 at 10:27:19AM +0200, Andreas Schildbach wrote: > On 07/22/2013 09:42 PM, Jeff Garzik wrote: > > > The general goal of the HTTP REST interface is to access > > unauthenticated, public blockchain information. There is no plan to > > add wallet interfacing/manipulation via this API. > > Is it planned to expose the UXTO set of a given address? That would be > useful for SPV wallets to be able to swipe a previously unknown private > key (e.g. paper wallet). The REST API has nothing to do with SPV clients; it's similar to the RPC interface and won't be exposed to the network as a whole. Increasing the resource usage by SPV clients on full nodes is undesirable; we have a lot of work to do regarding DoS attacks. John Dillon's comments here on using micro-transactions to compensate full-nodes for maintaining expensive blockchain indexes are worth reading: https://github.com/bitcoin/bitcoin/pull/2802#issuecomment-20232958 In any case UTXO data currently requires you to have full trust in whomever is providing you with it, and that situation will continue until UTXO commitments are implemented - if they are implemented. -- 'peter'[:-1]@petertodd.org 007bea8b46717ec4acb05830bcb6222497366dd72b02ddc80569 signature.asc Description: Digital signature -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote: > One additional URL makes this pretty much perfect: > > GET /rest/block-with-tx/TX-HASH > > Construction of the transaction-hash-to-block database is something the full > client's have to do anyway, so this query is no harder than the others for > them to supply; but suddenly makes it possible for an SPV client to trace the > providence of any transaction without needing to maintain the entire chain. There is actually no such index being maintained by default, and doing so is an unnecessary burden IMHO (you need to enable -txindex since 0.8 to get this). Of course, if enabled, it can be exposed. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 10:27:19AM +0200, Andreas Schildbach wrote: > On 07/22/2013 09:42 PM, Jeff Garzik wrote: > > > The general goal of the HTTP REST interface is to access > > unauthenticated, public blockchain information. There is no plan to > > add wallet interfacing/manipulation via this API. > > Is it planned to expose the UXTO set of a given address? That would be > useful for SPV wallets to be able to swipe a previously unknown private > key (e.g. paper wallet). Depends what you mean by expose. Maintaining an address/script-indexed UTXO is generally useful, in particular for things like sweeping addresses. I certainly have less problems with 'exposing' this than exposing a fully-indexed block chain history. However, and I expect that's what your question is about, this isn't really useful for SPV (or less) nodes, as there is no way to authenticate this data. If you can fake a UTXO entry, you can make a peer believe anything about their balance, potentially resulting in creating a valid transaction that sends change it didn't know was there as fee to miners. Other than for normal block chain data, there is no way to detect this without at least partial validation. The only way to do this safely at an SPV security assumption, is by having an address-indexed committed merkle UTXO-set tree, like the one proposed by Alan Reiner, and being implemented by Mark Friedenback. I know Michael Gronager has something similar implemented, but I don't know whether it is script-indexed. To be actually useful, it likely needs to be enforced by miners - putting a significant burden on validation nodes. Still, if it can be done efficiently, I think this would be worth it, but more research is needed first in any case. Regarding sweeping keys in the first place - I think using those, and relying on address-indexed UTXO sets or blockchains to import them, is an idea that doesn't scale very well in the first place. If it is for things like scratch card or physical coins, with a pre-set value, the obvious solution IMHO is storing the crediting transaction with its merkle path together with the key. If that's not possible, just the txid:vout of the credit output can suffice. Yes, that's more data than is necessary now, but it's so much more trivial to use. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Monday 22 July 2013 20:42:45 Jeff Garzik wrote: > URL: https://github.com/bitcoin/bitcoin/pull/2844 > > Adding an HTTP REST API for bitcoind has been occasionally tossed > about as a useful thing. Such an API would essentially provide a > decentralized block explorer capability, enabling easy external access > to transaction/address/block indices that we maintain. This is excellent. > The first two implemented API calls are simple, returning a block or > TX given a simple query string based on block hash, e.g. > > GET /rest/tx/TX-HASH > or > GET /rest/block/BLOCK-HASH One additional URL makes this pretty much perfect: GET /rest/block-with-tx/TX-HASH Construction of the transaction-hash-to-block database is something the full client's have to do anyway, so this query is no harder than the others for them to supply; but suddenly makes it possible for an SPV client to trace the providence of any transaction without needing to maintain the entire chain. Andy -- Dr Andy Parkins andypark...@gmail.com -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
Hi Andreas / Jeff, Access to the UTXO set can be done using libcoin (see the coinexplorer example), which also has a rest interface. Access to the UTXO set pr address/script requires indexing of all scripts, which was easy in libcoin as the blockchain is stored in a sqlite database. Integrating this in bitcoind would require setting up and maintaining such an index ad hoc. Michael On Jul 23, 2013, at 10:27 , Andreas Schildbach wrote: > On 07/22/2013 09:42 PM, Jeff Garzik wrote: > >> The general goal of the HTTP REST interface is to access >> unauthenticated, public blockchain information. There is no plan to >> add wallet interfacing/manipulation via this API. > > Is it planned to expose the UXTO set of a given address? That would be > useful for SPV wallets to be able to swipe a previously unknown private > key (e.g. paper wallet). > > > > -- > See everything from the browser to the database with AppDynamics > Get end-to-end visibility with application monitoring from AppDynamics > Isolate bottlenecks and diagnose root cause in seconds. > Start your free trial of AppDynamics Pro today! > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk > ___ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On 07/22/2013 09:42 PM, Jeff Garzik wrote: > The general goal of the HTTP REST interface is to access > unauthenticated, public blockchain information. There is no plan to > add wallet interfacing/manipulation via this API. Is it planned to expose the UXTO set of a given address? That would be useful for SPV wallets to be able to swipe a previously unknown private key (e.g. paper wallet). -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
