Re: [Bitcoin-development] HTTP REST API for bitcoind
On 07/22/2013 09:42 PM, Jeff Garzik wrote: The general goal of the HTTP REST interface is to access unauthenticated, public blockchain information. There is no plan to add wallet interfacing/manipulation via this API. Is it planned to expose the UXTO set of a given address? That would be useful for SPV wallets to be able to swipe a previously unknown private key (e.g. paper wallet). -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
Hi Andreas / Jeff, Access to the UTXO set can be done using libcoin (see the coinexplorer example), which also has a rest interface. Access to the UTXO set pr address/script requires indexing of all scripts, which was easy in libcoin as the blockchain is stored in a sqlite database. Integrating this in bitcoind would require setting up and maintaining such an index ad hoc. Michael On Jul 23, 2013, at 10:27 , Andreas Schildbach andr...@schildbach.de wrote: On 07/22/2013 09:42 PM, Jeff Garzik wrote: The general goal of the HTTP REST interface is to access unauthenticated, public blockchain information. There is no plan to add wallet interfacing/manipulation via this API. Is it planned to expose the UXTO set of a given address? That would be useful for SPV wallets to be able to swipe a previously unknown private key (e.g. paper wallet). -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Monday 22 July 2013 20:42:45 Jeff Garzik wrote: URL: https://github.com/bitcoin/bitcoin/pull/2844 Adding an HTTP REST API for bitcoind has been occasionally tossed about as a useful thing. Such an API would essentially provide a decentralized block explorer capability, enabling easy external access to transaction/address/block indices that we maintain. This is excellent. The first two implemented API calls are simple, returning a block or TX given a simple query string based on block hash, e.g. GET /rest/tx/TX-HASH or GET /rest/block/BLOCK-HASH One additional URL makes this pretty much perfect: GET /rest/block-with-tx/TX-HASH Construction of the transaction-hash-to-block database is something the full client's have to do anyway, so this query is no harder than the others for them to supply; but suddenly makes it possible for an SPV client to trace the providence of any transaction without needing to maintain the entire chain. Andy -- Dr Andy Parkins andypark...@gmail.com -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote: One additional URL makes this pretty much perfect: GET /rest/block-with-tx/TX-HASH Construction of the transaction-hash-to-block database is something the full client's have to do anyway, so this query is no harder than the others for them to supply; but suddenly makes it possible for an SPV client to trace the providence of any transaction without needing to maintain the entire chain. There is actually no such index being maintained by default, and doing so is an unnecessary burden IMHO (you need to enable -txindex since 0.8 to get this). Of course, if enabled, it can be exposed. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tuesday 23 July 2013 10:42:05 Pieter Wuille wrote: On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote: One additional URL makes this pretty much perfect: GET /rest/block-with-tx/TX-HASH Construction of the transaction-hash-to-block database is something the full client's have to do anyway, so this query is no harder than the others for them to supply; but suddenly makes it possible for an SPV client to trace the providence of any transaction without needing to maintain the entire chain. There is actually no such index being maintained by default, and doing so is an unnecessary burden IMHO (you need to enable -txindex since 0.8 to get this). Of course, if enabled, it can be exposed. Wow. I'm surprised at that. How does a newly received transaction have its inputs verified then? Multiple linear brute force searches of the block chain for every new transaction? Or is it that transactions are only recorded if they were in a block, and just their presence indicates they're valid? Andy -- Dr Andy Parkins andypark...@gmail.com -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
The only way to do this safely at an SPV security assumption, is by having an address-indexed committed merkle UTXO-set tree, like the one proposed by Alan Reiner, and being implemented by Mark Friedenback. I know Michael Gronager has something similar implemented, but I don't know whether it is script-indexed. The MerkleTrie I have in libcoin is indexed on UTXOs only. However, adding an extra index for scripts would be pretty easy (half day of coding), or even having the two merged into one index. The burden imposed on validating nodes for keeping such an index is really minimal. When using the UTXO MerkleTrie I switch off the sqlite index of these and vise versa, so there are hardly any measurable timing difference. However, the MerkleTrie index is currently re-build on startup (which takes ~30 sec on my laptop), keeping it synced with disk would be optimal and in the long run necessary as even the UTXO set will grow over time. To be actually useful, it likely needs to be enforced by miners - putting a significant burden on validation nodes. Still, if it can be done efficiently, I think this would be worth it, but more research is needed first in any case. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 11:52 AM, Andy Parkins andypark...@gmail.com wrote: There is actually no such index being maintained by default, and doing so is an unnecessary burden IMHO (you need to enable -txindex since 0.8 to get this). Of course, if enabled, it can be exposed. Wow. I'm surprised at that. How does a newly received transaction have its inputs verified then? Multiple linear brute force searches of the block chain for every new transaction? Or is it that transactions are only recorded if they were in a block, and just their presence indicates they're valid? The block chain is not involved at all to verify transactions, it's just a historical record to serve to other nodes, and to do wallet rescans with. For validation, a separate database with just unspent transaction outputs is used (around 230 MB now). -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tuesday 23 July 2013 10:47:03 Peter Todd wrote: On Tue, Jul 23, 2013 at 10:30:13AM +0100, Andy Parkins wrote: One additional URL makes this pretty much perfect: GET /rest/block-with-tx/TX-HASH Construction of the transaction-hash-to-block database is something the full client's have to do anyway, so this query is no harder than the others for them to supply; but suddenly makes it possible for an SPV client to trace the providence of any transaction without needing to maintain the entire chain. The REST API has nothing to do with SPV clients; it's similar to the RPC interface and won't be exposed to the network as a whole. Yes; I know that. I'm saying that it would make it easier for SPV (and other lightweight clients) for that matter. Increasing the resource usage by SPV clients on full nodes is undesirable; I don't think that's thinking big enough. What I imagine is that making it easier and easier to store a partial blockchain would result in lower demand on full nodes. I might run a client that has only fetched blocks that contain transactions needed to verify my balances, right back to the genesis block. That will be some small subset of the block chain and will take me very little resource to maintain. I join the network and am my client is willing to verify based on information I have, or supply (by REST or bitcoin protocol) blocks. Imagine then that everyone with a wallet were doing this. The blockchain would be distributed massively. Obviously the miners would still be keeping the entire chain, but we'd have a lot more nodes in the network, each contributing a little bit and so reducing the load on the full nodes. In any case UTXO data currently requires you to have full trust in whomever is providing you with it, and that situation will continue until UTXO commitments are implemented - if they are implemented. Almost; because you can go and ask someone else the same question, it's pretty easy to check if you're being lied to. Also, it's far easier to maintain a headers-only block chain. When you fetch your relevant block subset, you can easily see that they are real blocks in your headers-only blockchain; and so it's pretty much impossible to lie to give me the block containing transaction X. Andy -- Dr Andy Parkins andypark...@gmail.com -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tuesday 23 July 2013 10:56:02 Pieter Wuille wrote: The block chain is not involved at all to verify transactions, it's just a historical record to serve to other nodes, and to do wallet rescans with. It must be involved to some extent. Certainly during a temporary fork, there are two branches growing, and you have to be able, when verifying a new transaction, to say which branch it's one... which branch of the blockchain. Andy -- Dr Andy Parkins andypark...@gmail.com -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 12:00 PM, Andy Parkins andypark...@gmail.com wrote: The REST API has nothing to do with SPV clients; it's similar to the RPC interface and won't be exposed to the network as a whole. Yes; I know that. I'm saying that it would make it easier for SPV (and other lightweight clients) for that matter. In what way? Increasing the resource usage by SPV clients on full nodes is undesirable; I don't think that's thinking big enough. What I imagine is that making it easier and easier to store a partial blockchain would result in lower demand on full nodes. A partial blockchain is quite useless, as you can't build an UTXO set from it. If you're talking simply about the storage requirements for maintaining history, perhaps, but why rely on SPV nodes for this? Right now, those don't store any blocks at all, and there is no reason why they should. The only requirement is that old blocks remain available for new full nodes to be able to bootstrap. It's certainly not required that everyone keeps every block ever created. That's how the software currently works, but as soon as we get to a few protocol changes that would allow new full nodes to find specific peers with the data they need, we can have fully-verifying yet (partially) pruning nodes. I think that's a much better idea than conflating maintaining a wallet with maintaining a subset of historical block data. Obviously the miners would still be keeping the entire chain, but we'd have a lot more nodes in the network, each contributing a little bit and so reducing the load on the full nodes. I disagree strongly here. The rules of the network are enforced by full nodes, not by miners - they are just trying to satisfy the rules demaned by the network. And as far as I know, there is no way to do some partial validation using just the blocks you care about (and remember, SPV nodes currently have none at all). One interesting possibility here is fraud proofs, where the network can relay proofs that certain blocks or transactions are violating certain network rules. In this case, even SPV nodes can just communicate with eachother and get some herd immunity. But storing some blocks doesn't matter here - it is all about whether you can maintain the UTXO set or not. In any case UTXO data currently requires you to have full trust in whomever is providing you with it, and that situation will continue until UTXO commitments are implemented - if they are implemented. Almost; because you can go and ask someone else the same question, it's pretty easy to check if you're being lied to. Also, it's far easier to maintain a headers-only block chain. When you fetch your relevant block subset, you can easily see that they are real blocks in your headers-only blockchain; and so it's pretty much impossible to lie to give me the block containing transaction X. That's assuming there is no powerful enough attacker that can benefit from doing a sybil attack on you. For SPV nodes currently, that risk is limited to an attacker that can spend enough on faking a chain with valid proof-of-work, to make you accept a transaction that will be reversed. If you go let SPV nodes rely on unauthenticated UTXO set data, there is no such limitation, and they can let you believe *anything*. There are some safeguards, like combining it with merkle paths for all outputs that credit you (which again requires a more expensive index on the other side), but you can't ever guarantee that a particular outputs isn't spent yet. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 12:17 PM, Andreas Schildbach andr...@schildbach.de wrote: Sweeping paper wallets is a common feature request. People switch to centralized services just for getting that. That means they value convenience more than the trust-freeness of a decentralized solution. The only way to avoid that is by making sure the decentralized one is convenient enough. But relying on unauthenticated data itself is equally bad - it means you lose whatever benefit the decentralization had. It is my understanding that for the usecase, an address-indexed UXTO is enough. So you probably don't need to worry about script-indexed for now. The difference between script-indexed and address-indexed is absolutely trivial compared to the effort needed to implement and maintain such authenticated trees by all full nodes. Restricting things at the network level (which doesn't even know about a thing like an address) to address-based indexes is ridiculous IMHO. Security issues could be mitigated by applying trust to the REST server, e.g. because its your own or the one of your apps vendor. Of course, link-level security would be needed for this (e.g. SSL). Sure, once you introduce trust, a lot can be done. But it's not really Bitcoin anymore in that case - it's relying on a third party to do the heavy indexing for you. And if that is the best-scaling solution, sure - but I don't think we should encourage that. Or at least, we should first search for alternatives. And encourage infrastructure that doesn't require it. Paper wallets that include the necessary additional information is something I have been thinking about. I see some issues: - Paper wallets are already quite widespread. You still won't be able to sweep those. - Some people like to top up a paper wallet or even just sweep a portion of it. That would not be possible, and in some cases even lead to loss of coins because of the involuntary fee you described. Yeah, those are inherent problems with how there are used today. But today there is also little problem - the UTXO set is tiny. - Does the necessary info fit into a QR code? Absolutely, though a slightly bigger one. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 12:29 PM, Andreas Schildbach andr...@schildbach.de wrote: Yes, I understand that. For this reason, I would vote for adding the usual HTTP authentication/SSL stuff to the REST API. That way, SPV users can decide to run their own instance of the API (providing the needed resources themselves). Or, a trusted party can set up a server. For example, I would be willing to set it up for users of Bitcoin Wallet. I don't expect shitloads of paper wallets sweeps for the forseeable future. I don't object to using a trusted server for this if people want that, but I don't think the reference client should encourage this. Apart from that, exposing this HTTP-based interface publicly has its own problems, like security risks and potential DoS risks. If anything, we should be reducing the attack surface rather than increase it. IMHO, the only thing that should be exposed in the P2P protocol, which is inevitable, and already has some DoS protections. I like this HTTP interface, but it should really only be used for trusted local applications and debugging. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Distributing low POW headers
I was thinking about a change to the rules for distinguishing between forks
and maybe a BIP..
*Summary*
- Low POW headers should be broadcast by the network
If a header has more than 1/64 of the POW of a block, it should be
broadcast. This provides information on which fork is getting most of the
hashing power.
- Miners should use the header information to decide on longest chain
The fork selection rule for miners should be biased towards staying on the
fork that has the most hashing power.
This means that they might keep hashing on a fork that is 1-2 blocks
shorter.
If most miners follow the rule, then it is the best strategy for other
miners to also follow this rule.
- Advantages
This lowers the probability of natural and malicious reversals.
*Distributing low POW headers*
First block header messages that have more than 1/64 of the standard POW
requirements would be forwarded.
This means the client needs to maintain a short term view of the entire
header tree.
if (header extends header tree) {
if (header meets full POW) {
add to header tree;
forward to peers;
check if any blocks in storage now extend the header tree
} else {
if (header meets POW / 64) {
forward to peers;
}
} else {
if (header meets POW) {
add to orphan header storage
}
}
The storage could be limited and headers could be discarded after a while.
This has the extra advantage that it informs clients of forks that are
receiving hashing power.
This could be linked to a protocol version to prevent disconnects due to
invalid header messages.
*Determining the longest chain*
Each link would get extra credit for headers received.
Assume there are 2 forks starting at block A as the fork point.
A(63) - B(72) - C(37) - D(58)
and
A(63) - B'(6) - C'(9) - D'(4) - E(7) - F(6)
The numbers in brackets are the number of low POW headers received that
have those blocks as parent.
The new rule is that the POW for a block is equal to
POW * (1 + (headers / 16))
Only headers within some threshold of the end of the (shorter) chain
count. However, in most cases, that doesn't matter since the fork point
will act as the start point. As long as miners keep headers for 30-40
blocks, they will likely have all headers back to any reasonable fork point.
This means that the top fork is considered longer, since it has much more
headers, even though it has 2 less blocks.
If 75% of the miners follow this rule, then the top fork will eventually
catch up and win, so it is in the interests of the other 25% to follow the
rule too.
Even if there isn't complete agreement on headers received, the fork that
is getting the most hashing will naturally gain most of the headers, so
ties will be broken quickly.
--
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tuesday 23 July 2013 11:17:28 Peter Todd wrote: On Tue, Jul 23, 2013 at 11:00:24AM +0100, Andy Parkins wrote: Increasing the resource usage by SPV clients on full nodes is undesirable; I don't think that's thinking big enough. What I imagine is that making it easier and easier to store a partial blockchain would result in lower demand on full nodes. Read my proposal for Partial UTXO mode: http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg02 511.html Very interesting. I love the idea of the UTXO set being tied to a block. I might run a client that has only fetched blocks that contain transactions needed to verify my balances, right back to the genesis block. That will be some small subset of the block chain and will take me very little resource to maintain. I join the network and am my client is willing to verify based on information I have, or supply (by REST or bitcoin protocol) blocks. Imagine then that everyone with a wallet were doing this. The blockchain would be distributed massively. Obviously the miners would still be keeping the entire chain, but we'd have a lot more nodes in the network, each contributing a little bit and so reducing the load on the full nodes. Actually the really scary thing about partial UTXO mode is miners can get away without keeping the entire chain provided they don't (often) try to mine transactions spending UTXO's that they haven't verified themselves. You're right. That is scary. In any case UTXO data currently requires you to have full trust in whomever is providing you with it, and that situation will continue until UTXO commitments are implemented - if they are implemented. Almost; because you can go and ask someone else the same question, it's pretty How do you know they actually are someone else? You don't. You can't invalidate the lie if all you have access to is lies. But if you have access to just one honest node; that will reveal the liars. I'm not claiming that headers-only nodes can ever be made as secure as a full node. Just _more_ secure than they are now; and potentially able to act as one of those honest nodes. easy to check if you're being lied to. Also, it's far easier to maintain a headers-only block chain. When you fetch your relevant block subset, you can easily see that they are real blocks in your headers-only blockchain; and so it's pretty much impossible to lie to give me the block containing transaction X. Do you think you have SPV or full security in that situation? Do you know the difference? There is absolutely no need to get condescendingly shirty. I thought this was a friendly list; and we were having a discussion. If you don't want to respond to posts -- don't. I also didn't realise I had to pass an exam before I was allowed to speak. Yes: I know the difference between SPV and full security. SPV is headers only and so has no way to verify that the transaction outputs references as inputs to any new as-yet-unverified transaction are valid. Instead it relies on having some way of proving it's in the chain; and then looking for the number of blocks built on top of it as verification. Full security (which is itself a very poor name), is obviously just checking that every output referenced in the inputs is unspent; that necessarily requires full blocks. The difference in security being that in SPV there is no way to know if the referenced Unspent TransaXtion Output really is unspent -- it might have been spent elsewhere then referenced again in this new transaction. My suggestion was that we want to be able to fetch a block by transaction; and that simple nodes can all, in aggregate offer contribution to the network rather than just being parasitical on the full nodes. When I ask for a block that contains a transaction, and I do that repeatedly, I have part of the block chain. If lots of simple nodes are doing that, then the whole chain should be available if there are enough of them. They would then gain the ability to do transaction-forwarding in some cases. This is only possible if a few extra facilities are added to the protocol. One of which is the new feature I suggested: block-given-transaction. It's not enough on its own, but if you also add in the ability for a node to tell another about the output transactions (basically, what block spends it), _then_ the simple nodes are able to become much more secure -- not 100% of course, they're still not full nodes, because they have no way of knowing if they are being lied to when they are told (this transaction is unspent), but all it takes is one honest node to point them at the truth, and the lie is then exposed. That facility is just a drain on full nodes for the most part; except if you start encouraging it whole-sale. The simple node would keep cache both the incoming and outgoing transactions (or rather the blocks
Re: [Bitcoin-development] HTTP REST API for bitcoind
On Tue, Jul 23, 2013 at 4:36 AM, Pieter Wuille pieter.wui...@gmail.comwrote: Apart from that, exposing this HTTP-based interface publicly has its own problems, like security risks and potential DoS risks. If anything, we should be reducing the attack surface rather than increase it. IMHO, the only thing that should be exposed in the P2P protocol, which is inevitable, and already has some DoS protections. I like this HTTP interface, but it should really only be used for trusted local applications and debugging. We already have a good private HTTP interface. Most benefits of this REST interface come from exposing it publicly. As always, the challenge is balancing costs and benefits. I'm not confident that either can be known with certainty until a well-written prototype is running in the wild. As some nodes expose this interface, we'll gain concrete experience from which node operators can make informed security trade offs. -- Michael -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 4:01 PM, Mike Hearn m...@plan99.net wrote: Hi, Some of us have put together an open letter to the Linux packaging community, outlining why Bitcoin is different to other programs and asking them to not patch or modify the upstream sources. Please consider signing it if you agree (I think the wording by now is fine, so don't edit the contents - use the comment feature if you want to though). https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi The trigger for this is the discovery that Debian bitcoind's got split out of the consensus some time in April, for reasons that nobody yet figured out but is presumably related to a patch (eg it uses system leveldb). Hi Mike, Debian's bitcoin is maintained on an open and archived mailing list and git repo: Debian Bitcoin Packaging Team pkg-bitcoin-de...@lists.alioth.debian.org If there is a problem or question, getting an answer should be really easy. It would be good to include them in the discussion there (I CC:ed the list). If the upstream developers have a consensus that distribution packaging is not in the best interest of the project, then I personally would defer to their judgement and request removal. I'm leaving this open for other opinions from the Debian side. That said, let me summarize the arguments and see if we can figure out a permanent solution: 1) It appears that the consensus of upstream developers is that any distributed binary should only be linked against libraries that the bitcoin developers have tested and audited since any compatibility bug is a risk to both the user and the network. Response: Is there a way to certify the Debian libraries? Debian bitcoind/bitcoin-qt runs the compile test during all architectures. MIPS has been failing recently, but no one has looked into it yet. Perhaps it's not worth developers efforts yet, but at some point the technology should reach a point it can be redistributed. 2) Bitcoin is new technology, so any patches have the ability of harming both the network and user. Response: I, and I'm sure everyone else working on it, totally agrees. All patches are public [1], you can see that the patches are only to the build system (except for one adding a debug message). Is there a specific patch or bug that is problematic? This seems to be reiterating (1) above: don't patch the build system to use libraries that were not audited by the developers. The two solutions are: (1) no one besides the upstream developers compiles and distributes binaries, ever, or (2) upstream comes up with a system where someone besides them can compile working binaries for distribution. Most likely the best solution is to do (1) until upstream sets up a system to allow (2). I'm curious as to other's opinions. Thanks, Scott [1] http://anonscm.debian.org/gitweb/?p=collab-maint/bitcoin.git;a=tree;f=debian/patches;h=ba576f9f3ddad47a2f85dcbfb7a0b3482834f19f;hb=HEAD -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tuesday, July 23, 2013 10:02:28 PM Scott Howard wrote: 1) It appears that the consensus of upstream developers is that any distributed binary should only be linked against libraries that the bitcoin developers have tested and audited since any compatibility bug is a risk to both the user and the network. Response: Is there a way to certify the Debian libraries? Debian bitcoind/bitcoin-qt runs the compile test during all architectures. It doesn't need to be audited by any given person or team, just someone who understands the issues and can dedicate the time to doing a competent audit. Testing bitcoind/bitcoin-qt is not sufficient: you must guarantee that your libraries (especially LevelDB) are bug-for-bug compatible with the ones used by everyone else. And not only the current versions, but every future version to ever hit the repository. This means a lot of additional work for the maintainers of the library packages, and the security team; for example, the security team must understand that they *cannot* deploy a critical security bugfix to LevelDB until someone competent has reviewed that it is behaviourally (including bug behaviours!) compatible with the versions in use everywhere else/previously. I think it is likely all this additional work/delays will be considered unacceptable to your library/security teams, thus using the bundled/embedded LevelDB is probably the best solution. MIPS has been failing recently, but no one has looked into it yet. Perhaps it's not worth developers efforts yet, but at some point the technology should reach a point it can be redistributed. MIPS (and any other big endian architecture) has *always* failed on the Satoshi codebase, and will not be supported until someone has time to dedicate to fixing the numerous little-endian assumptions in the code. I have an incomplete port, but it isn't very high on my priority list to figure out what else it's missing. 2) Bitcoin is new technology, so any patches have the ability of harming both the network and user. Response: I, and I'm sure everyone else working on it, totally agrees. All patches are public [1], you can see that the patches are only to the build system (except for one adding a debug message). Is there a specific patch or bug that is problematic? This seems to be reiterating (1) above: don't patch the build system to use libraries that were not audited by the developers. The two solutions are: (1) no one besides the upstream developers compiles and distributes binaries, ever, or (2) upstream comes up with a system where someone besides them can compile working binaries for distribution. Most likely the best solution is to do (1) until upstream sets up a system to allow (2). Debian could probably get away with packaging Bitcoin-Qt and bitcoind as-is with no modifications, and not have any problems. It's only when you begin making modifications that it becomes a problem. There are also some concerns that it puts a much larger price on compromising Debian's build servers and/or repositories (suddenly the attacker can steal a LOT of money). The official binaries are not simply built by upstream developers: using Gitian, *anyone* can produce bit-for-bit identical binaries. Official releases are only published after 3 or more people have done an independent compile and signed the output. It would be excellent if the whole of Debian could work toward achieving this level of security eventually, which would make distributing Bitcoin node software much safer as well. Luke signature.asc Description: This is a digitally signed message part. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 10:01:55PM +0200, Mike Hearn wrote: The trigger for this is the discovery that Debian bitcoind's got split out of the consensus some time in April, for reasons that nobody yet figured out but is presumably related to a patch (eg it uses system leveldb). Just to make sure there are no misunderstandings, as far as I know there is no reason to assume the reported problem (comment on #2726) is: 1) a fork (it's an indeterministic and avoidable database corruption, it seems) 2) related to leveldb 3) reproducible by more than one person 4) debian's fault. That said, I think reaching out to packagers to educate them about the risks is a good idea - but let's not blame people before we understand our own problems. -- Pieter -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
I find it interesting that this is a linux packaging letter. How much of this applies to pkgsrc, FreeBSD ports, OpenBSD ports, and other non-Linux packaging systems (pkgsrc supports Linux as on of 20 operating systems, but is not a Linux packaging system)? Is the repeatable build infrastructure portable (to any reasonable mostly-POSIX-compliant system, with gcc or clang)? I have the vague impression it's Ubuntu only, but I am very unclear on this point. How does this repeatableness interact with building for multiple operating systems and cpu types (say 20 OS, with typically 3 versions of the OS for each, with 1-20 cpu types per OS, for a cross-product of perhaps 200 combinations)? Requiring precise library depdendencies is quite awkward. Certainly requiring new enough to avoid known bugs is understandable, but that should be caught at configure time and fail. Synchronous updates of multiple packages is difficult, and runs into A wants only n of C, while B wants only m. So if you are talking about running regression tests with the set of versions of a dependency that are considered reasonable, and there's therefore a solution to the multiple-package constraint problem, that seems ok. It seems like a bug that the package will build on BE systems and then fail tests. If it's known not to be ok, it seems that absent some configure-time flag the build should fail. Asking people not to patch should mean willingnesss to make accomodation in the master sources for build issues for multiple packaging systems. I haven't gotten around to packaging this for pkgsrc - so far I only have the energy to lurk (due to too many things on the todo list). But I often find that some changes are needed. If you're willing (in theory) to add in configure flags to control build behavior (in a way that you can audit and decide is safe), that's great, and of course we can discuss an actual situation when one gets figured out. Greg -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 4:23 PM, Greg Troxel g...@work.lexort.com wrote: Is the repeatable build infrastructure portable (to any reasonable mostly-POSIX-compliant system, with gcc or clang)? I have the vague It's portable to anything that can run the relevant VMs. Uh provided you don't mind cross compiling everything from an unbuntu VM. It certainly would be nice if the trusted-computing-base for gitian were a bit smaller, thats an area for long term improvement for sure. It may need some massaging. The tor project is beginning to use the same infrastructure, so this could be usefully conserved work. Likewise expanding the supported output targets would be great— though in the case of Bitcoin this is bounded by resources to adequately QA builds on alternative targets. Requiring precise library depdendencies is quite awkward. Certainly requiring new enough to avoid known bugs is understandable, but that should be caught at configure time and fail. In some cases packages solving bugs is problematic for Bitcoin. This is something that it seems to take a whiteboard to explain, so I apologize for the opacity of simple email here. From a technical perspective Bitcoin's whole purpose is getting a whole bunch of computers world wide to reach a bit identical agreement on the content of a database, subject to a whole pile of rules, in the face of potentially malicious input, without any trusted parties at all (even the guy you got the software from, assuming you have the resources to audit it). I'll walk through a simple example: Say Bitcoin used a backing database which had an unknown a bug where any item with a key that begins with 0xDEADBEEF returns not found when queried, even if its in the DB. Once discovered, any database library would want to fix that quickly and they'd fix it in a point release without reservation. They might not even release note that particular fix it if went along with some others, it could even be fixed accidentally. Now say that we have a state where half the Bitcoin network is running the old buggy version, and half is running the fixed version. Someone creates a transaction with ID 0xDEADBEEF... and then subsequently spends the output of that transaction. This could be by pure chance or it could be a malicious act. To half the network that spending transaction looks like someone spending coin from nowhere, a violation of the rules. The consensus would then fork, effectively partitioning the network. On each fork any coin could be spent twice, and the fork will only be resolvable by one side or the other abandoning their state (generally the more permissive side would need to be abandoned because the permissive one is tolerant of the restrictive one's behavior) by manually downgrading or patching software. As a result of this parties who believed some of their transactions were safely settled would find them reversed by people who exploited the inconsistent consensus. To deploy such a fix in Bitcoin without creating a risk for participants we need to make a staged revision of the network protocol rules: There would be a protocol update that fixed the database bug _and_ explicitly rejected 0xDEADBEEF transactions until either some far out future date or until triggered by quorum sensing (or both). The users of Bitcoin would all be advised that they had to apply fixes/workaround by the switchover point or be left out of service or vulnerable. At designated time / quorum nodes would simultaneously switch to the new behavior. (Or in some cases, we'd just move the 'bug' into the Bitcoin code so that it can be fixed in the database, and we'd then just keep it forever, depending on how harmful it was to Bitcoin, a one if 4 billion chance of having to rewrite a transaction wouldn't be a big deal) We've done these organized updates to solve problems (as various flaws in Bitcoin itself can present similar consensus risks) several times with great success, typical time horizons spanning for months to years. But it cannot work if the behavior is changed out from under the software. Fortunately, if the number of users running with an uncontrolled consensus relevant inconsistent behavior is small the danger is only to themselves (and, perhaps, their customers). I'm not happy to see anyone get harmed, but it's better if its few people than many. This is part of the reason that it's a linux packaging letter, since for Bitcoin the combination of uncoordinated patching and non-trivial userbases appears to be currently unique to GNU/Linux systems. Though indeed, the concerns do apply more broadly. multiple packages is difficult, and runs into A wants only n of C, while B wants only m. My understanding is that gentoo is actually able to handle this (and does, for Bitcoin)— and really I presume just about everything else could with enough effort. I certainly wouldn't ask anyone else to do that. If you're really getting into the rathole of building separate libraries
Re: [Bitcoin-development] Linux packaging letter
Honestly, until I read the quoted part of your response, I actually wasn't in favor of this whole thing since in general the types of issues being mentioned are, in large part, the types of issues that maintainers deal with all the time. On Jul 23, 2013, at 3:02 PM, Scott Howard showard...@gmail.com wrote: Response: Is there a way to certify the Debian libraries? Debian bitcoind/bitcoin-qt runs the compile test during all architectures. MIPS has been failing recently, but no one has looked into it yet. Perhaps it's not worth developers efforts yet, but at some point the technology should reach a point it can be redistributed. The fact that you're even trying to package and/or at some point have packaged and shipped big endian binaries is straight up *NEGLIGENT.* Stop that. Now. It won't work. Thanks for showing that this *is* necessary, I guess. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 9:45 PM, Douglas Huff dh...@jrbobdobbs.org wrote: Honestly, until I read the quoted part of your response, I actually wasn't in favor of this whole thing since in general the types of issues being mentioned are, in large part, the types of issues that maintainers deal with all the time. On Jul 23, 2013, at 3:02 PM, Scott Howard showard...@gmail.com wrote: Response: Is there a way to certify the Debian libraries? Debian bitcoind/bitcoin-qt runs the compile test during all architectures. MIPS has been failing recently, but no one has looked into it yet. Perhaps it's not worth developers efforts yet, but at some point the technology should reach a point it can be redistributed. The fact that you're even trying to package and/or at some point have packaged and shipped big endian binaries is straight up *NEGLIGENT.* Stop that. Now. It won't work. Thanks for showing that this *is* necessary, I guess. before people get too upset, I'm talking about little-endian MIPS (debian-mipsel) http://www.debian.org/ports/mips/ -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Linux packaging letter
On Tue, Jul 23, 2013 at 6:26 PM, Luke-Jr l...@dashjr.org wrote: This means a lot of additional work for the maintainers of the library packages, and the security team; for example, the security team must understand that they *cannot* deploy a critical security bugfix to LevelDB until someone competent has reviewed that it is behaviourally (including bug behaviours!) compatible with the versions in use everywhere else/previously. I think it is likely all this additional work/delays will be considered unacceptable to your library/security teams, thus using the bundled/embedded LevelDB is probably the best solution. The above is a key point, lukejr addressed it well I think it is likely all this additional work/delays will be considered unacceptable to your library/security teams, thus using the bundled/embedded LevelDB is probably the best solution. MIPS has been failing recently, but no one has looked into it yet. Perhaps it's not worth developers efforts yet, but at some point the technology should reach a point it can be redistributed. MIPS (and any other big endian architecture) has *always* failed on the Satoshi codebase, and will not be supported until someone has time to dedicate to fixing the numerous little-endian assumptions in the code. I have an incomplete port, but it isn't very high on my priority list to figure out what else it's missing. To be clear, bitcoind/bitcoin-qt is only built on little endian machines https://buildd.debian.org/status/package.php?p=bitcoin Debian could probably get away with packaging Bitcoin-Qt and bitcoind as-is with no modifications, and not have any problems. It's only when you begin making modifications that it becomes a problem. There are also some concerns that it puts a much larger price on compromising Debian's build servers and/or repositories (suddenly the attacker can steal a LOT of money). The only current modification is to use system leveldb instead of the packaged leveldb. (There is also a patch porting libmemenv.a to several other architectures, but that is only used in test suites - so it shouldn't pose a risk to users). The official binaries are not simply built by upstream developers: using Gitian, *anyone* can produce bit-for-bit identical binaries. Official releases are only published after 3 or more people have done an independent compile and signed the output. It would be excellent if the whole of Debian could work toward achieving this level of security eventually, which would make distributing Bitcoin node software much safer as well. Ironically, debian (in general) doesn't trust upstream security maintenance of third part libraries - that's why they typically get dropped in favor of use system libraries. In this case, upstream doesn't trust (rightfully) that some future debian security team bug fix to a stable library won't be tested properly against bitcoin, causing problems for users (since bitcoin might expect buggy library behavior). I'm not the original packager or maintainer - I just came across the package in really bad shape and helped bring it to something reasonable and have done the most recent uploads (since 0.8, I believe). Since updated libraries could pose a security risk because bitcoin may expect buggy behavior, I think that is a good argument for debian to use the included library. However, I'm just a recent helper - I still want to hear what people who have been doing this for longer think. ~Scott -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Endianness (was: Linux packaging letter)
On Wednesday, July 24, 2013 3:54:25 AM Wendell wrote:
Is there a substantial barrier to endian independence in the Bitcoin
codebase?
I got the obvious stuff ('endian' branch in my repo), but it still didn't work
when I moved on. I haven't had time to try to figure out why not yet.
Luke
--
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Endianness (was: Linux packaging letter)
On Tue, Jul 23, 2013 at 8:54 PM, Wendell w...@grabhive.com wrote: Forking for curiosity's sake: Is there a substantial barrier to endian independence in the Bitcoin codebase? Not really. The software was originally written to write out memory order to and from the wire, which is part of why the protocol is LE everywhere, so fixing that much is pretty typical endianness fixes. There is an extra kink in that almost everything Bitcoin sends and receives is an authenticated data structure— the stuff gets hashed for authentication. So that simply swizzling the byte order on immediately on input isn't enough because sometimes you'll go on to hash that data and it can't be in memory order for that. Luke gave an initial crack at it a long time ago: http://gitorious.org/~Luke-Jr/bitcoin/luke-jr-bitcoin/commits/endian But it wasn't enough yet. Seems like its just enough of an undertaking that absent a really good reason to care about it no real progress in fixing it is happening. -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Endianness (was: Linux packaging letter)
On Tue, Jul 23, 2013 at 9:07 PM, Gregory Maxwell gmaxw...@gmail.com wrote: order to and from the wire, which is part of why the protocol is LE everywhere, *before someone corrects me, it's not LE everywhere (I meant manywhere :P)— there is just enough BE to keep you on your toes. :P -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
