Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
As I see the BIP it is basically stressing that ver 1 transactions are malleable. It then addresses the need for unmalleable transactions for e.g. spending unconfirmed outputs in a deterministic way (i.e. no 3rd party can sabotage) - this transaction type is defined as ver 3. A lot of clients today spend unconfirmed outputs (even bitcoin-qt) and as such make an implicit assumption that this is kind of safe, which it is not - it can be intervened and sabotaged through tx malleability. What I suggested was to ensure that a subclass of version 1 transactions become unmalleable - namely those with sighash=all. Note that only the sender can modify the sighash as it is part of the hash signed. So instead of defining a version 3, we could constrain version 1 txes with sighash=all to have a unmalleable hash. If you e.g. would like to still have a sighash=all type of transaction with malleable features you can simply use that sighash=all today is checked for using sighash0x1f=sighash_all, so just OR'ing with 0x20 or 0x40 will get you the 'old' feature. I do however buy the argument of Peter and Gregory that there might exist unpublished transactions out there that does not even conform to the DER rules etc, and as such we cannot forbid them from being mined, nor can we timestamp them and include 'only the old ones'. Hence we cannot change the consensus rule for version 1 transactions - and only changing the relay rules will not provide a certain guarantee. So, I think the two line argument for the BIP is as follows: 1. We cannot change the consensus rules for version 1 transactions as that might invalidate unpublished non-standard transactions (= voiding peoples money, which is a line we don't want to cross) 2. The prime usecase for unmalleable transactions is being able to spend unconfirmed outputs - this is done today by almost all clients, but it is really broken. Hence a need for a fix asap. I am all in favor for the BIP, but I expect the realistic timeline for enforced version 3 transactions is roughly one year, which is better than two, but it would have been nice to get it faster... /M On Feb 19, 2014, at 10:11 PM, Pieter Wuille pieter.wui...@gmail.com wrote: On Wed, Feb 19, 2014 at 9:28 PM, Michael Gronager grona...@mac.com wrote: I think that we could guarantee fewer incidents by making version 1 transactions unmalleable and then optionally introduce a version 3 that supported the malleability feature. That way most existing problematic implementations would be fixed and no doors were closed for people experimenting with other stuff - tx v 3 would probably then be called experimental transactions. Just to be clear: this change is not directly intended to avoid incidents. It will take way too long to deploy this. Software should deal with malleability. This is a longer-term solution intended to provide non-malleability guarantees for clients that a) are upgraded to use them b) willing to restrict their functionality. As there are several intended use cases for malleable transactions (the sighash flags pretty directly are a way to signify what malleabilities are *wanted*), this is not about outlawing malleability. While we could right now make all these rules non-standard, and schedule a soft fork in a year or so to make them illegal, it would mean removing potential functionality that can only be re-enabled through a hard fork. This is significantly harder, so we should think about it very well in advance. About new transaction and block versions: this allows implementing and automatically scheduling a softfork without waiting for wallets to upgrade. The non-DER signature change was discussed for over two years, and implemented almost a year ago, and we still notice wallets that don't support it. We can't expect every wallet to be instantly modified (what about hardware wallets like the Trezor, for example? they may not just be able to be upgraded). Nor is it necessary: if your software only spends confirmed change, and tracks all debits correctly, there is no need. -- Pieter signature.asc Description: Message signed with OpenPGP using GPGMail -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
We've done forking changes before much faster than a year and that was with less experience. If we want, we can get this done within months. On 20 Feb 2014 16:30, Michael Gronager grona...@mac.com wrote: As I see the BIP it is basically stressing that ver 1 transactions are malleable. It then addresses the need for unmalleable transactions for e.g. spending unconfirmed outputs in a deterministic way (i.e. no 3rd party can sabotage) - this transaction type is defined as ver 3. A lot of clients today spend unconfirmed outputs (even bitcoin-qt) and as such make an implicit assumption that this is kind of safe, which it is not - it can be intervened and sabotaged through tx malleability. What I suggested was to ensure that a subclass of version 1 transactions become unmalleable - namely those with sighash=all. Note that only the sender can modify the sighash as it is part of the hash signed. So instead of defining a version 3, we could constrain version 1 txes with sighash=all to have a unmalleable hash. If you e.g. would like to still have a sighash=all type of transaction with malleable features you can simply use that sighash=all today is checked for using sighash0x1f=sighash_all, so just OR'ing with 0x20 or 0x40 will get you the 'old' feature. I do however buy the argument of Peter and Gregory that there might exist unpublished transactions out there that does not even conform to the DER rules etc, and as such we cannot forbid them from being mined, nor can we timestamp them and include 'only the old ones'. Hence we cannot change the consensus rule for version 1 transactions - and only changing the relay rules will not provide a certain guarantee. So, I think the two line argument for the BIP is as follows: 1. We cannot change the consensus rules for version 1 transactions as that might invalidate unpublished non-standard transactions (= voiding peoples money, which is a line we don't want to cross) 2. The prime usecase for unmalleable transactions is being able to spend unconfirmed outputs - this is done today by almost all clients, but it is really broken. Hence a need for a fix asap. I am all in favor for the BIP, but I expect the realistic timeline for enforced version 3 transactions is roughly one year, which is better than two, but it would have been nice to get it faster... /M On Feb 19, 2014, at 10:11 PM, Pieter Wuille pieter.wui...@gmail.com wrote: On Wed, Feb 19, 2014 at 9:28 PM, Michael Gronager grona...@mac.com wrote: I think that we could guarantee fewer incidents by making version 1 transactions unmalleable and then optionally introduce a version 3 that supported the malleability feature. That way most existing problematic implementations would be fixed and no doors were closed for people experimenting with other stuff - tx v 3 would probably then be called experimental transactions. Just to be clear: this change is not directly intended to avoid incidents. It will take way too long to deploy this. Software should deal with malleability. This is a longer-term solution intended to provide non-malleability guarantees for clients that a) are upgraded to use them b) willing to restrict their functionality. As there are several intended use cases for malleable transactions (the sighash flags pretty directly are a way to signify what malleabilities are *wanted*), this is not about outlawing malleability. While we could right now make all these rules non-standard, and schedule a soft fork in a year or so to make them illegal, it would mean removing potential functionality that can only be re-enabled through a hard fork. This is significantly harder, so we should think about it very well in advance. About new transaction and block versions: this allows implementing and automatically scheduling a softfork without waiting for wallets to upgrade. The non-DER signature change was discussed for over two years, and implemented almost a year ago, and we still notice wallets that don't support it. We can't expect every wallet to be instantly modified (what about hardware wallets like the Trezor, for example? they may not just be able to be upgraded). Nor is it necessary: if your software only spends confirmed change, and tracks all debits correctly, there is no need. -- Pieter -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Managing the Performance of Cloud-Based
Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
On Thu, Feb 20, 2014 at 6:08 AM, Mike Hearn m...@plan99.net wrote: We've done forking changes before much faster than a year and that was with less experience. If we want, we can get this done within months. You mean P2SH... which your implementation has only picked up support for in the last month or so? -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
I think we should get Pieter's proposal done and implemented quickly. I agree with Mike, it doesn't have to take a long time for the core network to fully support this. Getting wallets to start generating transaction.version=3 might take years, but that is OK. -- -- Gavin Andresen -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
On Thu, Feb 20, 2014 at 6:29 AM, Gavin Andresen gavinandre...@gmail.com wrote: I think we should get Pieter's proposal done and implemented quickly. I agree with Mike, it doesn't have to take a long time for the core network to fully support this. Getting wallets to start generating transaction.version=3 might take years, but that is OK. Sure I'm all for doing what Pieter suggested— it's basically the plan we've been executing for some time already but with the version check to make it sane to complete. My reserved sounding comments were relative to the proposals to do things with nversion=1 transactions, frankly I think thats completely insane. Though while we're on the subject of reservations, I am far from confident that we've uncovered all the possible malleability routes— that list gained a new, never before discussed entry, when Pieter was writing it a couple weeks ago. We also have no proof of the absence of further algebraic malleability in DSA (though I think its somewhat unlikely, a solid proof of it has been somewhat elusive). -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
Great, I'm hearing rough consensus to proceed with Pieter's plan. RE: far from confident on malleability routes: I'm reasonably confident that we can squash malleability for IsStandard, SIGHASH_ALL transactions. A proper proof of DSA signature un-malleability (or an lower bound for how much work it would be to create a valid doppleganger signature) would be great, but I don't think it is necessary to proceed. On Thu, Feb 20, 2014 at 9:36 AM, Gregory Maxwell gmaxw...@gmail.com wrote: On Thu, Feb 20, 2014 at 6:29 AM, Gavin Andresen gavinandre...@gmail.com wrote: I think we should get Pieter's proposal done and implemented quickly. I agree with Mike, it doesn't have to take a long time for the core network to fully support this. Getting wallets to start generating transaction.version=3 might take years, but that is OK. Sure I'm all for doing what Pieter suggested-- it's basically the plan we've been executing for some time already but with the version check to make it sane to complete. My reserved sounding comments were relative to the proposals to do things with nversion=1 transactions, frankly I think thats completely insane. Though while we're on the subject of reservations, I am far from confident that we've uncovered all the possible malleability routes-- that list gained a new, never before discussed entry, when Pieter was writing it a couple weeks ago. We also have no proof of the absence of further algebraic malleability in DSA (though I think its somewhat unlikely, a solid proof of it has been somewhat elusive). -- -- Gavin Andresen -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
I hereby request a BIP number. On Thu, Feb 20, 2014 at 3:58 PM, Gavin Andresen gavinandre...@gmail.com wrote: Great, I'm hearing rough consensus to proceed with Pieter's plan. RE: far from confident on malleability routes: I'm reasonably confident that we can squash malleability for IsStandard, SIGHASH_ALL transactions. A proper proof of DSA signature un-malleability (or an lower bound for how much work it would be to create a valid doppleganger signature) would be great, but I don't think it is necessary to proceed. On Thu, Feb 20, 2014 at 9:36 AM, Gregory Maxwell gmaxw...@gmail.com wrote: On Thu, Feb 20, 2014 at 6:29 AM, Gavin Andresen gavinandre...@gmail.com wrote: I think we should get Pieter's proposal done and implemented quickly. I agree with Mike, it doesn't have to take a long time for the core network to fully support this. Getting wallets to start generating transaction.version=3 might take years, but that is OK. Sure I'm all for doing what Pieter suggested-- it's basically the plan we've been executing for some time already but with the version check to make it sane to complete. My reserved sounding comments were relative to the proposals to do things with nversion=1 transactions, frankly I think thats completely insane. Though while we're on the subject of reservations, I am far from confident that we've uncovered all the possible malleability routes-- that list gained a new, never before discussed entry, when Pieter was writing it a couple weeks ago. We also have no proof of the absence of further algebraic malleability in DSA (though I think its somewhat unlikely, a solid proof of it has been somewhat elusive). -- -- Gavin Andresen -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
On Thu, Feb 20, 2014 at 7:11 AM, Pieter Wuille pieter.wui...@gmail.com wrote: I hereby request a BIP number. 62 assigned. -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Transaction malleability in the core code: update
A quick update on the state of transaction malleability work in Bitcoind/Bitcoin-Qt (aka Bitcoin Core). This is not about longer-term malleability issues, just the very short-term work being done (or already done) to the reference implementation. First, the problems: We've had a longstanding TODO to improve the way the core code deals with double-spends. From the core code's point of view, malleable transactions are just one particular form of double-spend. Improving double-spend handling never made it to the top of the TODO list, because the cases where it happened involved doing unsupported things (like copying your wallet.dat to another machine and then spending on both machines). And because there is a heavy-handed workaround if a wallet becomes confused because of a double-spend: restore all of the keys, rescan for transactions confirmed in the blockchain, and any outputs tied up in double-spends get released. Coins (really, unspent transaction outputs) were never permanently lost, but they could be tied up and unspendable when associated with a 0-confirmation transaction that would never confirm. So, work in progress or done: https://github.com/bitcoin/bitcoin/pull/3659 https://github.com/bitcoin/bitcoin/pull/3674 These implements a kinder, gentler sledgehammer (-zapwallettxes) to fix a confused wallet. If you have a wallet with 0-confirmation transactions that are tying up bitcoins these should fix it. https://github.com/bitcoin/bitcoin/pull/3651 https://github.com/bitcoin/bitcoin/pull/3657 https://github.com/bitcoin/bitcoin/pull/3676 These three merged pull requests implement a new command-line option: -nospendzeroconfchange . The best way to get a wallet confused is to spend zero-confirmation change outputs that you created yourself; if the transaction creating the change gets mutated, then the subsequent transaction is invalid and will never confirm. The core code spends unconfirmed change only as a last resort. If you are a service using bitcoind that generates a lot of transactions then best practice would be to run with -nospendzeroconfchange, and use sendmany to batch payments only after previous payments have confirmed. https://github.com/bitcoin/bitcoin/pull/3025 This tightens up the IsStandard() rule, so the easiest-to-implement method of mutating transactions is blocked. Many big mining pools are already running this patch. https://github.com/bitcoin/bitcoin/pull/3669 https://github.com/bitcoin/bitcoin/pull/3671 https://github.com/bitcoin/bitcoin/pull/3694 These three get at the root of the problem; they rework the core wallet code to implement handle double spends better. See the pull requests for details. How can you help: Testing and code review is, as always, the bottleneck for getting out a release with these changes. We have a chronic problem with people running Bitcoin services on top of the core code waiting until there is an official release, and then assuming that somebody else has done the hard work of reviewing and testing the changes. YOU SHOULD NOT BE MAKING THAT ASSUMPTION! Your particular RPC call usage might trigger some edge-case bug that was missed, or perhaps the size of your wallet triggers a performance problem introduced by a fix. Or, in other words: do not treat the core development team as if we were a commercial company that sold you a software library. That is not how open source works; if you are making a profit using the software, you are expected to help develop, debug, test, and review it. -- -- Gavin Andresen -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Base-32 error correction coding
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 What follows is a proposed BIP for human-friendly base-32 serialization with error correction encoding. A formatted version is viewable as part of a gist with related code: https://gist.github.com/maaku/8996338#file-bip-ecc32-mediawiki An implementation of this BIP and associated APIs is made available as a pull request, with comprehensive testing: https://github.com/bitcoin/bitcoin/pull/3713 This format is anticipated to be useful for helpdesk-related data (e.g. the proposed normalized transaction ID), and future wallet backup paper wallet serialization formats. == Abstract == The BIP proposes an human-centered encoding format for base-32 data serialization. It differs from the presumptive default hexadecimal or base58 encodings in the following ways: 1. Visually distinctive in that it includes the full range of alphanumeric digits in its base-32 encoding, except the characters 0, l, v, and 2. which are too easily confused with 1, i, u, r, or z in font or handwriting. 2. Automatic correction of up to 1 transcription error per 31 coded digits (130 bits of payload data). For a 256-bit hash or secret key, this enables seamless recovery from up to two transcription errors so long as they occur in separate halves of the coded representation. 3. Highly probable detection of errors beyond the error correction threshold, with a false negative rate on the order of 25 bits, or 1 in 33 million likelihood. 4. Case-insensitive encoding ensures that it may be displayed in an easier to read uniform case, and it is faster and more comfortable to vocally read off a base-32 encoded number than the alternatives of hexadecimal or base58. In addition to the error correction code transformation of base-32 data, a padding scheme is specified for extending numbers or bit vectors of any length to a multiple of 5 bits suitable for base-32 encoding. == z-base-32 == The bitcoin reference client already has one implementation of base-32 encoding following the RFC 3548 standard, using the following alphabet: const char *pbase32 = abcdefghijklmnopqrstuvwxyz234567; For error correction coded strings this BIP specifies usage of Phil Zimmermann's z-base-32 encoding alphabet[], which provides better resistance to transcriptive errors than the RFC 3548 standard: const char *pzbase32 = ybndrfg8ejkmcpqxot1uwisza345h769; The same RFC 3548 coder is used for z-base-32, except that unnecessary '=' padding characters are stripped before performing the alphabet substitution. For example, the hexadecimal string 'ae653be0049be3' is RFC 3548 encoded as 'vzstxyaetprq', and z-base-32 encoded as 'i31uzayruxto'. == CRC-5-USB error correction coding == Herein we describe an error correction encoding using cyclic redundancy check polynomial division[], which requires 5 error correction digits per 26 digits of input, instead of the theoretically optimal 4, but is much, much easier to implement correctly then available non-patented error correction codes. Cyclic redundancy check polynomial division provides a very straightforward, patent-free mechanism for reliably detecting transcription errors in input, and performing up to 1-digit corrections per 26 digit block. === Encoding === The input to this error correction encoder is a sequence of 26 base-32 digits. These digits are decoded into 5-bit unsigned integers with values equal to their offset into the base-32 alphabet string. If the input is less than 26 digits in length, it is extended with zero-valued digits. If For example, the string 'vzstxyaetprq' using the RFC 3548 alphabet becomes the code point sequence: 21 25 18 19 23 24 0 4 19 15 17 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0' |--input-|-padding--| Expositionally it helps to think of this array as a 26-element column vector of 5-bit binary integers: 0b10101 0b11001 0b10010 0b10011 0b10111 0b11000 0b0 0b00100 0b10011 0b0 0b10001 0b1 ... 14 zero elements ... If we explode the bits of each element into 5, 1-bit columns, we get a 26 x 5 matrix: 1 0 1 0 1 1 1 0 0 1 1 0 0 1 0 1 0 0 1 1 1 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 1 0 1 1 1 1 1 0 0 0 1 1 0 0 0 0 ... 14 x 5 zero elements ... The array is then transposed, such that we get a 5 x 26 matrix where each row represents the 5th, 4th, 3rd, 2nd, or 1st bit, respectfully, of each element: 1 1 1 1 1 1 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 |-input|--padding-| We then use each of these rows separately as input into a cyclic redundancy check polynomial division operation,
Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
No, I was thinking of the height in coinbase change. At any rate, p2sh was supported by the consensus code in bitcoinj for a long time already, since it was first written. Support for sending to such addresses in the wallet appeared once an app that wanted that support also appeared, which seems OK - the market for wallets is very competitive so there will always be some skew in what features are worked on in what order. V3 transactions are a consensus change that wallets will pick up at different times like any other feature. On 20 Feb 2014 19:45, Gregory Maxwell gmaxw...@gmail.com wrote: On Thu, Feb 20, 2014 at 6:08 AM, Mike Hearn m...@plan99.net wrote: We've done forking changes before much faster than a year and that was with less experience. If we want, we can get this done within months. You mean P2SH... which your implementation has only picked up support for in the last month or so? -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Bitcoin Core trial balloon: splitting blockchain engine and wallet
[Meta: Bitcoin Core is the newfangled branding of bitcoind / Bitcoin-Qt reference implementation, in case you wondering.] Several sites, including BitPay, use bitcoind outside the standard role of wallet software. bitcoind can be used purely for payment network access and management. I call this the border router role. Upcoming version 0.9 will feature the ability to disable the bitcoind wallet at compile time or runtime. This permits a more optimized border router profile, reducing process size by 40-200MB according to some reports. Recent IRC discussion have floated a rough proposal for a wallet next-step: Running the Bitcoin Core wallet as a separate process, a separate binary, from the blockchain engine. The wallet process would communicate with the blockchain engine using existing RPC and P2P channels, becoming a real SPV client. This accomplishes a longstanding security goal of sandboxing away wallet keys and sensitive data from the network-exposed P2P engine, in a separate process, among other benefits. Simple forking was explored a bit. I did some hacking in that direction, as it seemed potentially lightweight and somewhat easy to me: https://github.com/jgarzik/bitcoin/tree/fork fork+pipe is fine for Linux and OSX/BSD. However, Windows requires an exec-like solution to create a new process. MSDN does give us a Unix-pipe-like solution: http://msdn.microsoft.com/en-us/library/edze9h7e%28v=vs.80%29.aspx Others pointed to boost interprocess communication APIs, which come with their own set of caveats. Such a solution would involve a brand new IPC protocol, and lots of brand new glue code. Separate programs seems better. Windows forces us to achieve process separation via exec-like method. We already have IPC: RPC + P2P. Modern OS's make localhost sockets just about as fast as other IPCs methods. Linux, at least, employs zero-copy for localhost sockets in many situations, similar to the kernel's pipe tricks. Pieter has been working on headers-first sync: https://github.com/bitcoin/bitcoin/pull/2964 Moving along this wallet/blockchain engine split requires upping the reviewtest bandwidth on Pieter's PRs, such as https://github.com/bitcoin/bitcoin/pull/3514 Unsure how much of the separate-binary discussion Gavin saw, so cc'd for emphasis. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Core trial balloon: splitting blockchain engine and wallet
Bear in mind a separate process doesn't buy you anything without a sandbox, and those are expensive (in terms of complexity). On 21 Feb 2014 11:40, Jeff Garzik jgar...@bitpay.com wrote: [Meta: Bitcoin Core is the newfangled branding of bitcoind / Bitcoin-Qt reference implementation, in case you wondering.] Several sites, including BitPay, use bitcoind outside the standard role of wallet software. bitcoind can be used purely for payment network access and management. I call this the border router role. Upcoming version 0.9 will feature the ability to disable the bitcoind wallet at compile time or runtime. This permits a more optimized border router profile, reducing process size by 40-200MB according to some reports. Recent IRC discussion have floated a rough proposal for a wallet next-step: Running the Bitcoin Core wallet as a separate process, a separate binary, from the blockchain engine. The wallet process would communicate with the blockchain engine using existing RPC and P2P channels, becoming a real SPV client. This accomplishes a longstanding security goal of sandboxing away wallet keys and sensitive data from the network-exposed P2P engine, in a separate process, among other benefits. Simple forking was explored a bit. I did some hacking in that direction, as it seemed potentially lightweight and somewhat easy to me: https://github.com/jgarzik/bitcoin/tree/fork fork+pipe is fine for Linux and OSX/BSD. However, Windows requires an exec-like solution to create a new process. MSDN does give us a Unix-pipe-like solution: http://msdn.microsoft.com/en-us/library/edze9h7e%28v=vs.80%29.aspx Others pointed to boost interprocess communication APIs, which come with their own set of caveats. Such a solution would involve a brand new IPC protocol, and lots of brand new glue code. Separate programs seems better. Windows forces us to achieve process separation via exec-like method. We already have IPC: RPC + P2P. Modern OS's make localhost sockets just about as fast as other IPCs methods. Linux, at least, employs zero-copy for localhost sockets in many situations, similar to the kernel's pipe tricks. Pieter has been working on headers-first sync: https://github.com/bitcoin/bitcoin/pull/2964 Moving along this wallet/blockchain engine split requires upping the reviewtest bandwidth on Pieter's PRs, such as https://github.com/bitcoin/bitcoin/pull/3514 Unsure how much of the separate-binary discussion Gavin saw, so cc'd for emphasis. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] [RFC] [BIP proposal] Dealing with malleability
On Thu, Feb 20, 2014 at 10:07 PM, Mike Hearn m...@plan99.net wrote: No, I was thinking of the height in coinbase change. At any rate, p2sh was supported by the consensus code in bitcoinj for a long time already, since it was first written. Support for sending to such addresses in the wallet appeared once an app that wanted that support also appeared, which seems OK - the market for wallets is very competitive so there will always be some skew in what features are worked on in what order. V3 transactions are a consensus change that wallets will pick up at different times like any other feature. We're in agreement. I had mistakenly believed you were supporting the discussion about trying to force these constraints on current version transactions, in which case wallets will pick up at different times is an absolute deal breaker. :) -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Fwd: Bitcoin Core trial balloon: splitting blockchain engine and wallet
On Fri, Feb 21, 2014 at 7:27 AM, Mike Hearn m...@plan99.net wrote: Bear in mind a separate process doesn't buy you anything without a sandbox, and those are expensive (in terms of complexity). Sandboxing in user space is complex, agreed, The most straightforward way would be to run the blockchain daemon as a system service (with its own uid/gid and set of Apparmor/SELinux restrictions) and the wallet daemon as the user. This would also allow sharing one blockchain daemon between multiple users and wallet processes (not necessarily on the same machine), something I've wanted to be able to do for a long time. Wladimir -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Core trial balloon: splitting blockchain engine and wallet
RE doesn't buy you anything Today, when unlocked, plaintext private keys reside in the same address space as the blockchain engine (BCE). Process separation increases the difficulty of accessing key data from the BCE, even presuming a normal, no-chroot, same-uid, parent-child process relationship. The attack surface is clearly changed from one buffer overflow can touch this data. Regardless, the split makes sense given existing modularity and coding directions. I wouldn't micro-focus on the sandbox word. On Fri, Feb 21, 2014 at 1:27 AM, Mike Hearn m...@plan99.net wrote: Bear in mind a separate process doesn't buy you anything without a sandbox, and those are expensive (in terms of complexity). On 21 Feb 2014 11:40, Jeff Garzik jgar...@bitpay.com wrote: [Meta: Bitcoin Core is the newfangled branding of bitcoind / Bitcoin-Qt reference implementation, in case you wondering.] Several sites, including BitPay, use bitcoind outside the standard role of wallet software. bitcoind can be used purely for payment network access and management. I call this the border router role. Upcoming version 0.9 will feature the ability to disable the bitcoind wallet at compile time or runtime. This permits a more optimized border router profile, reducing process size by 40-200MB according to some reports. Recent IRC discussion have floated a rough proposal for a wallet next-step: Running the Bitcoin Core wallet as a separate process, a separate binary, from the blockchain engine. The wallet process would communicate with the blockchain engine using existing RPC and P2P channels, becoming a real SPV client. This accomplishes a longstanding security goal of sandboxing away wallet keys and sensitive data from the network-exposed P2P engine, in a separate process, among other benefits. Simple forking was explored a bit. I did some hacking in that direction, as it seemed potentially lightweight and somewhat easy to me: https://github.com/jgarzik/bitcoin/tree/fork fork+pipe is fine for Linux and OSX/BSD. However, Windows requires an exec-like solution to create a new process. MSDN does give us a Unix-pipe-like solution: http://msdn.microsoft.com/en-us/library/edze9h7e%28v=vs.80%29.aspx Others pointed to boost interprocess communication APIs, which come with their own set of caveats. Such a solution would involve a brand new IPC protocol, and lots of brand new glue code. Separate programs seems better. Windows forces us to achieve process separation via exec-like method. We already have IPC: RPC + P2P. Modern OS's make localhost sockets just about as fast as other IPCs methods. Linux, at least, employs zero-copy for localhost sockets in many situations, similar to the kernel's pipe tricks. Pieter has been working on headers-first sync: https://github.com/bitcoin/bitcoin/pull/2964 Moving along this wallet/blockchain engine split requires upping the reviewtest bandwidth on Pieter's PRs, such as https://github.com/bitcoin/bitcoin/pull/3514 Unsure how much of the separate-binary discussion Gavin saw, so cc'd for emphasis. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Bitcoin Core trial balloon: splitting blockchain engine and wallet
Running the network part of the core as a system service might make sense for server implementations, but it’s a pain in the rear for most users. That said, I think segregating the two processes is a great idea. Let’s just try to avoid some complicated scheme that involves necessarily running things under multiple users. Will On Feb 21, 2014, at 0:43, Wladimir laa...@gmail.com wrote: The most straightforward way would be to run the blockchain daemon as a system service (with its own uid/gid and set of Apparmor/SELinux restrictions) and the wallet daemon as the user. signature.asc Description: Message signed with OpenPGP using GPGMail -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Bitcoin Core trial balloon: splitting blockchain engine and wallet
On Fri, Feb 21, 2014 at 7:50 AM, William Yager will.ya...@gmail.com wrote: Running the network part of the core as a system service might make sense for server implementations, but it’s a pain in the rear for most users. Come on, making it a possibility doesn't affect other kinds of use cases in any way. Are you just arguing for the sake of arguing? Wladimir -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
