Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Thu, Jul 23, 2015 at 04:30:06PM +0200, Jorge Timón via bitcoin-dev wrote: I think there were some misunderstandings in our previous conversation about this topic. I completely agree with having a separated repository for libconsensus (that's the whole point, alternative implementations can be consensus-safe by using it, and in the event of a schism fork[1], they can fork just that smaller project without having to relay on Bitcoin Core [satoshi] at all). Indeed. But I thought you also wanted Bitcoin Core to use libconsensus instead of just having a subtree/subrepository like it currently does with libsecp256k1. I'm not sure if that would ever be accepted, but in any case we're certainly far away from that goal. Here are some things that need to happen first: I don't see any reason why Bitcoin Core would not use the consensus library. Eating our own dogfood and such. Biggest risk, as I've said before, is that the refactoring loading to a (more complete) consensus library will result in code that is no longer bug-for-bug compatible with previous versions, thus defeating its entire purpose and introducing fork risk. If that can be avoided - for example by going from here to there using pure code moves, as you're trying to do - I'm all for it. 2) Finish libconsensus's API: expose more things than VerifyScript, at the very least, also expose VerifyTx, VerifyHeader and VerifyBlock. Feedback from alternative implementations like libbitcoin is extremely valuable here. Some related closed-for-now PRs: Agreed. 3) Move libconsensus to a separate repository as a subtree/subrepository of Bitcoin Core. If the rest is done, this is the easy part :) Unfortunately and ironically again, safer, small and incremental changes are less interesting for reviewers. For example, I've been trying to move consensus code to the consensus folder for a long time. The correctness of a MOVEONLY change is trivial to review for anyone who knows how to copy/paste in its favorite editor and how to use git diff, but will I ever get answers to my questions in [1]? Code review capacity is still our greatest bottleneck. And I don't see any way out of that, unfortunately. I know there's many people who really care about this, Cory Fields, Wladimir and Pieter Wuille to name a few have reviewed many of this changes (I've just got used to publicly whine about lack of review on this front and policy encapsulation [very related fronts] as an attempt to get some attention: not always, but begging for review actually works some times). I do really care about this. Wladimir ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Mon, Jul 27, 2015 at 11:40:42PM -0700, Eric Voskuil via bitcoin-dev wrote: It's a performance sacrifice, and then there's the OpenSSL dependency, but these are both optional within our stack - so the application developer has the option. So the only downside is that we are maintaining the conditional compilation. Now that BIP66 became active, and only strict DER signatures are allowed, the OpenSSL dependency can be removed from consensus. Pieter Wuille will do an announcement on this soon. Wladimir ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Tue, Jul 28, 2015 at 10:43 AM, Wladimir J. van der Laan laa...@gmail.com wrote: On Thu, Jul 23, 2015 at 04:30:06PM +0200, Jorge Timón via bitcoin-dev wrote: But I thought you also wanted Bitcoin Core to use libconsensus instead of just having a subtree/subrepository like it currently does with libsecp256k1. I'm not sure if that would ever be accepted, but in any case we're certainly far away from that goal. Here are some things that need to happen first: I don't see any reason why Bitcoin Core would not use the consensus library. Eating our own dogfood and such. As explained to Eric, it's not that I don't want Bitcoin Core to use future-libconsensu through the API instead of a subtree: it's just that that's more long-term and more work. And I don't see why other implementations should really care about it. Biggest risk, as I've said before, is that the refactoring loading to a (more complete) consensus library will result in code that is no longer bug-for-bug compatible with previous versions, thus defeating its entire purpose and introducing fork risk. If that can be avoided - for example by going from here to there using pure code moves, as you're trying to do - I'm all for it. Well, pure movements will not be enough, parameters will have to change, incompatible dependencies have to be removed (ie util.h which contains globals), etc. But yes, I think we can do it with only low-risk and easy-to-review commits. 3) Move libconsensus to a separate repository as a subtree/subrepository of Bitcoin Core. If the rest is done, this is the easy part :) And still, this doesn't require Bitcoin Core to use the API, a subtree is enough at first. This easy step doesn't guarantee that Bitcoin Core is using future-libconsensus' API. Code review capacity is still our greatest bottleneck. And I don't see any way out of that, unfortunately. I really think these code separations help with this (ie there are many more people in the world with enough knowledge to review the qt or even policy parts than there's people able to review consensus changes). I do really care about this. I know and I said so. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
Ok, I'm going to separate terms: current-libconsensus from theoretical future-libconsensus (implementing ALL consensus rules). On Tue, Jul 28, 2015 at 8:40 AM, Eric Voskuil e...@voskuil.org wrote: libsecp256k1 has it's own repository, libbitcoinconsensus doesn't. A separate repository was what I considered as a requirement for us to use it. We want to complete future-libconsensus (decouple all the consensus rules from the rest of the bitcoin core code) first. Then we can move future-libconsensus to a subrepository/subtree like libsecp256k1 and I believe everybody wants this to eventually happen. Separating current-libconsensus now may make completing future-libconsensus harder. I'm not sure if that would ever be accepted, but in any case we're certainly far away from that goal. If it's not certain whether this would even be accepted, the commitment to a community consensus library is too weak to take a strong dependency on. But for us it's moot, as we have made the already accomplished that goal. What I mean is that once it is separated to a subtree, there's one more step: Make Bitcoin Core use future-libconsensus' API instead of a subtree. Decoupling future-libconsensus from Bitcoin Core is one thing, and Decoupling Bitcoin Core from future-libconsensus is another thing: you need to decouple Bitcoin Core from all future-libconsensus implementation internals. For example, script/sign (part of Bitcoin Core) depends on individual non-API-exposed classes in current-libconsensus. Moving from a subtree to a completely separated library is what I don't know will ever happen, but I don't think this is unfairly advantageous for Bitcoin Core or anything like that: other implementations can also use future-libcosensus as a subtree instead of the C API as well. You have accomplished the goal of separating curren-libconsensus, not future-libconsensus. In fact, if you complete the equivalent of future-libconsensus in libbitcoin and separate that, maybe that's a better place to start drafting a full API. 2) Finish libconsensus's API: expose more things than VerifyScript, at the very least, also expose VerifyTx, VerifyHeader and VerifyBlock. Feedback from alternative implementations like libbitcoin is extremely valuable here. Some related closed-for-now PRs: In our earlier discussion I believe you said that the library would not be undergoing significant change or feature creep. If this is the very least that's projected it would seem that constraint will not hold. future-libconsensus will not have significant changes *once it is completed*. Currently future-libconsensus is spread around many places inclusing src/main, so that obviously needs to change before it can be separated to an independent repo. 3) Move libconsensus to a separate repository as a subtree/subrepository of Bitcoin Core. Only after all that we can discuss whether Bitcoin Core itself should include libconsensus' code or just use its API directly. I don't think it's a question of whether it *should* use its own library as it is published for others - this is a practically self-evident conclusion. Well, Bitcoin Core is currently the only user of future-libconsensus since bitcoin core and future-libconsensus are currently mutually coupled. Bitcoin Core will always keep using future-libconsensus. The only question is whether it will use it through the C API or as a subtree/subrepository (both options are also available to other implementations). I don't know if decoupling Bitcoin Core from future-libconsensus' implementation details enough to be able to directly use the API is worth it or if anyone will be interested in doing so. For me this last step is not all that interesting: if we have an independent repo with a full API that other implementations can use, I don't really care about Bitcoin Core not going through the API and using including all the code directly instead. I hope that after all this, libbitcoin also reconsiders whether to reimplement its own libconsensus or use the official one directly instead. We use a fork of libsecp256k1 and would probably do the same with the consensus library if it was cleanly isolated. Great. In any case I agree with your stated need for this isolation (if not the means) for the reasons you state. The community needs to move beyond a largely singular and monolithic codebase that is holding that position in part due to fear about consensus bug forks. I completely agree. That's the goal of libconsensus (and an alternative implementation like libbitcoin being able to use it without sacrificing any of its current or future design differences from Bitcoin Core would be a sign of success in this reward). It's a performance sacrifice, and then there's the OpenSSL dependency, but these are both optional within our stack - so the application developer has the option. So the only downside is that we are maintaining the conditional compilation. As I told you
Re: [bitcoin-dev] Why Satoshi's temporary anti-spam measure isn't temporary
I agree that the historical reasons are irrelevant from an engineering perspective. But they still set a context for the discussion…and might help shed some insight into the motivations behind some of the participants. It’s also good to know these things to counter arguments that start with “But Satoshi said that…” What’s critically important to note is that several of the assumptions that were being made at the time this limit was decided have turned out wrong…and that these other issues should probably be of greater concern and more highly prioritized in any discussion considering the merits of deploying potentially incompatible consensus rule changes. It seems if these other issues were fixed perhaps no block size limit would be required at all (as was originally hoped). - Eric On Jul 28, 2015, at 5:46 PM, Mark Friedenbach m...@friedenbach.org wrote: Does it matter even in the slightest why the block size limit was put in place? It does not. Bitcoin is a decentralized payment network, and the relationship between utility (block size) and decentralization is empirical. Why the 1MB limit was put in place at the time might be a historically interesting question, but it bears little relevance to the present engineering issues. On Tue, Jul 28, 2015 at 5:43 PM, Jean-Paul Kogelman via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org mailto:bitcoin-dev@lists.linuxfoundation.org wrote: Enter a “temporary” anti-spam measure - a one megabyte block size limit. Let’s test this out, then increase it once we see how things work. So far so good… The block size limit was put in place as an anti-DoS measure (monster blocks), not anti-spam. It was never intended to have any economic effect, not on spam and not on any future fee market. jp ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org mailto:bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Why Satoshi's temporary anti-spam measure isn't temporary
Does it matter even in the slightest why the block size limit was put in place? It does not. Bitcoin is a decentralized payment network, and the relationship between utility (block size) and decentralization is empirical. Why the 1MB limit was put in place at the time might be a historically interesting question, but it bears little relevance to the present engineering issues. On Tue, Jul 28, 2015 at 5:43 PM, Jean-Paul Kogelman via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org wrote: Enter a “temporary” anti-spam measure - a one megabyte block size limit. Let’s test this out, then increase it once we see how things work. So far so good… The block size limit was put in place as an anti-DoS measure (monster blocks), not anti-spam. It was never intended to have any economic effect, not on spam and not on any future fee market. jp ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[bitcoin-dev] Disclosure: consensus bug indirectly solved by BIP66
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello all, I'd like to disclose a vulnerability I discovered in September 2014, which became unexploitable when BIP66's 95% threshold was reached earlier this month. ## Short description: A specially-crafted transaction could have forked the blockchain between nodes: * using OpenSSL on a 32-bit systems and on 64-bit Windows systems * using OpenSSL on non-Windows 64-bit systems (Linux, OSX, ...) * using some non-OpenSSL codebases for parsing signatures ## Upgrade instructions: None. Transactions that could trigger this problem have become invalid on the network since BIP66's deployment of version 3 blocks reached 95% on July 4th, 2015. ## Long description: The problem is related to the signature encoding rules. Bitcoin's signatures are ASN.1 BER encoded. BER is a complex standard that allows many different encodings for the same data. Since Bitcoin Core 0.8, a standardness rule has been in effect that only allowed subset of encodings (DER) for relay and mining, even though any BER remained valid in the blockchain - at least in theory. In practice, BER has many weird edge cases, and I have not found a single cryptographic codebase that can parse all of them correctly. This includes OpenSSL, Crypto++, BouncyCastle, btcec, and our own libsecp256k1 library. This on itself would not be a problem, as full nodes on the network currently use OpenSSL. However, while researching what was needed to make libsecp256k1 compatible with it, I discovered that OpenSSL is even inconsistent with itself across different platforms. One of the features of BER is the ability for internal structures to have a length descriptor whose size itself is up to 126 bytes (see X.690-0207 8.1.3.5). A 1 terabyte data structure would for example use a 5-byte length descriptor. However, there is no requirement to use the shortest possible descriptor, so even a 70-byte ECDSA signature could use a 5-byte length descriptor and be valid. Unfortunately, OpenSSL supports length descriptors only as long as their size is at most that of a C 'long int', a type whose size depends on the platform (Windows and 32-bit Linux/OSX have a 4-byte long int, 64-bit Linux/OSX have an 8-byte long int). See https://github.com/openssl/openssl/blob/bfa34f551c2d38e826deb44a269cb0f720f9f63b/crypto/asn1/asn1_lib.c#L178. Some non-OpenSSL based signature validation systems don't support such length descriptors at all, resulting in an extra forking risk on top for them if used for blockchain validation. This effectively means that a block chain containing a transaction with a valid signature using such a 5-byte length descriptor would be accepted by some systems and not by others, resulting in a fork if it were mined into a block. ## Timeline: * 2013-Feb-19: Bitcoin Core 0.8.0 was released, which made non-DER signatures non-standard. No release since then would relay or mine transactions that could trigger the vulnerability. However, such a transaction was still valid inside blocks. * 2014-Feb-10: I proposed BIP62 to deal with transaction malleability. The BIP62 draft includes a rule that would make version 2 transactions with non-DER signatures invalid. * 2014-Jul-18: In order to make Bitcoin's signature encoding rules not depend on OpenSSL's specific parser, I modified the BIP62 proposal to have its strict DER signatures requirement also apply to version 1 transactions. No non-DER signatures were being mined into blocks anymore at the time, so this was assumed to not have any impact. See https://github.com/bitcoin/bips/pull/90 and http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2014-July/006299.html. Unknown at the time, but if deployed this would have solved the vulnerability. * 2014-Sep-01: While analysing OpenSSL's source code for BER parsing, I discovered the architecture dependency listed above and the associated vulnerability. The best means to fix it at the time was by getting BIP62 adopted. * 2014-Sep-07, 2014-Oct-17, 2014-Oct-26, 2014-Dec-06, 2015-Jan-09: Several proposed changes to BIP62. See https://github.com/bitcoin/bips/pulls?utf8=%E2%9C%93q=is%3Apr+is%3Aclosed+bip62. * 2015-Jan-10: OpenSSL releases versions 1.0.0p and 1.0.1k, with a fix for CVE-2014-8275. The fix introduced a restriction on ECDSA signatures to be strict DER, which would have solved all problems related to signature encodings, except Bitcoin's consensus-critical nature requires bug-for-bug compatibility between nodes. Worse, it seemed that there was again a small (1%) number of blocks being created with non-DER signatures in it, resulting in actual forks. The only immediate solution that did not introduce more risk for forks was parsing and re-encoding signatures using OpenSSL itself before verification to bypass the restriction, making the problem persist. See http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-January/007097.html. * 2015-Jan-21: The new attention to signature encoding might have revealed the
Re: [bitcoin-dev] Why Satoshi's temporary anti-spam measure isn't temporary
On Jul 28, 2015, at 7:40 PM, Eric Lombrozo elombr...@gmail.com wrote: Note: many of these ideas are neither my own nor really all that new, but it seems in the past we’ve given up too easily on actually moving forward on them despite their critical importance. In retrospect I regret not having made this note more emphatic: GUYS, WE’VE KNOWN ABOUT THESE PROBLEMS AND HAVE TALKED ABOUT THEM FOR YEARS ALREADY…AND IT SEEMS PRACTICALLY NOTHING HAS HAPPENED…WTF?!?!?!? signature.asc Description: Message signed with OpenPGP using GPGMail ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Bitcoin Core and hard forks
That's not what I said. We don't seem able to communicate with each other efficiently, probably my fault since English is not my native language. But I don't want to use more of my time (or yours) in this discussion, since it's clearly unproductive. On Jul 28, 2015 6:45 PM, Tom Harding t...@thinlink.com wrote: Jorge, We obviously disagree fundamentally on the role of societal adoption, in the system that Satoshi designed. Adoption is well ahead of Satoshi's schedule, and the measure of this is the exchange rate. It is at once an imperfect measure, and one of the most perfect markets that has ever existed. As long as hardware, electric power, and bandwidth are priced in fiat currency, the exchange rate is a critical variable to security, capacity, and other metrics of network health. It's not inconsistent that you consider the exchange rate irrelevant. In fact it explains why you believe that Satoshi's timetable for transitioning to fee incentives can be summarily tossed aside and replaced with something you think is better. Here's an English saying I just invented. A bunch of geniuses can do a lot more damage than one fool. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On 07/23/2015 07:30 AM, Jorge Timón wrote: On Thu, Jul 23, 2015 at 2:49 AM, Eric Voskuil via bitcoin-dev wrote: On 07/22/2015 05:13 PM, Eric Lombrozo via bitcoin-dev wrote: Only being partly serious - I strongly am in favor of a sufficiently modularized codebase that swapping out consensus rules is fairly straightforward and easy to test... We (libbitcoin) have taken the time to publish and maintain bitcoind's libbitcoinconsensus source files as an independent C++ library... I think there were some misunderstandings in our previous conversation about this topic. I completely agree with having a separated repository for libconsensus (that's the whole point, alternative implementations can be consensus-safe by using it, and in the event of a schism fork[1], they can fork just that smaller project without having to relay on Bitcoin Core [satoshi] at all). But I thought you also wanted Bitcoin Core to use libconsensus instead of just having a subtree/subrepository like it currently does with libsecp256k1. libsecp256k1 has it's own repository, libbitcoinconsensus doesn't. A separate repository was what I considered as a requirement for us to use it. I'm not sure if that would ever be accepted, but in any case we're certainly far away from that goal. If it's not certain whether this would even be accepted, the commitment to a community consensus library is too weak to take a strong dependency on. But for us it's moot, as we have made the already accomplished that goal. Here are some things that need to happen first: 1) Finish encapsulating consensus code so that it can be built without any (we've done it only with script-related code so far). Here are some related PRs (other people have done other things that help with this as well): ... 2) Finish libconsensus's API: expose more things than VerifyScript, at the very least, also expose VerifyTx, VerifyHeader and VerifyBlock. Feedback from alternative implementations like libbitcoin is extremely valuable here. Some related closed-for-now PRs: In our earlier discussion I believe you said that the library would not be undergoing significant change or feature creep. If this is the very least that's projected it would seem that constraint will not hold. 3) Move libconsensus to a separate repository as a subtree/subrepository of Bitcoin Core. Only after all that we can discuss whether Bitcoin Core itself should include libconsensus' code or just use its API directly. I don't think it's a question of whether it *should* use its own library as it is published for others - this is a practically self-evident conclusion. I hope that after all this, libbitcoin also reconsiders whether to reimplement its own libconsensus or use the official one directly instead. We use a fork of libsecp256k1 and would probably do the same with the consensus library if it was cleanly isolated. In any case I agree with your stated need for this isolation (if not the means) for the reasons you state. The community needs to move beyond a largely singular and monolithic codebase that is holding that position in part due to fear about consensus bug forks. I completely agree. That's the goal of libconsensus (and an alternative implementation like libbitcoin being able to use it without sacrificing any of its current or future design differences from Bitcoin Core would be a sign of success in this reward). It's a performance sacrifice, and then there's the OpenSSL dependency, but these are both optional within our stack - so the application developer has the option. So the only downside is that we are maintaining the conditional compilation. Unfortunately any changes that touch consensus code are risky and therefore slow. And when consensus encapsulation changes conflict with other changes (not because the other changes need to change consensus but because consensus code is still coupled with policy and other bitcoind-specific code), refactors are never prioritized. Ironically, you need to encapsulate the consensus code to avoid such conflicts, which would make all non-consensus changes far less risky (reducing the consensus-critical review development bottleneck). Unfortunately and ironically again, safer, small and incremental changes are less interesting for reviewers. For example, I've been trying to move consensus code to the consensus folder for a long time. The correctness of a MOVEONLY change is trivial to review for anyone who knows how to copy/paste in its favorite editor and how to use git diff, but will I ever get answers to my questions in [1]? I think it's worthwhile work, especially if you are passionate about the longer term objectives. I haven't been involved in these reviews as I spend very little time with the satoshi client I know there's many people who really care about this, Cory Fields, Wladimir and Pieter Wuille to name a few have reviewed many of this changes (I've just got used to
