Re: [bitcoin-dev] Libconsensus phase 2
On Tue, Jan 12, 2016 at 8:17 PM, Eric Voskuilwrote: > Jorge, first, thanks again for your work on this. > > Without creating and using a public blockchain interface in phase 2, how > will you isolate the database dependency from consensus critical code? > Is it that the interface will exist but you will recommend against its use? The interface will exist but it will be a C++ interface that fits better with Bitcoin Core internals. See an initial draft of what could be the storage interface: https://github.com/jtimon/bitcoin/blob/1717db89c6db17ea65ddbd5eb19034315db0b059/src/consensus/storage_interfaces_cpp.h Phase 3 will consist on discussing and refining that interface to also define the C interfaces using structs of function pointers instead of classes (see https://github.com/jtimon/bitcoin/blob/2bcc07c014e5dd30e666be047dfa11f54c10/src/consensus/interfaces.h for an early draft) that is needed for the "final" C API. But since I think there will be more discussion and work defining those interfaces, I would rather start with ANY interface that allows us to decouple the consensus code from chain.o and coins.o, which we don't want to be built as part of the consensus building package (which is used for building both libbitcoinconsensus and Bitcoin Core). Future potential users are more than welcomed to draft their own C APIs and that experience should be useful for phase 3. I was expecting you, for example, to include the whole consensus code (even if it lacks a C API) in https://github.com/libbitcoin/libbitcoin-consensus for better testing of the equivalent code in libbitcoin. You are kind of taking the C API part out already, so this time you will just have less things to delete/ignore. > This work presumes that the users of the library reject the argument > that the database implementation is consensus critical code. Faithful > reproduction of stored data is a prerequisite for a validity. But a > common store implementation is only slightly more reasonable for this > library than a common RAM implementation. This is a concern that has been risen repeatedly. I am aware that faithful reproduction of the stored data is a prerequisite for consensus validity. On the other hand, my presumption is that a libbitcoinconsensus that forces its users to a given unifrom storage will likely had much less users and any alternative implementation that wants to implement its own custom storage would have to necessarily reimplement the consensus validation code. Doing it this way is more flexible. We can relatively easily implement another library (if I remember correctly, last time we talked about it we reffered to it as "libconsensus plus", but there's probably better names) also takes care of storage for the users that don't want to take the risks of reimplementing the storage (probably just using Bitcoin Core's structures). Unlike me, Luke Dashjr, for example, advocated for the storage-dependent version, but I believe that implementing both versions was an acceptable solution to him. It is certainly an acceptable solution for me. I don't want to force anyone that doesn't want or need to take the risks reimplementing the consensus storage part to do so. But at the same time I really believe that it would be a mistake to not allow it optionally. Does that make sense? ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus phase 2
On 01/13/2016 12:37 AM, Jorge Timón wrote: > On Tue, Jan 12, 2016 at 8:17 PM, Eric Voskuilwrote: >> Jorge, first, thanks again for your work on this. >> >> Without creating and using a public blockchain interface in phase 2, how >> will you isolate the database dependency from consensus critical code? >> Is it that the interface will exist but you will recommend against its use? > > The interface will exist but it will be a C++ interface that fits > better with Bitcoin Core internals. > See an initial draft of what could be the storage interface: > https://github.com/jtimon/bitcoin/blob/1717db89c6db17ea65ddbd5eb19034315db0b059/src/consensus/storage_interfaces_cpp.h > > Phase 3 will consist on discussing and refining that interface to also > define the C interfaces using structs of function pointers instead of > classes (see > https://github.com/jtimon/bitcoin/blob/2bcc07c014e5dd30e666be047dfa11f54c10/src/consensus/interfaces.h > for an early draft) that is needed for the "final" C API. > But since I think there will be more discussion and work defining > those interfaces, I would rather start with ANY interface that allows > us to decouple the consensus code from chain.o and coins.o, which we > don't want to be built as part of the consensus building package > (which is used for building both libbitcoinconsensus and Bitcoin > Core). Okay. > Future potential users are more than welcomed to draft their own C > APIs and that experience should be useful for phase 3. > I was expecting you, for example, to include the whole consensus code > (even if it lacks a C API) in > https://github.com/libbitcoin/libbitcoin-consensus for better testing > of the equivalent code in libbitcoin. You are kind of taking the C API > part out already, so this time you will just have less things to > delete/ignore. Generalization of the store interface may be more challenging than you anticipate, but the objective makes sense. >> This work presumes that the users of the library reject the argument >> that the database implementation is consensus critical code. Faithful >> reproduction of stored data is a prerequisite for a validity. But a >> common store implementation is only slightly more reasonable for this >> library than a common RAM implementation. > > This is a concern that has been risen repeatedly. > I am aware that faithful reproduction of the stored data is a > prerequisite for consensus validity. On the other hand, my presumption > is that a libbitcoinconsensus that forces its users to a given unifrom > storage will likely had much less users and any alternative > implementation that wants to implement its own custom storage would > have to necessarily reimplement the consensus validation code. > Doing it this way is more flexible. We can relatively easily implement > another library (if I remember correctly, last time we talked about it > we reffered to it as "libconsensus plus", but there's probably better > names) also takes care of storage for the users that don't want to > take the risks of reimplementing the storage (probably just using > Bitcoin Core's structures). > > Unlike me, Luke Dashjr, for example, advocated for the > storage-dependent version, but I believe that implementing both > versions was an acceptable solution to him. > It is certainly an acceptable solution for me. I don't want to force > anyone that doesn't want or need to take the risks reimplementing the > consensus storage part to do so. But at the same time I really believe > that it would be a mistake to not allow it optionally. > > Does that make sense? We would not offer libbitcoinconsensus integration if it required us to incorporate the store. These are distinct logical components, as are p2p networking and client-server networking (e.g. RPC), for example. I would not think of these as multiple versions of libbitcoinconsensus but instead as distinct components of a bitcoin node. It doesn't make sense to me that you would ship this as two consensus variants. I would work toward shipping independent component libraries (i.e. consensus and store). e signature.asc Description: OpenPGP digital signature ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus phase 2
Jorge, first, thanks again for your work on this. Without creating and using a public blockchain interface in phase 2, how will you isolate the database dependency from consensus critical code? Is it that the interface will exist but you will recommend against its use? This work presumes that the users of the library reject the argument that the database implementation is consensus critical code. Faithful reproduction of stored data is a prerequisite for a validity. But a common store implementation is only slightly more reasonable for this library than a common RAM implementation. e On 01/12/2016 09:53 AM, Jorge Timón wrote: > After talking to some people about libconsensus in the last Hong Kong > conference I realized that my initial plan of exposing one more thing > at a time would actually probably slow things down. > > There's still a promised pdf with pictures that will be released, and > actually drafting the UML pictures helped realize that the whole > explanation could be much simpler if #7091 was merged first as the > last step in phase 1 (that phase has so many contributors that I will > probably never get finished documenting it). Matt Corallo's idea of > exposing VerifyScript() through a C API certainly helped a lot in > cementing the more-minimal-than-earlier dependencies (thanks to Cory > Fields among many other people before him) that are not part of the > incomplete but existing libbitcoinconsensus library. > > Given this success in protecting encapsulation by exposing things in a > new library, my instinct was to expose more things: VerifyHeader(), > VerifyTx() and VerifyBlock() [in that order]. > But all those three new functions depend on storage in one way or > another. That was part of my reasoning to expose VerifyHeader() first, > because I believe there will be less discussion on a common interface > for the stored longest chain than for the utxo view (which may depend > on other transactions spent within the same block). > In any case, I realized we should finish putting all the consensus > critical code in the libconsensus lib and then worry about its "final" > API. > > Therefore I changed the goal of the phase 2 in my libconsensus > encapsulation planning from "expose VerifyHeader() in the existing > libconsensus library" to "build all the consensus critical code within > the existing libconsensus library, even if we don't expose anything > else". I believe this is much feasible for a single Bitcoin Core > release cycle and also more of a priority. Other implementations > experimenting with libconsensus like > https://github.com/libbitcoin/libbitcoin-consensus will have the > chance to compare their reimplementations with the future complete > libbitcoinconsensus without having to worry about the C API, which > ideally they will help to define. > > I repeat, the goal of phase 2 in my upcoming libconsensus > encapsulation plan is to fully decouple libconsensus from Bitcoin > Core. > In phase 3, we can refine the storage interfaces and focus on a > quasi-final C API. > In phase 4, we can refine and take out code that doesn't belong in > libconsensus like CTxOut::GetDustThreshold() in > primitives/transaction.h and move all those consensus files to the > consensus folder before creating a separate sub-repository like for > libsecp256k1. Note that most of the file-moving work can be in > parallel to phases 2 and 3 and, in fact, by any new developer that is > willing to exchange rebase-patience for meaningless github stats (I'll > do it if nobody else wants, but I'm more than happy to delegate there: > I have more than enough github meaningless stats already). > > As said, the document with pictures and the update to #6714 are still > promised, but until they're ready, merging/reviewing #7091, #7287, > #7310 and #7311 could do a great deal to make later steps in > libconsensus phase 2 more readable. > Most reviewers probably don't need to see any "big picture" to tell > whether certain functions on Bitcoin Core are consensus-critical or > not, or whether consensus critical code needs to depend on util.o or > not. > But I wouldn't be writing to the mailing list without a plan with > further words nor pictures if I didn't had what I believe is a > complete implementation of what I just defined as "libconsensus phase > 2". > > Phase 3 should finish long pending discussions like "should > libconsensus be C++14 or plain C" which should NOT delay phase 2. > Phase 4 should be mostly trivial: rename files to the target dir and > move the remaining unused code out of libconsensus. > Phase 5 should make Bitcoin Core eat its own dog food and use > libbitcoinconsensus oonly by its generic C API (I'm sorry if this > looks too far away for me to even think about detailing it). > > The work in progress branch (but hopefully being finished, nit and > merged within the 0.12.99 cycle) can be found in: > https://github.com/jtimon/bitcoin/commits/libconsensus-f2 > > Before sipa asks, signing
[bitcoin-dev] Libconsensus phase 2
After talking to some people about libconsensus in the last Hong Kong conference I realized that my initial plan of exposing one more thing at a time would actually probably slow things down. There's still a promised pdf with pictures that will be released, and actually drafting the UML pictures helped realize that the whole explanation could be much simpler if #7091 was merged first as the last step in phase 1 (that phase has so many contributors that I will probably never get finished documenting it). Matt Corallo's idea of exposing VerifyScript() through a C API certainly helped a lot in cementing the more-minimal-than-earlier dependencies (thanks to Cory Fields among many other people before him) that are not part of the incomplete but existing libbitcoinconsensus library. Given this success in protecting encapsulation by exposing things in a new library, my instinct was to expose more things: VerifyHeader(), VerifyTx() and VerifyBlock() [in that order]. But all those three new functions depend on storage in one way or another. That was part of my reasoning to expose VerifyHeader() first, because I believe there will be less discussion on a common interface for the stored longest chain than for the utxo view (which may depend on other transactions spent within the same block). In any case, I realized we should finish putting all the consensus critical code in the libconsensus lib and then worry about its "final" API. Therefore I changed the goal of the phase 2 in my libconsensus encapsulation planning from "expose VerifyHeader() in the existing libconsensus library" to "build all the consensus critical code within the existing libconsensus library, even if we don't expose anything else". I believe this is much feasible for a single Bitcoin Core release cycle and also more of a priority. Other implementations experimenting with libconsensus like https://github.com/libbitcoin/libbitcoin-consensus will have the chance to compare their reimplementations with the future complete libbitcoinconsensus without having to worry about the C API, which ideally they will help to define. I repeat, the goal of phase 2 in my upcoming libconsensus encapsulation plan is to fully decouple libconsensus from Bitcoin Core. In phase 3, we can refine the storage interfaces and focus on a quasi-final C API. In phase 4, we can refine and take out code that doesn't belong in libconsensus like CTxOut::GetDustThreshold() in primitives/transaction.h and move all those consensus files to the consensus folder before creating a separate sub-repository like for libsecp256k1. Note that most of the file-moving work can be in parallel to phases 2 and 3 and, in fact, by any new developer that is willing to exchange rebase-patience for meaningless github stats (I'll do it if nobody else wants, but I'm more than happy to delegate there: I have more than enough github meaningless stats already). As said, the document with pictures and the update to #6714 are still promised, but until they're ready, merging/reviewing #7091, #7287, #7310 and #7311 could do a great deal to make later steps in libconsensus phase 2 more readable. Most reviewers probably don't need to see any "big picture" to tell whether certain functions on Bitcoin Core are consensus-critical or not, or whether consensus critical code needs to depend on util.o or not. But I wouldn't be writing to the mailing list without a plan with further words nor pictures if I didn't had what I believe is a complete implementation of what I just defined as "libconsensus phase 2". Phase 3 should finish long pending discussions like "should libconsensus be C++14 or plain C" which should NOT delay phase 2. Phase 4 should be mostly trivial: rename files to the target dir and move the remaining unused code out of libconsensus. Phase 5 should make Bitcoin Core eat its own dog food and use libbitcoinconsensus oonly by its generic C API (I'm sorry if this looks too far away for me to even think about detailing it). The work in progress branch (but hopefully being finished, nit and merged within the 0.12.99 cycle) can be found in: https://github.com/jtimon/bitcoin/commits/libconsensus-f2 Before sipa asks, signing code may make it into a new library but SHOULDN'T BE PART OF LIBBITCOINCONSENSUS. Ideally, all exposed functions will return true or false and an error string. It is based on last-0.12.99 3cd836c1 but by popular demand I can open it as a "DEPENDENT-tagged" PR linking to smaller steps and keeping track of steps done. Analogous to the about to be replaced (for a simpler and more maintainable example of testchain) #6382. If people like Wladimir, Cory and Pieter cannot see that I've been able to reduce my overall cry-for-review noise thanks to github adoption of emacs' org-mode's [ ] vs [X] I can alwways leave those "big picture" branches as "private" branches out of the pull request count. I expect to publish a phase 3 branch very shortly. But as said I expect a lot of discussion on the