Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Sun, Aug 23, 2015 at 8:42 AM, Tamas Blummer ta...@bitsofproof.com wrote: I see the huge amount of sweat and love that went into core and it actually hurts to see that most is expended in friction and lack of a vision for the software architecture. To be concrete, this was my plan if dealing with the Core code base: 1) I'd consider the separation of networking and storage as suggested for a future extended libconsensus low priority, as their design should be (are) dominated by the need of the consensus logic only. 2) create an API to the consensus+networking+storage service that is not at the C++ language level but some scaleable cross-platform remoting, like eg. ZeroMQ. This API should be minimal and simple, assuming that one fully trusts the node answering it. This API would unlock user land development by distinct teams with diverse technologies. I plan to replicate the RPC API (or a subset of it) using ZMQ's req/rep pattern, but #6103 comes first. 3) move the wallet, QT and RPC and other backward compatibility stuff (if e.g. there is some mining support) in-top of the new API and into distinct source code repositories. Well, the RPC is the API. For libconsensus, its C API is the API. We've been talking about separating the wallet and qt to a different repository for long, but modularization is a prerequisite. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
I see the huge amount of sweat and love that went into core and it actually hurts to see that most is expended in friction and lack of a vision for the software architecture. To be concrete, this was my plan if dealing with the Core code base: 1) I'd consider the separation of networking and storage as suggested for a future extended libconsensus low priority, as their design should be (are) dominated by the need of the consensus logic only. 2) create an API to the consensus+networking+storage service that is not at the C++ language level but some scaleable cross-platform remoting, like eg. ZeroMQ. This API should be minimal and simple, assuming that one fully trusts the node answering it. This API would unlock user land development by distinct teams with diverse technologies. 3) move the wallet, QT and RPC and other backward compatibility stuff (if e.g. there is some mining support) in-top of the new API and into distinct source code repositories. Tamas Blummer On Aug 23, 2015, at 03:23, Eric Lombrozo elombr...@gmail.com wrote: I've been pushing for greater modularization since I first got into bitcoin. I got quickly frustrated when I was only able to get through very few things (i.e. moving core structure serialization classes to a separate unit not called main). Working on Bitcoin has an added layer of frustration that goes beyond most open source projects: even though we're clearly in userland working at the application layer, a good layered protocol design is still lacking. We have no standards process separate from what basically amount to updates to one specific reference implementation. And we all need to agree on any major change, since a blockchain that is easily forked in contentious ways pretty much defeats its own purpose. I went off to develop my own stack, where I could more easily avoid politics and focus on engineering. But I now understand the politics are inevitable. Bitcoin is inherently a cooperative project. Several people have poured themselves passionately into the reference codebase, most of whom did it (at least initially) purely as unpaid volunteers. There's a lot of love that's gone into this. But it's become pretty clear that the modularization is no longer merely a matter of good engineering - it is essential to resolving serious political challenges. Perhaps the most frustrating thing of all is watching people pushing for relatively superficial yet highly controversial changes while we still lack the proper infrastructure to handle these kinds of divergences of opinion without either stagnating or becoming polarized. I could continue working to reimplement an entire stack from scratch, as several others have also done - but besides the serious effort duplication this entails, it doesn't really seem like it will ultimately be a convergent process. It's too easy to let ego and habit dictate one's preferences rather than rational engineering considerations. I know that some might feel I'm just preaching to the choir, but we should probably take a step back from implementation hackery and try to specify some core protocol layers, focusing on interfaces. Specifically, we need a consensus layer that doesn't try to specify networking, storage, wallets, UI, etc. Let different people improve upon these things independently in their own implementations. What matters is that we all converge on a common history and state. At the same time, let's open up more competition on all these other things that are separate from the consensus layer. If only we were to dedicate a fraction of the effort we've put into this whole block size circus into what's actually important...and I blame myself as well... On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org mailto:bitcoin-dev@lists.linuxfoundation.org wrote: On Aug 21, 2015, at 21:46, Jorge Timón jti...@jtimon.cc mailto:jti...@jtimon.cc wrote: On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer ta...@bitsofproof.com mailto:ta...@bitsofproof.com wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. But you don't want something better, you want something functionally identical. You may want to watch sipa's explanation on why the implementation is the specification and the reasons to separate libconsensus: https://youtu.be/l3O4nh79CUU?t=764 https://youtu.be/l3O4nh79CUU?t=764 I do want something better, but not for the focus you have. Not because what you produce was not high quality, but because quality is achieved at a very high cost and is hard to uphold over generations of developer. You focus on a single use case while there are many out there for distributed ledgers. I think in an infrastructure for enterprise applications, building
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Aug 21, 2015, at 21:46, Jorge Timón jti...@jtimon.cc wrote: On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer ta...@bitsofproof.com mailto:ta...@bitsofproof.com wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. But you don't want something better, you want something functionally identical. You may want to watch sipa's explanation on why the implementation is the specification and the reasons to separate libconsensus: https://youtu.be/l3O4nh79CUU?t=764 https://youtu.be/l3O4nh79CUU?t=764 I do want something better, but not for the focus you have. Not because what you produce was not high quality, but because quality is achieved at a very high cost and is hard to uphold over generations of developer. You focus on a single use case while there are many out there for distributed ledgers. I think in an infrastructure for enterprise applications, building consensus on the ledger is a cornerstone there, but is only a piece of the solution. I built several commercially successful deployments where I delegated the consensus building to a border router, a Bitcoin Core, then interfaced that trusted peer with my implementation that accepted Core’s decisions in an SPV manner. One might think of this setup as wasteful and unsuitable for “small devices” therefore an example of centralization people here try to avoid. Enterprises have sufficient resources. Solving the business problem is valuable to them even at magnitudes higher cost than a hobbyist would bear. For mainstream adoption you need to get enterprises on board too, and that is what I care of. Enterprises want code that is not only high quality, but is easy to maintain with a development team with high attrition. One has to take whatever help is offered for that, and one is modern languages and runtimes. Bits of Proof’s own implementation of the scripts was not practically relevant in my commercially successful deployments, because of the use of a border router, but it helped development, enabling easier debug and precise error feedback esp. end even after Core had a reject message. I integrated libconsensus only for the hope that is significantly fastens application side tx verification, which it has turned out it does not, until secp265k1 is integrated. I would likely use an other extended libconsensus too, but do not think there was a dependency on that for enterprise development. It would help there more to have a slim protocol server, no wallet, no rpc, no qt but a high performance remoting API. Since you already depend on libconsensus for VerifyScript, wouldn't it be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock? You would still have complete control over storage, concurrency, networking, policy... My plan is for the C API to interface with the external storage by passing a function pointer to it. Storage and validation is non-trivially interconnected, but I now the separation can be done, since I did it. Excuse me, but function pointers is a pattern I used in the 80’s. I know that they are behind the curtain of modern abstractions with similar use, I still prefer not to see them again. Tamas Blummer signature.asc Description: Message signed with OpenPGP using GPGMail ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
I've been pushing for greater modularization since I first got into bitcoin. I got quickly frustrated when I was only able to get through very few things (i.e. moving core structure serialization classes to a separate unit not called main). Working on Bitcoin has an added layer of frustration that goes beyond most open source projects: even though we're clearly in userland working at the application layer, a good layered protocol design is still lacking. We have no standards process separate from what basically amount to updates to one specific reference implementation. And we all need to agree on any major change, since a blockchain that is easily forked in contentious ways pretty much defeats its own purpose. I went off to develop my own stack, where I could more easily avoid politics and focus on engineering. But I now understand the politics are inevitable. Bitcoin is inherently a cooperative project. Several people have poured themselves passionately into the reference codebase, most of whom did it (at least initially) purely as unpaid volunteers. There's a lot of love that's gone into this. But it's become pretty clear that the modularization is no longer merely a matter of good engineering - it is essential to resolving serious political challenges. Perhaps the most frustrating thing of all is watching people pushing for relatively superficial yet highly controversial changes while we still lack the proper infrastructure to handle these kinds of divergences of opinion without either stagnating or becoming polarized. I could continue working to reimplement an entire stack from scratch, as several others have also done - but besides the serious effort duplication this entails, it doesn't really seem like it will ultimately be a convergent process. It's too easy to let ego and habit dictate one's preferences rather than rational engineering considerations. I know that some might feel I'm just preaching to the choir, but we should probably take a step back from implementation hackery and try to specify some core protocol layers, focusing on interfaces. Specifically, we need a consensus layer that doesn't try to specify networking, storage, wallets, UI, etc. Let different people improve upon these things independently in their own implementations. What matters is that we all converge on a common history and state. At the same time, let's open up more competition on all these other things that are separate from the consensus layer. If only we were to dedicate a fraction of the effort we've put into this whole block size circus into what's actually important...and I blame myself as well... On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org wrote: On Aug 21, 2015, at 21:46, Jorge Timón jti...@jtimon.cc wrote: On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer ta...@bitsofproof.com wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. But you don't want something better, you want something functionally identical. You may want to watch sipa's explanation on why the implementation is the specification and the reasons to separate libconsensus: https://youtu.be/l3O4nh79CUU?t=764 I do want something better, but not for the focus you have. Not because what you produce was not high quality, but because quality is achieved at a very high cost and is hard to uphold over generations of developer. You focus on a single use case while there are many out there for distributed ledgers. I think in an infrastructure for enterprise applications, building consensus on the ledger is a cornerstone there, but is only a piece of the solution. I built several commercially successful deployments where I delegated the consensus building to a border router, a Bitcoin Core, then interfaced that trusted peer with my implementation that accepted Core’s decisions in an SPV manner. One might think of this setup as wasteful and unsuitable for “small devices” therefore an example of centralization people here try to avoid. Enterprises have sufficient resources. Solving the business problem is valuable to them even at magnitudes higher cost than a hobbyist would bear. For mainstream adoption you need to get enterprises on board too, and that is what I care of. Enterprises want code that is not only high quality, but is easy to maintain with a development team with high attrition. One has to take whatever help is offered for that, and one is modern languages and runtimes. Bits of Proof’s own implementation of the scripts was not practically relevant in my commercially successful deployments, because of the use of a border router, but it helped development, enabling easier debug and precise error feedback esp. end even after Core had a reject message. I integrated libconsensus only for the hope that is
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
One thing it occurs to me (and I don't know if this has been suggested before) we could do is separate the BIP process into at several distinct areas: 1) Commit structure changes/consensus rule change proposals - Consensus-building process (how are proposals debated, improved, vetted, and selected) - Update/deployment mechanisms for rule changes - Specific hard fork proposals - Specific soft fork proposals 2) Peer policies - Seeding and discovery mechanisms - Relay policies - p2p message support 3) RPC 4) Everything else On Sat, Aug 22, 2015, 6:28 PM Eric Lombrozo elombr...@gmail.com wrote: I've been pushing for greater modularization since I first got into bitcoin. I got quickly frustrated when I was only able to get through very few things (i.e. moving core structure serialization classes to a separate unit not called main). Working on Bitcoin has an added layer of frustration that goes beyond most open source projects: even though we're clearly in userland working at the application layer, a good layered protocol design is still lacking. We have no standards process separate from what basically amount to updates to one specific reference implementation. And we all need to agree on any major change, since a blockchain that is easily forked in contentious ways pretty much defeats its own purpose. I went off to develop my own stack, where I could more easily avoid politics and focus on engineering. But I now understand the politics are inevitable. Bitcoin is inherently a cooperative project. Several people have poured themselves passionately into the reference codebase, most of whom did it (at least initially) purely as unpaid volunteers. There's a lot of love that's gone into this. But it's become pretty clear that the modularization is no longer merely a matter of good engineering - it is essential to resolving serious political challenges. Perhaps the most frustrating thing of all is watching people pushing for relatively superficial yet highly controversial changes while we still lack the proper infrastructure to handle these kinds of divergences of opinion without either stagnating or becoming polarized. I could continue working to reimplement an entire stack from scratch, as several others have also done - but besides the serious effort duplication this entails, it doesn't really seem like it will ultimately be a convergent process. It's too easy to let ego and habit dictate one's preferences rather than rational engineering considerations. I know that some might feel I'm just preaching to the choir, but we should probably take a step back from implementation hackery and try to specify some core protocol layers, focusing on interfaces. Specifically, we need a consensus layer that doesn't try to specify networking, storage, wallets, UI, etc. Let different people improve upon these things independently in their own implementations. What matters is that we all converge on a common history and state. At the same time, let's open up more competition on all these other things that are separate from the consensus layer. If only we were to dedicate a fraction of the effort we've put into this whole block size circus into what's actually important...and I blame myself as well... On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org wrote: On Aug 21, 2015, at 21:46, Jorge Timón jti...@jtimon.cc wrote: On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer ta...@bitsofproof.com wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. But you don't want something better, you want something functionally identical. You may want to watch sipa's explanation on why the implementation is the specification and the reasons to separate libconsensus: https://youtu.be/l3O4nh79CUU?t=764 I do want something better, but not for the focus you have. Not because what you produce was not high quality, but because quality is achieved at a very high cost and is hard to uphold over generations of developer. You focus on a single use case while there are many out there for distributed ledgers. I think in an infrastructure for enterprise applications, building consensus on the ledger is a cornerstone there, but is only a piece of the solution. I built several commercially successful deployments where I delegated the consensus building to a border router, a Bitcoin Core, then interfaced that trusted peer with my implementation that accepted Core’s decisions in an SPV manner. One might think of this setup as wasteful and unsuitable for “small devices” therefore an example of centralization people here try to avoid. Enterprises have sufficient resources. Solving the business problem is valuable to them even at magnitudes higher cost than a hobbyist would bear. For mainstream
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
Unfortunately we have no way of rigorously proving functional equivalence other than code review and unit testing. The simpler the consensus code (and the more we can write it in a style that affords provability of correctness) the easier it will be in the future to compare implementations. Prior to swapping out implementations, we should at the least run it through the gauntlet and perhaps run both implementations side-by-side. All I/O should be treated abstractly in the API. In C++ I really like using a nearly bare-bones signal template for most async message handling, i.e. https://github.com/ciphrex/mSIGNA/blob/master/deps/Signals/src/Signals.h This greatly facilitates support for async bidirectional I/O, etc...with minimal overhead. But others might have other stylistic preferences. - Eric On Fri, Aug 21, 2015, 12:46 PM Jorge Timón bitcoin-dev@lists.linuxfoundation.org wrote: On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer ta...@bitsofproof.com wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. But you don't want something better, you want something functionally identical. You may want to watch sipa's explanation on why the implementation is the specification and the reasons to separate libconsensus: https://youtu.be/l3O4nh79CUU?t=764 On Aug 20, 2015, at 10:06, Jorge Timón jti...@jtimon.cc wrote: But the goal is not reimplementing the consensus rules but rather extract them from Bitcoin Core so that nobody needs to re-implement them again. My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins, but it is also imperative to be able to create and serve other block chains with other rules and for those I do not want to carry on the legacy of an antique tool set and a spaghetti style. Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus) and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were made because each piece appears best in what they do. Since you already depend on libconsensus for VerifyScript, wouldn't it be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock? You would still have complete control over storage, concurrency, networking, policy... My plan is for the C API to interface with the external storage by passing a function pointer to it. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer ta...@bitsofproof.com wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. But you don't want something better, you want something functionally identical. You may want to watch sipa's explanation on why the implementation is the specification and the reasons to separate libconsensus: https://youtu.be/l3O4nh79CUU?t=764 On Aug 20, 2015, at 10:06, Jorge Timón jti...@jtimon.cc wrote: But the goal is not reimplementing the consensus rules but rather extract them from Bitcoin Core so that nobody needs to re-implement them again. My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins, but it is also imperative to be able to create and serve other block chains with other rules and for those I do not want to carry on the legacy of an antique tool set and a spaghetti style. Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus) and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were made because each piece appears best in what they do. Since you already depend on libconsensus for VerifyScript, wouldn't it be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock? You would still have complete control over storage, concurrency, networking, policy... My plan is for the C API to interface with the external storage by passing a function pointer to it. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
Thinking in Bitcoins only on the level of technology unnecessarily narrows your view. OK, I hope to be pleasantly surprised. Tamas Blummer On Aug 20, 2015, at 23:35, Matt Corallo lf-li...@mattcorallo.com wrote: On 08/20/15 21:26, Tamas Blummer wrote: I know what you mean as I already have such a component with pluggable block store and networking. I'm not suggesting pluggable networking, I'm suggesting (and I think everyone thinks the design should be) NO networking. The API is ValidationResult libconsensus.HeyIFoundABlock(Block) and ListOfBlocksToDownloadNext libconsensus.HeyIFoundAHeaderList(ListOfHeaders). While you are at it you could aim for isolation of bitcoin specific decisions and algos from generic block chain code. Are you suggesting to support altcoins? I dont think anyone cares about supporting that. The magnitude of refactoring you would have to do to get there from main.cpp and the rest of the hairball is harder than a re-write from scratch, I think you'd be very pleasantly surprised. It sounds like you havent dug into Bitcoin Core validation code in years. and the result will not be impressive, just hopefully working. Hmm? The result would be an obviously correct consensus implementation that everyone could use, instead of everyone going off and writing their own and either being wrong, or never updating in the case of forks. Its a huge deal to allow people to focus on making their libraries have good APIs/Wallets/etc instead of focusing on making a working validation engine (though maybe for that the p2p layer needs to also be in a library). I think a slim API server was a lower hanging fruit in Core’s case. We have one, it just needs a few already obvious performance improvements. BTW, support for refactoring is an example where you see if your tool set is modern. There are a number of good development tools for C++ that allow this Tamas Blummer On Aug 20, 2015, at 19:44, Matt Corallo via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org mailto:bitcoin-dev@lists.linuxfoundation.org wrote: I dont think a libconsensus would have any kind of networking layer, nor is C++ an antique tool set (hopefully libconsensus can avoid a boost dependency, though thats not antique either). Ideally it would have a simple API to give it blocks and a simple API for it to inform you of what the current chain is. If you really want to get fancy maybe it has pluggable block storage, too, but I dont see why you couldnt use this in ~any client? On 08/20/15 08:35, Tamas Blummer via bitcoin-dev wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. On Aug 20, 2015, at 10:06, Jorge Timón jti...@jtimon.cc mailto:jti...@jtimon.cc wrote: But the goal is not reimplementing the consensus rules but rather extract them from Bitcoin Core so that nobody needs to re-implement them again. My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins, but it is also imperative to be able to create and serve other block chains with other rules and for those I do not want to carry on the legacy of an antique tool set and a spaghetti style. Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus) and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were made because each piece appears best in what they do. Tamas Blummer ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org mailto:bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org mailto:bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
I dont think a libconsensus would have any kind of networking layer, nor is C++ an antique tool set (hopefully libconsensus can avoid a boost dependency, though thats not antique either). Ideally it would have a simple API to give it blocks and a simple API for it to inform you of what the current chain is. If you really want to get fancy maybe it has pluggable block storage, too, but I dont see why you couldnt use this in ~any client? On 08/20/15 08:35, Tamas Blummer via bitcoin-dev wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. On Aug 20, 2015, at 10:06, Jorge Timón jti...@jtimon.cc wrote: But the goal is not reimplementing the consensus rules but rather extract them from Bitcoin Core so that nobody needs to re-implement them again. My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins, but it is also imperative to be able to create and serve other block chains with other rules and for those I do not want to carry on the legacy of an antique tool set and a spaghetti style. Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus) and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were made because each piece appears best in what they do. Tamas Blummer ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
I know what you mean as I already have such a component with pluggable block store and networking. While you are at it you could aim for isolation of bitcoin specific decisions and algos from generic block chain code. The magnitude of refactoring you would have to do to get there from main.cpp and the rest of the hairball is harder than a re-write from scratch, and the result will not be impressive, just hopefully working. I think a slim API server was a lower hanging fruit in Core’s case. BTW, support for refactoring is an example where you see if your tool set is modern. Tamas Blummer On Aug 20, 2015, at 19:44, Matt Corallo via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org wrote: I dont think a libconsensus would have any kind of networking layer, nor is C++ an antique tool set (hopefully libconsensus can avoid a boost dependency, though thats not antique either). Ideally it would have a simple API to give it blocks and a simple API for it to inform you of what the current chain is. If you really want to get fancy maybe it has pluggable block storage, too, but I dont see why you couldnt use this in ~any client? On 08/20/15 08:35, Tamas Blummer via bitcoin-dev wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. On Aug 20, 2015, at 10:06, Jorge Timón jti...@jtimon.cc wrote: But the goal is not reimplementing the consensus rules but rather extract them from Bitcoin Core so that nobody needs to re-implement them again. My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins, but it is also imperative to be able to create and serve other block chains with other rules and for those I do not want to carry on the legacy of an antique tool set and a spaghetti style. Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus) and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were made because each piece appears best in what they do. Tamas Blummer ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On 08/20/15 21:26, Tamas Blummer wrote: I know what you mean as I already have such a component with pluggable block store and networking. I'm not suggesting pluggable networking, I'm suggesting (and I think everyone thinks the design should be) NO networking. The API is ValidationResult libconsensus.HeyIFoundABlock(Block) and ListOfBlocksToDownloadNext libconsensus.HeyIFoundAHeaderList(ListOfHeaders). While you are at it you could aim for isolation of bitcoin specific decisions and algos from generic block chain code. Are you suggesting to support altcoins? I dont think anyone cares about supporting that. The magnitude of refactoring you would have to do to get there from main.cpp and the rest of the hairball is harder than a re-write from scratch, I think you'd be very pleasantly surprised. It sounds like you havent dug into Bitcoin Core validation code in years. and the result will not be impressive, just hopefully working. Hmm? The result would be an obviously correct consensus implementation that everyone could use, instead of everyone going off and writing their own and either being wrong, or never updating in the case of forks. Its a huge deal to allow people to focus on making their libraries have good APIs/Wallets/etc instead of focusing on making a working validation engine (though maybe for that the p2p layer needs to also be in a library). I think a slim API server was a lower hanging fruit in Core’s case. We have one, it just needs a few already obvious performance improvements. BTW, support for refactoring is an example where you see if your tool set is modern. There are a number of good development tools for C++ that allow this Tamas Blummer On Aug 20, 2015, at 19:44, Matt Corallo via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org mailto:bitcoin-dev@lists.linuxfoundation.org wrote: I dont think a libconsensus would have any kind of networking layer, nor is C++ an antique tool set (hopefully libconsensus can avoid a boost dependency, though thats not antique either). Ideally it would have a simple API to give it blocks and a simple API for it to inform you of what the current chain is. If you really want to get fancy maybe it has pluggable block storage, too, but I dont see why you couldnt use this in ~any client? On 08/20/15 08:35, Tamas Blummer via bitcoin-dev wrote: Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement, but also open the chance of doing the work better, in the sense of software engineering. On Aug 20, 2015, at 10:06, Jorge Timón jti...@jtimon.cc mailto:jti...@jtimon.cc wrote: But the goal is not reimplementing the consensus rules but rather extract them from Bitcoin Core so that nobody needs to re-implement them again. My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins, but it is also imperative to be able to create and serve other block chains with other rules and for those I do not want to carry on the legacy of an antique tool set and a spaghetti style. Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus) and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were made because each piece appears best in what they do. Tamas Blummer ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org mailto:bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org mailto:bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Thu, Jul 23, 2015 at 04:30:06PM +0200, Jorge Timón via bitcoin-dev wrote: I think there were some misunderstandings in our previous conversation about this topic. I completely agree with having a separated repository for libconsensus (that's the whole point, alternative implementations can be consensus-safe by using it, and in the event of a schism fork[1], they can fork just that smaller project without having to relay on Bitcoin Core [satoshi] at all). Indeed. But I thought you also wanted Bitcoin Core to use libconsensus instead of just having a subtree/subrepository like it currently does with libsecp256k1. I'm not sure if that would ever be accepted, but in any case we're certainly far away from that goal. Here are some things that need to happen first: I don't see any reason why Bitcoin Core would not use the consensus library. Eating our own dogfood and such. Biggest risk, as I've said before, is that the refactoring loading to a (more complete) consensus library will result in code that is no longer bug-for-bug compatible with previous versions, thus defeating its entire purpose and introducing fork risk. If that can be avoided - for example by going from here to there using pure code moves, as you're trying to do - I'm all for it. 2) Finish libconsensus's API: expose more things than VerifyScript, at the very least, also expose VerifyTx, VerifyHeader and VerifyBlock. Feedback from alternative implementations like libbitcoin is extremely valuable here. Some related closed-for-now PRs: Agreed. 3) Move libconsensus to a separate repository as a subtree/subrepository of Bitcoin Core. If the rest is done, this is the easy part :) Unfortunately and ironically again, safer, small and incremental changes are less interesting for reviewers. For example, I've been trying to move consensus code to the consensus folder for a long time. The correctness of a MOVEONLY change is trivial to review for anyone who knows how to copy/paste in its favorite editor and how to use git diff, but will I ever get answers to my questions in [1]? Code review capacity is still our greatest bottleneck. And I don't see any way out of that, unfortunately. I know there's many people who really care about this, Cory Fields, Wladimir and Pieter Wuille to name a few have reviewed many of this changes (I've just got used to publicly whine about lack of review on this front and policy encapsulation [very related fronts] as an attempt to get some attention: not always, but begging for review actually works some times). I do really care about this. Wladimir ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Mon, Jul 27, 2015 at 11:40:42PM -0700, Eric Voskuil via bitcoin-dev wrote: It's a performance sacrifice, and then there's the OpenSSL dependency, but these are both optional within our stack - so the application developer has the option. So the only downside is that we are maintaining the conditional compilation. Now that BIP66 became active, and only strict DER signatures are allowed, the OpenSSL dependency can be removed from consensus. Pieter Wuille will do an announcement on this soon. Wladimir ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Tue, Jul 28, 2015 at 10:43 AM, Wladimir J. van der Laan laa...@gmail.com wrote: On Thu, Jul 23, 2015 at 04:30:06PM +0200, Jorge Timón via bitcoin-dev wrote: But I thought you also wanted Bitcoin Core to use libconsensus instead of just having a subtree/subrepository like it currently does with libsecp256k1. I'm not sure if that would ever be accepted, but in any case we're certainly far away from that goal. Here are some things that need to happen first: I don't see any reason why Bitcoin Core would not use the consensus library. Eating our own dogfood and such. As explained to Eric, it's not that I don't want Bitcoin Core to use future-libconsensu through the API instead of a subtree: it's just that that's more long-term and more work. And I don't see why other implementations should really care about it. Biggest risk, as I've said before, is that the refactoring loading to a (more complete) consensus library will result in code that is no longer bug-for-bug compatible with previous versions, thus defeating its entire purpose and introducing fork risk. If that can be avoided - for example by going from here to there using pure code moves, as you're trying to do - I'm all for it. Well, pure movements will not be enough, parameters will have to change, incompatible dependencies have to be removed (ie util.h which contains globals), etc. But yes, I think we can do it with only low-risk and easy-to-review commits. 3) Move libconsensus to a separate repository as a subtree/subrepository of Bitcoin Core. If the rest is done, this is the easy part :) And still, this doesn't require Bitcoin Core to use the API, a subtree is enough at first. This easy step doesn't guarantee that Bitcoin Core is using future-libconsensus' API. Code review capacity is still our greatest bottleneck. And I don't see any way out of that, unfortunately. I really think these code separations help with this (ie there are many more people in the world with enough knowledge to review the qt or even policy parts than there's people able to review consensus changes). I do really care about this. I know and I said so. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
Ok, I'm going to separate terms: current-libconsensus from theoretical future-libconsensus (implementing ALL consensus rules). On Tue, Jul 28, 2015 at 8:40 AM, Eric Voskuil e...@voskuil.org wrote: libsecp256k1 has it's own repository, libbitcoinconsensus doesn't. A separate repository was what I considered as a requirement for us to use it. We want to complete future-libconsensus (decouple all the consensus rules from the rest of the bitcoin core code) first. Then we can move future-libconsensus to a subrepository/subtree like libsecp256k1 and I believe everybody wants this to eventually happen. Separating current-libconsensus now may make completing future-libconsensus harder. I'm not sure if that would ever be accepted, but in any case we're certainly far away from that goal. If it's not certain whether this would even be accepted, the commitment to a community consensus library is too weak to take a strong dependency on. But for us it's moot, as we have made the already accomplished that goal. What I mean is that once it is separated to a subtree, there's one more step: Make Bitcoin Core use future-libconsensus' API instead of a subtree. Decoupling future-libconsensus from Bitcoin Core is one thing, and Decoupling Bitcoin Core from future-libconsensus is another thing: you need to decouple Bitcoin Core from all future-libconsensus implementation internals. For example, script/sign (part of Bitcoin Core) depends on individual non-API-exposed classes in current-libconsensus. Moving from a subtree to a completely separated library is what I don't know will ever happen, but I don't think this is unfairly advantageous for Bitcoin Core or anything like that: other implementations can also use future-libcosensus as a subtree instead of the C API as well. You have accomplished the goal of separating curren-libconsensus, not future-libconsensus. In fact, if you complete the equivalent of future-libconsensus in libbitcoin and separate that, maybe that's a better place to start drafting a full API. 2) Finish libconsensus's API: expose more things than VerifyScript, at the very least, also expose VerifyTx, VerifyHeader and VerifyBlock. Feedback from alternative implementations like libbitcoin is extremely valuable here. Some related closed-for-now PRs: In our earlier discussion I believe you said that the library would not be undergoing significant change or feature creep. If this is the very least that's projected it would seem that constraint will not hold. future-libconsensus will not have significant changes *once it is completed*. Currently future-libconsensus is spread around many places inclusing src/main, so that obviously needs to change before it can be separated to an independent repo. 3) Move libconsensus to a separate repository as a subtree/subrepository of Bitcoin Core. Only after all that we can discuss whether Bitcoin Core itself should include libconsensus' code or just use its API directly. I don't think it's a question of whether it *should* use its own library as it is published for others - this is a practically self-evident conclusion. Well, Bitcoin Core is currently the only user of future-libconsensus since bitcoin core and future-libconsensus are currently mutually coupled. Bitcoin Core will always keep using future-libconsensus. The only question is whether it will use it through the C API or as a subtree/subrepository (both options are also available to other implementations). I don't know if decoupling Bitcoin Core from future-libconsensus' implementation details enough to be able to directly use the API is worth it or if anyone will be interested in doing so. For me this last step is not all that interesting: if we have an independent repo with a full API that other implementations can use, I don't really care about Bitcoin Core not going through the API and using including all the code directly instead. I hope that after all this, libbitcoin also reconsiders whether to reimplement its own libconsensus or use the official one directly instead. We use a fork of libsecp256k1 and would probably do the same with the consensus library if it was cleanly isolated. Great. In any case I agree with your stated need for this isolation (if not the means) for the reasons you state. The community needs to move beyond a largely singular and monolithic codebase that is holding that position in part due to fear about consensus bug forks. I completely agree. That's the goal of libconsensus (and an alternative implementation like libbitcoin being able to use it without sacrificing any of its current or future design differences from Bitcoin Core would be a sign of success in this reward). It's a performance sacrifice, and then there's the OpenSSL dependency, but these are both optional within our stack - so the application developer has the option. So the only downside is that we are maintaining the conditional compilation. As I told you
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On 07/23/2015 07:30 AM, Jorge Timón wrote: On Thu, Jul 23, 2015 at 2:49 AM, Eric Voskuil via bitcoin-dev wrote: On 07/22/2015 05:13 PM, Eric Lombrozo via bitcoin-dev wrote: Only being partly serious - I strongly am in favor of a sufficiently modularized codebase that swapping out consensus rules is fairly straightforward and easy to test... We (libbitcoin) have taken the time to publish and maintain bitcoind's libbitcoinconsensus source files as an independent C++ library... I think there were some misunderstandings in our previous conversation about this topic. I completely agree with having a separated repository for libconsensus (that's the whole point, alternative implementations can be consensus-safe by using it, and in the event of a schism fork[1], they can fork just that smaller project without having to relay on Bitcoin Core [satoshi] at all). But I thought you also wanted Bitcoin Core to use libconsensus instead of just having a subtree/subrepository like it currently does with libsecp256k1. libsecp256k1 has it's own repository, libbitcoinconsensus doesn't. A separate repository was what I considered as a requirement for us to use it. I'm not sure if that would ever be accepted, but in any case we're certainly far away from that goal. If it's not certain whether this would even be accepted, the commitment to a community consensus library is too weak to take a strong dependency on. But for us it's moot, as we have made the already accomplished that goal. Here are some things that need to happen first: 1) Finish encapsulating consensus code so that it can be built without any (we've done it only with script-related code so far). Here are some related PRs (other people have done other things that help with this as well): ... 2) Finish libconsensus's API: expose more things than VerifyScript, at the very least, also expose VerifyTx, VerifyHeader and VerifyBlock. Feedback from alternative implementations like libbitcoin is extremely valuable here. Some related closed-for-now PRs: In our earlier discussion I believe you said that the library would not be undergoing significant change or feature creep. If this is the very least that's projected it would seem that constraint will not hold. 3) Move libconsensus to a separate repository as a subtree/subrepository of Bitcoin Core. Only after all that we can discuss whether Bitcoin Core itself should include libconsensus' code or just use its API directly. I don't think it's a question of whether it *should* use its own library as it is published for others - this is a practically self-evident conclusion. I hope that after all this, libbitcoin also reconsiders whether to reimplement its own libconsensus or use the official one directly instead. We use a fork of libsecp256k1 and would probably do the same with the consensus library if it was cleanly isolated. In any case I agree with your stated need for this isolation (if not the means) for the reasons you state. The community needs to move beyond a largely singular and monolithic codebase that is holding that position in part due to fear about consensus bug forks. I completely agree. That's the goal of libconsensus (and an alternative implementation like libbitcoin being able to use it without sacrificing any of its current or future design differences from Bitcoin Core would be a sign of success in this reward). It's a performance sacrifice, and then there's the OpenSSL dependency, but these are both optional within our stack - so the application developer has the option. So the only downside is that we are maintaining the conditional compilation. Unfortunately any changes that touch consensus code are risky and therefore slow. And when consensus encapsulation changes conflict with other changes (not because the other changes need to change consensus but because consensus code is still coupled with policy and other bitcoind-specific code), refactors are never prioritized. Ironically, you need to encapsulate the consensus code to avoid such conflicts, which would make all non-consensus changes far less risky (reducing the consensus-critical review development bottleneck). Unfortunately and ironically again, safer, small and incremental changes are less interesting for reviewers. For example, I've been trying to move consensus code to the consensus folder for a long time. The correctness of a MOVEONLY change is trivial to review for anyone who knows how to copy/paste in its favorite editor and how to use git diff, but will I ever get answers to my questions in [1]? I think it's worthwhile work, especially if you are passionate about the longer term objectives. I haven't been involved in these reviews as I spend very little time with the satoshi client I know there's many people who really care about this, Cory Fields, Wladimir and Pieter Wuille to name a few have reviewed many of this changes (I've just got used to
Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
On Thu, Jul 23, 2015 at 4:57 PM, Milly Bitcoin via bitcoin-dev bitcoin-dev@lists.linuxfoundation.org wrote: On 7/23/2015 10:30 AM, Jorge Timón via bitcoin-dev wrote: [4] http://lmgtfy.com/?q=mike+hearn+dictatorl=1 Mike has sincerely said that he would like Bitcoin Core to have a benevolent dictator like other free software projects, and I wanted to make clear that I wasn't putting words in his mouth but it's actually something very easy to find on the internet. But I now realize that the search can be interpreted as me calling him dictator or something of the sort. That wasn't my intention. In fact, Mike's point of view on Bitcoin Core development wasn't even relevant for my example so I shouldn't even have mentioned him in the first place. I apologize for both mistakes, but please let's keep this thread focused on libconsensus. You spend too much time on reddit. I actually don't spend much time on reddit: I don't particularly like it. But I do spend some time in reddit so, I agree: I spend too much time on reddit. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
