Re: [bitcoin-dev] Long-term vision for bitcoind (was libconsensus and bitcoin development process)
I should also add that the mempool should exist in (2). This way the peer services layer can manage all relay policy and mempool management. -- Original Message -- From: "Eric Lombrozo" To: "Jorge Timón" ; "Wladimir J. van der Laan" Cc: "Bitcoin development mailing list" Sent: 9/22/2015 5:07:22 PM Subject: Re: [bitcoin-dev] Long-term vision for bitcoind (was libconsensus and bitcoin development process) Here's what I propose as a long-term plan: 1) libconsensus == We should probably start by defining an API for libconsensus. It should support an abstract DB model, track chain state, provide query mechanisms for blocks and transactions with optional pruning and indexing, expose a subscription mechanism for events such as NEW_TIP, REORG, etc, and contain a script interpreter. We can develop the library in parallel with Bitcoin Core without too much refactoring of Bitcoin Core itself...just moving pieces of Bitcoin Core's consensus code into the new library, tracking code movements to make merging easier. Yes, this is a bit ugly as it requires code duplication...but it will temporarily avoid much of the downstream pushback we're getting. The idea is that we can prove out the library with some simple projects, then start removing the consensus stuff from Bitcoin Core once we have greater acceptance of the library and better documentation. 2) peer services == We develop a peer services library that performs the tasks of peer discovery and relay, with the ability to connect to appropriate peers and queue messages. It uses libconsensus for all validation functionality and as a datastore for the consensus state but maintains its own database for peer history and statistics. I believe Cory has been working on this already using libevent. I've already developed an async library for this as well. 3) API/RPC === We provide high level calls and pub/sub mechanisms. ZMQ has been implemented and added already, but we could support other transports as well. 4) Wallet == The wallet is split out into a separate process that connects to the stack via the API/RPC layer. - Eric -- Original Message -- From: "Jorge Timón" To: "Wladimir J. van der Laan" Cc: "Bitcoin development mailing list" Sent: 9/22/2015 11:36:14 AM Subject: [bitcoin-dev] Long-term vision for bitcoind (was libconsensus and bitcoin development process) On Fri, Sep 18, 2015 at 2:07 AM, Wladimir J. van der Laan via bitcoin-dev wrote: My long-term vision of bitcoind is a P2P node with validation and blockchain store, with a couple of data sources that can be subscribed to or pulled from. I agree with this long term vision. Here's how I think it could happen: 1) Libconsensus is completed and moved to a subtree (which has libsecp as an internal subtree) 2) Bitcoind becomes a subtree of bitcoin-wallet (which has bitcoin-wallet and bitcoin-qt) Without aggressively changing it for this purpose, libconsensus should tend to become C, like libsecp, which is better for proving correctness. Hopefully at some point it won't take much to move to C. Upper layers should move to C++11 Don't focus on the git subtrees, the basic architecture is bitcoin-qt on top of bitcoin-wallet, bitcoin-wallet on top of bitcoind (and friends like bitcoin-cli and bitcoin-tx), bitcoind on top of libconsensus on top of libsecp256k1. I believe this would maximize the number of people who can safely contribute to the project. I also believe this is the architecture most contributors have in mind for the long term, but I may be wrong about it. Criticisms to this plan? ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Long-term vision for bitcoind (was libconsensus and bitcoin development process)
Here's what I propose as a long-term plan: 1) libconsensus == We should probably start by defining an API for libconsensus. It should support an abstract DB model, track chain state, provide query mechanisms for blocks and transactions with optional pruning and indexing, expose a subscription mechanism for events such as NEW_TIP, REORG, etc, and contain a script interpreter. We can develop the library in parallel with Bitcoin Core without too much refactoring of Bitcoin Core itself...just moving pieces of Bitcoin Core's consensus code into the new library, tracking code movements to make merging easier. Yes, this is a bit ugly as it requires code duplication...but it will temporarily avoid much of the downstream pushback we're getting. The idea is that we can prove out the library with some simple projects, then start removing the consensus stuff from Bitcoin Core once we have greater acceptance of the library and better documentation. 2) peer services == We develop a peer services library that performs the tasks of peer discovery and relay, with the ability to connect to appropriate peers and queue messages. It uses libconsensus for all validation functionality and as a datastore for the consensus state but maintains its own database for peer history and statistics. I believe Cory has been working on this already using libevent. I've already developed an async library for this as well. 3) API/RPC === We provide high level calls and pub/sub mechanisms. ZMQ has been implemented and added already, but we could support other transports as well. 4) Wallet == The wallet is split out into a separate process that connects to the stack via the API/RPC layer. - Eric -- Original Message -- From: "Jorge Timón" To: "Wladimir J. van der Laan" Cc: "Bitcoin development mailing list" Sent: 9/22/2015 11:36:14 AM Subject: [bitcoin-dev] Long-term vision for bitcoind (was libconsensus and bitcoin development process) On Fri, Sep 18, 2015 at 2:07 AM, Wladimir J. van der Laan via bitcoin-dev wrote: My long-term vision of bitcoind is a P2P node with validation and blockchain store, with a couple of data sources that can be subscribed to or pulled from. I agree with this long term vision. Here's how I think it could happen: 1) Libconsensus is completed and moved to a subtree (which has libsecp as an internal subtree) 2) Bitcoind becomes a subtree of bitcoin-wallet (which has bitcoin-wallet and bitcoin-qt) Without aggressively changing it for this purpose, libconsensus should tend to become C, like libsecp, which is better for proving correctness. Hopefully at some point it won't take much to move to C. Upper layers should move to C++11 Don't focus on the git subtrees, the basic architecture is bitcoin-qt on top of bitcoin-wallet, bitcoin-wallet on top of bitcoind (and friends like bitcoin-cli and bitcoin-tx), bitcoind on top of libconsensus on top of libsecp256k1. I believe this would maximize the number of people who can safely contribute to the project. I also believe this is the architecture most contributors have in mind for the long term, but I may be wrong about it. Criticisms to this plan? ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] libconsensus and bitcoin development process
If I'm reading this situation correctly, Jeff is basically pointing out that developers need more links (hooks, rungs, handholds, data points, whatever you want to call them) so that they can see all the things his email insinuated are missing (a plan, order, sense, etc.). He didn't say these things were missing, but that it kind of feels like it from the 10,000 foot view. If you use Google to search the list, as in <> you DO NOT get the page Jorge gave. He wrote that page, so he had a good idea what to search for to find it again. I just want to recommend that when you describe the work you're doing on bitcoin, imagine several different ways people might try to find this description in the future and make them work. In other words, Jorge could have put "A plan for abstracting out libconsensus" in the email where he wrote "Here are some things that need to happen first..." Likewise, if Jeff had searched for <> (maybe he did, but he didn't list any results), he may have found enough clues to see Jorge's overall plan. The "site:" keyword on Google fascinated me when I discovered it, so I let it inspire this email :-) Maybe someone can explain this if I have it wrong: A few people are able to pull code into Bitcoin/bitcoin. Isn't is possible that those few people can agree to merge in a lot of refactor-hell PRs for those making the requests, but postpone them to that one-week-per-month that someone suggested? The idea of letting that "hell" come in (predictable) waves is excellent and I was hoping to see some agreement. But I don't know who those few are, so even if they all wrote "Yeah, we'll do that," I wouldn't recognize that I got what I wanted. notplato On Tue, Sep 22, 2015 at 11:12 AM, Jorge Timón < bitcoin-dev@lists.linuxfoundation.org> wrote: > On Tue, Sep 15, 2015 at 6:10 AM, Jeff Garzik via bitcoin-dev > wrote: > > [collating a private mail and a github issue comment, moving it to a > > better forum] > > > > On libconsensus > > --- > > In general there exists the reasonable goal to move consensus state > > and code to a specific, separate lib. > > > > To someone not closely reviewing the seemingly endless stream of > > libconsensus refactoring PRs, the 10,000 foot view is that there is a > > rather random stream of refactors that proceed in fits and starts > > without apparent plan or end other than a one sentence "isolate > > consensus state and code" summary. > > > > I am hoping that > > * There is some plan > > * We will not see a five year stream of random consensus code movement > > patches causing lots of downstream developer headaches. > > > > I read every code change in every pull request that comes into > > github/bitcoin/bitcoin with three exceptions: > > * consensus code movement changes - too big, too chaotic, too > > frequent, too unfocused, laziness guarantees others will inevitably > > ACK it without me. > > * some non-code changes (docs) > > * ignore 80% of the Qt changes > > > > As with any sort of refactoring, they are easy to prove correct, easy > > to reason, and therefore quick and easy to ACK and merge. > > > > Refactors however have a very real negative impact. > > bitcoin/bitcoin.git is not only the source tree in the universe. > > Software engineers at home, at startups, and at major companies are > > maintaining branches of their own. > > > > It is very very easy to fall into a trap where a project is merging > > lots of cosmetic changes and not seeing the downstream ripple effects. > > Several people complained to me at the conference about all the code > > movement changes breaking their own work, causing them to stay on > > older versions of bitcoin due to the effort required to rebase to each > > new release version - and I share those complaints. > > > > Complex code changes with longer development cycles than simple code > > movement patches keep breaking. It is very frustrating, and causes > > folks to get trapped between a rock and a hard place: > > - Trying to push non-trivial changes upstream is difficult, for normal > > and reasonable reasons (big important changes need review etc.). > > - Maintaining non-trivial changes out of tree is also painful, for the > > aforementioned reasons. > > > > Reasonable work languishes in constant-rebase hell, and incentivizes > > against keeping up with the latest tree. > > > > > > Aside from the refactor, libconsensus appears to be engineering in the > > dark. Where is any sort of plan? I have low standards - a photo of a > > whiteboard or youtube clip will do. > > Just because you don't understand the changes proposed it doesn't mean > that they are random. > I may have done a poor job in communicating "my plan for libconsensus" > but I have tried many times and in many ways. > #bitcoin-dev logs show that I have not worked "in the dark" at all, on > the contrary, I've been very tenacious when asking for review and > opinions, to the point that several people (at least @laanwj and > @theuni have co
[bitcoin-dev] Long-term vision for bitcoind (was libconsensus and bitcoin development process)
On Fri, Sep 18, 2015 at 2:07 AM, Wladimir J. van der Laan via bitcoin-dev wrote: > My long-term vision of bitcoind is a P2P node with validation and blockchain > store, with a couple of data sources that can be subscribed to or pulled from. I agree with this long term vision. Here's how I think it could happen: 1) Libconsensus is completed and moved to a subtree (which has libsecp as an internal subtree) 2) Bitcoind becomes a subtree of bitcoin-wallet (which has bitcoin-wallet and bitcoin-qt) Without aggressively changing it for this purpose, libconsensus should tend to become C, like libsecp, which is better for proving correctness. Hopefully at some point it won't take much to move to C. Upper layers should move to C++11 Don't focus on the git subtrees, the basic architecture is bitcoin-qt on top of bitcoin-wallet, bitcoin-wallet on top of bitcoind (and friends like bitcoin-cli and bitcoin-tx), bitcoind on top of libconsensus on top of libsecp256k1. I believe this would maximize the number of people who can safely contribute to the project. I also believe this is the architecture most contributors have in mind for the long term, but I may be wrong about it. Criticisms to this plan? ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] libconsensus and bitcoin development process
On Tue, Sep 15, 2015 at 6:10 AM, Jeff Garzik via bitcoin-dev wrote: > [collating a private mail and a github issue comment, moving it to a > better forum] > > On libconsensus > --- > In general there exists the reasonable goal to move consensus state > and code to a specific, separate lib. > > To someone not closely reviewing the seemingly endless stream of > libconsensus refactoring PRs, the 10,000 foot view is that there is a > rather random stream of refactors that proceed in fits and starts > without apparent plan or end other than a one sentence "isolate > consensus state and code" summary. > > I am hoping that > * There is some plan > * We will not see a five year stream of random consensus code movement > patches causing lots of downstream developer headaches. > > I read every code change in every pull request that comes into > github/bitcoin/bitcoin with three exceptions: > * consensus code movement changes - too big, too chaotic, too > frequent, too unfocused, laziness guarantees others will inevitably > ACK it without me. > * some non-code changes (docs) > * ignore 80% of the Qt changes > > As with any sort of refactoring, they are easy to prove correct, easy > to reason, and therefore quick and easy to ACK and merge. > > Refactors however have a very real negative impact. > bitcoin/bitcoin.git is not only the source tree in the universe. > Software engineers at home, at startups, and at major companies are > maintaining branches of their own. > > It is very very easy to fall into a trap where a project is merging > lots of cosmetic changes and not seeing the downstream ripple effects. > Several people complained to me at the conference about all the code > movement changes breaking their own work, causing them to stay on > older versions of bitcoin due to the effort required to rebase to each > new release version - and I share those complaints. > > Complex code changes with longer development cycles than simple code > movement patches keep breaking. It is very frustrating, and causes > folks to get trapped between a rock and a hard place: > - Trying to push non-trivial changes upstream is difficult, for normal > and reasonable reasons (big important changes need review etc.). > - Maintaining non-trivial changes out of tree is also painful, for the > aforementioned reasons. > > Reasonable work languishes in constant-rebase hell, and incentivizes > against keeping up with the latest tree. > > > Aside from the refactor, libconsensus appears to be engineering in the > dark. Where is any sort of plan? I have low standards - a photo of a > whiteboard or youtube clip will do. Just because you don't understand the changes proposed it doesn't mean that they are random. I may have done a poor job in communicating "my plan for libconsensus" but I have tried many times and in many ways. #bitcoin-dev logs show that I have not worked "in the dark" at all, on the contrary, I've been very tenacious when asking for review and opinions, to the point that several people (at least @laanwj and @theuni have complained about their github inboxes being full of "spam"). This is a relatively recent thread where I describe my plan: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009568.html Not my first attempt on this list. It is very frustrating that everybody seems to agree that separating libconsensus is a priority to maximize the number of people that can safely contribute to the project, but at the same time, nobody thinks that reviewing the necessary refactors to do so is a priority. I tried creating big PRs for people to see "the big picture" #5946 but those were too many commits and nobody wanted to read it. Gavin asked for an API. So I tried a smaller step: exposing just VerifyHeader in libconsensus and leave VerifyTx and VerifyBlock for later #5995 Again, this was "too big" and "a moving target". In the meantime I always had smaller one-little-step PRs that were part of a longer branch: ** [8/8] MERGED Consensus - [X] Consensus: Decouple pow from chainparams #5812 [consensuspow] - [X] MOVEONLY: Move constants and globals to consensus.h #5696 [consensus_policy0] - [X] Chainparams: Refactor: Decouple IsSuperMajority from Params() #5968 [params_consensus] - [X] Remove redundant getter CChainParams::SubsidyHalvingInterval() #5996 [params_subsidy] - [X] Separate CValidationState from main #5669 [consensus] - [X] Consensus: Decouple ContextualCheckBlockHeader from checkpoints #5975 [consensus_checkpoints] - [X] Separate Consensus::CheckTxInputs and GetSpendHeight in CheckInputs #6061 [consensus_inputs] - [X] Bugfix: Don't check the genesis block header before accepting it #6299 [5975-quick-fix] ** [5/5] DELETED *** DELETED Refactor: Create CCoinsViewEfficient interface for CCoinsViewCache #5747 [coins] *** DELETED Chainparams: Explicit Consensus::Params arg in consensus functions #6024 [params_consensus2] *** DELETED MOVEONLY: Move most of consensus functions (pre-block) #6051 [co
Re: [bitcoin-dev] Fill-or-kill transaction
On Fri, Sep 18, 2015 at 12:44 AM, Peter Todd wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > > > On 17 September 2015 12:14:38 GMT-07:00, "Jorge Timón via bitcoin-dev" > wrote: >>Fill or kill us normally used for trades and I think it can be >>confusing. >>Previous times this has been discussed it has been discussed under >>nExpiryTime or op_height (which enables expiration), for example, in >>the >>freimarkets white paper. >> >>As Mark points out this can be made safe by requiring that all the >>outputs >>of a transaction that can expire have op_maturity/csv/rcltv of 100. >>That >>makes them as reorg-safe as coinbase transactions. Unfortunately this >>doesn't play very well with p2sh... > > Why wouldn't that work with p2sh? It can be implemented by a "treat like > Coinbase" flag in the UTXO set, set when the output is created. That would work. I was thinking about requiring OP_MATURITY 100 (or greater than 100) in all the scriptPubKey's of the expiry transaction. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[bitcoin-dev] Weekly development meetings on IRC: schedule
Hello, There was overwhelming response that weekly IRC meetings are a good thing. Thanks to the doodle site we were able to select a time slot that everyone (that voted) is available: Thursday 19:00-20:00 UTC, every week, starting September 24 (next Thursday) I created a shared Google Calendar here: https://www.google.com/calendar/embed?src=MTFwcXZkZ3BkOTlubGliZjliYTg2MXZ1OHNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ The timezone of this calendar is Reykyavik (Iceland) which is UTC+0. However, you can use the button on the lower right to add the calendar to your own calendar, which will then show the meeting in your own timezone. See you then, Wladimir ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Weekly development meetings on IRC
On Mon, Sep 21, 2015 at 03:51:29PM +0200, gb wrote: > > Although the planning for this a bit far along now, one consideration I > might add from experience on working with other transglobal IT projects Nah, we can always change the scheduling later... But let's first try it out with one time. W ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
