Re: [Bitcoin-development] Recent EvalScript() changes mean CHECKLOCKTIMEVERIFY can't be merged
On Mon, Dec 15, 2014 at 10:20 AM, Jeff Garzik jgar...@bitpay.com wrote: On Mon, Dec 15, 2014 at 9:57 AM, Btc Drak btcd...@gmail.com wrote: We all want to see more modular code, but the first steps should just be to relocate blocks of code so everything is more logically organised in smaller files (especially for consensus critical code). Refactoring should come in a second wave preferably after a stable release. This is my opinion as well. In the Linux kernel, we often were faced with a situation where you have a One Big File driver with 1MB of source code. The first step was -always- raw code movement, a brain-dead breaking up of code into logical source code files. Refactoring of data structures comes after that. While not always money-critical, these drivers Had To Keep Working. We had several situations where we had active users, but zero hardware access for debugging, and zero access to the vendor knowledge (hardware documentation, engineers). Failure was not an option. ;p Performing the dumb Break Up Files step first means that future, more invasive data structures are easier to review, logically segregated, and not obscured by further code movement changes down the line. In code such as Bitcoin Core, it is important to think about the _patch stream_ and how to optimize for reviewer bandwidth. The current stream of refactoring is really a turn-off in terms of reviewing, sapping reviewer bandwidth by IMO being reviewer-unfriendly. It is a seemingly never-ending series of tiny refactor-and-then-stuff-in-a-class-and-make-it-pretty-and-do-all-the-work. Some change is in order, gentlemen. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ That's exactly what happened during the modularization process, with the exception that the code movement and refactors happened in parallel rather than in series. But they _were_ done in separate logical chunks for the sake of easier review. The commit tag MOVEONLY developed organically out of this process, and a grep of the 0.10 branch for MOVEONLY is a testament to exactly how much code moved 1:1 out of huge files and into logically separated places and/or new files. Perhaps it's worth making MOVEONLY (which as the name implies, means that code has been copied 1:1 to a new location) use an official dev guideline for use in future refactors. Cory -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Recent EvalScript() changes mean CHECKLOCKTIMEVERIFY can't be merged
On Mon, Dec 15, 2014 at 7:47 AM, Peter Todd p...@petertodd.org wrote: BtcDrak was working on rebasing my CHECKLOCKTIMEVERIFY¹ patch to master a few days ago and found a fairly large design change that makes merging it currently impossible. Pull-req #4890², specifically commit c7829ea7, changed the EvalScript() function to take an abstract SignatureChecker object, removing the txTo and nIn arguments that used to contain the transaction the script was in and the txin # respectively. CHECKLOCKTIMEVERIFY needs txTo to obtain the nLockTime field of the transaction, and it needs nIn to obtain the nSequence of the txin. We need to fix this if CHECKLOCKTIMEVERIFY is to be merged. Secondly, that this change was made, and the manner in which is was made, is I think indicative of a development process that has been taking significant risks with regard to refactoring the consensus critical codebase. I know I personally have had a hard time keeping up with the very large volume of code being moved and changed for the v0.10 release, and I know BtcDrak - who is keeping Viacoin up to date with v0.10 - has also had a hard time giving the changes reasonable review. The #4890 pull-req in question had no ACKs at all, and only two untested utACKS, which I find worrying for something that made significant consensus critical code changes. While it would be nice to have a library encapsulating the consensus code, this shouldn't come at the cost of safety, especially when the actual users of that library or their needs is still uncertain. This is after all a multi-billion project where a simple fork will cost miners alone tens of thousands of dollars an hour; easily much more if it results in users being defrauded. That's also not taking into account the significant negative PR impact and loss of trust. I personally would recommend *not* upgrading to v0.10 due to these issues. A much safer approach would be to keep the code changes required for a consensus library to only simple movements of code for this release, accept that the interface to that library won't be ideal, and wait until we have feedback from multiple opensource projects with publicly evaluatable code on where to go next with the API. 1) https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki 2) https://github.com/bitcoin/bitcoin/pull/4890 -- 'peter'[:-1]@petertodd.org 1b18a596ecadd07c0e49620fb71b16f9e41131df9fc52fa6 It would appear as though you're trying to drum up controversy here, but the argument is quite a stretch, and contrary to some other arguments you're making in parallel. There seem to be three themes in your above complaint, so I'd like to address them individually. 1. Pr #4890 specifically. The argument seems to be that this was not properly reviewed/tested, and that it is an unnecessary risk to the consensus codebase. Looking at the PR at github, while I certainly don't agree with those conclusions, I suppose I can understand where they're coming from. There's plenty of context missing, as well as sidebar discussions on IRC and other PRs. To an outside observer, these changes may look under-tested and unnecessary. The context that's missing is the flurry of work that was going on in parallel to modularize this (and surrounding code). #4890 was one of the first pieces necessary for that, so some of the discussion about it was happening in dependent pull requests. You can point to a lack ACKs in one place for that PR, but that doesn't mean that the changes weren't tested/reviewed/necessary. You could also argue that ACKs should've been mirrored on the PR in question for posterity, which would be a perfectly reasonable argument that I would agree with. 2. These changes conflict with a rebased version of your CHECKLOCKTIMEVERIFY changes. OK? You have a tree that's a few months old, and you find that you have conflicts when rebasing to master. It happens to all of us. Do as the rest of us do and update your changes to fit. If you missed the review of #4890 and think it should be reverted, then call for a revert. But please give a concrete reason other than I've picked this commit series for a crusade because it gave me merge conflicts. What is the conspiracy here? There's a signature cache that is implementation-specific, and in a parallel universe, you might be arguing that we should rip it out because it adds unnecessary complexity to the consensus code. The PR provides a path around that complexity. For some reason, your reaction is to cry foul months later because you missed reviewing it at the time, rather than cheering for the reduced complexity. 3. You seem to think that 1. and 2. seem to point to a systemic failure of the review process because modularization shouldn't come at the cost of safety. I agree that it shouldn't come at the cost of safety, but I see no failure here. There has been a HUGE effort to modularize the code with a combination of
Re: [Bitcoin-development] Recent EvalScript() changes mean CHECKLOCKTIMEVERIFY can't be merged
On Mon, Dec 15, 2014 at 2:35 PM, Jeff Garzik jgar...@bitpay.com wrote: On Mon, Dec 15, 2014 at 1:42 PM, Cory Fields li...@coryfields.com wrote: That's exactly what happened during the modularization process, with the exception that the code movement and refactors happened in parallel rather than in series. But they _were_ done in separate logical chunks for the sake of easier review. That's exactly what was done except it wasn't Yes, in micro, at the pull request level, this happened * Code movement * Refactor At a macro level, that cycle was repeated many times, leading to the opposite end result: a lot of tiny movement/refactor/movement/refactor producing the review and patch annoyances described. It produces a blizzard of new files and new data structures, breaking a bunch of out-of-tree patches, complicating review quite a bit. If the vast majority of code movement is up front, followed by algebraic simplifications, followed by data structure work, further patches are easy to review/apply with less impact on unrelated code. I won't argue that at all because it's perfectly logical, but in practice that doesn't translate from the macro level to the micro level very well. At the micro level, minor code changes are almost always needed to accommodate useful code movement. Even if they're not required, it's often hard to justify code movement for the sake of code movement with the promise that it will be useful later. Rather than arguing hypotheticals, let's use a real example: https://github.com/bitcoin/bitcoin/pull/5118 . That one's pretty simple. The point of the PR was to unchain our openssl wrapper so that key operations could be performed by the consensus lib without dragging in bitcoind's structures. The first commit severs the dependencies. The second commit does the code movement from the now-freed wrapper. I'm having a hard time coming up with a workflow that would handle these two changes as _separate_ events, while making review easier. Note that I'm not attempting to argue with you here, rather I'm genuinely curious as to how you'd rather see this specific example (which is representative of most of my other code movement for the libbitcoinconsensus work, i believe) handled. Using your model above, I suppose we'd do the code movement first with the dependencies still intact as a pull request. At some later date, we'd sever the dependencies in the new files. I suppose you'd also prefer that I group a bunch of code-movement changes together into a single PR which needs little scrutiny, only verification that it's move-only. Once the code-movement PRs are merged, I can begin the cleanups which actually fix something. In practice, though, that'd be a massive headache for different reasons. Lots in flux with seemingly no benefits until some later date. My PR's can't depend on eachother because they don't actually fix issues in a linear fashion. That means that other devs can't depend on my PRs either for the same reason. And what have we gained? Do you find that assessment unreasonable? Cory -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP process
Sounds like this is what you're after, it's a fairly new feature: https://github.com/blog/1375%0A-task-lists-in-gfm-issues-pulls-comments I've been meaning to use it in a PR to try it out. Cory On Wed, Oct 15, 2014 at 5:36 AM, Wladimir laa...@gmail.com wrote: This all makes a lot of sense to me, and would help a lot with the workflow. Unfortunately github pulls and issues really have nothing to faciltate a multistage workflow... e.g. where something can go through several steps. Indeed, pull requests don't have a status. It would be possible to (ab)use labels for this. The drawback of labels is that only the repository team can set these, there is no way to delegate. But I suppose it'd be possible to build something on top of the github API that handles this. We're also having problems with people failing to comment on things, not even I looked at this and have no opinion, which is really obstructing things. Well - the only way to avoid that is to set a reasonable deadline, after which there is a default decision. You'd hope this would motivate people to get involved in time. Wladimir -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
