[Bitcoin-development] How to create a pull tester JAR
I just checked in a change to bitcoinj git master that makes it much easier to create a pull tester jar. Here are instructions for how to do it. You will need: - A Java Development Kit (JDK), version 6 or up should work. As Java 6 was released eight years ago, this should not be a challenging requirement. If you have a Mac just running java from the command line should give you a GUI prompt to install it automatically. Otherwise apt-get or fetch the latest from the interwebs. - Apache Maven. This is a rough equivalent of autotools, except it does dependency resolution for you. Grab it from http://maven.apache.org/download.cgi then unzip it and make sure the bin directory is in your PATH. You may need to set the JAVA_HOME environment variable if you installed Java to an odd place. - git Make sure you can run javac from the command line, then make sure you can run mvn, it should complain it can't find a POM (this is a build config file) and not, say, that it can't find Java. Now grab bitcoinj from git master: git clone https://github.com/bitcoinj/bitcoinj.git ... and build cd bitcoinj mvn -DskipTests package It will go off and download the libraries needed, compile, and create a bundled executable JAR called core/target/pull-tests.jar. This is sort of analogous to static linking in the Java world. It should be fast - expect a full build plus downloads to take less than a minute. You can use it either with the QA scripts in the bitcoin core qa/pull-tester directory or just run things directly: ./bitcoind -regtest -connect=0.0.0.0 -listen -whitelist=127.0.0.1 -datadir=/tmp/pulltester java -jar core/target/pull-tests.jar It should go ahead and print lots of debug spew, then at the end say it's happy. Let me know if you encounter any problems with this. Java JARs (which are just zip files renamed) are easily reproduced if you use the same version of javac and the same bitcoinj version. The ZIP container has timestamps, but unzipping them and simply diffing the files between two builds should reveal no differences. I am happy to provide a pull-tests.jar from my local machine if anyone would like to do this. -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] How to create a pull tester JAR
Oh, I forgot to mention something important. Ridiculously, the default package repository Maven uses was not protected by SSL up until a few days ago. They made it available via SSL now, but you have to tell Maven about the new URL. I guess they'll do a new release where SSL is the default soon. But for now before you run mvn save the following magic incantation to the path ~/.m2/settings.xml: (side note: yes maven's love of XML is widely ridiculed and more modern build tools have much better config languages, but we didn't upgrade yet) settings activeProfiles !--make the profile active all the time -- activeProfilesecurecentral/activeProfile /activeProfiles profiles profile idsecurecentral/id !--Override the repository (and pluginRepository) central from the Maven Super POM -- repositories repository idcentral/id urlhttps://repo1.maven.org/maven2/url releases enabledtrue/enabled /releases /repository /repositories pluginRepositories pluginRepository idcentral/id urlhttps://repo1.maven.org/maven2/url releases enabledtrue/enabled /releases /pluginRepository /pluginRepositories /profile /profiles /settings -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] How to create a pull tester JAR
Thanks for posting that (and implicitly archiving the knowledge). Anything that makes test improvement easier is welcomed. On Tue, Aug 5, 2014 at 11:00 AM, Mike Hearn m...@plan99.net wrote: I just checked in a change to bitcoinj git master that makes it much easier to create a pull tester jar. Here are instructions for how to do it. You will need: - A Java Development Kit (JDK), version 6 or up should work. As Java 6 was released eight years ago, this should not be a challenging requirement. If you have a Mac just running java from the command line should give you a GUI prompt to install it automatically. Otherwise apt-get or fetch the latest from the interwebs. - Apache Maven. This is a rough equivalent of autotools, except it does dependency resolution for you. Grab it from http://maven.apache.org/download.cgi then unzip it and make sure the bin directory is in your PATH. You may need to set the JAVA_HOME environment variable if you installed Java to an odd place. - git Make sure you can run javac from the command line, then make sure you can run mvn, it should complain it can't find a POM (this is a build config file) and not, say, that it can't find Java. Now grab bitcoinj from git master: git clone https://github.com/bitcoinj/bitcoinj.git ... and build cd bitcoinj mvn -DskipTests package It will go off and download the libraries needed, compile, and create a bundled executable JAR called core/target/pull-tests.jar. This is sort of analogous to static linking in the Java world. It should be fast - expect a full build plus downloads to take less than a minute. You can use it either with the QA scripts in the bitcoin core qa/pull-tester directory or just run things directly: ./bitcoind -regtest -connect=0.0.0.0 -listen -whitelist=127.0.0.1 -datadir=/tmp/pulltester java -jar core/target/pull-tests.jar It should go ahead and print lots of debug spew, then at the end say it's happy. Let me know if you encounter any problems with this. Java JARs (which are just zip files renamed) are easily reproduced if you use the same version of javac and the same bitcoinj version. The ZIP container has timestamps, but unzipping them and simply diffing the files between two builds should reveal no differences. I am happy to provide a pull-tests.jar from my local machine if anyone would like to do this. -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] How to create a pull tester JAR
On 08/05/2014 05:11 PM, Mike Hearn wrote: Oh, I forgot to mention something important. Ridiculously, the default package repository Maven uses was not protected by SSL up until a few days ago. They made it available via SSL now, but you have to tell Maven about the new URL. I guess they'll do a new release where SSL is the default soon. FWIW, I filed a wishlist item here: https://jira.codehaus.org/browse/MNG-5672 and here, for the old Ubuntu versions of Maven: https://bugs.launchpad.net/ubuntu/+source/maven/+bug/1352418 -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] How to create a pull tester JAR
No problem. The pull tester entry point can be found here: https://github.com/bitcoinj/bitcoinj/blob/master/core/src/test/java/com/google/bitcoin/core/BitcoindComparisonTool.java (nb: in the near future I will be re-namespacing the library from com.google.bitcoin to org.bitcoinj to reflect that it no longer has anything to do with Google and then this link will break). The code itself is a rather bad example of copy/paste coding and I can say that, because Matt knows it and already plans to refactor things ;) So if anyone is thinking of adding tests to the framework coordinate with him first to ensure you don't end up conflicting with a big refactor/rewrite. -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] deterministic transaction expiration
It would make more sense to introduce a new script opcode that pushes the current block height onto the operand stack. Then you could implement arbitrary logic about which blocks the transaction can be valid in. This would require that the client revalidate all transactions in its mempool (really, only those making use of this opcode) whenever the chain tip changes. I have to say I like this idea, this would allow someone to prove they can't spend funds before a given date, and vice versa, prove that the funds can't ever be spent after a given date (and this is provably prunable, isn't it?). Of course, there are some risks associated with that, but nobody is forced to use it. flooding the network with unrelated high fee transactions in order to push a transaction out to where it can never be mined at all. This becomes increasingly expensive as the deadline is further away, so not very hard to mitigate. On Sat, Aug 2, 2014 at 1:36 AM, Tom Harding t...@thinlink.com wrote: On 7/31/2014 5:58 PM, Kaz Wesley wrote: 1. start setting nLockTime to the current height by default in newly created transactions (or slightly below the current height, for reorg-friendliness) Reorg-frendliness is the opposite of the rationale behind #2340, which proposes setting nLockTime at current-height + 1 to prevent fee-sniping reorgs... 2. once users have had some time to upgrade to clients that set nLockTime, start discouraging transactions without nLockTime -- possibly with a slightly higher fee required for relay 3. start rate-limiting relay of transactions without an nLockTime (maybe this alone could be used to achieve [2]) 4. add a new IsStandard rule rejecting transactions with an nLockTime more than N blocks behind the current tip (for some fixed value N, to be determined) One way to proceed is implement #3753 (mempool janitor) in such a way that transactions with nLockTime are allowed to live a bit longer in the mempool (say 500 blocks) than those without (72 hours). In other words, as a first step, just actually start expiring things from the mempool in bitcoin core, and leave any relay fee adjustments or rate limiting for later. The isStandard change would be a good complement to #3753, to avoid relaying a tx that will soon expire by the nLockTime rule anyway. -- Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] deterministic transaction expiration
Glad this was brought up. Transaction expiration is something that I have wanted to see happen in bitcoin for a long, long time. The user experience of unconfirming transactions setting around in limbo is just horrible. Bitcoin software by necessity has gotten better about attaching fees so this sort of behavior is uncommon, but that does not eliminate the problem. Of course, we cannot presume that a transaction will truly disappear -- The Internet Never Forgets -- but given a bit of mempool adjusting, we can achieve the next best thing: the majority of the network forgets the transaction and becomes willing to relay a respend of some or all of the inputs. This uses existing client logic where the client must rebroadcast a transaction until it is confirmed. In general, if a transaction has not made it into a block within 144*X blocks, there is _some_ reason it is getting rejected by the miners. The mempool janitor is a garbage collector design. This is inferior to the superblock model described at https://github.com/bitcoin/bitcoin/issues/3723 Other models can also achieve similar results. There are a lot of issues tied together here: transaction expiration, the desire to cap the mempool ram usage, scalability, DoS prevention, ... mempool ties a lot together. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] deterministic transaction expiration
A distinction there is that they can only become invalid via a conflict— replaced by another transaction authored by the prior signers. If no other transaction could be created (e.g. you're a multisigner and won't sign it again) then there is no such risk. You need to check transaction's dependencies up to a certain depth to know whether it is safe: If one of inputs depends on transaction which is signed by parties with unknown trustworthiness, then it isn't safe. It now introduces chance events (act of god) into the mix where they they didn't exist before. You need to check transaction's dependencies up to a certain depth to know whether it is safe: If one of inputs depends on transaction time-locked script (or other unrecognized script), then it isn't safe. Situation is identical, you might need several extra lines of code. I think it would matter only if we had deterministic, reliable mempool and reorganization behavior. But it's not something we can depend on. -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] deterministic transaction expiration
The user experience of unconfirming transactions setting around in limbo is just horrible. Bitcoin software by necessity has gotten better about attaching fees so this sort of behavior is uncommon, but that does not eliminate the problem. Yes, indeed. I suspect there's a quick hack that could make this problem a lot better though. I think I brought up this idea before, but can't quite remember. Anyway I'm willing to bet that if we analysed the data some more, we'd discover that most legitimate i.e. non-DoS unconfirmed transactions that sit around for ages are linked back to the block chain within two hops and not more. That is people send a transaction that uses up their coin age, and then immediately those coins are immediately respent again, but then those final new coins are not spent. On the other hand DoS attacks look like bouncing your coins around over and over forever, i.e. more than two or three hops back to the chain. So I wonder if making priority look back two or three transactions but not more would help real users a lot, whilst not opening up any significant new potential for DoS. -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] deterministic transaction expiration
I feel like a lot of this will be driven by implementation, and costs of changing the implementation. Additional look-backs are of course doable, but they incur some disk I/O costs. The fields available in memory for each mempool TX are uint256 tx_hash; // hash of next field CTransaction tx; int64_t nFee; // Cached to avoid expensive parent-transaction lookups size_t nTxSize; // ... and avoid recomputing tx size int64_t nTime; // Local time when entering the mempool double dPriority; // Priority when entering the mempool unsigned int nHeight; // Chain height when entering the mempool As a first pass, we may prune the mempool without additional db lookups quite easily based on time criteria. Or, additional in-memory indexes may be constructed to maintain hashes ordered by priority/fees. Those techniques seem likely to be attempted before resorting to looking back two or three TXs deep at coin age -- which I admit is an interesting metric. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] deterministic transaction expiration
In general, if a transaction has not made it into a block within 144*X blocks, there is _some_ reason it is getting rejected by the miners. This is the crux of my concern: relay policy and miner priorities will not necessarily always be in sync, and node resource management shouldn't rely on them being compatible. There are other solutions than transaction expiration; I think Gavin's idea from the block-squashing thread, in which miners explicitly provide information about their policies, would go a long way to address this. But even when mechanisms for reconciling nodes' expectations about miners' behavior with miners' actual behavior are available, it may be desirable to keep an expiry mechanism in place in case of glitches between node understanding of policy and actual miner policy. Any approach based on beginning a transaction expiry countdown when a transaction is received (as in mempool janitor) seems unviable to me: once a node has forgotten a transaction, it must be susceptible to reaccepting it; all the functionality of such an expiry mechanism depends on the network not containing any nodes with slightly different relay behavior from bitcoind. I could accidentally debilitate mempool janitors across the entire network if I set up two nodes to exchange mempools whenever they reconnected to each other, and restarted each frequently. That's why I think including information in the transaction itself, as with my nLockTime/IsStandard proposal, is necessary for transactions to reliably eventually die off from mempools. There's a modification I've been thinking about: allow a transaction's lifetime to be refreshed (even after expiry) by a child transaction, along the lines of child-pays-for-parent fee policy. This would eliminate the need to reuse a key to make a replacement for an expired transaction (when submitting the tx directly to a miner is not an option), as well as alleviating the potential inconvenience in cases like Peter brought up, where nLockTime is used for exchanged locked transactions as part of a multi-transaction contract. With child-refreshes-parent, a transaction's receivers and senders would have the ability to keep trying to get their payment confirmed, but anyone on the network can't just keep all transactions alive. On Tue, Aug 5, 2014 at 10:48 AM, Jeff Garzik jgar...@bitpay.com wrote: Glad this was brought up. Transaction expiration is something that I have wanted to see happen in bitcoin for a long, long time. The user experience of unconfirming transactions setting around in limbo is just horrible. Bitcoin software by necessity has gotten better about attaching fees so this sort of behavior is uncommon, but that does not eliminate the problem. Of course, we cannot presume that a transaction will truly disappear -- The Internet Never Forgets -- but given a bit of mempool adjusting, we can achieve the next best thing: the majority of the network forgets the transaction and becomes willing to relay a respend of some or all of the inputs. This uses existing client logic where the client must rebroadcast a transaction until it is confirmed. In general, if a transaction has not made it into a block within 144*X blocks, there is _some_ reason it is getting rejected by the miners. The mempool janitor is a garbage collector design. This is inferior to the superblock model described at https://github.com/bitcoin/bitcoin/issues/3723 Other models can also achieve similar results. There are a lot of issues tied together here: transaction expiration, the desire to cap the mempool ram usage, scalability, DoS prevention, ... mempool ties a lot together. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] deterministic transaction expiration
On Tue, Aug 5, 2014 at 3:10 PM, Kaz Wesley kezi...@gmail.com wrote: Any approach based on beginning a transaction expiry countdown when a transaction is received (as in mempool janitor) seems unviable to me: ... That's why I think including information in the transaction itself, as with my nLockTime/IsStandard proposal, is necessary for transactions to reliably eventually die off from mempools. reliably die off from mempools leads into the land of tightly synchronizing memory pools across the network which is a problem of... large scope and much debate. :) For the moment, simply capping the mempool's size at each local node is a much more reachable goal. Capping, then, implies some culling policy. In general, bitcoind Tx mempool size is rather open ended, and that needs sorting out. -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] deterministic transaction expiration
On 8/5/2014 12:10 PM, Kaz Wesley wrote: Any approach based on beginning a transaction expiry countdown when a transaction is received (as in mempool janitor) seems unviable to me: once a node has forgotten a transaction, it must be susceptible to reaccepting it; It's hard to argue with that logic. If nLockTime is used for expiration, transaction creator can't lie to help tx live longer without pushing initial confirmation eligibility into the future. Very pretty. It would also enable fill or kill transactions with a backdated nLockTime, which must be confirmed in a few blocks, or start vanishing from mempools. -- Infragistics Professional Build stunning WinForms apps today! Reboot your WinForms applications with our WinForms controls. Build a bridge from your legacy apps to the future. http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
