Re: [Bitcoin-development] Bitcoin Core 0.10 release schedule
On Sun, Oct 26, 2014 at 9:53 AM, Luke Dashjr l...@dashjr.org wrote: On Sunday, October 26, 2014 7:57:12 AM Wladimir wrote: Let me know if there is anything else you think is ready (and not too risky) to be in 0.10. At the very least, we need: #5106 Bugfix: submitblock: Use a temporary CValidationState to determine ... #5103 CreateNewBlock and miner_tests: Also check generated template is ... #5078 Bugfix: CreateNewBlock: Check that active chain has a valid tip ... (or at least some conclusion for the problem discussed therein) OK Harmless/No reason not to have: #3727 RPC: submitblock: Support for returning specific rejection reasons #1816 Support for BIP 23 block proposal #5144 Qt: Elaborate on signverify message dialog warning #5071 Introduce CNodePolicy for putting isolated node policy code and ... (futher commits exist that should ideally get in after this is merged) ACK on the UI change, I think it would be best to let the full-blown miner policy class wait for 0.11. Debatable (but harmless, and miners seem to want it): #5077 Enable customising node policy for datacarrier data size with a ... OK, that's a low-risk change, it just makes what is now a constant configurable. Wladimir -- ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Core 0.10 release schedule
On Sun, Oct 26, 2014 at 12:44 PM, Melvin Carvalho melvincarva...@gmail.com wrote: Firstly, apologies in coming in late to the conversation. As I am also working on a REST API for electronic coins. Some questions: 1. Is there a BIP, or some other doc (e.g. gist), outlining the REST output e.g. the response format and MIME types. Or just compile from source? See the opening post from @jgarzik, it has some documentation on how to use the API. It would be nice to have some write-up about the current functionality in the release notes, but there currently is none. It's a RPC-side change, not a P2P-side change so it doesn't require a BIP. 2. How set in stone is v1 of the the going forward? PS I support @maaku's comments re: /api/v1/ -- tho I guess it is too late for that now. 3. Would there be any support to develop this interface into something that would be W3C standards compliant, or reviewed by the REST community. So for example a context can be provided to self document the terms (something I've almost completed) and would allow standardization of block explorer and bitcoind outputs. Right now every explorer seems to have a different JSON output. It's not too late, it's not been merged yet. Though a W3C standard takes a long time to pan out, and it may be more useful to have this available rather than wait for the API to be standardized (which means this will need to be postponed at least one version). As you say, a new interface be added later under another URI. Note that we're only interested in exposing read-only, public data which is already available in Bitcoin Core's internals. We're not aiming to add a fully-fledged block explorer with (say) address indexes. Although that could be part of the standard if it allows implementations to support just a subset. Anyhow - please coordinate this with Jeff Garzik, it's better to work together here. Wladimir -- ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Core 0.10 release schedule
On 27 October 2014 08:49, Wladimir laa...@gmail.com wrote: On Sun, Oct 26, 2014 at 12:44 PM, Melvin Carvalho melvincarva...@gmail.com wrote: Firstly, apologies in coming in late to the conversation. As I am also working on a REST API for electronic coins. Some questions: 1. Is there a BIP, or some other doc (e.g. gist), outlining the REST output e.g. the response format and MIME types. Or just compile from source? See the opening post from @jgarzik, it has some documentation on how to use the API. It would be nice to have some write-up about the current functionality in the release notes, but there currently is none. It's a RPC-side change, not a P2P-side change so it doesn't require a BIP. Thanks. Yes, I worked this out after looking at the code. I would be happy to help with documentation. 2. How set in stone is v1 of the the going forward? PS I support @maaku's comments re: /api/v1/ -- tho I guess it is too late for that now. 3. Would there be any support to develop this interface into something that would be W3C standards compliant, or reviewed by the REST community. So for example a context can be provided to self document the terms (something I've almost completed) and would allow standardization of block explorer and bitcoind outputs. Right now every explorer seems to have a different JSON output. It's not too late, it's not been merged yet. Though a W3C standard takes a long time to pan out, and it may be more useful to have this available rather than wait for the API to be standardized (which means this will need to be postponed at least one version). As you say, a new interface be added later under another URI. Agree. I think these changes are great for 0.10. Note that we're only interested in exposing read-only, public data which is already available in Bitcoin Core's internals. We're not aiming to add a fully-fledged block explorer with (say) address indexes. Although that could be part of the standard if it allows implementations to support just a subset. Got it thanks. Anyhow - please coordinate this with Jeff Garzik, it's better to work together here. Will do. Work in this area is ongoing, both in terms of coding/patches/testing and documentation. Do you think it would a reasonable idea to put down some thoughts and proposals in a BIP? Wladimir -- ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Core 0.10 release schedule
On Mon, Oct 27, 2014 at 7:24 AM, Melvin Carvalho melvincarva...@gmail.com wrote: Do you think it would a reasonable idea to put down some thoughts and proposals in a BIP? It would certainly be nice to start with a document that reflects the new REST interface. That makes a good starting point for further discussion. In general the interface exports what information is already available. As Wladimir notes, there is no plan to turn this into a full fledged block explorer, if that implies adding indices etc. Feedback on the HTTP headers and form, and additional thoughts proposals are welcome. My pull request is intended to present something minimal, that is easy to review and merge. My own list of further to-dos includes * last-modified and etag headers * export UTXOs a la Mike Hearn's getutxos query * eventually rebuild the RPC server to something multithreaded a la https://github.com/jgarzik/rpcsrv PR #2844 @ https://github.com/bitcoin/bitcoin/pull/2844 -- Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/ -- ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Reworking the policy estimation code (fee estimates)
I've been playing around with the code for estimating fees and found a few issues with the existing code. I think this will address several observations that the estimates returned by the existing code appear to be too high. For instance see @cozz in Issue 4866 https://github.com/bitcoin/bitcoin/issues/4866. Here's what I found: 1) We're trying to answer the question of what fee X you need in order to be confirmed within Y blocks. The existing code tries to do that by calculating the median fee for each possible Y instead of gathering statistics for each possible X. That approach is statistically incorrect. In fact since certain X's appear so frequently, they tend to dominate the statistics at all possible Y's (a fee rate of about 40k satoshis) 2) The existing code then sorts all of the data points in all of the buckets together by fee rate and then reassigns buckets before calculating the medians for each confirmation bucket. The sorting forces a relationship where there might not be one. Imagine some other variable, such as first 2 bytes of the transaction hash. If we sorted these and then used them to give estimates, we'd see a clear but false relationship where transactions with low starting bytes in their hashes took longer to confirm. 3) Transactions which don't have all their inputs available (because they depend on other transactions in the mempool) aren't excluded from the calculations. This skews the results. I rewrote the code to follow a different approach. I divided all possible fee rates up into fee rate buckets (I spaced these logarithmically). For each transaction that was confirmed, I updated the appropriate fee rate bucket with how many blocks it took to confirm that transaction. The hardest part of doing this fee estimation is to decide what the question really is that we're trying to answer. I took the approach that if you are asking what fee rate I need to be confirmed within Y blocks, then what you would like to know is the lowest fee rate such that a relatively high percentage of transactions of that fee rate are confirmed within Y blocks. Since even the highest fee transactions are confirmed within the first block only 90-93% of the time, I decided to use 80% as my cutoff. So now to answer estimatefee Y, I scan through all of the fee buckets from the most expensive down until I find the last bucket with 80% of the transactions confirmed within Y blocks. Unfortunately we still have the problem of not having enough data points for non-typical fee rates, and so it requires gathering a lot of data to give reasonable answers. To keep all of these data points in a circular buffer and then sort them for every analysis (or after every new block) is expensive. So instead I adopted the approach of keeping an exponentially decaying moving average for each bucket. I used a decay of .998 which represents a half life of 374 blocks or about 2.5 days. Also if a bucket doesn't have very many transactions, I combine it with the next bucket. Here is a link https://github.com/morcos/bitcoin/pull/3 to the code. I can create an actual pull request if there is consensus that it makes sense to do so. I've attached a graph comparing the estimates produced for 1-3 confirmations by the new code and the old code. I did apply the patch to fix issue 3 above to the old code first. The new code is in green and the fixed code is in purple. The Y axis is a log scale of feerate in satoshis per KB and the X axis is chain height. The new code produces the same estimates for 2 and 3 confirmations (the answers are effectively quantized by bucket). I've also completely reworked smartfees.py. It turns out to require many many more transactions are put through in order to have statistically significant results, so the test is quite slow to run (about 3 mins on my machine). I've also been running a real world test, sending transactions of various fee rates and seeing how long they took to get confirmed. After almost 200 tx's at each fee rate, here are the results so far: Fee rate 1100 Avg blocks to confirm 2.30 NumBlocks:% confirmed 1: 0.528 2: 0.751 3: 0.870 Fee rate 2500 Avg blocks to confirm 2.22 NumBlocks:% confirmed 1: 0.528 2: 0.766 3: 0.880 Fee rate 5000 Avg blocks to confirm 1.93 NumBlocks:% confirmed 1: 0.528 2: 0.782 3: 0.891 Fee rate 1 Avg blocks to confirm 1.67 NumBlocks:% confirmed 1: 0.569 2: 0.844 3: 0.943 Fee rate 2 Avg blocks to confirm 1.33 NumBlocks:% confirmed 1: 0.715 2: 0.963 3: 0.989 Fee rate 3 Avg blocks to confirm 1.27 NumBlocks:% confirmed 1: 0.751 2: 0.974 3: 1.0 Fee rate 4 Avg blocks to confirm 1.25 NumBlocks:% confirmed 1: 0.792 2: 0.953 3: 0.994 Fee rate 6 Avg blocks to confirm 1.12 NumBlocks:% confirmed 1: 0.875 2: 1.0 3: 1.0 Fee rate 10 Avg blocks to confirm 1.09 NumBlocks:% confirmed 1: 0.901 2: 1.0 3: 1.0 Fee rate 30 Avg blocks to confirm 1.12 NumBlocks:% confirmed 1: 0.886 2: 0.989 3: 1.0 Alex
[Bitcoin-development] DS Deprecation Window
Greetings Bitcoin Dev, This is a proposal to improve the ability of bitcoin users to rely on unconfirmed transactions. It can be adopted incrementally, with no hard or soft fork required. https://github.com/dgenr8/out-there/blob/master/ds-dep-win.md Your thoughtful feedback would be very much appreciated. It is not yet implemented anywhere. Cheers, Tom Harding CA, USA -- ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] DS Deprecation Window
Matt, You're right, thanks. Without double-spend relay, miner won't know that some txes conflict with anything. I'll add that first-double-spends are relayed per #4570. Miner has to be very careful including a double-spend in his block -- he hopes: - that based on his measured time offset from the first spend he received, at most a tiny fraction of the network will delay his block - that not too many nodes saw an earlier spend that he didn't see, which could increase that fraction - that most other nodes saw his tx. Any who didn't will only learn about it by receiving his block, and they will assign it the time when they receive the block. That's likely to be more than T (30 seconds) after an earlier spend, so they would delay the block. The best course of action is intended to be for miner to exclude fast ( 2 hours) double spends completely. On 10/27/2014 1:17 PM, Matt Corallo wrote: miners are incentivized to go connect to everyone on the network and look for double-spends On 10/27/14 19:58, Tom Harding wrote: https://github.com/dgenr8/out-there/blob/master/ds-dep-win.md -- ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] DS Deprecation Window
On Tue, Oct 28, 2014 at 2:26 AM, Tom Harding t...@thinlink.com wrote: Matt, You're right, thanks. Without double-spend relay, miner won't know that some txes conflict with anything. Even with that, the miner cannot tell, his only safe option is to include no transactions at all. Consider a malicious miner can concurrently flood all other miners with orthogonal double spends (which he doesn't mine himself). These other miners will all be spending some amount of their time mining on these transactions before realizing others consider them double-spends. -- ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
