Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
Hi all, I've also been spending a few months coding upon the change's Pieter has been making with the headersfirst8 pull request. My code updates are also ready to test, and are available on github at https://github.com/rebroad/bitcoin/ and the branch is sipa-headersfirst8-patches. I've made a number of improvement. Namely that it tracks the block as it downloads and won't disconnect if the block is downloading at a reasonable speed. The current stall logic of Pieter's is broken in that it will continue to disconnect a peer that is providing a block - particularly the next block needed to advance the current tip. I've raised this issue, but so far haven't been able to communicate the problem in a way that's been understood. I've also added logic to avoid the node stalling due to many blocks being added to the ActiveTip (which would cause timeouts both from our node, and nodes we are connected to). It will also log and determine bandwidth per node, and the next changes I will be adding will be to make it prefer to download from the faster nodes (coming shortly). I have also added code ready to adapt the window size for the download. Currently the start setting for blocks in flight is 3 per node, but for early on on the block chain this will be too small, so once it realises this after a few downloads and determines the average block size and speed, it will automatically adjust the number of blocks to request per node and revise this each minute. Please do take a look at my code, and feel free to test it. It also improves some of the debug.log output to make it easier to read and provide useful information about concurrent downloads, etc. Edmund On Sun, Oct 12, 2014 at 7:34 AM, Pieter Wuille pieter.wui...@gmail.com wrote: Hi all, I believe that a large change that I've been working on for Bitcoin Core is ready for review and testing: headers-first synchronization. In short, it changes the way the best chain is discovered, downloaded and verified, with several advantages: * Parallel block downloading (much faster sync on typical network connections). * No more stalled downloads. * Much more robust against unresponsive or slow peers. * Removes a class of DoS attacks related to peers feeding you low-difficulty valid large blocks on a side branch. * Reduces the need for checkpoints in the code. * No orphan blocks stored in memory anymore (reducing memory usage during sync). * A major step step towards an SPV mode using the reference codebase. Historically, this mode of operation has been known for years (Greg Maxwell wrote up a description of a very similar method in https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync in early 2012, but it was known before that), but it took a long time to refactor these code enough to support it. Technically, it works by replacing the single-peer blocks download by a single-peer headers download (which typically takes seconds/minutes) and verification, and simultaneously fetching blocks along the best known headers chain from all peers that are known to have the relevant blocks. Downloading is constrained to a moving window to avoid unbounded unordering of blocks on disk (which would interfere with pruning later). At the protocol level, it increases the minimally supported version for peers to 31800 (corresponding to bitcoin v3.18, released in december 2010), as earlier versions did not support the getheaders P2P message. So, the code is available as a github pull request (https://github.com/bitcoin/bitcoin/pull/4468), or packaged on http://bitcoin.sipa.be/builds/headersfirst, where you can also find binaries to test with. Known issues: * At the very start of the sync, especially before all headers are processed, downloading is very slow due to a limited number of blocks that are requested per peer simultaneously. The policies around this will need some experimentation can certainly be improved. * Blocks will be stored on disk out of order (in the order they are received, really), which makes it incompatible with some tools or other programs. Reindexing using earlier versions will also not work anymore as a result of this. * The block index database will now hold headers for which no block is stored on disk, which earlier versions won't support. If you are fully synced, it may still be possible to go back to an earlier version. Unknown issues: * Who knows, maybe it will replace your familiy pictures with Nyan Cat? Use at your own risk. TL;DR: Review/test https://github.com/bitcoin/bitcoin/pull/4468 or http://bitcoin.sipa.be/builds/headersfirst. -- Pieter -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5
[Bitcoin-development] Decreasing block propagation time
https://bitcointalk.org/index.php?topic=145066.0 The idea proposed in the above article seemed like an excellent idea. What is holding this up from being implemented? Does someone need to code it, or write a BIP first? -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://pubads.g.doubleclick.net/gampad/clk?id=154622311iu=/4140/ostg.clktrk___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Fwd: Service bits for pruned nodes
As part of a roadmap for block downloading, I think this may be a good time to look into providing an HTTP/HTTPS protocol for block downloading - this would also allow web proxies to cache blocks and thus make it more accessible, as well as cater for resumeable downloads. -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] block difficulty - inherently stable or inherently unstable?
I was just thinking about the way block difficulty is calculated, and how people may (in future) decide whether to mine or not. Is it possible that when the difficulty is low, many will decide to mine, producing blocks every 3 or 5 minutes, and then in 1 week, bitcoin will increase the difficulty, and many will decide it is no longer cost effective, stop mining, and find blocks are being produced every 30 minutes for the next 6 weeks, before the returning to a previous difficulty where people decide to mine again, etc, etc. Seems reasonably possible in my mind, and I'm wondering if the stability so far has been purely due to people mining without thinking too much about the immediate cost-effectiveness (or mining by zombie-farms). Thoughts, anyone? Ed -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Potential network split when individual tx used as coinbase?
Hi, Looking at: https://github.com/bitcoin/bitcoin/commit/3e52aaf2121d597ab1ed012b65e37f9cb5f2754e#src/main.cpp-P52 It appears that 8 months ago the code was changed to DoS(100) nodes sending on txs that use individual txs as the coinbase. Does this mean txs that are 0 confirmed? If so, then, is this a risk of a network split, as I'm sure I've read about services popping up using bitcoin that are specifically allowing 0 confirmed transactions, and therefore there must be peers around that accept these. Or have I misread the code? Cheers, Ed PS. Would a BIP have been applicable for the above-mentioned change? -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] free tx rate limiter potentially causes more traffic not less?
I recently was dabbling with AskFor() and changing the time waited from 2 minutes to 10 seconds (I think perhaps this value may change in future versions when network tuning is implemented). I noticed that, when used with the -limitfreerelay option that is causes significant increase in traffic between peers. This is because the tx gets asked for (from all connected peers usually), but AlwaysHave never becomes true as it's never stored, always rejected, so it effectively getdatas the transaction from every single connected peer. Would it perhaps be better to set up a memory pool for rejected txs and blocks (perhaps keeping only the hash) so that these rejected items can continue being ignored? I hope these observations are ok - I consider myself at the trying to understand the code/protocol/algorithm stage so might sometimes make false assumptions of what the code intends to do. Ed -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] BIP to improve the availability of blocks
Dear Bitcoin developers, In brief, the proposal I have is to extend the protocol to allow partial block download and upload. This is for people with intermittent connectivity or restricted connectivity. e.g. my own internet connection is quite slow, and my ISP routinely sends RSTs to both sides of connections to severe them. This often happens during block download and upload. I also often encounter the reception of blocks I have already received, further wasting bandwidth. This happens as quite often it can be far more than 2 minutes before block reception occurs following the getdata request, by which time my node has already sent a new getdata to another node requesting the same block. My proposal is that in addition to the size (which is advertised in the header), the hash is also advertised in the header (of a block). This would help nodes to determine whether they wanted to reject the download. (e.g. if it already had a block matching that hash). This of course wouldn't prevent a rogue node from sending an incorrect hash, but this would aid in saving bandwidth amongst behaving nodes. The other part of the proposal is to allow nodes to request upload and download blocks that have already been partially downloaded. This could be done by modifying the existing methods of upload, download, or by adding a new method, perhaps even using HTTP/HTTPS or something similar. This would also help nodes to obtain the blockchain who have restrictive ISPs, especially if they are being served on port 80 or 443. This could perhaps also allow web caches to keep caches of the blockchain, thereby making it also more available also. Currently, without this functionality, nodes with restrictive (or slow) internet have some options, such as going via a tor proxy, but due to the latency, the problem with multiple receptions of the same block still occur. Hopefully, not too clueless a post for my first post to this mailing list. Regards, Ed -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
