Re: [bitcoin-dev] Optimized Header Sync
On Thu, Mar 29, 2018 at 05:50:30PM -0700, Jim Posen via bitcoin-dev wrote:
> Taken a step further though, I'm really interested in treating the checkpoints
> as commitments to chain work [...]
In that case, shouldn't the checkpoints just be every 2016 blocks and
include the corresponding bits value for that set of blocks?
That way every node commits to (approximately) how much work their entire
chain has by sending something like 10kB of data (currently), and you
could verify the deltas in each node's chain's target by downloading the
2016 headers between those checkpoints (~80kB with the proposed compact
encoding?) and checking the timestamps and proof of work match both the
old target and the new target from adjacent checkpoints.
(That probably still works fine even if there's a hardfork that allows
difficulty to adjust more frequently: a bits value at block n*2016 will
still enforce *some* lower limit on how much work blocks n*2016+{1..2016}
will have to contribute; so will still allow you to estimate how much work
will have been done, it may just be less precise than the estimate you could
generate now)
Cheers,
aj
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Optimized Header Sync
Thanks for giving it a read and for sparking the discussion with your observation about the 40% savings from dropping prev_hash! > Thought this wasn't effective in case overt asic boost get widely adopted, > but then I understood that at the moment only two bits of version get > scrambled by that technique so this looks fine, maybe add a comment about > this so the reader doesn't get the same initial doubt I got. > I still need to compute for historical blocks how many could have an omitted version. Will post back with that when I get results. If overt ASIC Boost made this less effective, that would be unfortunate, but so be it. > My feeling is that encoding of the headers and checkpoints/parallel > download are separate subjects for two BIPS. > About the checkpoints I don't grasp why they are useful since an attacker > could lie about them but maybe I am missing something... > Yeah, I guess the background wasn't explained in the BIP itself. After your original post on the mailing list, there were suggestions that instead of modifying the format of existing messages, it would be better do create a new headers message. And as long as we're designing a new headers message, we should change the semantics to allow parallel download. But if you want to download from peers in parallel, you need to get a summary of the blocks that they have. Hence the checkpoints message. So that is why both of these messages are in the same BIP -- only together can they perform an efficient sync. Regarding the reliability of the checkpoints, I think it's strictly better than what we have now. Let's say a node is connected to 6 honest peers and 2 malicious peers. Even if the node does not know which ones are good or bad until it validates the headers, it sees that 6 of the peers are on the same chain, and can download those headers in parallel from 6 different sources. So that's already a win. Taken a step further though, I'm really interested in treating the checkpoints as commitments to chain work and using random sampling to detect lying peers before downloading all of their headers. So imagine you are connected to two peers, one good one bad, where the good one claims a chain with X total work and the bad one claims a chain with Y total work. To determine quickly which is correct, you can randomly sample ranges of headers and check the proofs of work to see whether it matches what the peer claimed. So basically you pick a checkpoint at random (weighted by the work delta) which commits to a total amount of work from the last checkpoint, then request all headers in between. If the peer responds with headers with the correct start hash, end hash, and start height (from the coinbase tx of the first header), then you can be somewhat more confident their total PoW matches the claimed amount. How many times do you need to sample? I don't know yet, but I've heard Benedikt Bunz is exploring this question with his research on FlyClients [1], which was an inspiration for this. > Bitfield allows great savings, however the encoding depends on the headers > height a client ask for, this cause a little computational burden on the > node and the undesirable side effect of difficult caching. Variable length > encoding cause caching difficulties too... > A simpler approach could be to encode the headers in groups of 2016 > headers (the difficulty period) where the first header is complete and the > others 2015 are missing the previous hash and the difficulty, this achieve > comparable savings ~45%, allows better caching and has fixed length > encoding. This could be useful for the node by caching headers on a single > file on disk and simply stream out the relative range when requested or to > serve the same encoded headers format in other context like http, > leveraging http caching infrastructure. > I don't see too much of a problem with caching. Most node implementations I know of keep all headers in memory anyway, often in contiguous segments of RAM for historical headers, so it should be fairly inexpensive to serve queries. Beyond that, the response for a particular query (start_height, end_hash, encoding version) can be cached, so if some service wants to precompute max size responses for all start_height multiples of 1,000, they could cache those. -Jim [1] https://www.youtube.com/watch?time_continue=8400&v=BPNs9EVxWrA > 2018-03-28 1:31 GMT+02:00 Jim Posen via bitcoin-dev linuxfoundation.org>: > >> Based on some ideas that were thrown around in this thread ( >> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/ >> 2017-December/015385.html), I have been working on a P2P extension that >> will allow faster header sync mechanisms. The one-sentence summary is that >> by encoding headers more efficiently (eg. omitting prev_hash) and >> downloading evenly spaced checkpoints throughout history (say every >> 1,000th) from all peers first, we could speed up header sync, which would >> be a huge improvement for light client
Re: [bitcoin-dev] Soft Fork Activation & Enforcement w/o Signaling?
Excellent - Thanks for your response Jorge. This helps us plan out the future upgrades properly. Since I see 0.15 and 0.16 use block versions as 0x2000, whereas the current deployed codebase (based on bitcoin 0.9.4) makes versions 0x0002 (as seen by a 0.15 client), it appears safe to activate soft forks which require a minimum of version 3 and 4 blocks (0x0003 and 0x0004, respectively). Would you agree? On Wed, Mar 28, 2018 at 7:55 AM, Jorge Timón wrote: > Yes, you can activate softforks at a given height. > I don't see any reason why you couldn't rebase to 0.16 directly. > The block version bumping was a mistake in bip34, you don't really > need to bump the version number. In any case, I would recommend > reading bip34 and what it activates in the code. IIRC the last thing > was bip65. > > On Wed, Mar 21, 2018 at 11:04 PM, Samad Sajanlal via bitcoin-dev > wrote: > > Is it possible to activate soft forks such as BIP65 and BIP66 without > prior > > signaling from miners? I noticed in chainparams.cpp that there are block > > heights where the enforcement begins. > > > > I understand this is already active on bitcoin. I'm working on a project > > that is a clone of a clone of bitcoin, and we currently do not have > BIP65 or > > BIP66 enforced - no signaling of these soft forks either (most of the > > network is on a source code fork of bitcoin 0.9). This project does not > and > > never intends to attempt to replace bitcoin - we know that without > bitcoin > > our project could never exist, so we owe a great deal of gratitude to the > > bitcoin developers. > > > > If the entire network upgrades to the correct version of the software > (based > > on bitcoin 0.15), which includes the block height that has enforcement, > can > > we simply skip over the signaling and go straight into > > activation/enforcement? > > > > At this time we are lucky that our network is very small, so it is > > reasonable to assume that the whole network will upgrade their clients > > within a short window (~2 weeks). We would schedule the activation ~2 > months > > out from when the client is released, just to ensure everyone has time to > > upgrade. > > > > We have been stuck on the 0.9 code branch and my goal is to bring it up > to > > 0.15 at least, so that we can implement Segwit and other key features > that > > bitcoin has introduced. The 0.15 client currently works with regards to > > sending and receiving transactions but the soft forks are not active. I > > understand that activating them will segregate the 0.15 clients onto > their > > own fork, which is why I'd like to understand the repercussions of doing > it > > without any signaling beforehand. I also would prefer not to have to make > > intermediate releases such as 0.10, 0.11.. etc to get the soft forks > > activated. > > > > Another related question - does the block version get bumped up > > automatically at the time that a soft fork activates, or is there > additional > > stuff that I need to do within the code to ensure it bumps up at the same > > time? From what I saw in the code it appears that it will bump up > > automatically, but I would like some confirmation on that. > > > > Regards, > > Samad > > > > > > > > ___ > > bitcoin-dev mailing list > > bitcoin-dev@lists.linuxfoundation.org > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > > ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Optimized Header Sync
Hi Jim, | version[i] = version[i - ((bit[3] << 2) + (bit[4] << 1) + bit[5])] Thought this wasn't effective in case overt asic boost get widely adopted, but then I understood that at the moment only two bits of version get scrambled by that technique so this looks fine, maybe add a comment about this so the reader doesn't get the same initial doubt I got. ...downloading evenly spaced checkpoints throughout history (say every > 1,000th) from all peers first... My feeling is that encoding of the headers and checkpoints/parallel download are separate subjects for two BIPS. About the checkpoints I don't grasp why they are useful since an attacker could lie about them but maybe I am missing something... To take advantage of these possible savings, this document defines a > variable-sized ''compressed encoding'' of block headers that occur in a > range. Note that no savings are possible when serializing a single header; > it should only be used for vectors of sequential headers. The full headers > are reconstructed using data from previous headers in the range. The > serialization begins with an ''encoding indicator'', which is a bitfield > specifying how each field is serialized. The bits of the indicator have the > following semantics: Bitfield allows great savings, however the encoding depends on the headers height a client ask for, this cause a little computational burden on the node and the undesirable side effect of difficult caching. Variable length encoding cause caching difficulties too... A simpler approach could be to encode the headers in groups of 2016 headers (the difficulty period) where the first header is complete and the others 2015 are missing the previous hash and the difficulty, this achieve comparable savings ~45%, allows better caching and has fixed length encoding. This could be useful for the node by caching headers on a single file on disk and simply stream out the relative range when requested or to serve the same encoded headers format in other context like http, leveraging http caching infrastructure. 2018-03-28 1:31 GMT+02:00 Jim Posen via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org>: > Based on some ideas that were thrown around in this thread (https://lists. > linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015385.html), I > have been working on a P2P extension that will allow faster header sync > mechanisms. The one-sentence summary is that by encoding headers more > efficiently (eg. omitting prev_hash) and downloading evenly spaced > checkpoints throughout history (say every 1,000th) from all peers first, we > could speed up header sync, which would be a huge improvement for light > clients. Here is a draft of the BIP: https://github.com/jimpo/ > bips/blob/headers-sync/headersv2.mediawiki. The full text is below as > well. > > I'd love to hear any feedback people have. > > -- > > == Abstract == > > This BIP describes a P2P network extension enabling faster, more reliable > methods for syncing the block header chain. New P2P messages are proposed as > more efficient replacements for getheaders and > headers during initial block download. The proposed header > download protocol reduces bandwidth usage by ~40%-50% and supports > downloading headers ranges from multiple peers in parallel, which is not > possible with the current mechanism. This also enables sync strategies with > better resistance to denial-of-service attacks. > > == Motivation == > > Since 2015, optimized Bitcoin clients fetch all block headers before blocks > themselves in order to avoid downloading ones that are not part of the most > work chain. The protocol currently in use for fetching headers leaves room > for further optimization, specifically by compressing header data and > downloading more headers > simulaneouslyhttps://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015385.html. > Any savings here should have a large impact given that both full nodes and > light clients must sync the header chain as a first step, and that the time > to validate and index the headers is negligible compared to the time spent > downloading them from the network. Furthermore, some current implementations > of headers syncing rely on preconfigured checkpoints to discourage attackers > attempting to fill up a victim's disk space with low-work headers. The > proposed messages enable sync strategies that are resilient against these > types of attacks. The P2P messages are designed to be flexible, supporting > multiple header sync strategies and leaving room for future innovations, > while also compact. > > == Definitions == > > ''double-SHA256'' is a hash algorithm defined by two invocations of SHA-256: > double-SHA256(x) = SHA256(SHA256(x)). > > == Specification == > > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", > "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this do
[bitcoin-dev] Electrum Personal Server beta release
Electrum Personal Server is an implementation of the Electrum wallet server protocol that allows users to point their Electrum wallet at their own full node. It is compatible resource-saving features like pruning, blocksonly and disabled txindex. It is much less resource-intensive than other Electrum servers because it only stores the user's own addresses, instead of every address that was ever used. As such it makes tradeoffs, namely losing Electrum's "instant on" feature. Right now using Electrum Personal Server is the easiest way to use a hardware wallet backed by your own full node. It is very lightweight, being a small python script that barely uses any CPU or RAM; much less than the full node it's connected to. Hopefully Electrum Personal Server can be part of the solution in putting full node wallets into the hands of as many people as possible. The project is now in beta release: https://github.com/chris-belcher/electrum-personal-server It now has all the essential features to make it practical for use; Merkle proofs, deterministic wallets, bech32 addresses, SSL, Core's multi-wallet support. Along with the features that were in the alpha release of tracking new transactions, confirmations, block headers, importing addresses. There is a caveat about pruning. Electrum Personal Server obtains merkle proofs using the `gettxoutproof` RPC call, if pruning is enabled and that block has been deleted then the RPC will return null and so the Electrum wallet will display `Not Verified`. Everything else will still work, and this shouldn't be a problem in most situations because Electrum usually only requests merkle proofs for recent transactions and pruning keeps recent blocks. But in the long term it needs some thought on the best way to fix this. I've been thinking about adding code for Bitcoin Core that stores merkle proofs for each of the wallet's own transactions in wallet.dat. Further Reading: * https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-February/015707.html * https://bitcointalk.org/index.php?topic=3167572.0 ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
