Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size
> The additional benefit of the input script/outpoint filter is to watch for > unexpected spends (coins getting stolen or spent from another wallet) or > transactions without a unique change or output address. I think this is a > reasonable implementation, and it would be nice to be able to download that > filter without any input elements. As someone who's implemented a complete integration of the filtering technique into an existing wallet, and a higher application I disagree. There's not much gain to be had in splitting up the filters: it'll result in additional round trips (to fetch these distinct filter) during normal operation, complicate routine seed rescanning logic, and also is detrimental to privacy if one is fetching blocks from the same peer as they've downloaded the filters from. However, I'm now convinced that the savings had by including the prev output script (addr re-use and outputs spent in the same block as they're created) outweigh the additional booking keeping required in an implementation (when extracting the precise tx that matched) compared to using regular outpoint as we do currently. Combined with the recently proposed re-parametrization of the gcs parameters[1], the filter size should shrink by quite a bit! I'm very happy with the review the BIPs has been receiving as of late. It would've been nice to have this 1+ year ago when the draft was initially proposed, but better late that never! Based on this thread, [1], and discussions on various IRC channels, I plan to make the following modifications to the BIP: 1. use P=2^19 and M=784931 as gcs parameters, and also bind these to the filter instance, so future filter types may use distinct parameters 2. use the prev output script rather than the prev input script in the regular filter 3. remove the txid from the regular filter(as with some extra book-keeping the output script is enough) 4. do away with the extended filter all together, as our original use case for it has been nerfed as the filter size grew too large when doing recursive parsing. instead we watch for the outpoint being spent and extract the pre-image from it if it matches now The resulting changes should slash the size of the filters, yet still ensure that they're useful enough for our target use case. [1]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-May/016029.html -- Laolu ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size
> Is there an application that requires watching for output scripts that > doesn't also require watching for input scrips (or, less efficiently, > input outpoints)? > Certain wallets may be able to use only the output script filter by using output scripts to watch for confirmations on sent transactions, assuming that application is the only one with access to the private keys. The additional benefit of the input script/outpoint filter is to watch for unexpected spends (coins getting stolen or spent from another wallet) or transactions without a unique change or output address. I think this is a reasonable implementation, and it would be nice to be able to download that filter without any input elements. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size
On Tue, May 29, 2018 at 2:42 AM, Jim Posen wrote: > Certain wallets may be able to use only the output script filter by using > output scripts to watch for confirmations on sent transactions, assuming > that application is the only one with access to the private keys. The > additional benefit of the input script/outpoint filter is to watch for > unexpected spends (coins getting stolen or spent from another wallet) or > transactions without a unique change or output address. I think this is a > reasonable implementation, and it would be nice to be able to download that > filter without any input elements. In this configuration there is little need to access historical blocks though, since you're assuming that you'll never recover from a backup. No? ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size
Is there an application that requires watching for output scripts that doesn't also require watching for input scrips (or, less efficiently, input outpoints)? Any of the wallet application that I'm aware of need to see coins being spent as well as created, else they may try to spend already spent coins. If we're not aware of any applications that wouldnt use both, there isn't much reason to separate them and if input scripts are used instead of input outpoints there is additional savings from combining them (due to the same scripts being spent as were created in the block-- due to reuse and chaining). I still am of the belief, based on Matt's argument, that there is no use for txid what-so-ever (instead just watch for an outpoint). On Mon, May 28, 2018 at 6:28 PM, Tamas Blummer wrote: > Forgot to mention: The link I sent is to a branch that is patched to produce > the filter stats. > This is the main project and the BIP158 implementation: > https://github.com/rust-bitcoin/rust-bitcoin-spv/blob/master/src/blockfilter.rs > > Tamas Blummer > >> On May 28, 2018, at 20:18, Tamas Blummer wrote: >> >> Hi Jim, >> >> A “basic” combined filter would mean up to 0.5 GB filter data per month >> (with 100% segwith use). Considering that 1 GB is the usual data quota for >> an entry level mobile phone contract, this could be a too high barrier for >> adoption. >> >> I repeated your calculations and produced a slightly different graph that >> shows the fraction of cummulative filter size to cummulative blockchain >> size. This is less noisy but otherwise confirms your measurement. >> >> I think that the data supports separation of filters as a combined filter >> does not seem to come with significant savings. (basic size ~= txid + input >> points + output scripts sizes) >> >> My calculations are repeatable with: >> >> https://github.com/tamasblummer/rust-bitcoin-spv/blob/blockfilterstats/src/bin/blockfilterstats.rs >> >> that is using a Rust implementation of an SPV client built on top of other >> libraries we work on in the rust-bitcoin GitHub community >> (https://github.com/rust-bitcoin). Yes, this is a shameles plug for the >> project hoping to attract more developer. >> >> Tamas Blummer >> >> >> >> >>> On May 24, 2018, at 05:48, Jim Posen via bitcoin-dev >>> wrote: >>> >>> Greg, I've attached a graph including the input scripts. >>> >>> In the top graph, we can see how the input script filter compares to the >>> input outpoint filter. It is definitely smaller as a result of address >>> reuse. The bottom graph shows the ratio over time of combining the input >>> prev script and output script filters vs keeping them separate. In more >>> recent blocks, it appears that there are decreasing savings. >>> >> > ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size
Forgot to mention: The link I sent is to a branch that is patched to produce the filter stats. This is the main project and the BIP158 implementation: https://github.com/rust-bitcoin/rust-bitcoin-spv/blob/master/src/blockfilter.rs Tamas Blummer > On May 28, 2018, at 20:18, Tamas Blummer wrote: > > Hi Jim, > > A “basic” combined filter would mean up to 0.5 GB filter data per month (with > 100% segwith use). Considering that 1 GB is the usual data quota for an entry > level mobile phone contract, this could be a too high barrier for adoption. > > I repeated your calculations and produced a slightly different graph that > shows the fraction of cummulative filter size to cummulative blockchain size. > This is less noisy but otherwise confirms your measurement. > > I think that the data supports separation of filters as a combined filter > does not seem to come with significant savings. (basic size ~= txid + input > points + output scripts sizes) > > My calculations are repeatable with: > > https://github.com/tamasblummer/rust-bitcoin-spv/blob/blockfilterstats/src/bin/blockfilterstats.rs > > that is using a Rust implementation of an SPV client built on top of other > libraries we work on in the rust-bitcoin GitHub community > (https://github.com/rust-bitcoin). Yes, this is a shameles plug for the > project hoping to attract more developer. > > Tamas Blummer > > > > >> On May 24, 2018, at 05:48, Jim Posen via bitcoin-dev >> wrote: >> >> Greg, I've attached a graph including the input scripts. >> >> In the top graph, we can see how the input script filter compares to the >> input outpoint filter. It is definitely smaller as a result of address >> reuse. The bottom graph shows the ratio over time of combining the input >> prev script and output script filters vs keeping them separate. In more >> recent blocks, it appears that there are decreasing savings. >> > ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
