Re: [bitcoin-dev] Making OP_CODESEPARATOR and FindAndDelete in non-segwit scripts non-standard

[Matt Corallo via bitcoin-dev]

Indeed, the PR in question does *not* change the semantics of
OP_CODESEPARATOR within SegWit redeemScripts, where it is still allowed
(and Nicolas Dorier pointed out that he was using it in TumbleBit), so
there are still ways to use it, but only in places, like SegWit, where
the potential validation complexity blowup is massively reduced.

I am not sure that OP_CODESEPARATOR is entirely useless in pre-SegWit
scripts (I believe Nicolas' construction may still be relevant
pre-SegWit), though I strongly believe FindAndDelete is.

I don't think CODESEPARATOR rises to the threshold of it being "widely
known to be useless", but certainly the historical use of it (to
separate the scriptSig and the scriptPubKey in the scriptCode, which was
run as a single concatenated thing in the original design is no longer
relevant). FindAndDelete is equally irrelevant if not significantly more
irrelevant.

Matt

On 11/27/17 16:06, Mark Friedenbach wrote:
> It is relevant to note that BIP 117 makes an insecure form of
> CODESEPARATOR delegation possible, which could be made secure if some
> sort of CHECKSIGFROMSTACK opcode is added at a later point in time. It
> is not IMHO a very elegant way to achieve delegation, however, so I hope
> that one way or another this could be resolved quickly so it doesn’t
> hold up either one of those valuable additions.
> 
> I have no objections to making them nonstandard, or even to make them
> invalid if someone with a better grasp of history can attest that
> CODESEPARATOR was known to be entirely useless before the introduction
> of P2SH—not the same as saying it was useless, but that it was widely
> known to not accomplish what a early-days script author might think it
> was doing—and the UTXO set contains no scriptPubKeys making use of the
> opcode, even from the early days. Although a small handful could be
> special cased, if they exist.
> 
>> On Nov 27, 2017, at 8:33 AM, Matt Corallo > > wrote:
>>
>> I strongly disagree here - we don't only soft-fork out transactions that
>> are "fundamentally insecure", that would be significantly too
>> restrictive. We have generally been willing to soft-fork out things
>> which clearly fall outside of best-practices, especially rather
>> "useless" fields in the protocol eg soft-forking behavior into OP_NOPs,
>> soft-forking behavior into nSequence, etc.
>>
>> As a part of setting clear best-practices, making things non-standard is
>> the obvious step, though there has been active discussion of
>> soft-forking out FindAndDelete and OP_CODESEPARATOR for years now. I
>> obviously do not claim that we should be proposing a soft-fork to
>> blacklist FindAndDelete and OP_CODESEPARATOR usage any time soon, and
>> assume that it would take at least a year or three from when it was made
>> non-standard to when a soft-fork to finally remove them was proposed.
>> This should be more than sufficient time for folks using such weird (and
>> largely useless) parts of the protocol to object, which should be
>> sufficient to reconsider such a soft-fork.
>>
>> Independently, making them non-standard is a good change on its own, and
>> if nothing else should better inform discussion about the possibility of
>> anyone using these things.
>>
>> Matt
>>
>> On 11/15/17 14:54, Mark Friedenbach via bitcoin-dev wrote:
>>> As good of an idea as it may or may not be to remove this feature from
>>> the code base, actually doing so would be crossing a boundary that we
>>> have not previously been willing to do except under extraordinary
>>> duress. The nature of bitcoin is such that we do not know and cannot
>>> know what transactions exist out there pre-signed and making use of
>>> these features.
>>>
>>> It may be a good idea to make these features non standard to further
>>> discourage their use, but I object to doing so with the justification of
>>> eventually disabling them for all transactions. Taking that step has the
>>> potential of destroying value and is something that we have only done in
>>> the past either because we didn’t understand forks and best practices
>>> very well, or because the features (now disabled) were fundamentally
>>> insecure and resulted in other people’s coins being vulnerable. This
>>> latter concern does not apply here as far as I’m aware.
>>>
>>> On Nov 15, 2017, at 8:02 AM, Johnson Lau via bitcoin-dev
>>> >> 
>>> > wrote:
>>>
 In https://github.com/bitcoin/bitcoin/pull/11423 I propose to
 make OP_CODESEPARATOR and FindAndDelete in non-segwit scripts
 non-standard

 I think FindAndDelete() is one of the most useless and complicated
 functions in the script language. It is omitted from segwit (BIP143),
 but we still need to support it in non-segwit scripts. Actually,
 FindAndDelete() would only be triggered in some weird 

Re: [bitcoin-dev] Making OP_CODESEPARATOR and FindAndDelete in non-segwit scripts non-standard

[Mark Friedenbach via bitcoin-dev]

It is relevant to note that BIP 117 makes an insecure form of CODESEPARATOR 
delegation possible, which could be made secure if some sort of 
CHECKSIGFROMSTACK opcode is added at a later point in time. It is not IMHO a 
very elegant way to achieve delegation, however, so I hope that one way or 
another this could be resolved quickly so it doesn’t hold up either one of 
those valuable additions.

I have no objections to making them nonstandard, or even to make them invalid 
if someone with a better grasp of history can attest that CODESEPARATOR was 
known to be entirely useless before the introduction of P2SH—not the same as 
saying it was useless, but that it was widely known to not accomplish what a 
early-days script author might think it was doing—and the UTXO set contains no 
scriptPubKeys making use of the opcode, even from the early days. Although a 
small handful could be special cased, if they exist.

> On Nov 27, 2017, at 8:33 AM, Matt Corallo  wrote:
> 
> I strongly disagree here - we don't only soft-fork out transactions that
> are "fundamentally insecure", that would be significantly too
> restrictive. We have generally been willing to soft-fork out things
> which clearly fall outside of best-practices, especially rather
> "useless" fields in the protocol eg soft-forking behavior into OP_NOPs,
> soft-forking behavior into nSequence, etc.
> 
> As a part of setting clear best-practices, making things non-standard is
> the obvious step, though there has been active discussion of
> soft-forking out FindAndDelete and OP_CODESEPARATOR for years now. I
> obviously do not claim that we should be proposing a soft-fork to
> blacklist FindAndDelete and OP_CODESEPARATOR usage any time soon, and
> assume that it would take at least a year or three from when it was made
> non-standard to when a soft-fork to finally remove them was proposed.
> This should be more than sufficient time for folks using such weird (and
> largely useless) parts of the protocol to object, which should be
> sufficient to reconsider such a soft-fork.
> 
> Independently, making them non-standard is a good change on its own, and
> if nothing else should better inform discussion about the possibility of
> anyone using these things.
> 
> Matt
> 
> On 11/15/17 14:54, Mark Friedenbach via bitcoin-dev wrote:
>> As good of an idea as it may or may not be to remove this feature from
>> the code base, actually doing so would be crossing a boundary that we
>> have not previously been willing to do except under extraordinary
>> duress. The nature of bitcoin is such that we do not know and cannot
>> know what transactions exist out there pre-signed and making use of
>> these features.
>> 
>> It may be a good idea to make these features non standard to further
>> discourage their use, but I object to doing so with the justification of
>> eventually disabling them for all transactions. Taking that step has the
>> potential of destroying value and is something that we have only done in
>> the past either because we didn’t understand forks and best practices
>> very well, or because the features (now disabled) were fundamentally
>> insecure and resulted in other people’s coins being vulnerable. This
>> latter concern does not apply here as far as I’m aware.
>> 
>> On Nov 15, 2017, at 8:02 AM, Johnson Lau via bitcoin-dev
>> > > >> wrote:
>> 
>>> In https://github.com/bitcoin/bitcoin/pull/11423 I propose to
>>> make OP_CODESEPARATOR and FindAndDelete in non-segwit scripts non-standard
>>> 
>>> I think FindAndDelete() is one of the most useless and complicated
>>> functions in the script language. It is omitted from segwit (BIP143),
>>> but we still need to support it in non-segwit scripts. Actually,
>>> FindAndDelete() would only be triggered in some weird edge cases like
>>> using out-of-range SIGHASH_SINGLE.
>>> 
>>> Non-segwit scripts also use a FindAndDelete()-like function to remove
>>> OP_CODESEPARATOR from scriptCode. Note that in BIP143, only executed
>>> OP_CODESEPARATOR are removed so it doesn’t have the
>>> FindAndDelete()-like function. OP_CODESEPARATOR in segwit scripts are
>>> useful for Tumblebit so it is not disabled in this proposal
>>> 
>>> By disabling both, it guarantees that scriptCode serialized inside
>>> SignatureHash() must be constant
>>> 
>>> If we use a softfork to remove FindAndDelete() and OP_CODESEPARATOR
>>> from non-segwit scripts, we could completely remove FindAndDelete()
>>> from the consensus code later by whitelisting all blocks before the
>>> softfork block. The first step is to make them non-standard in the
>>> next release.
>>> 
>>> 
>>>  
>>> ___
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org 
>>> 
>>> 

Re: [bitcoin-dev] Block compression

[Jonas Schnelli via bitcoin-dev]

Hi Jeff

There where previous discussions about similar approaches [1] [2].

I’m not sure if compression should be built into the protocol.
My humble understanding of it, is, that it should be built into different 
layers.

If bandwidth is a concern, then on the fly gzip compression like apaches 
mod_deflate could be something. But I expect fast propagation is often more 
important then a ~30% bandwidth reduction.
Bandwidth may be a concern for historical blocks transmission. If you continue 
the proposal, I think you should focus on historical blocks.

If disk space is a concern, then the database layer should handle the 
compression.

Thanks
—



[1] 
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/011692.html
 

[2] https://github.com/bitcoin/bitcoin/pull/6973 




> Am 26.11.2017 um 16:11 schrieb Jeff Johnson via bitcoin-dev 
> :
> 
> I'm new to this mailing list and apologize if this has been suggested before. 
> I was directed from the Bitcoin core github to this mailing list for 
> suggestions.
> 
> I'd just like to post a possible solution that increases the amount of data 
> in a block without actually increasing the size on disk or the size in memory 
> or the size transmitted over the Internet. Simply applying various 
> compression algorithms, I was able to achieve about a 50% compression ratio. 
> Here are my findings on a recent Bitcoin block using max compression for all 
> methods:
> 
> Raw block
> 998,198 bytes
> 
> Gzip
> 521,212 bytes (52% ratio)
> (needs 2MB to decompress).
> 
> LZMA
> 415,308 bytes (41% ratio)
> (1MB dictionary, needs 3MB to decompress)
> 
> - ZStandard: 469,179 bytes (47% ratio)
> (1MB memory to decompress)
> 
> - LZ4: 641,063 bytes (64% ratio)
> (32-64K to decompress)
> 
> The compression time on my modest laptop (2 years old) was "instant". I ran 
> all from the command line and did not notice any lag as I pressed enter to do 
> the compression, so easily less than a second. But compression time doesn't 
> matter, decompression time is what matters as blocks will be decompressed 
> billions of times more than they will be compressed. Decompression speed for 
> LZ4 is the fastest of the above methods, at 3.3GB / second, slightly less 
> than half the speed of memcpy, see char at (https://github.com/lz4/lz4 
> ).
> 
> If decompression speed, CPU and memory usage is a concern, LZ4 is a no 
> brainer. You basically get a 33% larger block size for "free". But ZStandard, 
> in my opinion, makes the most sense as it offers greater than 50% compression 
> ratio with a very good decompression ratio of 900MB / second.
> 
> If this were implemented in the Bitcoin protocol, there would need to be a 
> place to specify the compression type in a set of bits somewhere, so that 
> future compression algorithms could potentially be added.
> 
> Miners could do nothing and keep sending blocks as is, and these blocks would 
> have "no compression" as the type of compression, just as today. Or they 
> could opt in to compress blocks and choose how many transactions they want to 
> stuff into the block, keeping the compressed size under the limit.
> 
> The bitcoin client code would also need to be able to handle the appropriate 
> compression bits, and limits of signature data, etc. modified to deal with 
> the compression.
> 
> I understand schnorr signatures are on the roadmap as a 25% compression gain 
> which is great, I suspect that schnorr signatures would compress even further 
> when compressed with the above compression methods.
> 
> Here is a link to the block that I compressed: 
> https://mega.nz/#!YPIF2KTa!4FxxLvqzjqIftrkhXwSC2h4G4Dolk8dLteNUolEtq98 
> 
> 
> Thanks for reading, best wishes to all.
> 
> -- Jeff Johnson
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev



signature.asc
Description: Message signed with OpenPGP
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Block compression

[Marco Pontello via bitcoin-dev]

Hi Jeff!


On Mon, Nov 27, 2017 at 3:11 AM, Jeff Johnson via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:


> Raw block
> 998,198 bytes
>
> Gzip
> 521,212 bytes (52% ratio)
> (needs 2MB to decompress).
>

I don't know how you got that raw block, but it seems a bit odd.
If you look at it in an hex editor, you'll notice that every odd byte is 0,
and that explain the unusual high compression ratio.

Bye!
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev