Re: [bitcoin-dev] BIP 174 thoughts
Hi,
On June 26, 2018 8:33 AM, matejcik via bitcoin-dev
wrote:
>
>
> hello,
>
> in general, I agree with my colleague Tomas, the proposed changes are
>
> good and achieve the most important things that we wanted. We'll review
>
> the proposal in more detail later.
>
> For now a couple minor things I have run into:
>
> - valid test vector 2 ("one P2PKH input and one P2SH-P2WPKH input")
>
> seems broken, at least its hex version; a delimiter seems to be missing,
>
> misplaced or corrupted
Fixed
>
> - at least the first signing vector is not updated, but you probably
>
> know that
I updated all of the tests yesterday so they should be correct now. I will be
adding more tests
this week.
>
> - BIP32 derivation fields don't specify size of the "master public key",
>
> which would make it un-parsable :)
Oops, that's actually supposed to be master key fingerprint, not master public
key. I have updated
the BIP to reflect this.
>
> - "Transaction format is specified as follows" and its table need to be
>
> updated.
Done.
>
> I'm still going to argue against the key-value model though.
>
> It's true that this is not significant in terms of space. But I'm more
>
> concerned about human readability, i.e., confusing future implementers.
>
> At this point, the key-value model is there "for historical reasons",
>
> except these aren't valid even before finalizing the format. The
>
> original rationale for using key-values seems to be gone (no key-based
>
> lookups are necessary). As for combining and deduplication, whether key
>
> data is present or not is now purely a stand-in for a "repeatable" flag.
>
> We could just as easily say, e.g., that the high bit of "type" specifies
>
> whether this record can be repeated.
>
> (Moreover, as I wrote previously, the Combiner seems like a weirdly
>
> placed role. I still don't see its significance and why is it important
>
> to correctly combine PSBTs by agents that don't understand them. If you
>
> have a usecase in mind, please explain.
There is a diagram in the BIP that explains this. The combiner's job is to
combine two PSBTs that
are for the same transaction but contain different data such as signatures. It
is easier to implement
a combiner that does not need to understand the types at all, and such
combiners are forwards compatible,
so new types do not require new combiner implementations.
>
> ISTM a Combiner could just as well combine based on whole-record
>
> uniqueness, and leave the duplicate detection to the Finalizer. In case
>
> the incoming PSBTs have incompatible unique fields, the Combiner would
>
> have to fail anyway, so the Finalizer might as well do it. Perhaps it
>
> would be good to leave out the Combiner role entirely?)
A transaction that contains duplicate keys would be completely invalid.
Furthermore, in the set of typed
records model, having more than one redeemScript and witnessScript should be
invalid, so a combiner
would still need to understand what types are in order to avoid this situation.
Otherwise it would produce
an invalid PSBT.
I also dislike the idea of having type specific things like "only one
redeemScript" where a more generic
thing would work.
>
> There's two remaining types where key data is used: BIP32 derivations
>
> and partial signatures. In case of BIP32 derivation, the key data is
>
> redundant ( pubkey = derive(value) ), so I'd argue we should leave that
>
> out and save space.
I think it is still necessary to include the pubkey as not all signers who can
sign for a given pubkey may
know the derivation path. For example, if a privkey is imported into a wallet,
that wallet does not necessarily
know the master key fingerprint for the privkey nor would it necessarily have
the master key itself in
order to derive the privkey. But it still has the privkey and can still sign
for it.
>
> Re breaking change, we are proposing breaking changes anyway, existing
>
> code will need to be touched, and given that this is a hand-parsed
>
> format, changing `parse_keyvalue` to `parse_record` seems like a small
>
> matter?
The point is to not make it difficult for existing implementations to change.
Mostly what has been done now is just
moving things around, not an entire format change itself. Changing to a set of
typed records model require more
changes and is a complete restructuring of the format, not just moving things
around.
Additionally, I think that the current model is fairly easy to hand parse. I
don't think a record set model would make
it easier to follow. Furthermore, moving to Protobuf would make it harder to
hand parse as varints are slightly more
confusing in protobuf than with C
Re: [bitcoin-dev] BIP 174 thoughts
On Tue, Jun 26, 2018 at 8:33 AM, matejcik via bitcoin-dev wrote: > I'm still going to argue against the key-value model though. > > It's true that this is not significant in terms of space. But I'm more > concerned about human readability, i.e., confusing future implementers. > At this point, the key-value model is there "for historical reasons", > except these aren't valid even before finalizing the format. The > original rationale for using key-values seems to be gone (no key-based > lookups are necessary). As for combining and deduplication, whether key > data is present or not is now purely a stand-in for a "repeatable" flag. > We could just as easily say, e.g., that the high bit of "type" specifies > whether this record can be repeated. I understand this is a philosophical point, but to me it's the opposite. The file conveys "the script is X", "the signature for key X is Y", "the derivation for key X is Y" - all extra metadata added to inputs of the form "the X is Y". In a typed record model, you still have Xes, but they are restricted to a single number (the record type). In cases where that is insufficient, your solution is adding a repeatable flag to switch from "the first byte needs to be unique" to "the entire record needs to be unique". Why just those two? It seems much more natural to have a length that directly tells you how many of the first bytes need to be unique (which brings you back to the key-value model). Since the redundant script hashes were removed by making the scripts per-input, I think the most compelling reason (size advantages) for a record based model is gone. > (Moreover, as I wrote previously, the Combiner seems like a weirdly > placed role. I still don't see its significance and why is it important > to correctly combine PSBTs by agents that don't understand them. If you > have a usecase in mind, please explain. Forward compatibility with new script types. A transaction may spend inputs from different outputs, with different script types. Perhaps some of these are highly specialized things only implemented by some software (say HTLCs of a particular structure), in non-overlapping ways where no piece of software can handle all scripts involved in a single transaction. If Combiners cannot deal with unknown fields, they won't be able to deal with unknown scripts. That would mean that combining must be done independently by Combiner implementations for each script type involved. As this is easily avoided by adding a slight bit of structure (parts of the fields that need to be unique - "keys"), this seems the preferable option. > ISTM a Combiner could just as well combine based on whole-record > uniqueness, and leave the duplicate detection to the Finalizer. In case > the incoming PSBTs have incompatible unique fields, the Combiner would > have to fail anyway, so the Finalizer might as well do it. Perhaps it > would be good to leave out the Combiner role entirely?) No, a Combiner can pick any of the values in case different PSBTs have different values for the same key. That's the point: by having a key-value structure the choice of fields can be made such that Combiners don't need to care about the contents. Finalizers do need to understand the contents, but they only operate once at the end. Combiners may be involved in any PSBT passing from one entity to another. > There's two remaining types where key data is used: BIP32 derivations > and partial signatures. In case of BIP32 derivation, the key data is > redundant ( pubkey = derive(value) ), so I'd argue we should leave that > out and save space. In case of partial signatures, it's simple enough to > make the pubkey part of the value. In case of BIP32 derivation, computing the pubkeys is possibly expensive. A simple signer can choose to just sign with whatever keys are present, but they're not the only way to implement a signer, and even less the only software interacting with this format. Others may want to use a matching approach to find keys that are relevant; without pubkeys in the format, they're forced to perform derivations for all keys present. And yes, it's simple enough to make the key part of the value everywhere, but in that case it becomes legal for a PSBT to contain multiple signatures for a key, for example, and all software needs to deal with that possibility. With a stronger uniqueness constraint, only Combiners need to deal with repetitions. > Thing is: BIP174 *is basically protobuf* (v2) as it stands. If I'm > succesful in convincing you to switch to a record set model, it's going > to be "protobuf with different varint". If you take the records model, and then additionally drop the whole-record uniqueness constraint, yes, though that seems pushing it a bit by moving even more guarantees from the file format to application level code. I'd like to hear opinions of other people who have worked on implementations about changing this. Cheers, -- Pieter
Re: [bitcoin-dev] BIP 174 thoughts
On Tue, Jun 26, 2018 at 6:58 PM, William Casarin via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > > seems a bit overkill for how simple the format is, and pulling in a > large dependency just for this is a bit silly. Although making it > protobuf-compatible is an interesting idea, but I fear would be more > work than is worth? I haven't looked closed enough at the protobuf > encoding to be sure. > > > ...while at the same time, implementing "protobuf-based-BIP174" by > > hand is roughly equally difficult as implementing the current BIP174. > > as a data point, I was able to build a simple serializer[1] in about an > afternoon. I would much prefer to use this lib in say, clightning (my > original goal), without having to have the larger protobuf dependency. > > That was exactly matejcik's point; you can easily write protobuf-compatible encoder/decoder for such simple structure in about an afternoon, if you need. Or you can use existing protobuf parsers in matter of minute, if you don't care about dependencies. Also, many projects already have protobuf parsers, so it work in other way, too; you need BIP174 parser as extra dependency/library, although you already use protobuf library (like Trezor device does). For needs of BIP174, the difference between ad-hoc format and protobuf is neglible, so it is a mistake to introduce yet another format. slush ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP 174 thoughts
matejcik via bitcoin-dev writes: > BIP174 is a ad-hoc format, simple to parse by hand; but that results > in _having to_ parse it by hand. In contrast, protobuf has a huge > collection of implementations that will do the job of sorting record > types into relevant struct fields, proper delimiting of records, etc. seems a bit overkill for how simple the format is, and pulling in a large dependency just for this is a bit silly. Although making it protobuf-compatible is an interesting idea, but I fear would be more work than is worth? I haven't looked closed enough at the protobuf encoding to be sure. > ...while at the same time, implementing "protobuf-based-BIP174" by > hand is roughly equally difficult as implementing the current BIP174. as a data point, I was able to build a simple serializer[1] in about an afternoon. I would much prefer to use this lib in say, clightning (my original goal), without having to have the larger protobuf dependency. Cheers, [1] https://github.com/jb55/libpsbt -- https://jb55.com ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[bitcoin-dev] BIP039 - How to add a Portuguese wordlist?
Hello, Since Portuguese is considered the 6th most spoken language in the world and is an official language in 10 countries, I'd like to propose the expansion of the BIP039 wordlist to Portuguese or help if someone had already proposed it. What should I do? Regards, Breno ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP 174 thoughts
hello,
in general, I agree with my colleague Tomas, the proposed changes are
good and achieve the most important things that we wanted. We'll review
the proposal in more detail later.
For now a couple minor things I have run into:
- valid test vector 2 ("one P2PKH input and one P2SH-P2WPKH input")
seems broken, at least its hex version; a delimiter seems to be missing,
misplaced or corrupted
- at least the first signing vector is not updated, but you probably
know that
- BIP32 derivation fields don't specify size of the "master public key",
which would make it un-parsable :)
- "Transaction format is specified as follows" and its table need to be
updated.
I'm still going to argue against the key-value model though.
It's true that this is not significant in terms of space. But I'm more
concerned about human readability, i.e., confusing future implementers.
At this point, the key-value model is there "for historical reasons",
except these aren't valid even before finalizing the format. The
original rationale for using key-values seems to be gone (no key-based
lookups are necessary). As for combining and deduplication, whether key
data is present or not is now purely a stand-in for a "repeatable" flag.
We could just as easily say, e.g., that the high bit of "type" specifies
whether this record can be repeated.
(Moreover, as I wrote previously, the Combiner seems like a weirdly
placed role. I still don't see its significance and why is it important
to correctly combine PSBTs by agents that don't understand them. If you
have a usecase in mind, please explain.
ISTM a Combiner could just as well combine based on whole-record
uniqueness, and leave the duplicate detection to the Finalizer. In case
the incoming PSBTs have incompatible unique fields, the Combiner would
have to fail anyway, so the Finalizer might as well do it. Perhaps it
would be good to leave out the Combiner role entirely?)
There's two remaining types where key data is used: BIP32 derivations
and partial signatures. In case of BIP32 derivation, the key data is
redundant ( pubkey = derive(value) ), so I'd argue we should leave that
out and save space. In case of partial signatures, it's simple enough to
make the pubkey part of the value.
Re breaking change, we are proposing breaking changes anyway, existing
code *will* need to be touched, and given that this is a hand-parsed
format, changing `parse_keyvalue` to `parse_record` seems like a small
matter?
---
At this point I'm obliged to again argue for using protobuf.
Thing is: BIP174 *is basically protobuf* (v2) as it stands. If I'm
succesful in convincing you to switch to a record set model, it's going
to be "protobuf with different varint".
I mean this very seriously: (the relevant subset of) protobuf is a set
of records in the following format:
Record types can repeat, the schema specifies whether a field is
repeatable or not - if it's not, the last parsed value is used.
BIP174 is a ad-hoc format, simple to parse by hand; but that results in
_having to_ parse it by hand. In contrast, protobuf has a huge
collection of implementations that will do the job of sorting record
types into relevant struct fields, proper delimiting of records, etc.
...while at the same time, implementing "protobuf-based-BIP174" by hand
is roughly equally difficult as implementing the current BIP174.
N.B., it's possible to write a parser for protobuf-BIP174 without
needing a general protobuf library. Protobuf formats are designed with
forwards- and backwards- compatibility in mind, so having a hand-written
implementation should not lead to incompatibilities.
I did an experiment with this and other variants of the BIP174 format.
You can see them here:
[1] https://github.com/matejcik/bip174-playground
see in particular:
[2] https://github.com/matejcik/bip174-playground/blob/master/bip174.proto
The tool at [1] does size comparisons. On the test vectors, protobuf is
slightly smaller than key-value, and roughly equal to record-set, losing
out a little when BIP32 fields are used.
(I'm also leaving out key-data for BIP32 fields.)
There's some technical points to consider about protobuf, too:
- I decided to structure the message as a single "PSBT" type, where
"InputType" and "OutputType" are repeatable embedded fields. This seemed
better in terms of extensibility - we could add more sections in the
future. But the drawback is that you need to know the size of
Input/OutputType record in advance.
The other option is sending the messages separated by NUL bytes, same as
now, in which case you don't need to specify size.
- in InputType, i'm using "uint32" for sighash. This type is not
length-delimited, so non-protobuf consumers would have to understand it
specially. We could also declare that all fields must be
length-delimited[1] and implement sighash as a separate message type
with one field.
- non-protobuf consumers will also need to understand both protobuf
varint and bitcoin compact uint, which is a little ugly
Re: [bitcoin-dev] BetterHash status
Things go into production when people decide to adopt them, not before. You're welcome to contribute to the implementation at https://github.com/TheBlueMatt/mining-proxy On June 26, 2018 2:32:06 PM UTC, "Casciano, Anthony via bitcoin-dev" wrote: >What is the status of Matt Corallo's "BetterHash" BIP?? I recommend >it > >goes into production sooner than later. Any 2nd's ? > > >Thanks in advance! > >Tony Cash ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[bitcoin-dev] BetterHash status
What is the status of Matt Corallo's "BetterHash" BIP?? I recommend it goes into production sooner than later. Any 2nd's ? Thanks in advance! Tony Cash ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
