Re: [Bitcoin-development] Missing fRelayTxes in version message

[Jeff Garzik]

On Thu, Jun 20, 2013 at 3:13 AM, Addy Yeow  wrote:
> I personally don't treat the relay field as optional, i.e. it is there as
> 0x01 if it is set. Otherwise, it is simply a trailing zero byte. Hence, the
> right way of reading the packet as with any network packet is to first
> retrieve the header information, get the actual payload length, then parse
> the payload accordingly. I can also choose to include 0x00 for my relay
> field in my outgoing packet and reflect that accordingly in my length field
> in the header.

That works until another field is added in the same manner.

Implementations are expected to follow the standard logic of

if (more data)
   parse field X
if (more data)
   parse field X+1
if (more data)
   parse field X+2
etc.

Ugly or not, there is no point in changing now.  Updating the version
message to a radically different design, for simply cleanliness
reasons, is not sufficient cause to migrate the entire bitcoin
universe to a new and different version/feature negotiation setup.

--
Jeff Garzik
Senior Software Engineer and open source evangelist
BitPay, Inc.  https://bitpay.com/

--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version

[Mike Hearn]

As I said, there's no benefit. Even if we do that on the C++ side, you
still have to handle connections from bitcoinj clients which will send the
field with the old version number. You can't assume they'll all be updated
simultaneously, even though both the Android app and MultiBit do have
update notifications these days and eventually old versions will presumably
disappear.

Re: flexibility. Let's say version V+1 adds a complicated new set of data
to some messages. Not every client wants or needs the feature enabled by
them.

Now version V+2 adds a simple extension to a basic message that everyone
wants/needs.

To get the latter feature, all clients now have to support the first
feature as well because the version number is monotonic.

OK, we can use a service bit to handle these cases, if we anticipate that
not all clients will want the first feature. But then again, we can also
use the presence of the additional data as the ground truth instead of
duplicating that fact. I don't really mind either way. It just seems that
parsing always requires you to be able to handle truncated messages anyway
(without asserting or crashing), because a bogus client can always send you
partial data. So I don't see what effort is saved.



On Thu, Jun 20, 2013 at 12:52 PM, Pieter Wuille wrote:

> Let's just increase the version number and be done with this discussion.
> It's a small benefit, but it simplifies things and it's trivial to do.
>
> I don't understand how a policy of requiring version increases could limit
> future extensions: after the version/verack exchange, the protocol version
> is negotiated between peers, and there is no need for anything optional
> anymore.
>
> Note thay this is just about parsing, not about relaying - you should
> still relay parts of a message you haven't parsed. But that doesn't apply
> to the version message anyway, which is the only place where this matters.
>
> --
> Pieter
>  On 20 Jun 2013 12:38, "Turkey Breast"  wrote:
>
>> I don't get why this is such a contentious change?
>>
>> Before I was able to use asserts to check the expected length of length
>> of messages per protocol version, I could pass in dumb iterators that just
>> parse the byte stream and I could serialize and deserialize a message to
>> check the parser is correct (in debug mode).
>>
>> This 'simple' change causes all that behaviour to be lost. You can no
>> longer just use iterators but must know the remaining length (or if you use
>> std::distance, you can only use specific std containers - not just anything
>> with an iterator and an operator++). You cannot check the deserialization
>> process by serializing the deserialized message and comparing it to the
>> original data (because the bool is always present in the serializer).
>>
>> It's a bit stupid you call it buggy code when this behaviour has never
>> been present in Bitcoin. The BIP doesn't introduce any unwanted
>> side-effects and is a trivial reasonable change.
>>
>> If you want optional fields then the proper way to do it, is to either
>> set a flag in the Services field of the "version" message to indicate
>> different formats for messages (i.e use this template structure for a
>> message, not that one), introduce a new message (if the changes are big),
>> to approve/improve Stefan's BIP 32 for custom services or to have a value
>> in the byte stream indicating which fields are present (maybe a bitfield or
>> so).
>>
>> Using a quirk of an implementation is just bad form and sloppy coding.
>> Optional fields should have their own mechanism that allows them to remain
>> as optional fields between protocol version upgrades.
>>
>> The bitcoind software can probably be improved too, by checking that the
>> length of the version message is consistent for the protocol version given
>> by the connected node. Right now it makes no assumptions based on that
>> which is a mistake (new clients can broadcast older version messages that
>> don't have all the fields required). Probably the software should penalise
>> hosts which do that.
>>
>> What's the big deal to update the protocol version number from 70001 to
>> 70002? It's not like we'll run out of integers. The field has now gone from
>> optional to required now anyway - that's a behaviour change. It'd be good
>> to enforce that. I see this as a bug.
>>
>>   --
>>  *From:* Mike Hearn 
>> *To:* Pieter Wuille 
>> *Cc:* Bitcoin Dev ; Tamas
>> Blummer 
>> *Sent:* Thursday, June 20, 2013 11:17 AM
>> *Subjec

Re: [Bitcoin-development] Missing fRelayTxes in version

[Pieter Wuille]

Let's just increase the version number and be done with this discussion.
It's a small benefit, but it simplifies things and it's trivial to do.

I don't understand how a policy of requiring version increases could limit
future extensions: after the version/verack exchange, the protocol version
is negotiated between peers, and there is no need for anything optional
anymore.

Note thay this is just about parsing, not about relaying - you should still
relay parts of a message you haven't parsed. But that doesn't apply to the
version message anyway, which is the only place where this matters.

-- 
Pieter
 On 20 Jun 2013 12:38, "Turkey Breast"  wrote:

> I don't get why this is such a contentious change?
>
> Before I was able to use asserts to check the expected length of length of
> messages per protocol version, I could pass in dumb iterators that just
> parse the byte stream and I could serialize and deserialize a message to
> check the parser is correct (in debug mode).
>
> This 'simple' change causes all that behaviour to be lost. You can no
> longer just use iterators but must know the remaining length (or if you use
> std::distance, you can only use specific std containers - not just anything
> with an iterator and an operator++). You cannot check the deserialization
> process by serializing the deserialized message and comparing it to the
> original data (because the bool is always present in the serializer).
>
> It's a bit stupid you call it buggy code when this behaviour has never
> been present in Bitcoin. The BIP doesn't introduce any unwanted
> side-effects and is a trivial reasonable change.
>
> If you want optional fields then the proper way to do it, is to either set
> a flag in the Services field of the "version" message to indicate different
> formats for messages (i.e use this template structure for a message, not
> that one), introduce a new message (if the changes are big), to
> approve/improve Stefan's BIP 32 for custom services or to have a value in
> the byte stream indicating which fields are present (maybe a bitfield or
> so).
>
> Using a quirk of an implementation is just bad form and sloppy coding.
> Optional fields should have their own mechanism that allows them to remain
> as optional fields between protocol version upgrades.
>
> The bitcoind software can probably be improved too, by checking that the
> length of the version message is consistent for the protocol version given
> by the connected node. Right now it makes no assumptions based on that
> which is a mistake (new clients can broadcast older version messages that
> don't have all the fields required). Probably the software should penalise
> hosts which do that.
>
> What's the big deal to update the protocol version number from 70001 to
> 70002? It's not like we'll run out of integers. The field has now gone from
> optional to required now anyway - that's a behaviour change. It'd be good
> to enforce that. I see this as a bug.
>
>   --
>  *From:* Mike Hearn 
> *To:* Pieter Wuille 
> *Cc:* Bitcoin Dev ; Tamas
> Blummer 
> *Sent:* Thursday, June 20, 2013 11:17 AM
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version
>
> There's no problem, but there's no benefit either. It also locks us in to
> a potentially problematic guarantee - what if in future we want to have,
> say, two optional new pieces of data in two different messages. We don't
> want to require that if version > X then you have to implement all features
> up to and including that point.
>
> Essentially the number of fields in a message is like a little version
> number, just for that message. It adds flexibility to keep it that way, and
> there's no downside, seeing as that bridge was already crossed and people
> with parsers that can't handle it need to fix their code anyway.
>
> So I have a slight preference for keeping things the way they are, it
> keeps things flexible for future and costs nothing.
>
>
>
> On Thu, Jun 20, 2013 at 11:06 AM, Pieter Wuille 
> wrote:
>
> On Thu, Jun 20, 2013 at 09:36:40AM +0200, Mike Hearn wrote:
> > Sure but why not do that when there's an actual new field to add? Does
> > anyone have a proposal for a feature that needs a new version field at
> the
> > moment? There's no point changing the protocol now unless there's
> actually
> > a new field to add.
> >
> > Anyway I still don't see why anyone cares about this issue. The Bitcoin
> > protocol does not and never has required that all messages have a fixed
> > number of fields per version. Any parser written on the assum

Re: [Bitcoin-development] Missing fRelayTxes in version

[Mike Hearn]

Sure, the issue isn't running out of integers, it's that you have to handle
the case of truncated messages whether you like it or not so it doesn't add
any simplicity. Even if Bitcoin-Qt starts only sending the new field with a
new version number, there are tens of thousands of bitcoinj based wallets
out there now that send the current version number and the fRelayTx field
as well, so you cannot assume anything about whether the field will exist
or not based on the version number regardless of what is changed on the C++
side. Assuming you care about your code being able to serve Bloom-filtering
clients of course.

With regards to relying on quirks, etc, this is the old "is the protocol
defined by Satoshi's code" debate again ... as I said, version messages
have always had a variable number of fields. You didn't notice before
because it was a long time since any fields were added. Perhaps it's indeed
not ideal, perhaps if Bitcoin was designed in 2013 it'd be using protobufs
or some other pre-packaged serialization system. But it is what it is.



On Thu, Jun 20, 2013 at 12:37 PM, Turkey Breast wrote:

> I don't get why this is such a contentious change?
>
> Before I was able to use asserts to check the expected length of length of
> messages per protocol version, I could pass in dumb iterators that just
> parse the byte stream and I could serialize and deserialize a message to
> check the parser is correct (in debug mode).
>
> This 'simple' change causes all that behaviour to be lost. You can no
> longer just use iterators but must know the remaining length (or if you use
> std::distance, you can only use specific std containers - not just anything
> with an iterator and an operator++). You cannot check the deserialization
> process by serializing the deserialized message and comparing it to the
> original data (because the bool is always present in the serializer).
>
> It's a bit stupid you call it buggy code when this behaviour has never
> been present in Bitcoin. The BIP doesn't introduce any unwanted
> side-effects and is a trivial reasonable change.
>
> If you want optional fields then the proper way to do it, is to either set
> a flag in the Services field of the "version" message to indicate different
> formats for messages (i.e use this template structure for a message, not
> that one), introduce a new message (if the changes are big), to
> approve/improve Stefan's BIP 32 for custom services or to have a value in
> the byte stream indicating which fields are present (maybe a bitfield or
> so).
>
> Using a quirk of an implementation is just bad form and sloppy coding.
> Optional fields should have their own mechanism that allows them to remain
> as optional fields between protocol version upgrades.
>
> The bitcoind software can probably be improved too, by checking that the
> length of the version message is consistent for the protocol version given
> by the connected node. Right now it makes no assumptions based on that
> which is a mistake (new clients can broadcast older version messages that
> don't have all the fields required). Probably the software should penalise
> hosts which do that.
>
> What's the big deal to update the protocol version number from 70001 to
> 70002? It's not like we'll run out of integers. The field has now gone from
> optional to required now anyway - that's a behaviour change. It'd be good
> to enforce that. I see this as a bug.
>
>   --
>  *From:* Mike Hearn 
> *To:* Pieter Wuille 
> *Cc:* Bitcoin Dev ; Tamas
> Blummer 
> *Sent:* Thursday, June 20, 2013 11:17 AM
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version
>
> There's no problem, but there's no benefit either. It also locks us in to
> a potentially problematic guarantee - what if in future we want to have,
> say, two optional new pieces of data in two different messages. We don't
> want to require that if version > X then you have to implement all features
> up to and including that point.
>
> Essentially the number of fields in a message is like a little version
> number, just for that message. It adds flexibility to keep it that way, and
> there's no downside, seeing as that bridge was already crossed and people
> with parsers that can't handle it need to fix their code anyway.
>
> So I have a slight preference for keeping things the way they are, it
> keeps things flexible for future and costs nothing.
>
>
>
> On Thu, Jun 20, 2013 at 11:06 AM, Pieter Wuille 
> wrote:
>
> On Thu, Jun 20, 2013 at 09:36:40AM +0200, Mike Hearn wrote:
> > Sure but why not do that when there's an actual new field to add? Does
&g

Re: [Bitcoin-development] Missing fRelayTxes in version

[Turkey Breast]

I don't get why this is such a contentious change?

Before I was able to use asserts to check the expected length of length of 
messages per protocol version, I could pass in dumb iterators that just parse 
the byte stream and I could serialize and deserialize a message to check the 
parser is correct (in debug mode).

This 'simple' change causes all that behaviour to be lost. You can no longer 
just use iterators but must know the remaining length (or if you use 
std::distance, you can only use specific std containers - not just anything 
with an iterator and an operator++). You cannot check the deserialization 
process by serializing the deserialized message and comparing it to the 
original data (because the bool is always present in the serializer).

It's a bit stupid you call it buggy code when this behaviour has never been 
present in Bitcoin. The BIP doesn't introduce any unwanted side-effects and is 
a trivial reasonable change.

If you want optional fields then the proper way to do it, is to either set a 
flag in the Services field of the "version" message to indicate different 
formats for messages (i.e use this template structure for a message, not that 
one), introduce a new message (if the changes are big), to approve/improve 
Stefan's BIP 32 for custom services or to have a value in the byte stream 
indicating which fields are present (maybe a bitfield or so).

Using a quirk of an implementation is just bad form and sloppy coding. Optional 
fields should have their own mechanism that allows them to remain as optional 
fields between protocol version upgrades.


The bitcoind software can probably be improved too, by checking that the length 
of the version message is consistent for the protocol version given by the 
connected node. Right now it makes no assumptions based on that which is a 
mistake (new clients can broadcast older version messages that don't have all 
the fields required). Probably the software should penalise hosts which do that.

What's the big deal to update the protocol version number from 70001 to 70002? 
It's not like we'll run out of integers. The field has now gone from optional 
to required now anyway - that's a behaviour change. It'd be good to enforce 
that. I see this as a bug.




 From: Mike Hearn 
To: Pieter Wuille  
Cc: Bitcoin Dev ; Tamas Blummer 
 
Sent: Thursday, June 20, 2013 11:17 AM
Subject: Re: [Bitcoin-development] Missing fRelayTxes in version
 


There's no problem, but there's no benefit either. It also locks us in to a 
potentially problematic guarantee - what if in future we want to have, say, two 
optional new pieces of data in two different messages. We don't want to require 
that if version > X then you have to implement all features up to and including 
that point.

Essentially the number of fields in a message is like a little version number, 
just for that message. It adds flexibility to keep it that way, and there's no 
downside, seeing as that bridge was already crossed and people with parsers 
that can't handle it need to fix their code anyway.

So I have a slight preference for keeping things the way they are, it keeps 
things flexible for future and costs nothing.




On Thu, Jun 20, 2013 at 11:06 AM, Pieter Wuille  wrote:

On Thu, Jun 20, 2013 at 09:36:40AM +0200, Mike Hearn wrote:
>> Sure but why not do that when there's an actual new field to add? Does
>> anyone have a proposal for a feature that needs a new version field at the
>> moment? There's no point changing the protocol now unless there's actually
>> a new field to add.
>>
>> Anyway I still don't see why anyone cares about this issue. The Bitcoin
>> protocol does not and never has required that all messages have a fixed
>> number of fields per version. Any parser written on the assumption it did
>> was just buggy. Look at how tx messages are relayed for the most obvious
>> example of that pattern in action - it's actually the raw byte stream
>> that's stored and relayed to ensure that fields added in new versions
>> aren't dropped during round-tripping. Old versions are supposed to preserve
>> fields from the future.
>
>Actually, that is not the same issue. What is being argued for here is that
>the version in the version message itself should indicate which fields are
>present, so a parser doesn't need to look at the length of the message. That
>seems like a minor but very reasonable request to me, and it's trivial to do.
>That doesn't mean that you may receive versions higher than what you know of,
>and thus messages with fields you don't know about. That doesn't matter, you
>can just ignore them.
>
>I see no problem with raising the protocol version number to indicate
>"a

Re: [Bitcoin-development] Missing fRelayTxes in version

[Mike Hearn]

There's no problem, but there's no benefit either. It also locks us in to a
potentially problematic guarantee - what if in future we want to have, say,
two optional new pieces of data in two different messages. We don't want to
require that if version > X then you have to implement all features up to
and including that point.

Essentially the number of fields in a message is like a little version
number, just for that message. It adds flexibility to keep it that way, and
there's no downside, seeing as that bridge was already crossed and people
with parsers that can't handle it need to fix their code anyway.

So I have a slight preference for keeping things the way they are, it keeps
things flexible for future and costs nothing.



On Thu, Jun 20, 2013 at 11:06 AM, Pieter Wuille wrote:

> On Thu, Jun 20, 2013 at 09:36:40AM +0200, Mike Hearn wrote:
> > Sure but why not do that when there's an actual new field to add? Does
> > anyone have a proposal for a feature that needs a new version field at
> the
> > moment? There's no point changing the protocol now unless there's
> actually
> > a new field to add.
> >
> > Anyway I still don't see why anyone cares about this issue. The Bitcoin
> > protocol does not and never has required that all messages have a fixed
> > number of fields per version. Any parser written on the assumption it did
> > was just buggy. Look at how tx messages are relayed for the most obvious
> > example of that pattern in action - it's actually the raw byte stream
> > that's stored and relayed to ensure that fields added in new versions
> > aren't dropped during round-tripping. Old versions are supposed to
> preserve
> > fields from the future.
>
> Actually, that is not the same issue. What is being argued for here is that
> the version in the version message itself should indicate which fields are
> present, so a parser doesn't need to look at the length of the message.
> That
> seems like a minor but very reasonable request to me, and it's trivial to
> do.
> That doesn't mean that you may receive versions higher than what you know
> of,
> and thus messages with fields you don't know about. That doesn't matter,
> you
> can just ignore them.
>
> I see no problem with raising the protocol version number to indicate
> "all fields up to fRelayTxes are required, if the announced nVersion is
> above N".
> In fact, I believe (though haven't checked) all previous additions to the
> version
> message were accompanied with a protocol version (then: client version)
> increase
> as well.
>
> --
> Pieter
>
>
--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version

[Pieter Wuille]

On Thu, Jun 20, 2013 at 09:36:40AM +0200, Mike Hearn wrote:
> Sure but why not do that when there's an actual new field to add? Does
> anyone have a proposal for a feature that needs a new version field at the
> moment? There's no point changing the protocol now unless there's actually
> a new field to add.
> 
> Anyway I still don't see why anyone cares about this issue. The Bitcoin
> protocol does not and never has required that all messages have a fixed
> number of fields per version. Any parser written on the assumption it did
> was just buggy. Look at how tx messages are relayed for the most obvious
> example of that pattern in action - it's actually the raw byte stream
> that's stored and relayed to ensure that fields added in new versions
> aren't dropped during round-tripping. Old versions are supposed to preserve
> fields from the future.

Actually, that is not the same issue. What is being argued for here is that
the version in the version message itself should indicate which fields are
present, so a parser doesn't need to look at the length of the message. That
seems like a minor but very reasonable request to me, and it's trivial to do.
That doesn't mean that you may receive versions higher than what you know of,
and thus messages with fields you don't know about. That doesn't matter, you
can just ignore them.

I see no problem with raising the protocol version number to indicate
"all fields up to fRelayTxes are required, if the announced nVersion is above 
N".
In fact, I believe (though haven't checked) all previous additions to the 
version
message were accompanied with a protocol version (then: client version) increase
as well.

-- 
Pieter


--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version

[Tamas Blummer]

Yes it is trivial. I do not think greater complexity in the system should keep 
us from addressing low complexity issues.
You can't blame me or others not trying to simplify scripts, if there is such a 
headwind simplifying a version message.
You are right there is too much fuss about this.

Tamás Blummer
Founder, CEO
http://bitsofproof.com

On 20.06.2013, at 10:31, Mike Hearn  wrote:

> You can't eliminate the complexity (yet), otherwise you wouldn't be able to 
> talk to old nodes. You'll have to wait until versions prior to a particular 
> version are hard-forked off and can be safely dropped at connect time.
> 
> That said the reason I'm being so grumpy about this is that compared to the 
> complexity in the rest of the system, this is such a trivial and minor 
> detail. It's hardly even worth thinking about. I mean, we have a scripting 
> language full of opcodes nobody ever figured out how to use and the protocol 
> uses a mixture of byte orders, so an optional field in the version message is 
> really not such a big deal :)
> 
> 
> On Thu, Jun 20, 2013 at 10:17 AM, Tamas Blummer  wrote:
> I agree that this can be deferred until there is an actual new field without 
> any harm. But then remember to update the BIP37 too saying that it is 
> optional only if flag added in BIPXX is not present.
> 
> Your argument is that this complexity is already there so why not preserve 
> it. I think eliminating complexity (that has no benefit) strengthens the 
> system.
> 
> Tamás Blummer
> http://bitsofproof.com
> 
> On 20.06.2013, at 09:36, Mike Hearn  wrote:
> 
>> Sure but why not do that when there's an actual new field to add? Does 
>> anyone have a proposal for a feature that needs a new version field at the 
>> moment? There's no point changing the protocol now unless there's actually a 
>> new field to add.
>> 
>> Anyway I still don't see why anyone cares about this issue. The Bitcoin 
>> protocol does not and never has required that all messages have a fixed 
>> number of fields per version. Any parser written on the assumption it did 
>> was just buggy. Look at how tx messages are relayed for the most obvious 
>> example of that pattern in action - it's actually the raw byte stream that's 
>> stored and relayed to ensure that fields added in new versions aren't 
>> dropped during round-tripping. Old versions are supposed to preserve fields 
>> from the future.
>> 
>> 
>> 
>> On Thu, Jun 20, 2013 at 9:30 AM, Tamas Blummer  wrote:
>> Hi Mike,
>> 
>> The issue with the current parser is that those fields are conditionally 
>> optional on that there will be no subsequent fields added.
>> If there will be further fields they will become manadory. 
>>  
>> Why not bump the version and parse the fields as mandatory from then on? 
>> This would be backward compatible and cleaner
>> going forward.
>> 
>> Tamas Blummer
>> http://bitsofproof.com
>> 
>> 
>> --
>> This SF.net email is sponsored by Windows:
>> 
>> Build for Windows Store.
>> 
>> http://p.sf.net/sfu/windows-dev2dev
>> ___
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>> 
>> 
> 
> 

--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version

[Mike Hearn]

You can't eliminate the complexity (yet), otherwise you wouldn't be able to
talk to old nodes. You'll have to wait until versions prior to a particular
version are hard-forked off and can be safely dropped at connect time.

That said the reason I'm being so grumpy about this is that compared to the
complexity in the rest of the system, this is such a trivial and minor
detail. It's hardly even worth thinking about. I mean, we have a scripting
language full of opcodes nobody ever figured out how to use and the
protocol uses a mixture of byte orders, so an optional field in the version
message is really not such a big deal :)


On Thu, Jun 20, 2013 at 10:17 AM, Tamas Blummer wrote:

> I agree that this can be deferred until there is an actual new field
> without any harm. But then remember to update the BIP37 too saying that it
> is optional only if flag added in BIPXX is not present.
>
> Your argument is that this complexity is already there so why not preserve
> it. I think eliminating complexity (that has no benefit) strengthens the
> system.
>
> *Tamás Blummer*
> http://bitsofproof.com
>  
>
> On 20.06.2013, at 09:36, Mike Hearn  wrote:
>
> Sure but why not do that when there's an actual new field to add? Does
> anyone have a proposal for a feature that needs a new version field at the
> moment? There's no point changing the protocol now unless there's actually
> a new field to add.
>
> Anyway I still don't see why anyone cares about this issue. The Bitcoin
> protocol does not and never has required that all messages have a fixed
> number of fields per version. Any parser written on the assumption it did
> was just buggy. Look at how tx messages are relayed for the most obvious
> example of that pattern in action - it's actually the raw byte stream
> that's stored and relayed to ensure that fields added in new versions
> aren't dropped during round-tripping. Old versions are supposed to preserve
> fields from the future.
>
>
>
> On Thu, Jun 20, 2013 at 9:30 AM, Tamas Blummer wrote:
>
>>  Hi Mike,
>>
>> The issue with the current parser is that those fields are conditionally
>> optional on that there will be no subsequent fields added.
>> If there will be further fields they will become manadory.
>>
>> Why not bump the version and parse the fields as mandatory from then on?
>> This would be backward compatible and cleaner
>> going forward.
>>
>> Tamas Blummer
>> http://bitsofproof.com
>>  
>>
>>
>>
>> --
>> This SF.net email is sponsored by Windows:
>>
>> Build for Windows Store.
>>
>> http://p.sf.net/sfu/windows-dev2dev
>> ___
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>>
>
>
--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version

[Tamas Blummer]

I agree that this can be deferred until there is an actual new field without 
any harm. But then remember to update the BIP37 too saying that it is optional 
only if flag added in BIPXX is not present.

Your argument is that this complexity is already there so why not preserve it. 
I think eliminating complexity (that has no benefit) strengthens the system.

Tamás Blummer
http://bitsofproof.com

On 20.06.2013, at 09:36, Mike Hearn  wrote:

> Sure but why not do that when there's an actual new field to add? Does anyone 
> have a proposal for a feature that needs a new version field at the moment? 
> There's no point changing the protocol now unless there's actually a new 
> field to add.
> 
> Anyway I still don't see why anyone cares about this issue. The Bitcoin 
> protocol does not and never has required that all messages have a fixed 
> number of fields per version. Any parser written on the assumption it did was 
> just buggy. Look at how tx messages are relayed for the most obvious example 
> of that pattern in action - it's actually the raw byte stream that's stored 
> and relayed to ensure that fields added in new versions aren't dropped during 
> round-tripping. Old versions are supposed to preserve fields from the future.
> 
> 
> 
> On Thu, Jun 20, 2013 at 9:30 AM, Tamas Blummer  wrote:
> Hi Mike,
> 
> The issue with the current parser is that those fields are conditionally 
> optional on that there will be no subsequent fields added.
> If there will be further fields they will become manadory. 
>  
> Why not bump the version and parse the fields as mandatory from then on? This 
> would be backward compatible and cleaner
> going forward.
> 
> Tamas Blummer
> http://bitsofproof.com
> 
> 
> --
> This SF.net email is sponsored by Windows:
> 
> Build for Windows Store.
> 
> http://p.sf.net/sfu/windows-dev2dev
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 
> 

--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version

[Mike Hearn]

Sure but why not do that when there's an actual new field to add? Does
anyone have a proposal for a feature that needs a new version field at the
moment? There's no point changing the protocol now unless there's actually
a new field to add.

Anyway I still don't see why anyone cares about this issue. The Bitcoin
protocol does not and never has required that all messages have a fixed
number of fields per version. Any parser written on the assumption it did
was just buggy. Look at how tx messages are relayed for the most obvious
example of that pattern in action - it's actually the raw byte stream
that's stored and relayed to ensure that fields added in new versions
aren't dropped during round-tripping. Old versions are supposed to preserve
fields from the future.



On Thu, Jun 20, 2013 at 9:30 AM, Tamas Blummer wrote:

> Hi Mike,
>
> The issue with the current parser is that those fields are conditionally
> optional on that there will be no subsequent fields added.
> If there will be further fields they will become manadory.
>
> Why not bump the version and parse the fields as mandatory from then on?
> This would be backward compatible and cleaner
> going forward.
>
> Tamas Blummer
> http://bitsofproof.com
>  
>
>
>
> --
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version

[Tamas Blummer]

Hi Mike,

The issue with the current parser is that those fields are conditionally 
optional on that there will be no subsequent fields added.
If there will be further fields they will become manadory. 
 
Why not bump the version and parse the fields as mandatory from then on? This 
would be backward compatible and cleaner
going forward.

Tamas Blummer
http://bitsofproof.com

--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Addy Yeow]

I personally don't treat the relay field as optional, i.e. it is there as
0x01 if it is set. Otherwise, it is simply a trailing zero byte. Hence, the
right way of reading the packet as with any network packet is to first
retrieve the header information, get the actual payload length, then parse
the payload accordingly. I can also choose to include 0x00 for my relay
field in my outgoing packet and reflect that accordingly in my length field
in the header.


On Thu, Jun 20, 2013 at 4:20 PM, Turkey Breast wrote:

> I never said that Bitcoin message field lengths should always be the same.
> But before this change they certainly were constant per protocol version.
> All I'm saying is that optional lengths shouldn't be used (a field exists
> or not) and for every field change, the protocol version should be upgraded.
>
> Now that fRelayTxes is part of the protocol, the version number should be
> upgraded to reflect this fact.
>
>   --
>  *From:* Mike Hearn 
> *To:* Paul Lyon 
> *Cc:* Turkey Breast ; "
> bitcoin-development@lists.sourceforge.net" <
> bitcoin-development@lists.sourceforge.net>
> *Sent:* Wednesday, June 19, 2013 3:20 PM
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version message
>
> If you want to criticise the Bitcoin protocol for sloppyness, the variable
> length of some messages isn't where I'd start.
>
> Note that ping has the same issue, its length has changed over time to
> include the nonce.
>
> If your parser can't handle that kind of thing, you need to fix it. The
> protocol has always worked that way.
>
>
>
> On Wed, Jun 19, 2013 at 3:03 PM, Paul Lyon  wrote:
>
>  I’m also running into this exact same issue with my parser, now I
> understand why the relay field behavior I was seeing doesn’t match the wiki.
>
> So to parse a version message, you can’t rely on the protocol version? You
> have to know how long the payload is, and then parse the message
> accordingly? I agree with Turkey Breast, this seems a bit sloppy to me.
>
> Paul
>
> P.S. I’ve never used a dev mailing list before and I want to get involved
> with the Bitcoin dev community, so let me know if I’m horribly violating
> any mailing list etiquette. 😊
>
> *From:* Mike Hearn
> *Sent:* ‎Wednesday‎, ‎June‎ ‎19‎, ‎2013 ‎7‎:‎43‎ ‎AM
> *To:* Turkey Breast
> *Cc:* bitcoin-development@lists.sourceforge.net
>
> Bitcoin-Qt on master does send it now although it doesn't affect anything,
> but as old pre-filtering versions will continue to exist, you'll always
> have to be able to deserialize version messages without it.
>
> Bitcoin version messages have always had variable length, look at how the
> code is written in main.cpp. If you didn't experience issues until now all
> it means is that no sufficiently old nodes were talking to yours.
>
> The standard does not say it should appear. Read it again - BIP 37 says
> about the new version message field:
> If false then broadcast transactions will not be announced until a
> filter{load,add,clear} command is received. *If missing or true*, no
> change in protocol behaviour occurs.
>
>
> On Wed, Jun 19, 2013 at 12:33 PM, Turkey Breast wrote:
>
> It's a problem if you work with iterators to deserialize the byte stream.
> Even failing that, it's just sloppy programming. What happens in the future
> when new fields are added to the version message? It's not a big deal to
> say that this protocol version has X number of fields, that (higher)
> protocol version message has X + N number of fields. Deterministic number
> of fields per protocol version is sensical and how Bitcoin has been for a
> long time.
>
> And yes, it was a problem for me that caused a lot of confusion why this
> byte didn't exist in many version messages despite the standard saying it
> should and the code in bitcoind indicating it should. Nowhere was this
> written. It doesn't help other implementations to have an unclear behaviour
> that depends on some magic from one implementation.
>
>   --
>  *From:* Mike Hearn 
> *To:* Turkey Breast 
> *Cc:* "bitcoin-development@lists.sourceforge.net" <
> bitcoin-development@lists.sourceforge.net>
> *Sent:* Wednesday, June 19, 2013 11:39 AM
>
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version message
>
> It has to be optional because old clients don't send it, obviously.
>
> Why is this even an issue? There's no problem with variable length
> messages in any codebase that I'm aware of. Is this solving some actual
> problem?
>
>
> On Wed, Jun 19, 2013 at 12:30 AM, Turkey Breast wrote:
>
> 

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Mike Hearn]

The protocol version was bumped when Bloom filtering was added so there's
not much point bumping it again - you have to handle the old clients no
matter what. Nobody brought this up as an issue when the BIP or code was
first written and as you can see from main.cpp, it was done this way to be
consistent with how other version fields are handled:

if (!vRecv.empty())
vRecv >> addrFrom >> nNonce;
if (!vRecv.empty())
vRecv >> pfrom->strSubVer;
if (!vRecv.empty())
vRecv >> pfrom->nStartingHeight;
if (!vRecv.empty())
vRecv >> pfrom->fRelayTxes; // set to true after we get the
first filter* message

The existence of the nStartingHeight field for instance depends on the
message length and not anything else.

Anyway, are you really asking for the protocol to be changed to work around
an issue specific to how you wrote your parsing code? This is the first
time anyone has suggested this minor detail is a problem. It doesn't
present any issues for the C++ code or bitcoinj where message objects know
their own length at parse time.



On Thu, Jun 20, 2013 at 8:20 AM, Turkey Breast wrote:

> I never said that Bitcoin message field lengths should always be the same.
> But before this change they certainly were constant per protocol version.
> All I'm saying is that optional lengths shouldn't be used (a field exists
> or not) and for every field change, the protocol version should be upgraded.
>
> Now that fRelayTxes is part of the protocol, the version number should be
> upgraded to reflect this fact.
>
>   --
>  *From:* Mike Hearn 
> *To:* Paul Lyon 
> *Cc:* Turkey Breast ; "
> bitcoin-development@lists.sourceforge.net" <
> bitcoin-development@lists.sourceforge.net>
> *Sent:* Wednesday, June 19, 2013 3:20 PM
>
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version message
>
> If you want to criticise the Bitcoin protocol for sloppyness, the variable
> length of some messages isn't where I'd start.
>
> Note that ping has the same issue, its length has changed over time to
> include the nonce.
>
> If your parser can't handle that kind of thing, you need to fix it. The
> protocol has always worked that way.
>
>
>
> On Wed, Jun 19, 2013 at 3:03 PM, Paul Lyon  wrote:
>
>  I’m also running into this exact same issue with my parser, now I
> understand why the relay field behavior I was seeing doesn’t match the wiki.
>
> So to parse a version message, you can’t rely on the protocol version? You
> have to know how long the payload is, and then parse the message
> accordingly? I agree with Turkey Breast, this seems a bit sloppy to me.
>
> Paul
>
> P.S. I’ve never used a dev mailing list before and I want to get involved
> with the Bitcoin dev community, so let me know if I’m horribly violating
> any mailing list etiquette. 😊
>
> *From:* Mike Hearn
> *Sent:* ‎Wednesday‎, ‎June‎ ‎19‎, ‎2013 ‎7‎:‎43‎ ‎AM
> *To:* Turkey Breast
> *Cc:* bitcoin-development@lists.sourceforge.net
>
> Bitcoin-Qt on master does send it now although it doesn't affect anything,
> but as old pre-filtering versions will continue to exist, you'll always
> have to be able to deserialize version messages without it.
>
> Bitcoin version messages have always had variable length, look at how the
> code is written in main.cpp. If you didn't experience issues until now all
> it means is that no sufficiently old nodes were talking to yours.
>
> The standard does not say it should appear. Read it again - BIP 37 says
> about the new version message field:
> If false then broadcast transactions will not be announced until a
> filter{load,add,clear} command is received. *If missing or true*, no
> change in protocol behaviour occurs.
>
>
> On Wed, Jun 19, 2013 at 12:33 PM, Turkey Breast wrote:
>
> It's a problem if you work with iterators to deserialize the byte stream.
> Even failing that, it's just sloppy programming. What happens in the future
> when new fields are added to the version message? It's not a big deal to
> say that this protocol version has X number of fields, that (higher)
> protocol version message has X + N number of fields. Deterministic number
> of fields per protocol version is sensical and how Bitcoin has been for a
> long time.
>
> And yes, it was a problem for me that caused a lot of confusion why this
> byte didn't exist in many version messages despite the standard saying it
> should and the code in bitcoind indicating it should. Nowhere was this
> written. It doesn't help other implementations to have an unclear behaviour
> that depends on some magic from one imple

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Turkey Breast]

I never said that Bitcoin message field lengths should always be the same. But 
before this change they certainly were constant per protocol version. All I'm 
saying is that optional lengths shouldn't be used (a field exists or not) and 
for every field change, the protocol version should be upgraded.

Now that fRelayTxes is part of the protocol, the version number should be 
upgraded to reflect this fact.




 From: Mike Hearn 
To: Paul Lyon  
Cc: Turkey Breast ; 
"bitcoin-development@lists.sourceforge.net" 
 
Sent: Wednesday, June 19, 2013 3:20 PM
Subject: Re: [Bitcoin-development] Missing fRelayTxes in version message
 


If you want to criticise the Bitcoin protocol for sloppyness, the variable 
length of some messages isn't where I'd start.

Note that ping has the same issue, its length has changed over time to include 
the nonce.

If your parser can't handle that kind of thing, you need to fix it. The 
protocol has always worked that way.




On Wed, Jun 19, 2013 at 3:03 PM, Paul Lyon  wrote:

I’m also running into this exact same issue with my parser, now I understand 
why the relay field behavior I was seeing doesn’t match the wiki.
> 
>So to parse a version message, you can’t rely on the protocol version? You 
>have to know how long the payload is, and then parse the message accordingly? 
>I agree with Turkey Breast, this seems a bit sloppy to me.
> 
>Paul
> 
>P.S. I’ve never used a dev mailing list before and I want to get involved with 
>the Bitcoin dev community, so let me know if I’m horribly violating any 
>mailing list etiquette. 😊
> 
>From: Mike Hearn
>Sent: ‎Wednesday‎, ‎June‎ ‎19‎, ‎2013 ‎7‎:‎43‎ ‎AM
>To: Turkey Breast
>Cc: bitcoin-development@lists.sourceforge.net
> 
>Bitcoin-Qt on master does send it now although it doesn't affect anything, but 
>as old pre-filtering versions will continue to exist, you'll always have to be 
>able to deserialize version messages without it.
>
>
>Bitcoin version messages have always had variable length, look at how the code 
>is written in main.cpp. If you didn't experience issues until now all it means 
>is that no sufficiently old nodes were talking to yours.
>
>
>The standard does not say it should appear. Read it again - BIP 37 says about 
>the new version message field:
>If false then broadcast transactions will not be announced until a 
>filter{load,add,clear} command is received. If missing or true, no change in 
>protocol behaviour occurs.
> 
>
>
>
>On Wed, Jun 19, 2013 at 12:33 PM, Turkey Breast  wrote:
>
>It's a problem if you work with iterators to deserialize the byte stream. Even 
>failing that, it's just sloppy programming. What happens in the future when 
>new fields are added to the version message? It's not a big deal to say that 
>this protocol version has X number of fields, that (higher) protocol version 
>message has X + N number of fields. Deterministic number of fields per 
>protocol version is sensical and how Bitcoin has been for a long time.
>>
>>
>>And yes, it was a problem for me that caused a lot of confusion why this byte 
>>didn't exist in many version messages despite the standard saying it should 
>>and the code in bitcoind indicating it should. Nowhere was this written. It 
>>doesn't help other implementations to have an unclear behaviour that depends 
>>on some magic from one implementation.
>>
>>
>>
>>
>>
>> From: Mike Hearn 
>>To: Turkey Breast  
>>Cc: "bitcoin-development@lists.sourceforge.net" 
>> 
>>Sent: Wednesday, June 19, 2013 11:39 AM
>>
>>Subject: Re: [Bitcoin-development] Missing fRelayTxes in version message
>> 
>>
>>
>>It has to be optional because old clients don't send it, obviously.
>>
>>
>>Why is this even an issue? There's no problem with variable length messages 
>>in any codebase that I'm aware of. Is this solving some actual problem?
>>
>>
>>
>>On Wed, Jun 19, 2013 at 12:30 AM, Turkey Breast  
>>wrote:
>>
>>That's me. I never said to make all messages fixed length. I said to make a 
>>fixed number of fields per protocol. So given a protocol version number, you 
>>know the number of fields in a message. This is not only easier for parsing 
>>messages, but just good practice. I don't see why a 1 byte flag needs to be 
>>optional anyway.
>>>
>>>
>>>
>>>
>>>
>>> From: Mike Hearn 
>>>To: Turkey Breast  
>>>Cc: Bitcoin Dev  
>>>Sent: Tuesday, June 18, 2013 9:48 PM
>

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Paul Lyon]

I’m also running into this exact same issue with my parser, now I understand 
why the relay field behavior I was seeing doesn’tmatch the wiki.


So to parse a version message, you can’t rely on the protocol version? You have 
to know how long the payload is, and then parse the message accordingly? I 
agree with Turkey Breast, this seems a bit sloppy to me.


Paul


P.S. I’ve never used a dev mailing list before and I want to get involved with 
the Bitcoin dev community, so let me know if I’m horribly violating any mailing 
list etiquette. 😊



From: Mike Hearn
Sent: ‎Wednesday‎, ‎June‎ ‎19‎, ‎2013 ‎7‎:‎43‎ ‎AM
To: Turkey Breast
Cc: bitcoin-development@lists.sourceforge.net


Bitcoin-Qt on master does send it now although it doesn't affect anything, but 
as old pre-filtering versions will continue to exist, you'll always have to be 
able to deserialize version messages without it.



Bitcoin version messages have always had variable length, look at how the code 
is written in main.cpp. If you didn't experience issues until now all it means 
is that no sufficiently old nodes were talking to yours.




The standard does not say it should appear. Read it again - BIP 37 says about 
the new version message field:


If false then broadcast transactions will not be announced until a 
filter{load,add,clear} command is received. If missing or true, no change in 
protocol behaviour occurs.





On Wed, Jun 19, 2013 at 12:33 PM, Turkey Breast  wrote:




It's a problem if you work with iterators to deserialize the byte stream. Even 
failing that, it's just sloppy programming. What happens in the future when new 
fields are added to the version message? It's not a big deal to say that this 
protocol version has X number of fields, that (higher) protocol version message 
has X + N number of fields. Deterministic number of fields per protocol version 
is sensical and how Bitcoin has been for a long time.




And yes, it was a problem for me that caused a lot of confusion why this byte 
didn't exist in many version messages despite the standard saying it should and 
the code in bitcoind indicating it should. Nowhere was this written. It doesn't 
help other implementations to have an unclear behaviour that depends on some 
magic from one implementation.










From: Mike Hearn 
To: Turkey Breast  

Cc: "bitcoin-development@lists.sourceforge.net" 
 
Sent: Wednesday, June 19, 2013 11:39 AM


Subject: Re: [Bitcoin-development] Missing fRelayTxes in version message

 





It has to be optional because old clients don't send it, obviously.



Why is this even an issue? There's no problem with variable length messages in 
any codebase that I'm aware of. Is this solving some actual problem?




On Wed, Jun 19, 2013 at 12:30 AM, Turkey Breast  wrote:




That's me. I never said to make all messages fixed length. I said to make a 
fixed number of fields per protocol. So given a protocol version number, you 
know the number of fields in a message. This is not only easier for parsing 
messages, but just good practice. I don't see why a 1 byte flag needs to be 
optional anyway.













From: Mike Hearn 
To: Turkey Breast  
Cc: Bitcoin Dev  
Sent: Tuesday, June 18, 2013 9:48 PM
Subject: Re: [Bitcoin-development] Missing fRelayTxes in version message
 






It's not a bug (although there was recently a change to make bitcoind/qt always 
send this field anyway). 



I don't know where Amir is going with BIP 60. Version messages have always been 
variable length. There's nothing inherent in the Bitcoin protocol that says all 
messages are fixed length, indeed, tx messages are allowed to have arbitrary 
data appended after them that gets relayed.




On Tue, Jun 18, 2013 at 7:45 PM, Turkey Breast  wrote:




See this BIP. I'm not sure if this is a bug or what, but it would be good if 
messages always had a fixed number of fields per protocol version.




https://en.bitcoin.it/wiki/BIP_0060#Code_Updates




This BIP details everything that needs to be done and proposes a protocol 
upgrade.


--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development





--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development





--
This SF.ne

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Mike Hearn]

If you want to criticise the Bitcoin protocol for sloppyness, the variable
length of some messages isn't where I'd start.

Note that ping has the same issue, its length has changed over time to
include the nonce.

If your parser can't handle that kind of thing, you need to fix it. The
protocol has always worked that way.



On Wed, Jun 19, 2013 at 3:03 PM, Paul Lyon  wrote:

> I’m also running into this exact same issue with my parser, now I
> understand why the relay field behavior I was seeing doesn’t match the wiki.
>
> So to parse a version message, you can’t rely on the protocol version? You
> have to know how long the payload is, and then parse the message
> accordingly? I agree with Turkey Breast, this seems a bit sloppy to me.
>
> Paul
>
> P.S. I’ve never used a dev mailing list before and I want to get involved
> with the Bitcoin dev community, so let me know if I’m horribly violating
> any mailing list etiquette. 😊
>
> *From:* Mike Hearn
> *Sent:* ‎Wednesday‎, ‎June‎ ‎19‎, ‎2013 ‎7‎:‎43‎ ‎AM
> *To:* Turkey Breast
> *Cc:* bitcoin-development@lists.sourceforge.net
>
> Bitcoin-Qt on master does send it now although it doesn't affect anything,
> but as old pre-filtering versions will continue to exist, you'll always
> have to be able to deserialize version messages without it.
>
> Bitcoin version messages have always had variable length, look at how the
> code is written in main.cpp. If you didn't experience issues until now all
> it means is that no sufficiently old nodes were talking to yours.
>
> The standard does not say it should appear. Read it again - BIP 37 says
> about the new version message field:
> If false then broadcast transactions will not be announced until a
> filter{load,add,clear} command is received. *If missing or true*, no
> change in protocol behaviour occurs.
>
>
> On Wed, Jun 19, 2013 at 12:33 PM, Turkey Breast wrote:
>
>> It's a problem if you work with iterators to deserialize the byte stream.
>> Even failing that, it's just sloppy programming. What happens in the future
>> when new fields are added to the version message? It's not a big deal to
>> say that this protocol version has X number of fields, that (higher)
>> protocol version message has X + N number of fields. Deterministic number
>> of fields per protocol version is sensical and how Bitcoin has been for a
>> long time.
>>
>> And yes, it was a problem for me that caused a lot of confusion why this
>> byte didn't exist in many version messages despite the standard saying it
>> should and the code in bitcoind indicating it should. Nowhere was this
>> written. It doesn't help other implementations to have an unclear behaviour
>> that depends on some magic from one implementation.
>>
>>   ----------
>>  *From:* Mike Hearn 
>> *To:* Turkey Breast 
>> *Cc:* "bitcoin-development@lists.sourceforge.net" <
>> bitcoin-development@lists.sourceforge.net>
>> *Sent:* Wednesday, June 19, 2013 11:39 AM
>>
>> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version
>> message
>>
>> It has to be optional because old clients don't send it, obviously.
>>
>> Why is this even an issue? There's no problem with variable length
>> messages in any codebase that I'm aware of. Is this solving some actual
>> problem?
>>
>>
>> On Wed, Jun 19, 2013 at 12:30 AM, Turkey Breast 
>> wrote:
>>
>> That's me. I never said to make all messages fixed length. I said to make
>> a fixed number of fields per protocol. So given a protocol version number,
>> you know the number of fields in a message. This is not only easier for
>> parsing messages, but just good practice. I don't see why a 1 byte flag
>> needs to be optional anyway.
>>
>>   --
>>  *From:* Mike Hearn 
>> *To:* Turkey Breast 
>> *Cc:* Bitcoin Dev 
>> *Sent:* Tuesday, June 18, 2013 9:48 PM
>> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version
>> message
>>
>> It's not a bug (although there was recently a change to make bitcoind/qt
>> always send this field anyway).
>>
>> I don't know where Amir is going with BIP 60. Version messages have
>> always been variable length. There's nothing inherent in the Bitcoin
>> protocol that says all messages are fixed length, indeed, tx messages are
>> allowed to have arbitrary data appended after them that gets relayed.
>>
>>
>> On Tue, Jun 18, 2013 at 7:45 PM, Turkey Breast wrote:
>>
>> See this BIP. I'm not s

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Mike Hearn]

Bitcoin-Qt on master does send it now although it doesn't affect anything,
but as old pre-filtering versions will continue to exist, you'll always
have to be able to deserialize version messages without it.

Bitcoin version messages have always had variable length, look at how the
code is written in main.cpp. If you didn't experience issues until now all
it means is that no sufficiently old nodes were talking to yours.

The standard does not say it should appear. Read it again - BIP 37 says
about the new version message field:
If false then broadcast transactions will not be announced until a
filter{load,add,clear} command is received. *If missing or true*, no change
in protocol behaviour occurs.


On Wed, Jun 19, 2013 at 12:33 PM, Turkey Breast wrote:

> It's a problem if you work with iterators to deserialize the byte stream.
> Even failing that, it's just sloppy programming. What happens in the future
> when new fields are added to the version message? It's not a big deal to
> say that this protocol version has X number of fields, that (higher)
> protocol version message has X + N number of fields. Deterministic number
> of fields per protocol version is sensical and how Bitcoin has been for a
> long time.
>
> And yes, it was a problem for me that caused a lot of confusion why this
> byte didn't exist in many version messages despite the standard saying it
> should and the code in bitcoind indicating it should. Nowhere was this
> written. It doesn't help other implementations to have an unclear behaviour
> that depends on some magic from one implementation.
>
>   --
>  *From:* Mike Hearn 
> *To:* Turkey Breast 
> *Cc:* "bitcoin-development@lists.sourceforge.net" <
> bitcoin-development@lists.sourceforge.net>
> *Sent:* Wednesday, June 19, 2013 11:39 AM
>
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version message
>
> It has to be optional because old clients don't send it, obviously.
>
> Why is this even an issue? There's no problem with variable length
> messages in any codebase that I'm aware of. Is this solving some actual
> problem?
>
>
> On Wed, Jun 19, 2013 at 12:30 AM, Turkey Breast wrote:
>
> That's me. I never said to make all messages fixed length. I said to make
> a fixed number of fields per protocol. So given a protocol version number,
> you know the number of fields in a message. This is not only easier for
> parsing messages, but just good practice. I don't see why a 1 byte flag
> needs to be optional anyway.
>
>   ------
>  *From:* Mike Hearn 
> *To:* Turkey Breast 
> *Cc:* Bitcoin Dev 
> *Sent:* Tuesday, June 18, 2013 9:48 PM
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version message
>
> It's not a bug (although there was recently a change to make bitcoind/qt
> always send this field anyway).
>
> I don't know where Amir is going with BIP 60. Version messages have always
> been variable length. There's nothing inherent in the Bitcoin protocol that
> says all messages are fixed length, indeed, tx messages are allowed to have
> arbitrary data appended after them that gets relayed.
>
>
> On Tue, Jun 18, 2013 at 7:45 PM, Turkey Breast wrote:
>
> See this BIP. I'm not sure if this is a bug or what, but it would be good
> if messages always had a fixed number of fields per protocol version.
>
> https://en.bitcoin.it/wiki/BIP_0060#Code_Updates
>
> This BIP details everything that needs to be done and proposes a protocol
> upgrade.
>
>
> --
> This SF.net <http://sf.net/> email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
>
>
>
> --
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
>
>
>
> --
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> ___
> Bitcoin-de

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Turkey Breast]

It's a problem if you work with iterators to deserialize the byte stream. Even 
failing that, it's just sloppy programming. What happens in the future when new 
fields are added to the version message? It's not a big deal to say that this 
protocol version has X number of fields, that (higher) protocol version message 
has X + N number of fields. Deterministic number of fields per protocol version 
is sensical and how Bitcoin has been for a long time.

And yes, it was a problem for me that caused a lot of confusion why this byte 
didn't exist in many version messages despite the standard saying it should and 
the code in bitcoind indicating it should. Nowhere was this written. It doesn't 
help other implementations to have an unclear behaviour that depends on some 
magic from one implementation.




 From: Mike Hearn 
To: Turkey Breast  
Cc: "bitcoin-development@lists.sourceforge.net" 
 
Sent: Wednesday, June 19, 2013 11:39 AM
Subject: Re: [Bitcoin-development] Missing fRelayTxes in version message
 


It has to be optional because old clients don't send it, obviously.

Why is this even an issue? There's no problem with variable length messages in 
any codebase that I'm aware of. Is this solving some actual problem?



On Wed, Jun 19, 2013 at 12:30 AM, Turkey Breast  wrote:

That's me. I never said to make all messages fixed length. I said to make a 
fixed number of fields per protocol. So given a protocol version number, you 
know the number of fields in a message. This is not only easier for parsing 
messages, but just good practice. I don't see why a 1 byte flag needs to be 
optional anyway.
>
>
>
>
>
> From: Mike Hearn 
>To: Turkey Breast  
>Cc: Bitcoin Dev  
>Sent: Tuesday, June 18, 2013 9:48 PM
>Subject: Re: [Bitcoin-development] Missing fRelayTxes in version message
> 
>
>
>It's not a bug (although there was recently a change to make bitcoind/qt 
>always send this field anyway). 
>
>
>I don't know where Amir is going with BIP 60. Version messages have always 
>been variable length. There's nothing inherent in the Bitcoin protocol that 
>says all messages are fixed length, indeed, tx messages are allowed to have 
>arbitrary data appended after them that gets relayed.
>
>
>
>On Tue, Jun 18, 2013 at 7:45 PM, Turkey Breast  wrote:
>
>See this BIP. I'm not sure if this is a bug or what, but it would be good if 
>messages always had a fixed number of fields per protocol version.
>>
>>
>>https://en.bitcoin.it/wiki/BIP_0060#Code_Updates
>>
>>
>>This BIP details everything that needs to be done and proposes a protocol 
>>upgrade.
>>
>>--
>>This SF.net email is sponsored by Windows:
>>
>>Build for Windows Store.
>>
>>http://p.sf.net/sfu/windows-dev2dev
>>___
>>Bitcoin-development mailing list
>>Bitcoin-development@lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>>
>
>
>
>--
>This SF.net email is sponsored by Windows:
>
>Build for Windows Store.
>
>http://p.sf.net/sfu/windows-dev2dev
>___
>Bitcoin-development mailing list
>Bitcoin-development@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Mike Hearn]

It has to be optional because old clients don't send it, obviously.

Why is this even an issue? There's no problem with variable length messages
in any codebase that I'm aware of. Is this solving some actual problem?


On Wed, Jun 19, 2013 at 12:30 AM, Turkey Breast wrote:

> That's me. I never said to make all messages fixed length. I said to make
> a fixed number of fields per protocol. So given a protocol version number,
> you know the number of fields in a message. This is not only easier for
> parsing messages, but just good practice. I don't see why a 1 byte flag
> needs to be optional anyway.
>
>   --
>  *From:* Mike Hearn 
> *To:* Turkey Breast 
> *Cc:* Bitcoin Dev 
> *Sent:* Tuesday, June 18, 2013 9:48 PM
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version message
>
> It's not a bug (although there was recently a change to make bitcoind/qt
> always send this field anyway).
>
> I don't know where Amir is going with BIP 60. Version messages have always
> been variable length. There's nothing inherent in the Bitcoin protocol that
> says all messages are fixed length, indeed, tx messages are allowed to have
> arbitrary data appended after them that gets relayed.
>
>
> On Tue, Jun 18, 2013 at 7:45 PM, Turkey Breast wrote:
>
> See this BIP. I'm not sure if this is a bug or what, but it would be good
> if messages always had a fixed number of fields per protocol version.
>
> https://en.bitcoin.it/wiki/BIP_0060#Code_Updates
>
> This BIP details everything that needs to be done and proposes a protocol
> upgrade.
>
>
> --
> This SF.net <http://sf.net/> email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
>
>
>
> --
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Turkey Breast]

That's me. I never said to make all messages fixed length. I said to make a 
fixed number of fields per protocol. So given a protocol version number, you 
know the number of fields in a message. This is not only easier for parsing 
messages, but just good practice. I don't see why a 1 byte flag needs to be 
optional anyway.




 From: Mike Hearn 
To: Turkey Breast  
Cc: Bitcoin Dev  
Sent: Tuesday, June 18, 2013 9:48 PM
Subject: Re: [Bitcoin-development] Missing fRelayTxes in version message
 


It's not a bug (although there was recently a change to make bitcoind/qt always 
send this field anyway). 

I don't know where Amir is going with BIP 60. Version messages have always been 
variable length. There's nothing inherent in the Bitcoin protocol that says all 
messages are fixed length, indeed, tx messages are allowed to have arbitrary 
data appended after them that gets relayed.



On Tue, Jun 18, 2013 at 7:45 PM, Turkey Breast  wrote:

See this BIP. I'm not sure if this is a bug or what, but it would be good if 
messages always had a fixed number of fields per protocol version.
>
>
>https://en.bitcoin.it/wiki/BIP_0060#Code_Updates
>
>
>This BIP details everything that needs to be done and proposes a protocol 
>upgrade.
>
>--
>This SF.net email is sponsored by Windows:
>
>Build for Windows Store.
>
>http://p.sf.net/sfu/windows-dev2dev
>___
>Bitcoin-development mailing list
>Bitcoin-development@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Missing fRelayTxes in version message

[Mike Hearn]

It's not a bug (although there was recently a change to make bitcoind/qt
always send this field anyway).

I don't know where Amir is going with BIP 60. Version messages have always
been variable length. There's nothing inherent in the Bitcoin protocol that
says all messages are fixed length, indeed, tx messages are allowed to have
arbitrary data appended after them that gets relayed.


On Tue, Jun 18, 2013 at 7:45 PM, Turkey Breast wrote:

> See this BIP. I'm not sure if this is a bug or what, but it would be good
> if messages always had a fixed number of fields per protocol version.
>
> https://en.bitcoin.it/wiki/BIP_0060#Code_Updates
>
> This BIP details everything that needs to be done and proposes a protocol
> upgrade.
>
>
> --
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> ___
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[Bitcoin-development] Missing fRelayTxes in version message

[Turkey Breast]

See this BIP. I'm not sure if this is a bug or what, but it would be good if 
messages always had a fixed number of fields per protocol version.

https://en.bitcoin.it/wiki/BIP_0060#Code_Updates

This BIP details everything that needs to be done and proposes a protocol 
upgrade.
--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev___
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development