Re: [bitcoin-dev] Torrent-style new-block propagation on Merkle trees

[Tier Nolan via bitcoin-dev]

On Thu, Sep 24, 2015 at 12:12 AM, Jonathan Toomim (Toomim Bros) via
bitcoin-dev  wrote:

>
>
> As I understand it, the current block propagation algorithm is this:
>
> 1. A node mines a block.
> 2. It notifies its peers that it has a new block with an *inv*. Typical
> nodes have 8 peers.
> 3. The peers respond that they have not seen it, and request the block
> with *getdata* [hash].
> 4. The node sends out the block in parallel to all 8 peers simultaneously.
> If the node's upstream bandwidth is limiting, then all peers will receive
> most of the block before any peer receives all of the block. The block is
> sent out as the small header followed by a list of transactions.
> 5. Once a peer completes the download, it verifies the block, then enters
> step 2.
>

Mining pools currently connect to the "fast relay network".  This is
optimised for fast block distribution.  It does no validation and is only
for low latency propagation.  The normal network is used as a fallback.

My understanding is that it works as follows:

Each miner runs a normal full node and a relay node on the same computer.

The full node tells the relay node whenever it receives a new transaction
via the inv message and the node requests the full transaction.

The relay node tells its relay peers that it knows about the transaction
(hash only) and its 4 byte key. This is not forwarded onwards, since the
relay peer only gets the hash of the transaction and doesn't do validation
anyway.  The key is just a 4 byte counter.

Each relay node keeps a mapping of txid to key for each of its peer.  There
is some garbage collection and entries are removed once the transaction is
included in a block (there might be a confirm threshold).

When a block is found, the local node sends it to the relay node.  The
relay node then forwards it to all of its peers in a compact form.

The block is sent as a list of keys for that peer and full transactions are
only sent for unknown transactions.

When a relay node receives a block, it just verifies the POW, checks that
it is new and recent.  It does not do tx validation.  It forwards the block
to its local full node, which does the validation.  Since the relay node is
on localhost, it never gets kicked due to sending invalid blocks.  This
prevents a DOS attack where you could send invalid blocks to the relay node
and cause the local full node to kick it.

If all the transactions are already known, then it can forward a block for
only 4 bytes per transactions.  I think it has an optimisation, so that is
compressed to 1 byte per tx.
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[bitcoin-dev] Bitcoin Core 0.12.0 release schedule

[Wladimir J. van der Laan via bitcoin-dev]

Hello all,

The next major release of Bitcoin Core, 0.12.0 is planned for the end of the 
year. Let's propose a more detailed schedule:

2015-11-01
---
- Open Transifex translations for 0.12
- Soft translation string freeze (no large or unnecessary changes)
- Finalize and close translation for 0.10

2015-12-01
---
- Feature freeze
- Translation string freeze

In December at least I will probably not get much done code-wise (Scaling 
Bitcoin Hongkong, 32C3, end of year festivities, etc), and I'm sure I'm not the 
only one, so let's leave that for last pre-RC bugfixes and polishing.

2016-01-06
---
- Split off `0.12` branch from `master`
- Start RC cycle, tag and release `0.12.0rc1`
- Start merging for 0.13 on master branch

2016-02-01
---
- Release 0.12.0 final (aim)

Wladimir


___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] CI Build for Bitcoin - Some Basic Questions about Gitian and other stuff

[Jonas Schnelli via bitcoin-dev]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi Roy

> So who physically manually uploads the gitian build to bitcoin.org 
> ?

It's done by the core devs (very likely the project maintainer).
I think it doesn't matter who did the upload to bitcoin.org.

What really would matter is – if users deciding to run a pre-compiled
version of bitcoin-core – that they verify the binary against the
available gitian sigs.

Signatur repository:
https://github.com/bitcoin/gitian.sigs

PGP Pubkeys (mostly also available over gpg key servers):
https://github.com/bitcoin/bitcoin/tree/master/contrib/gitian-downloader

Instruction available here:
https://www.reddit.com/r/Bitcoin/wiki/verifying_bitcoin_core


/jonas
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJWA7/YAAoJECnUvLZBb1PspoEQAKDs672Vn/EFEn02oYbo41Xi
tiHLpZMePiL1GV+XAQrqA91x5Q4RY2oJBmBzq2Bnllr+fm5zodcTP80lDyYeizb7
+2MPuoF8ICMYxUDJ3tz163Y4hHforFyMMHgr/NPXQcsMEEnEuAQIMxRFMqexhkn9
W/YVT2ow/5illYmZ9EAoSreaD+1ShVTxkZY2ltY79ZATTcVU85mEXau9Jv6qq4sN
jHCWAJYrsFQay4t42Pp4ciclz3A7W//OoFhwxUR2sNcDZ6NkOSnYOZSwMPM4uBAL
z1OpSWVOSQZeGXFOzEF07oyLaCVKm6ygND4IT7eFcvlrgXMIYE0bvzfqPymBArRJ
ZSKR0bmTkUsuw3NCcV23HF8yW+G/Y/SKPRl5n1AmmXppzklSSojDzDT3h2xKGNTd
DYoIQhwsNcm7sLlmHZa+VZ7peeCzdQ9z+OTG1ZDwDsFRBDY9kTbP2pD1tHEkEvxR
z2DJg8iP9V4ZUUFEuAdOOCInl1v+0RJu8DxWa6BZmLYJ/SVuKqSEZ0HYBqPOJOUO
4ct+NLyY7fNQurq0VzIjpZN9/L+SEePSIP4bFwOiGBJosUEHyS3VZ7JZZMOs5jxx
1GDI964Pi54Z/XPzV2+X1GjWLeReJ6WUznX7zv/LNER2yI6XQBw2w5/rmomI+ifl
vk5pfinRCIP3t6q+K0ge
=vP2n
-END PGP SIGNATURE-
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Weekly development meetings on IRC: schedule

[Micha Bailey via bitcoin-dev]

That's suboptimal for Europe etc., starting at midnight in the UK, 1 AM in
CET, 2 AM in EET (an hour earlier once DST ends).

On Wednesday, September 23, 2015, Vincent Truong via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> All,
>
> Current meeting time visualised globally.
>
> http://everytimezone.com/#2015-9-24,420,4ia
>
> jl,
>
> I think I found a good compromise: if the US want to accommodate Asia and
> willing to sacrifice preference, 23:00 to 00:00 UTC might work.
>
> http://everytimezone.com/#2015-9-24,660,4ia
>
> It isn't easy to grab everyone's preference and accommodate everyone, so
> this might work in theory but people may not be free to show up. US should
> be ok. UK can participate or catch some nice z. Asia will need a bit of
> early bird time but it's not as crazy as 3am. AU also fits in there nicely.
>
> A meeting like this once a month should be enough probably (say, do this
> on the first week of the month, and run every other week on the main
> schedule). But I don't know whether there are enough people in Asia/AU to
> make it worth it. Asia/AU people, thoughts?
> On Sep 23, 2015 9:01 PM, "jl2012 via bitcoin-dev" <
> bitcoin-dev@lists.linuxfoundation.org
> >
> wrote:
>
>> There could not be a worse timing than this for those in China (3-4am),
>> Japan/Korea (4-5am), and Australia (3-6am depends on which part of the
>> country). Maybe we have no dev in this part of the planet? Is there any
>> chance to review the timing in a weekly or monthly basis (also with a
>> doodle vote?)
>>
>> Will there be any agenda published before the meetings? If I'm really
>> interested in the topics, I'll have some reasons to get up in the middle of
>> the night.
>>
>> Wladimir J. van der Laan via bitcoin-dev 於 2015-09-22 10:36 寫到:
>>
>>> Hello,
>>>
>>> There was overwhelming response that weekly IRC meetings are a good
>>> thing.
>>>
>>> Thanks to the doodle site we were able to select a time slot that
>>> everyone (that voted) is available:
>>>
>>> Thursday 19:00-20:00 UTC, every week, starting September 24 (next
>>> Thursday)
>>>
>>> I created a shared Google Calendar here:
>>>
>>> https://www.google.com/calendar/embed?src=MTFwcXZkZ3BkOTlubGliZjliYTg2MXZ1OHNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ
>>>
>>> The timezone of this calendar is Reykyavik (Iceland) which is UTC+0.
>>> However, you can use the button on the lower right to add the calendar
>>> to your own calendar, which will then show the meeting in your own
>>> timezone.
>>>
>>> See you then,
>>>
>>> Wladimir
>>>
>>> ___
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> 
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>
>> ___
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> 
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Suhas Daftuar via bitcoin-dev]

Hi,

I'm proposing the addition of a new, optional p2p message to help improve
the way blocks are announced on the network.  The draft BIP is available
here and pasted below:
https://gist.github.com/sdaftuar/465bf008f0a4768c0def

The goal of this p2p message is to facilitate nodes being able to
optionally announce blocks with headers messages rather than with inv's,
which is particularly beneficial since the introduction of headers-first
download in Bitcoin Core 0.10.  In particular, this allows for more
efficient propagation of reorgs as it would eliminate a round trip in
network communication.

The implementation of this BIP (which includes code to directly fetch
blocks based on announced headers) is in
https://github.com/bitcoin/bitcoin/pull/6494.  For additional background,
please also see https://github.com/bitcoin/bitcoin/issues/5982.

Note that this new p2p message is optional; nodes can feel free to ignore
and continue to use inv messages to announce new blocks.

Thanks to Pieter Wuille for suggesting this idea.

Draft BIP text:


  BIP: 
  Title: sendheaders message
  Author: Suhas Daftuar 
  Status: Draft
  Type: Standards Track
  Created: 2015-05-08


==Abstract==

Add a new message, "sendheaders", which indicates that a node prefers to
receive new block announcements via a "headers" message rather than an
"inv".

==Motivation==

Since the introduction of "headers-first" downloading of blocks in 0.10,
blocks will not be processed unless
they are able to connect to a (valid) headers chain.  Consequently, block
relay generally works
as follows:
# A node (N) announces the new tip with an "inv" message, containing the
block hash
# A peer (P) responds to the "inv" with a "getheaders" message (to request
headers up to the new tip) and a "getdata" message for the new tip itself
# N responds with a "headers" message (with the header for the new block
along with any preceding headers unknown to P) and a "block" message
containing the new block

However, in the case where a new block is being announced that builds on
the tip, it would be generally more efficient if the node N just announced
the block header for the new block, rather than just the block hash, and
saved the peer from generating and transmitting the getheaders message (and
the required block locator).

In the case of a reorg, where 1 or more blocks are disconnected, nodes
currently just send an "inv" for the new tip.  Peers currently are able to
request the new tip immediately, but wait until the headers for the
intermediate blocks are delivered before requesting those blocks.  By
announcing headers from the last fork point leading up to the new tip in
the block announcement, peers are able to request all the intermediate
blocks immediately.

==Specification==

# The sendheaders message is defined as an empty message where pchCommand
== "sendheaders"
# Upon receipt of a "sendheaders" message, the node will be permitted, but
not required, to announce new blocks by sending the header of the new block
(along with any other blocks that a node believes a peer might need in
order for the block to connect).
# Feature discovery is enabled by checking protocol version >= 70012

==Backward compatibility==

Older clients remain fully compatible and interoperable after this change.

==Implementation==

https://github.com/bitcoin/bitcoin/pull/6494
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Peter Todd via bitcoin-dev]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512



On 24 September 2015 14:56:23 GMT-04:00, Suhas Daftuar  
wrote:
>I considered that as well, but it seemed to me that other software on
>the
>network (say, different wallet implementations) might prefer the option
>of
>being able to bump up their protocol version in the future to pick up
>some
>other change, without having to also opt-in to receiving
>headers-announcements for blocks.
>
>In particular, inv-based block announcements aren't going away (even in
>my
>implementation of headers announcements, there are some edge cases
>where
>the code would need to fall back to an inv announcement), so forcing
>all
>software on the network to upgrade to supporting headers announcements,
>whether now or in the future, seems too drastic -- I could imagine some
>software not being very concerned about optimizing block relay in this
>particular way.

Block headers are so small - 80 bytes - that it may be reasonable to just stop 
using the inv mechanism for them in favor of always sending headers. IIRC a inv 
is 32 bytes of digest and another four bytes or something of the inv string 
itself - that's already nearly half of the header.

Meanwhile reducing the amount of state in the protocol does have some value, 
and decreasing overall latency for headers to get around the network certainely 
isnt a bad thing.
-BEGIN PGP SIGNATURE-

iQE9BAEBCgAnIBxQZXRlciBUb2RkIDxwZXRlQHBldGVydG9kZC5vcmc+BQJWBE60
AAoJEMCF8hzn9Lncz4MH/jybITh0VWtf+2MotWZOdMIiQtmWZ6Ly2yiDXwi3atu+
MEA6yx9vPFV8P1ZKIZzVtr/4Iu3gBHBdDxAzQW0SjreTLdzZ1+d28/A2kYD4+es7
MFD8rDV/kPtnu8ajMkS9bfmrU0WfkgSSB2fUheT+kqgH/ejIJBISo8BpQZbz7f4B
M+D+hoNadcqWcZZKBHT+o5o7v3jJwxh8qpJgMMZrtN/QfFJK5UVdU4I/hEd89XP9
XD/y29ykWAFQPDdBKMGIUj1csUGlyS5kFXp6ZLVtAZWHIgfZ1R/qOhIUcRwRxZjc
JXZEWrMGTIXr2zkX9mtLzfjAzDc6ZULoEAHCV3sVa0M=
=SLUT
-END PGP SIGNATURE-

___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Suhas Daftuar via bitcoin-dev]

I considered that as well, but it seemed to me that other software on the
network (say, different wallet implementations) might prefer the option of
being able to bump up their protocol version in the future to pick up some
other change, without having to also opt-in to receiving
headers-announcements for blocks.

In particular, inv-based block announcements aren't going away (even in my
implementation of headers announcements, there are some edge cases where
the code would need to fall back to an inv announcement), so forcing all
software on the network to upgrade to supporting headers announcements,
whether now or in the future, seems too drastic -- I could imagine some
software not being very concerned about optimizing block relay in this
particular way.

On Thu, Sep 24, 2015 at 2:41 PM, Peter Todd  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
>
>
> On 24 September 2015 14:37:40 GMT-04:00, Suhas Daftuar via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
> >On Thu, Sep 24, 2015 at 2:17 PM, Tier Nolan via bitcoin-dev <
> >bitcoin-dev@lists.linuxfoundation.org> wrote:
> >
> >>
> >> Is there actually a requirement for the new message?  New nodes could
> >just
> >> unilaterally switch to sending headers and current nodes would be
> >> compatible.
> >>
> >
> >I don't believe that unilaterally switching to headers announcements
> >would
> >work for all network participants -- both for users running older
> >Bitcoin
> >Core versions (anything before 0.10, which I believe all ignore headers
> >messages) and for non-Bitcoin Core software that participates on the
> >network (which may ignore headers messages too, I'm not sure what all
> >is
> >out there).
>
> You can enable the behaviour based on advertised p2p network version.
> -BEGIN PGP SIGNATURE-
>
> iQE9BAEBCgAnIBxQZXRlciBUb2RkIDxwZXRlQHBldGVydG9kZC5vcmc+BQJWBEO5
> AAoJEMCF8hzn9Lncz4MH/3ztGWdFvMWWcwQsjIRH+eP6PH57WaEru1smmFYOmKrj
> djdiRVdxfChxRqP3adO21RUKKchjl8DNjrFJHPFz75FSM0cDcD0QAGAHilVdnICE
> LEIlTEoiIc0f1z9f/EJHSHPhiUXMnjpl/l7PYJFZV3Lt2Bl30yLsNnrp9qxjR30n
> 3nykZjyRad4JSavdTP6Evd3qaqwGXNUWsdObXNI+WPKlrw6hczlhFDKQ7RC1FPQU
> Rbgb21pavtqLUTwbBZGUisAAc94e2Gama1p3ioUFklbVtLTdw+FtxPgV/0ZS75OR
> V9pCXIbg9VM6QY4+9gYnP635+qCkqAJ4tBsYGmsT8yA=
> =cF4B
> -END PGP SIGNATURE-
>
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Tier Nolan via bitcoin-dev]

On Thu, Sep 24, 2015 at 7:02 PM, Suhas Daftuar via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi,
>
> I'm proposing the addition of a new, optional p2p message to help improve
> the way blocks are announced on the network.  The draft BIP is available
> here and pasted below:
> https://gist.github.com/sdaftuar/465bf008f0a4768c0def
>
> The goal of this p2p message is to facilitate nodes being able to
> optionally announce blocks with headers messages rather than with inv's,
> which is particularly beneficial since the introduction of headers-first
> download in Bitcoin Core 0.10.  In particular, this allows for more
> efficient propagation of reorgs as it would eliminate a round trip in
> network communication.
>

Is there actually a requirement for the new message?  New nodes could just
unilaterally switch to sending headers and current nodes would be
compatible.

It looks like the only DOS misbehaving penalty is if the header is invalid
or if the headers don't form a chain.
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] [BIP Proposal] New "sendheaders" p2p message

[Peter Todd via bitcoin-dev]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512



On 24 September 2015 14:37:40 GMT-04:00, Suhas Daftuar via bitcoin-dev 
 wrote:
>On Thu, Sep 24, 2015 at 2:17 PM, Tier Nolan via bitcoin-dev <
>bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>>
>> Is there actually a requirement for the new message?  New nodes could
>just
>> unilaterally switch to sending headers and current nodes would be
>> compatible.
>>
>
>I don't believe that unilaterally switching to headers announcements
>would
>work for all network participants -- both for users running older
>Bitcoin
>Core versions (anything before 0.10, which I believe all ignore headers
>messages) and for non-Bitcoin Core software that participates on the
>network (which may ignore headers messages too, I'm not sure what all
>is
>out there).

You can enable the behaviour based on advertised p2p network version.
-BEGIN PGP SIGNATURE-

iQE9BAEBCgAnIBxQZXRlciBUb2RkIDxwZXRlQHBldGVydG9kZC5vcmc+BQJWBEO5
AAoJEMCF8hzn9Lncz4MH/3ztGWdFvMWWcwQsjIRH+eP6PH57WaEru1smmFYOmKrj
djdiRVdxfChxRqP3adO21RUKKchjl8DNjrFJHPFz75FSM0cDcD0QAGAHilVdnICE
LEIlTEoiIc0f1z9f/EJHSHPhiUXMnjpl/l7PYJFZV3Lt2Bl30yLsNnrp9qxjR30n
3nykZjyRad4JSavdTP6Evd3qaqwGXNUWsdObXNI+WPKlrw6hczlhFDKQ7RC1FPQU
Rbgb21pavtqLUTwbBZGUisAAc94e2Gama1p3ioUFklbVtLTdw+FtxPgV/0ZS75OR
V9pCXIbg9VM6QY4+9gYnP635+qCkqAJ4tBsYGmsT8yA=
=cF4B
-END PGP SIGNATURE-

___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[bitcoin-dev] 2015-09-24 #bitcoin-dev Weekly Development Meeting Minutes

[Daniel Stadulis via bitcoin-dev]

If you weren't able to attend the first, weekly development meeting, the
following are the minutes:

Meeting Title:

#bitcoin-dev Weekly Development Meeting

Meeting Date:

2015-09-24

Meeting Time:

19:00-20:00 UTC

Participants in Attendance:

luke-jr

CodeShark

sipa

morcos

sdaftuar

dstadulis

jtimon

wumpus

jgarzik

kanzure

gmaxwell

cfields

gavinandresen

IRC Chat Logs:

http://bitcoinstats.com/irc/bitcoin-dev/logs/2015/09/24#l1443121200.0

--

Topics Discussed:


   1.

   libconsensus and refactoring
   2.

   All goals for 0.12 release
   1.

  libsecp256k1 is ready for 0.12?
  1.

 libsecp256k1 needs a native OSX travis build
 1.

cfields has work that moves to the new Travis infrastructure
2.

PROPOSAL: propose libsecp256k1 validation PR as soon as all
currently-in-pipeline API changes are merged
2.

  OP_CHECKSEQUENCEVERIFY
  3.

  mempool limiting
  4.

  version bits
  3.

   BIP process
   4.

   Split off script base classes/execution for use in consensus?
   5.

   Current/near-term “what are you working on”
   1.

  versionbits: Codeshark has been working on an implementation
  2.

  gavinandresen: simple benchmarking framework then plan on optimizing
  new block relay/broadcast.

--

Meeting Conclusions:

Mempool limiting discussion will be delayed until 2015-10-1 meeting

#

Action items

Responsible Parties

ETA

1

Please review 6557 (starting Saturday), 6673 and any other mempool pulls
for concept

Everyone

Next Thurs. meeting (2015-10-01)

2

libsecp256k1 needs a native OSX travis build


3

Propose libsecp256k1 validation PR as soon as all currently-in-pipeline API
changes are merged


4

Review BIP 68, review #6312, #6564


5

versionbits BIP number assignment

gmaxwell

2015-09-25

Google Doc:
https://docs.google.com/document/d/1zsWVaf5H9ECrN1zPutMdD_2ky3fnhQUM411NDrRrc-M/edit
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev