Re: [Lightning-dev] Direct Message draft

2020-02-23 Thread Rusty Russell
Christian Decker  writes:
> Rusty Russell  writes:
>
>> I like it!  The lack of "reply" function eliminates all storage
>> requirements for the intermediaries.  Unfortunately it's not currently
>> possible to fit the reply onion inside the existing onion, but I know
>> Christian has a rabbit in his hat for this?
>
> I think circular payment really means an onion that is
>
>> A -> ... -> B -> ... -> A
>
> and not a reply onion inside of a forward onion.
>
> The problem with the circular path is that the "recipient" cannot add
> any reply without invalidating the HMACs on the return leg of the
> onion. The onion is fully predetermined by the sender, any malleability
> introduced in order to allow the recipient to reply poses a threat to
> the integrity of the onion routing, e.g., it opens us up to probing by
> fiddling with parts of the onion until the attacker identifies the
> location the recipient is supposed to put his reply into.
>
> As Rusty mentioned I have a construction of the onion routing packet
> that allows us to compress it in such a way that it fits inside of the
> payload itself.

I think this has the same problem though, that there's no way Alice can
send Bob an onion to use with an arbitrary message?

> Another advantage is that the end-to-end payload is not covered by the
> HMACs in the header, meaning that the recipient can construct a reply
> without having to modify (and invalidate) the routing onion. I guess
> this is going back to the roots of the Sphinx paper :-)

Good point, and it's trivial.  The paper suggests the payload be "final
key" followed by the desired data, providing a simple validation scheme.

We could potentially generalize the HTLC messages like this, but it's
unnecessary at this point.

Thanks,
Rusty.
___
Lightning-dev mailing list
Lightning-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev


Re: [Lightning-dev] Direct Message draft

2020-02-23 Thread ZmnSCPxj via Lightning-dev
Good morning Christian, and Rusty,

> > > Would it not be better to create a circular path? By this I mean,
> > > Alice constructs an onion that overall creates a path from herself to
> > > Bob and back, ensuring different nodes on the forward and return
> > > directions. The onion hop at Bob reveals that Bob is the chosen
> > > conversation partner, and Bob forwards its reply via the onion return
> > > path (that Alice prepared herself to get back to her via another
> > > path).
> >
> > I like it! The lack of "reply" function eliminates all storage
> > requirements for the intermediaries. Unfortunately it's not currently
> > possible to fit the reply onion inside the existing onion, but I know
> > Christian has a rabbit in his hat for this?
>
> I think circular payment really means an onion that is
>
> > A -> ... -> B -> ... -> A
>
> and not a reply onion inside of a forward onion.
>
> The problem with the circular path is that the "recipient" cannot add
> any reply without invalidating the HMACs on the return leg of the
> onion. The onion is fully predetermined by the sender, any malleability
> introduced in order to allow the recipient to reply poses a threat to
> the integrity of the onion routing, e.g., it opens us up to probing by
> fiddling with parts of the onion until the attacker identifies the
> location the recipient is supposed to put his reply into.

At the risk of constructing a novel cryptosystem, I think we can separate the 
request/response from the onion.
We effectively treat the onion as establishing a *non*-encrypted temporary 
tunnel, and add an asymmetric encryption between Alice and Bob for the request 
and response.

The onion is kept as-is (except without information about HTLC amounts and 
timelocks).
We add a field *uotside* the onion which contains the request/response.
At each hop, we derive a key from the shared secret between the ephemeral 
keypair and the hop keypair, and use the derived key to generate a symmetric 
stream cipher to encrypt the separated request/response.

Alice first creates a *separate* ephemeral key just for communication with Bob.
It encrypts the request using this level 2 ephemeral key and adds a MAC tag as 
well, and treat the encrypted request plus the MAC tag as a binary blob.
Then, for each hop, it derives the symmetric stream cipher using the onion 
(level 1) ephemeral key with that hop and applies the stream cipher on it.
Then it sends out the completed onion plus this encrypted request/response blob.

Each hop effectively peels a layer of encryption, because it is a symmetric 
stream cipher.
On reaching Bob, the encrypted request/response message plus MAC tag is 
revealed to Bob.
Bob learns it is the true destination by some TLV in the onion part, including 
the (level 2) ephemeral key, then validates the MAC using the Alice and Bob 
shared secret and if it is valid, decrypts the request part and processes the 
request.

Bob then generates its reply, and encrypts the reply with the shared secret 
between its static key and the level 2 ephemeral key, then creates a MAC using 
the level 2 ephemeral key and its static key.
Then it sends it together with the rest of the onion onward.

Each hop after Bob effectively adds a layer of encryption, because it is a 
symmetric stream cipher.
On reaching Alice, Alice (who knows the entire route, since it was the one who 
established the route) can peel every layer between Bob and Alice on the return 
route.
Then it should get the binary blob that is the (level 2) encryption of the 
reply plus a MAC, validates the MAC, then decrypts the reply.

We treat the tunnel as unencrypted (i.e. we have a level 2 asymmetric 
encryption between Alice and Bob) because the request/response is outside the 
onion.
We still bother to do a multilayer encryption so that, in case of a route like 
A->I->J->K->L->B, where I and L are controlled by the same surveillor, the 
request/response is different at each hop and I and L cannot be certain they 
are on the same route (though of course timing and message size can improve 
their accuracy --- we can have a fixed size for the request/response to hide 
message size, but we can do nothing about timing).

My *vague* understanding is that HORNET is effectively a better version of this 
--- it uses a "full" onion to establish a circuit, then uses simpler symmetric 
ciphers during circuit operation.
This effectively tears down the circuit as soon as the message passes through.


>
> As Rusty mentioned I have a construction of the onion routing packet
> that allows us to compress it in such a way that it fits inside of the
> payload itself. I'll write up a complete proposal over the coming days,
> but the basic idea is to initialize the unused part of the onion in such
> a way that it cancels out the layers of encryption and the fully wrapped
> onion consists of all `0x00` bytes. These can then be removed resulting
> in a compressed onion, and the sender can simply add the padding 0x00

Re: [Lightning-dev] Direct Message draft

2020-02-21 Thread Christian Decker
Rusty Russell  writes:

>> Would it not be better to create a circular path?  By this I mean,
>> Alice constructs an onion that overall creates a path from herself to
>> Bob and back, ensuring different nodes on the forward and return
>> directions.  The onion hop at Bob reveals that Bob is the chosen
>> conversation partner, and Bob forwards its reply via the onion return
>> path (that Alice prepared herself to get back to her via another
>> path).
>
> I like it!  The lack of "reply" function eliminates all storage
> requirements for the intermediaries.  Unfortunately it's not currently
> possible to fit the reply onion inside the existing onion, but I know
> Christian has a rabbit in his hat for this?

I think circular payment really means an onion that is

> A -> ... -> B -> ... -> A

and not a reply onion inside of a forward onion.

The problem with the circular path is that the "recipient" cannot add
any reply without invalidating the HMACs on the return leg of the
onion. The onion is fully predetermined by the sender, any malleability
introduced in order to allow the recipient to reply poses a threat to
the integrity of the onion routing, e.g., it opens us up to probing by
fiddling with parts of the onion until the attacker identifies the
location the recipient is supposed to put his reply into.

As Rusty mentioned I have a construction of the onion routing packet
that allows us to compress it in such a way that it fits inside of the
payload itself. I'll write up a complete proposal over the coming days,
but the basic idea is to initialize the unused part of the onion in such
a way that it cancels out the layers of encryption and the fully wrapped
onion consists of all `0x00` bytes. These can then be removed resulting
in a compressed onion, and the sender can simply add the padding 0x00
bytes back to get the original, fully HMACd onion, and then send it like
normal (there is an obfuscation step to hide the `0x00` bytes from the
next hop, but more on this in the full rendez-vous proposal later).

This rendez-vous construction is a bit more involved since we want to
fit an onion into another onion of the same size. If we design a
completely new messaging system, requiring end-to-end communication, it
might be worth re-introducing the end-to-end payload which we removed in
the routing onion. It's a simply, variable or fixed length, payload,
that is onion-decrypted at each hop and its contents are revealed to the
destination (this is how onion routing usually works). Since that
payload doesn't have to adhere to the constraints of the routing onions
(multiple payloads, one for each hop, and no special larger payload
destined for the final recipient) this is both simpler, and would allow
us to store a full, unmodified, return-onion in the end-to-end payload.

Another advantage is that the end-to-end payload is not covered by the
HMACs in the header, meaning that the recipient can construct a reply
without having to modify (and invalidate) the routing onion. I guess
this is going back to the roots of the Sphinx paper :-)


Might be worth a consideration, as it seems to me like it'd be simpler
:-) The downside of course is that we'd end up with two different onion
constructions for different use-cases.


>> After Alice receives the first message from Bob the circular
>> "circuit" is established and they can continue to communicate using
>> the same circuit: timing attacks are now "impossible" since Alice and
>> Bob can be anywhere along the circle, even if two of the nodes in the
>> circuit are surveillors cooperating with each other, the timing
>> information they get is the distance between the surveillor nodes.
>>
>> Of course, if a node in the circular path drops the circuit is
>> disrupted, so any higher-level protocols on top of that should
>> probably be willing to resume the conversation on another circular
>> circuit.
>
> My immediate purpose for this is for "offers" which cause a invoice
> request, followed by an invoice reply.  This will probably be reused
> once for the payment itself.  2 uses is not sufficient to justify
> setting up a circuit, AFAICT.

I know someone who is itching to implement hornet for these use-cases
;-)

Cheers,
Christian
___
Lightning-dev mailing list
Lightning-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev


Re: [Lightning-dev] Direct Message draft

2020-02-21 Thread ZmnSCPxj via Lightning-dev
Good morning Rusty,

> > Would it not be better to create a circular path?
> > By this I mean, Alice constructs an onion that overall creates a path from 
> > herself to Bob and back, ensuring different nodes on the forward and return 
> > directions.
> > The onion hop at Bob reveals that Bob is the chosen conversation partner, 
> > and Bob forwards its reply via the onion return path (that Alice prepared 
> > herself to get back to her via another path).
>
> I like it! The lack of "reply" function eliminates all storage
> requirements for the intermediaries. Unfortunately it's not currently
> possible to fit the reply onion inside the existing onion, but I know
> Christian has a rabbit in his hat for this?

Why not the same onion-in-an-onion we use in trampoline?
We could probably optimize this a little since we do not need to send data like 
outgoing amounts and timelocks and stuff but basically the same construction 
seems to work?

Without the ability to send messages in the reverse direction, a failure would 
be silent.
In particular, if I send out a direct message and then not get a response after 
a while, I might be tempted to send out another direct message over a different 
path.
Then whatever was stuck gets unstuck and the recipient gets two messages (and I 
might get two responses back after a while).
This implies that any higher-level protocol built on this must treat this like 
a UDP request-response: unreliable delivery, possible multiple copies of the 
same message getting received, probably have to send multiple copies for 
reliable delivery, etc.
In particular, requests that change the world must be idempotent.

But I suppose that also applies for your original formulation as well.

>
> > After Alice receives the first message from Bob the circular "circuit" is 
> > established and they can continue to communicate using the same circuit: 
> > timing attacks are now "impossible" since Alice and Bob can be anywhere 
> > along the circle, even if two of the nodes in the circuit are surveillors 
> > cooperating with each other, the timing information they get is the 
> > distance between the surveillor nodes.
> > Of course, if a node in the circular path drops the circuit is disrupted, 
> > so any higher-level protocols on top of that should probably be willing to 
> > resume the conversation on another circular circuit.
>
> My immediate purpose for this is for "offers" which cause a invoice
> request, followed by an invoice reply. This will probably be reused
> once for the payment itself. 2 uses is not sufficient to justify
> setting up a circuit, AFAICT.
>
> > I believe I even tied this to an HTLC in an attempt to provide a
> > spam-limit as well:
> > https://lists.linuxfoundation.org/pipermail/lightning-dev/2019-November/002294.html
>
> This part was deeply unclear. Eventually we will have to charge
> up-front for forwarding HTLCs (say 5% of existing fee, plus 1msat), and
> then we could use the same scheme with lesser amounts (say, 1msat) for
> forwarding messages.

Apparently, since nobody replied to it.

>
> But I have been unable to come up with an upfront scheme which doesn't
> leak information badly.
>
> The best I can do is some hashcash scheme, combined with the ability to
> buy a single-use token to weaken it. Under load, a node would raise
> their hashcash difficulty, and you could either find another route,
> grind your onion more to meet it, or send a payment for a token from
> that node which would let your HTLC through: the preimage could even be
> the XOR of some secret you send with the HTLC, and a shachain key which
> gives you 1000 tokens, and you can use them in order, etc.

In principle any use of hashcash can be replaced with a spend of Bitcoin, hmmm.

>
> (Really want to use some kind of Chaumian token here, but it's probably
> overkill).

No, Chaumian tokens are cool, we should totally add it for nerd points.

But I mean: if it works better that way, why not?
Implementation complexity I suppose?
waxwing showed a PoDLE-based Chaumian token protocol which, at least to my 
naive eyes, is not susceptible to Wagnerian attack the way blind Schnorr 
signatures are, so it seems to be something we can use as a way to issue tokens 
in a manner very much like Chaumian banks do.
Though it does require getting the reciprocal of a scalar, an operation which 
is not exposed in secp256k1.h for some reason.
https://privacypass.github.io/protocol

>
> > Finally: what does this improve over, say, requiring that all
> > Lightning nodes have a Tor .onion address and just doing direct
> > messaging over Tor?
>
> That would be far better! But that's not happening: lnurl over https is
> happening. Using lightning to tunnel messages is a strict improvement
> over that, at least.



Regards,
ZmnSCPxj

___
Lightning-dev mailing list
Lightning-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev


Re: [Lightning-dev] Direct Message draft

2020-02-20 Thread Rusty Russell
René Pickhardt  writes:
> Hey Rusty,
>
> I was very delighted to read your proposal. But I don't see how you prevent
> message spam. If I understand you correctly you suggest that I can
> communicate to any node along a path of peer connections (not necessarily
> backed by payment channels but kind of only known through channel
> announcements of gossip) via onions and these onions which are send within
> a new gossip message are not bound to any fees or payments.

It doesn't handle spam, but OTOH it's far cheaper than the HTLC system
(which also doesn't handle spam).  I'd be delighted to add an up-front
1msat payment, as soon as we can figure out how to do that.

The non-persistent storage costs for remembering how to forward the
reply are the 200 bytes in my implementation (one pointer to the source
peer, two SHA256s, and the shared secret).

> Let's assume I just missed some spam prevention mechanism or that we can
> fix them. Do I understand the impact of your suggestion correctly that I
> could use this protocol to
>
> 1.) create a fee free rebalancing protocol? Because I could also attach a
> new lightning message inside the onions that would allow nodes without
> direct peer connection to set up a circular rebalancing path.
> 2.) have the ability to communicate with nodes further away than just my
> peers - for example to exchange information for pathfinding and / or
> autopilots?

Indeed.  I haven't prevented it, precisely because we *can't*.  This
proposal merely gives a more efficient method than encoding via HTLCs.

Cheers,
Rusty.
___
Lightning-dev mailing list
Lightning-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev


Re: [Lightning-dev] Direct Message draft

2020-02-20 Thread Rusty Russell
ZmnSCPxj  writes:
> Good morning Rusty,
>
> A concern I have brought up in the past is that if this is more than just a 
> single request-response, i.e. if this is a conversation where Alice sends to 
> Bob, Bob sends back to Alice, Alice sends back to Bob, and so on, then if the 
> same path is used each time Alice sends to Bob, the timing from Bob response 
> to Alice to the next Alice sends to Bob can help an intermediate node guess 
> how far away Alice is from itself.
> Obviously the timing from Alice sending to Bob and Bob replying gives a hint 
> as well as to the distance the intermediate node is to Bob already.
>
> It may be good to at least recommend that direct messages use different paths 
> if they are part of a larger conversation between the two parties.

This already applies to HTLCs, no?

> Would it not be better to create a circular path?
> By this I mean, Alice constructs an onion that overall creates a path from 
> herself to Bob and back, ensuring different nodes on the forward and return 
> directions.
> The onion hop at Bob reveals that Bob is the chosen conversation partner, and 
> Bob forwards its reply via the onion return path (that Alice prepared herself 
> to get back to her via another path).

I like it!  The lack of "reply" function eliminates all storage
requirements for the intermediaries.  Unfortunately it's not currently
possible to fit the reply onion inside the existing onion, but I know
Christian has a rabbit in his hat for this?

> After Alice receives the first message from Bob the circular "circuit" is 
> established and they can continue to communicate using the same circuit: 
> timing attacks are now "impossible" since Alice and Bob can be anywhere along 
> the circle, even if two of the nodes in the circuit are surveillors 
> cooperating with each other, the timing information they get is the distance 
> between the surveillor nodes.
>
> Of course, if a node in the circular path drops the circuit is disrupted, so 
> any higher-level protocols on top of that should probably be willing to 
> resume the conversation on another circular circuit.

My immediate purpose for this is for "offers" which cause a invoice
request, followed by an invoice reply.  This will probably be reused
once for the payment itself.  2 uses is not sufficient to justify
setting up a circuit, AFAICT.

> I believe I even tied this to an HTLC in an attempt to provide a
> spam-limit as well:
> https://lists.linuxfoundation.org/pipermail/lightning-dev/2019-November/002294.html

This part was deeply unclear.  Eventually we will have to charge
up-front for forwarding HTLCs (say 5% of existing fee, plus 1msat), and
then we could use the same scheme with lesser amounts (say, 1msat) for
forwarding messages.

But I have been unable to come up with an upfront scheme which doesn't
leak information badly.

The best I can do is some hashcash scheme, combined with the ability to
buy a single-use token to weaken it.  Under load, a node would raise
their hashcash difficulty, and you could either find another route,
grind your onion more to meet it, or send a payment for a token from
that node which would let your HTLC through: the preimage could even be
the XOR of some secret you send with the HTLC, and a shachain key which
gives you 1000 tokens, and you can use them in order, etc.

(Really want to use some kind of Chaumian token here, but it's probably
overkill).

> Finally: what does this improve over, say, requiring that all
> Lightning nodes have a Tor .onion address and just doing direct
> messaging over Tor?

That would be far better!  But that's not happening: lnurl over https is
happening.  Using lightning to tunnel messages is a strict improvement
over that, at least.

Cheers!
Rusty.
___
Lightning-dev mailing list
Lightning-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev


Re: [Lightning-dev] Direct Message draft

2020-02-20 Thread ZmnSCPxj via Lightning-dev
Good morning Rusty,

A concern I have brought up in the past is that if this is more than just a 
single request-response, i.e. if this is a conversation where Alice sends to 
Bob, Bob sends back to Alice, Alice sends back to Bob, and so on, then if the 
same path is used each time Alice sends to Bob, the timing from Bob response to 
Alice to the next Alice sends to Bob can help an intermediate node guess how 
far away Alice is from itself.
Obviously the timing from Alice sending to Bob and Bob replying gives a hint as 
well as to the distance the intermediate node is to Bob already.

It may be good to at least recommend that direct messages use different paths 
if they are part of a larger conversation between the two parties.

Would it not be better to create a circular path?
By this I mean, Alice constructs an onion that overall creates a path from 
herself to Bob and back, ensuring different nodes on the forward and return 
directions.
The onion hop at Bob reveals that Bob is the chosen conversation partner, and 
Bob forwards its reply via the onion return path (that Alice prepared herself 
to get back to her via another path).
After Alice receives the first message from Bob the circular "circuit" is 
established and they can continue to communicate using the same circuit: timing 
attacks are now "impossible" since Alice and Bob can be anywhere along the 
circle, even if two of the nodes in the circuit are surveillors cooperating 
with each other, the timing information they get is the distance between the 
surveillor nodes.

Of course, if a node in the circular path drops the circuit is disrupted, so 
any higher-level protocols on top of that should probably be willing to resume 
the conversation on another circular circuit.

I believe I even tied this to an HTLC in an attempt to provide a spam-limit as 
well: 
https://lists.linuxfoundation.org/pipermail/lightning-dev/2019-November/002294.html

Finally: what does this improve over, say, requiring that all Lightning nodes 
have a Tor .onion address and just doing direct messaging over Tor?

Regards,
ZmnSCPxj

> Hi all!
>
> It seems that messaging over lightning is A Thing, and I want to
> use it for the offers protocol anyway. So I've come up with the
> simplest proposal I can, and even implemented it.
>
> Importantly, it's unreliable. Our implementation doesn't
> remember across restarts, limits us to 1000 total remembered forwards
> with random drop, and the protocol doesn't (yet?) include a method for
> errors.
>
> This is much friendlier on nodes than using an HTLC (which
> requires 2 round trips, signature calculations and db commits), so is an
> obvious candidate for much more than just invoice requests.
>
> The WIP patch is small enough I've pasted it below, but it's
> also at https://github.com/lightningnetwork/lightning-rfc/pull/748
>
> diff --git a/01-messaging.md b/01-messaging.md
> index 40d1909..faa5b18 100644
> --- a/01-messaging.md
> +++ b/01-messaging.md
> @@ -56,7 +56,7 @@ The messages are grouped logically into five groups, 
> ordered by the most signifi
>
> -   Setup & Control (types `0`-`31`): messages related to connection setup, 
> control, supported features, and error reporting (described below)
> -   Channel (types `32`-`127`): messages used to setup and tear down 
> micropayment channels (described in BOLT #2)
> -   Commitment (types `128`-`255`): messages related to updating the current 
> commitment transaction, which includes adding, revoking, and settling HTLCs 
> as well as updating fees and exchanging signatures (described in BOLT #2)
> -   -   Routing (types `256`-`511`): messages containing node and channel 
> announcements, as well as any active route exploration (described in BOLT #7)
>
> -   -   Routing (types `256`-`511`): messages containing node and channel 
> announcements, as well as any active route exploration or forwarding 
> (described in BOLT #7)
> -   Custom (types `32768`-`65535`): experimental and application-specific 
> messages
>
> The size of the message is required by the transport layer to fit 
> into a 2-byte unsigned int; therefore, the maximum possible size is 65535 
> bytes.
> diff --git a/04-onion-routing.md b/04-onion-routing.md
> index 8d0f343..84eff9a 100644
> --- a/04-onion-routing.md
> +++ b/04-onion-routing.md
> @@ -51,6 +51,7 @@ A node:
> -   Legacy HopData Payload Format
> -   TLV Payload Format
> -   Basic Multi-Part Payments
> -   -   Directed Messages
>
> -   Accepting and Forwarding a Payment
> -   Payload for the Last Node
> -   Non-strict Forwarding
> @@ -62,6 +63,7 @@ A node:
>
> -   Returning Errors
> -   Failure Messages
> -   Receiving Failure Codes
>
> -   -   Directed Message Replies
> -   Test Vector
> -   Returning Errors
> -   References
> @@ -366,6 +368,13 @@ otherwise meets the amount criterion (eg. some 
> other failure, or
> invoice 

Re: [Lightning-dev] Direct Message draft

2020-02-20 Thread René Pickhardt via Lightning-dev
Hey Rusty,

I was very delighted to read your proposal. But I don't see how you prevent
message spam. If I understand you correctly you suggest that I can
communicate to any node along a path of peer connections (not necessarily
backed by payment channels but kind of only known through channel
announcements of gossip) via onions and these onions which are send within
a new gossip message are not bound to any fees or payments.

Let's assume I just missed some spam prevention mechanism or that we can
fix them. Do I understand the impact of your suggestion correctly that I
could use this protocol to

1.) create a fee free rebalancing protocol? Because I could also attach a
new lightning message inside the onions that would allow nodes without
direct peer connection to set up a circular rebalancing path.
2.) have the ability to communicate with nodes further away than just my
peers - for example to exchange information for pathfinding and / or
autopilots?


With kind regards Rene

Rusty Russell  schrieb am Do., 20. Feb. 2020, 10:37:

> Hi all!
>
> It seems that messaging over lightning is A Thing, and I want to
> use it for the offers protocol anyway.  So I've come up with the
> simplest proposal I can, and even implemented it.
>
> Importantly, it's unreliable.  Our implementation doesn't
> remember across restarts, limits us to 1000 total remembered forwards
> with random drop, and the protocol doesn't (yet?) include a method for
> errors.
>
> This is much friendlier on nodes than using an HTLC (which
> requires 2 round trips, signature calculations and db commits), so is an
> obvious candidate for much more than just invoice requests.
>
> The WIP patch is small enough I've pasted it below, but it's
> also at https://github.com/lightningnetwork/lightning-rfc/pull/748
>
> diff --git a/01-messaging.md b/01-messaging.md
> index 40d1909..faa5b18 100644
> --- a/01-messaging.md
> +++ b/01-messaging.md
> @@ -56,7 +56,7 @@ The messages are grouped logically into five groups,
> ordered by the most signifi
>- Setup & Control (types `0`-`31`): messages related to connection
> setup, control, supported features, and error reporting (described below)
>- Channel (types `32`-`127`): messages used to setup and tear down
> micropayment channels (described in [BOLT #2](02-peer-protocol.md))
>- Commitment (types `128`-`255`): messages related to updating the
> current commitment transaction, which includes adding, revoking, and
> settling HTLCs as well as updating fees and exchanging signatures
> (described in [BOLT #2](02-peer-protocol.md))
> -  - Routing (types `256`-`511`): messages containing node and channel
> announcements, as well as any active route exploration (described in [BOLT
> #7](07-routing-gossip.md))
> +  - Routing (types `256`-`511`): messages containing node and channel
> announcements, as well as any active route exploration or forwarding
> (described in [BOLT #7](07-routing-gossip.md))
>- Custom (types `32768`-`65535`): experimental and application-specific
> messages
>
>  The size of the message is required by the transport layer to fit into a
> 2-byte unsigned int; therefore, the maximum possible size is 65535 bytes.
> diff --git a/04-onion-routing.md b/04-onion-routing.md
> index 8d0f343..84eff9a 100644
> --- a/04-onion-routing.md
> +++ b/04-onion-routing.md
> @@ -51,6 +51,7 @@ A node:
>  * [Legacy HopData Payload Format](#legacy-hop_data-payload-format)
>  * [TLV Payload Format](#tlv_payload-format)
>  * [Basic Multi-Part Payments](#basic-multi-part-payments)
> +* [Directed Messages](#directed-messages)
>* [Accepting and Forwarding a
> Payment](#accepting-and-forwarding-a-payment)
>  * [Payload for the Last Node](#payload-for-the-last-node)
>  * [Non-strict Forwarding](#non-strict-forwarding)
> @@ -62,6 +63,7 @@ A node:
>* [Returning Errors](#returning-errors)
>  * [Failure Messages](#failure-messages)
>  * [Receiving Failure Codes](#receiving-failure-codes)
> +  * [Directed Message Replies](#directed-message-replies)
>* [Test Vector](#test-vector)
>  * [Returning Errors](#returning-errors)
>* [References](#references)
> @@ -366,6 +368,13 @@ otherwise meets the amount criterion (eg. some other
> failure, or
>  invoice timeout), however if it were to fulfill only some of them,
>  intermediary nodes could simply claim the remaining ones.
>
> +### Directed Messages
> +
> +Directed messages have an onion with an alternate `hop_payload`
> +format.  If this node is not the intended recipient, the payload is
> +simply a 33-byte pubkey indicating the next recipient.  Otherwise, the
> +payload is the message for this node.
> +
>  # Accepting and Forwarding a Payment
>
>  Once a node has decoded the payload it either accepts the payment
> locally, or forwards it to the peer indicated as the next hop in the
> payload.
> @@ -1142,6 +1151,11 @@ The _origin node_:
>- MAY use the data specified in the various 

[Lightning-dev] Direct Message draft

2020-02-20 Thread Rusty Russell
Hi all!

It seems that messaging over lightning is A Thing, and I want to
use it for the offers protocol anyway.  So I've come up with the
simplest proposal I can, and even implemented it.

Importantly, it's unreliable.  Our implementation doesn't
remember across restarts, limits us to 1000 total remembered forwards
with random drop, and the protocol doesn't (yet?) include a method for
errors.

This is much friendlier on nodes than using an HTLC (which
requires 2 round trips, signature calculations and db commits), so is an
obvious candidate for much more than just invoice requests.

The WIP patch is small enough I've pasted it below, but it's
also at https://github.com/lightningnetwork/lightning-rfc/pull/748

diff --git a/01-messaging.md b/01-messaging.md
index 40d1909..faa5b18 100644
--- a/01-messaging.md
+++ b/01-messaging.md
@@ -56,7 +56,7 @@ The messages are grouped logically into five groups, ordered 
by the most signifi
   - Setup & Control (types `0`-`31`): messages related to connection setup, 
control, supported features, and error reporting (described below)
   - Channel (types `32`-`127`): messages used to setup and tear down 
micropayment channels (described in [BOLT #2](02-peer-protocol.md))
   - Commitment (types `128`-`255`): messages related to updating the current 
commitment transaction, which includes adding, revoking, and settling HTLCs as 
well as updating fees and exchanging signatures (described in [BOLT 
#2](02-peer-protocol.md))
-  - Routing (types `256`-`511`): messages containing node and channel 
announcements, as well as any active route exploration (described in [BOLT 
#7](07-routing-gossip.md))
+  - Routing (types `256`-`511`): messages containing node and channel 
announcements, as well as any active route exploration or forwarding (described 
in [BOLT #7](07-routing-gossip.md))
   - Custom (types `32768`-`65535`): experimental and application-specific 
messages
 
 The size of the message is required by the transport layer to fit into a 
2-byte unsigned int; therefore, the maximum possible size is 65535 bytes.
diff --git a/04-onion-routing.md b/04-onion-routing.md
index 8d0f343..84eff9a 100644
--- a/04-onion-routing.md
+++ b/04-onion-routing.md
@@ -51,6 +51,7 @@ A node:
 * [Legacy HopData Payload Format](#legacy-hop_data-payload-format)
 * [TLV Payload Format](#tlv_payload-format)
 * [Basic Multi-Part Payments](#basic-multi-part-payments)
+* [Directed Messages](#directed-messages)
   * [Accepting and Forwarding a Payment](#accepting-and-forwarding-a-payment)
 * [Payload for the Last Node](#payload-for-the-last-node)
 * [Non-strict Forwarding](#non-strict-forwarding)
@@ -62,6 +63,7 @@ A node:
   * [Returning Errors](#returning-errors)
 * [Failure Messages](#failure-messages)
 * [Receiving Failure Codes](#receiving-failure-codes)
+  * [Directed Message Replies](#directed-message-replies)
   * [Test Vector](#test-vector)
 * [Returning Errors](#returning-errors)
   * [References](#references)
@@ -366,6 +368,13 @@ otherwise meets the amount criterion (eg. some other 
failure, or
 invoice timeout), however if it were to fulfill only some of them,
 intermediary nodes could simply claim the remaining ones.
 
+### Directed Messages
+
+Directed messages have an onion with an alternate `hop_payload`
+format.  If this node is not the intended recipient, the payload is
+simply a 33-byte pubkey indicating the next recipient.  Otherwise, the
+payload is the message for this node.
+
 # Accepting and Forwarding a Payment
 
 Once a node has decoded the payload it either accepts the payment locally, or 
forwards it to the peer indicated as the next hop in the payload.
@@ -1142,6 +1151,11 @@ The _origin node_:
   - MAY use the data specified in the various failure types for debugging
   purposes.
 
+## Directed Message Replies
+
+Directed message replies are encoded the same way as failure messages,
+except the contents is a directed message for the originator.
+
 # Test Vector
 
 ## Returning Errors
diff --git a/07-routing-gossip.md b/07-routing-gossip.md
index ec1a8f0..4c2b836 100644
--- a/07-routing-gossip.md
+++ b/07-routing-gossip.md
@@ -1,4 +1,4 @@
-# BOLT #7: P2P Node and Channel Discovery
+# BOLT #7: P2P Node and Channel Discovery and Directed Messages
 
 This specification describes simple node discovery, channel discovery, and 
channel update mechanisms that do not rely on a third-party to disseminate the 
information.
 
@@ -31,6 +31,7 @@ multiple `node_announcement` messages, in order to update the 
node information.
   * [HTLC Fees](#htlc-fees)
   * [Pruning the Network View](#pruning-the-network-view)
   * [Recommendations for Routing](#recommendations-for-routing)
+  * [Directed Messages](#directed-messages)
   * [References](#references)
 
 ## Definition of `short_channel_id`
@@ -1103,6 +1104,37 @@ A->D's `update_add_htlc` message would be:
 And D->C's `update_add_htlc` would again be the same as B->C's direct