Re: [Lightning-dev] INTEROPERABILITY
yes i do have a solution that is easily integrated to the different impl for pumping up live loglines, what's needed there for this very purpose would be a html ws UI with side by side windows representing a impl instance version, also a query for download files of range unix.ts per instance. (need be sponsored !) ohhh i did not know You are the CL maintainer responsible for signing stable version releases. You are welcome to partake after You talked to the beancounters at Blockstream that probably can give You a budget ? /xraid On Wed, Nov 24, 2021 at 2:17 PM ZmnSCPxj wrote: > Good morning x raid, > > > We are talkin interoperability among impl not individual node operators > version management of chosen impl. > > > > where Pierre of Acinq says > > "So we eat our own dog food and will experience force closes before our > users do.." > > hahaha made my day ... > > > > a node operator do tests live in its continuous integration efforts > would be expected and should be able do so with a by impl assured latest > stable release version. > > > > what is suggested for dialog is the different impl maintainers before > sign off a stable release do a extra test live on mainnet with liquidity in > channels towards the other impl versions and by doing so can catch > unforseen glitches that tests of impl in isolation can not catch. > > > ***We developers already ARE running nodes that are connected to other > implementations, on mainnet, 24/7.*** > In practice we have large release windows before we actually make a final > release, precisely to catch these kinds of bugs that are not easily visible > in isolation, but need real data on the network. > My own node (C-Lightning) has channels to LNBIG (lnd), and I suspect at > least some of the unpublished channels made to me are Electrum, for > example, so I already have a node that is already running against other > implementations. > > We have been telling you that over several emails already. > > > Is there any specific thing you can offer? > Put up hardware and coins yourself if you really want this to happen. > I can give you an SSH key so I can install C-Lightning and CLBOSS on your > hardware myself, then give you the addresses of the C-Lightning node so you > can provide coins for CLBOSS to manage. > > > Regards, > ZmnSCPxj > ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
Good morning x raid, > We are talkin interoperability among impl not individual node operators > version management of chosen impl. > > where Pierre of Acinq says > "So we eat our own dog food and will experience force closes before our users > do.." > hahaha made my day ... > > a node operator do tests live in its continuous integration efforts would be > expected and should be able do so with a by impl assured latest stable > release version. > > what is suggested for dialog is the different impl maintainers before sign > off a stable release do a extra test live on mainnet with liquidity in > channels towards the other impl versions and by doing so can catch unforseen > glitches that tests of impl in isolation can not catch. ***We developers already ARE running nodes that are connected to other implementations, on mainnet, 24/7.*** In practice we have large release windows before we actually make a final release, precisely to catch these kinds of bugs that are not easily visible in isolation, but need real data on the network. My own node (C-Lightning) has channels to LNBIG (lnd), and I suspect at least some of the unpublished channels made to me are Electrum, for example, so I already have a node that is already running against other implementations. We have been telling you that over several emails already. Is there any specific thing you can offer? Put up hardware and coins yourself if you really want this to happen. I can give you an SSH key so I can install C-Lightning and CLBOSS on your hardware myself, then give you the addresses of the C-Lightning node so you can provide coins for CLBOSS to manage. Regards, ZmnSCPxj ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
We are talkin interoperability among impl not individual node operators version management of chosen impl. where Pierre of Acinq says "So we eat our own dog food and will experience force closes before our users do.." hahaha made my day ... a node operator do tests live in its continuous integration efforts would be expected and should be able do so with a by impl assured latest stable release version. what is suggested for dialog is the different impl maintainers before sign off a stable release do a extra test live on mainnet with liquidity in channels towards the other impl versions and by doing so can catch unforseen glitches that tests of impl in isolation can not catch. what also suggested was a collection point where one could fetch loglines from the instances partaking in the interoperability assurance test. ex. impl (a) try open channel to impl (b) and fails --can then ask central logline point for the loglines of ( a < -- > b ) at unix.ts(start) - range - unix.ts(end). this then would help assert a new release of a impl is interoperable and can be recommende for install as considered be latest stable release version. LN a Network System Class "Finacial" need in preparation for onboarding 1B by 2026 start to grow up *now*. thanks /xraid On Wed, Nov 24, 2021 at 12:14 AM ZmnSCPxj wrote: > Good morning x-raid, > > > so You propose Acinq / Blockstream / Lightning Labs do not have funds to > run a box or 2 ? > > Not at all, I am proposing that these people, who have already done the > effort to release working Lightning Network Node implementations free of > charge to you, are not obligated to *also* devote more hardware and > resources. > > Let me tell a little story... > > Some years ago, during the SegWit wars, there was a sentiment "when are > **they** going to implement Lightning??" > Both anti-SegWit and pro-SegWit asked this: > > * anti-SegWit: Yeah, you need bigblocks, Lightning is vaporware, when are > **they** going to implement Lightning? > * pro-SegWit: Lightning is so totes kool, this is why we SegWit, when are > **they** going to implement Lightning? > > After some time participating in the SegWit wars, I realized that I was, > in fact, a programmer (LOL). > So why should **I** be asking when **they** are going to implement > Lightning? > As a programmer, **I** could implement Lightning myself! > I should be asking myself why **I** was not implementing it. > > Thus I started contributing to the Lightning implementation written in a > language I could understand, C-Lightning. > > > My question to you is: obviously you are a node operator as otherwise the > issue you raise would not be relevant to you, but what can *you* do to > advance your goal? > > (In any case: C-Lightning and Eclair devs have already mentioned they > already run mainnet nodes tracking our respective master branches (i.e. we > already eat our own dog food, because duh --- for many of us, the reason we > are developing this is because for *other* reasons, we *have to* run > Lightning nodes), is there any particular implementation you are concerned > about? > Maybe ask them directly?) > > > Regards, > ZmnSCPxj > ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
Good morning x-raid, > so You propose Acinq / Blockstream / Lightning Labs do not have funds to run > a box or 2 ? Not at all, I am proposing that these people, who have already done the effort to release working Lightning Network Node implementations free of charge to you, are not obligated to *also* devote more hardware and resources. Let me tell a little story... Some years ago, during the SegWit wars, there was a sentiment "when are **they** going to implement Lightning??" Both anti-SegWit and pro-SegWit asked this: * anti-SegWit: Yeah, you need bigblocks, Lightning is vaporware, when are **they** going to implement Lightning? * pro-SegWit: Lightning is so totes kool, this is why we SegWit, when are **they** going to implement Lightning? After some time participating in the SegWit wars, I realized that I was, in fact, a programmer (LOL). So why should **I** be asking when **they** are going to implement Lightning? As a programmer, **I** could implement Lightning myself! I should be asking myself why **I** was not implementing it. Thus I started contributing to the Lightning implementation written in a language I could understand, C-Lightning. My question to you is: obviously you are a node operator as otherwise the issue you raise would not be relevant to you, but what can *you* do to advance your goal? (In any case: C-Lightning and Eclair devs have already mentioned they already run mainnet nodes tracking our respective master branches (i.e. we already eat our own dog food, because duh --- for many of us, the reason we are developing this is because for *other* reasons, we *have to* run Lightning nodes), is there any particular implementation you are concerned about? Maybe ask them directly?) Regards, ZmnSCPxj ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
> so You propose Acinq / Blockstream / Lightning Labs do not have funds to run a box or 2 ? Fwiw we already do that with our main node. It's even better than a demo one because it has real economic activity, and has a lot of channels to a variety of nodes implementations and versions. Sometimes our node is behind eclair master branch, but it's always up-to-date for critical commits and releases. So we eat our own dog food and will experience force closes before our users do. Pierre ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
Each dev does tests on a local machine before push to repo, repo maintainers do tests and when satisfied do an acceptance test in the proposed system to verify against all other implementations partaking before public release. Has not anything to do with You relly, not Your private machine but dedicated test machines that do have nano granularity in that request against runs can specify unix.ts(start)-range-unix.ts(end) of box xyz and independently se if others and own impl behaves as it is expected to. On Tue, Nov 23, 2021 at 5:31 PM x raid wrote: > so You propose Acinq / Blockstream / Lightning Labs do not have funds to > run a box or 2 ? > > contributions can be made towards each impl by ppl while the project still > do need put liquidity on mainnet to be a viable test before a public > sanctioned release. > > I find You choose misread what the text is supposed to convey - those with > the write credentials to repo, when do a public release could have been > tested, before, against the network, and that in no way hinders PR toward > said repo. > > On Tue, Nov 23, 2021 at 1:44 PM ZmnSCPxj wrote: > >> Good morning x-raid, >> >> > what i can imagine is each team should provide boxes and channel >> liquidity as stake on mainnet for tests before announce a public realise as >> to feel the pain first hand instead of having several K´s of plebs confused >> and at worst have funds in channelclosed etc. but mostly for helping in >> smooth transitioning into future envisioned mass. >> >> Not all members of all teams are independently wealthy, and cannot afford >> significant liquidity on mainnet, or can afford good Internet connection >> and keeping a device operational 24/7. >> For example, for some time I was a C-Lightning core developer, yet did >> not run a C-Lightning node myself, relying on sheer code review, because I >> could not afford to run a node. >> What you imagine would raise the barrier towards contribution (i.e. I >> might not have been able to start contributing to C-Lightning in the first >> place, for example). >> >> I think you misunderstand the open-source model. >> If you have the skill, but not the money, you can contribute directly. >> If you do not have the skill, but do have the money, you can contribute >> that by hiring developers to work on the project you want. >> >> So, if you are using a particular open-source implementation and storing >> your funds with it, either: >> >> * You have the 1337 skillz0rs: you contribute review and actual code. >> * You do not have the 1337 skillz0rs: you contribute hardware and testing >> reports and possibly money. >> >> If the several Ks of plebs are confused, they can aggregate their >> resources and fund one or two developers to review and contribute to the >> project they are using, and maybe some hardware and coins for boxes they >> keep running. >> >> At my point of view, the Real Issue (TM) here is how to aggregate the >> will of a group of people, without risking that some centralized "manager" >> of resources gets incentives that diverge from the group of people and >> starts allocating resources in ways that the group of people would, in >> aggregate, disagree with. >> >> > >> > If teams rather outsource the running of boxes with channels on mainnet >> for impl release and rc versions they would of course be able to, but close >> to home for managing analysis of the team impl themselves is what I would >> recommend. >> > >> > Can also see that each box loglines are collected at one central point >> whereby requests can be made for comparing interoperability per unix.ts >> identified by box. >> > (thats alot of data You say --not really in Big Data terms, question is >> where to set a proper cap in time for collections ? a week ? a month ?) >> > I think i might have a solution for the central point collector that >> could be run by an outside of impl teams perimeter. (sponsored?) >> >> See, if the money on the node is my own, and not contributed by the group >> that is going to receive the logs, I am not going to send the logs verbatim >> to them, nope not nada. >> I do not want to become a target, because logs leak information like who >> my channel counterparties are and how often I forward HTLCs and exact dates >> and times of each event, and thus can be used to locate my node, and >> location is the first step to targeted attack. >> I mean I use a frikkin set of 8 random letters, come on. >> Possibly if the logs had sensitive information redacted (even dates and >> times??), but we need to automate that redaction, and in particular, if the >> implementation changes log messages, we need to ensure that changed log >> messages do not leak information that gets past the automated redaction. >> >> >> Regards, >> ZmnSCPxj >> > ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
so You propose Acinq / Blockstream / Lightning Labs do not have funds to run a box or 2 ? contributions can be made towards each impl by ppl while the project still do need put liquidity on mainnet to be a viable test before a public sanctioned release. I find You choose misread what the text is supposed to convey - those with the write credentials to repo, when do a public release could have been tested, before, against the network, and that in no way hinders PR toward said repo. On Tue, Nov 23, 2021 at 1:44 PM ZmnSCPxj wrote: > Good morning x-raid, > > > what i can imagine is each team should provide boxes and channel > liquidity as stake on mainnet for tests before announce a public realise as > to feel the pain first hand instead of having several K´s of plebs confused > and at worst have funds in channelclosed etc. but mostly for helping in > smooth transitioning into future envisioned mass. > > Not all members of all teams are independently wealthy, and cannot afford > significant liquidity on mainnet, or can afford good Internet connection > and keeping a device operational 24/7. > For example, for some time I was a C-Lightning core developer, yet did not > run a C-Lightning node myself, relying on sheer code review, because I > could not afford to run a node. > What you imagine would raise the barrier towards contribution (i.e. I > might not have been able to start contributing to C-Lightning in the first > place, for example). > > I think you misunderstand the open-source model. > If you have the skill, but not the money, you can contribute directly. > If you do not have the skill, but do have the money, you can contribute > that by hiring developers to work on the project you want. > > So, if you are using a particular open-source implementation and storing > your funds with it, either: > > * You have the 1337 skillz0rs: you contribute review and actual code. > * You do not have the 1337 skillz0rs: you contribute hardware and testing > reports and possibly money. > > If the several Ks of plebs are confused, they can aggregate their > resources and fund one or two developers to review and contribute to the > project they are using, and maybe some hardware and coins for boxes they > keep running. > > At my point of view, the Real Issue (TM) here is how to aggregate the will > of a group of people, without risking that some centralized "manager" of > resources gets incentives that diverge from the group of people and starts > allocating resources in ways that the group of people would, in aggregate, > disagree with. > > > > > If teams rather outsource the running of boxes with channels on mainnet > for impl release and rc versions they would of course be able to, but close > to home for managing analysis of the team impl themselves is what I would > recommend. > > > > Can also see that each box loglines are collected at one central point > whereby requests can be made for comparing interoperability per unix.ts > identified by box. > > (thats alot of data You say --not really in Big Data terms, question is > where to set a proper cap in time for collections ? a week ? a month ?) > > I think i might have a solution for the central point collector that > could be run by an outside of impl teams perimeter. (sponsored?) > > See, if the money on the node is my own, and not contributed by the group > that is going to receive the logs, I am not going to send the logs verbatim > to them, nope not nada. > I do not want to become a target, because logs leak information like who > my channel counterparties are and how often I forward HTLCs and exact dates > and times of each event, and thus can be used to locate my node, and > location is the first step to targeted attack. > I mean I use a frikkin set of 8 random letters, come on. > Possibly if the logs had sensitive information redacted (even dates and > times??), but we need to automate that redaction, and in particular, if the > implementation changes log messages, we need to ensure that changed log > messages do not leak information that gets past the automated redaction. > > > Regards, > ZmnSCPxj > ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
Good morning x-raid, > what i can imagine is each team should provide boxes and channel liquidity as > stake on mainnet for tests before announce a public realise as to feel the > pain first hand instead of having several K´s of plebs confused and at worst > have funds in channelclosed etc. but mostly for helping in smooth > transitioning into future envisioned mass. Not all members of all teams are independently wealthy, and cannot afford significant liquidity on mainnet, or can afford good Internet connection and keeping a device operational 24/7. For example, for some time I was a C-Lightning core developer, yet did not run a C-Lightning node myself, relying on sheer code review, because I could not afford to run a node. What you imagine would raise the barrier towards contribution (i.e. I might not have been able to start contributing to C-Lightning in the first place, for example). I think you misunderstand the open-source model. If you have the skill, but not the money, you can contribute directly. If you do not have the skill, but do have the money, you can contribute that by hiring developers to work on the project you want. So, if you are using a particular open-source implementation and storing your funds with it, either: * You have the 1337 skillz0rs: you contribute review and actual code. * You do not have the 1337 skillz0rs: you contribute hardware and testing reports and possibly money. If the several Ks of plebs are confused, they can aggregate their resources and fund one or two developers to review and contribute to the project they are using, and maybe some hardware and coins for boxes they keep running. At my point of view, the Real Issue (TM) here is how to aggregate the will of a group of people, without risking that some centralized "manager" of resources gets incentives that diverge from the group of people and starts allocating resources in ways that the group of people would, in aggregate, disagree with. > > If teams rather outsource the running of boxes with channels on mainnet for > impl release and rc versions they would of course be able to, but close to > home for managing analysis of the team impl themselves is what I would > recommend. > > Can also see that each box loglines are collected at one central point > whereby requests can be made for comparing interoperability per unix.ts > identified by box. > (thats alot of data You say --not really in Big Data terms, question is where > to set a proper cap in time for collections ? a week ? a month ?) > I think i might have a solution for the central point collector that could be > run by an outside of impl teams perimeter. (sponsored?) See, if the money on the node is my own, and not contributed by the group that is going to receive the logs, I am not going to send the logs verbatim to them, nope not nada. I do not want to become a target, because logs leak information like who my channel counterparties are and how often I forward HTLCs and exact dates and times of each event, and thus can be used to locate my node, and location is the first step to targeted attack. I mean I use a frikkin set of 8 random letters, come on. Possibly if the logs had sensitive information redacted (even dates and times??), but we need to automate that redaction, and in particular, if the implementation changes log messages, we need to ensure that changed log messages do not leak information that gets past the automated redaction. Regards, ZmnSCPxj ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
ZmnSCPxj via Lightning-dev writes: > Are you proposing as well to provide the hardware and Internet > connection for these boxes? Having implemented and operated the lightning integration testing framework [1,2] in the past this is something near and dear to my heart. However I have since become convinced that this kind of artificial setup is unlikely to catch any but the most egregious issues, given their different usage pattern. Much like the bitcoin testnet is not representative of what happens on the mainnet, I don't think a separate network would be able to reproduce all the issues that occur on the lightning mainnet. I agree with ZmnSCPxj that testing on the mainnet is much more likely to catch more involved issues, and therefore a solid process with release candidates and nightly builds, in combination with lnprototest [3] to test the nodes for spec adherence in isolation is the way to go. > I know of one person at least who runs a node that tracks the > C-Lightning master (I think they do a nightly build?), and I run a > node that I update every release of C-Lightning (and runs CLBOSS as > well). I do not know the actual implementations of what they connect > to, but LND is very popular on the network and LNBIG is known to be an > LND shop, and LNBIG is so pervasive that nearly every long-lived > forwarding node has at least one channel with *some* LNBIG node. I > consider this "good enough" in practice to catch interop bugs, but > some interop bugs are deeper than just direct node-to-node > communications. For example, we had bugs in our interop with LND > `keysend` before, by my memory. We should differentiate between spec compliance, and compatibility of extensions. `keysend` wasn't and still isn't specd which resulted in us having to reverse engineer the logic from the first experimental implementation, and I did get some details wrong. For example I expected nodes to explicitly opt-into keysend via featurebit 55, but they just yolo it... As these primitives become more widespread and more users rely on them, I think it is paramount that we actually spec them out (something that the new bLIP process should cover), but until we do there is no way of saying what's correct and what isn't. Cheers, Christian [1] https://github.com/cdecker/lightning-integration [2] https://cdecker.github.io/lightning-integration/ [3] https://github.com/rustyrussell/lnprototest ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
what i can imagine is each team should provide boxes and channel liquidity as stake on mainnet for tests before announce a public realise as to feel the pain first hand instead of having several K´s of plebs confused and at worst have funds in channelclosed etc. but mostly for helping in smooth transitioning into future envisioned mass. If teams rather outsource the running of boxes with channels on mainnet for impl release and rc versions they would of course be able to, but close to home for managing analysis of the team impl themselves is what I would recommend. Can also see that each box loglines are collected at one central point whereby requests can be made for comparing interoperability per unix.ts identified by box. (thats alot of data You say --not really in Big Data terms, question is where to set a proper cap in time for collections ? a week ? a month ?) I think i might have a solution for the central point collector that could be run by an outside of impl teams perimeter. (sponsored?) /xraid On Tue, Nov 23, 2021 at 11:35 AM ZmnSCPxj wrote: > Good morning again x-raid, > > Are you proposing as well to provide the hardware and Internet connection > for these boxes? > > I know of one person at least who runs a node that tracks the C-Lightning > master (I think they do a nightly build?), and I run a node that I update > every release of C-Lightning (and runs CLBOSS as well). > I do not know the actual implementations of what they connect to, but LND > is very popular on the network and LNBIG is known to be an LND shop, and > LNBIG is so pervasive that nearly every long-lived forwarding node has at > least one channel with *some* LNBIG node. > I consider this "good enough" in practice to catch interop bugs, but some > interop bugs are deeper than just direct node-to-node communications. > For example, we had bugs in our interop with LND `keysend` before, by my > memory. > > Regards, > ZmnSCPxj > ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
Good morning again x-raid, Are you proposing as well to provide the hardware and Internet connection for these boxes? I know of one person at least who runs a node that tracks the C-Lightning master (I think they do a nightly build?), and I run a node that I update every release of C-Lightning (and runs CLBOSS as well). I do not know the actual implementations of what they connect to, but LND is very popular on the network and LNBIG is known to be an LND shop, and LNBIG is so pervasive that nearly every long-lived forwarding node has at least one channel with *some* LNBIG node. I consider this "good enough" in practice to catch interop bugs, but some interop bugs are deeper than just direct node-to-node communications. For example, we had bugs in our interop with LND `keysend` before, by my memory. Regards, ZmnSCPxj ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
Re: [Lightning-dev] INTEROPERABILITY
Good morning x-raid, > I propose a dialog of the below joint effort ... > > thanks > /xraid > > *** > A decentralised integration lab where CL Eclair LDK LND (++ ?) runs each the > latest release on "one box" rBOX and master.rc on "another box" rcBOX. I believe Electrum also has its own bespoke implementation. There was also Ptarmigan. Regards, ZmnSCPxj ___ Lightning-dev mailing list Lightning-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
