Re: BloomFilter issue with segwit addresses
I was able to isolate an issue, here's a transaction sent at Jun 8 whose 2nd output of 0.00436685 BTC belongs to a wallet: https://www.smartbit.com.au/tx/6e0a1a32c981703b4df3210deabd1c9cc87ac8683fdfc904f6db7db443b3bb2b Here's a second transaction sent at Jun 11 which spends that output without change: https://www.smartbit.com.au/tx/513833bad28407a1a63740a136d5534603f3e6a68af69678ebc0cec273079176 If I start a blockchain resync and do nothing, then I can see an incoming tx on Jun 8 but no spending tx is found and 0.00436685 BTC is seen as unspent. If, however, I quickly close an app once I see an incoming transaction on Jun 8 then after restarting it catches a Jun 11 spending tx just fine and output is marked as spent. I was able to reproduce this a few times in a row so seems like this is not a coincidence. I plan to investigate this closely very soon but reporting about this in case if it sounds familiar. суббота, 23 июня 2018 г., 21:14:03 UTC+3 пользователь Andreas Schildbach написал: > > Damn, in this case we need to find the root cause of this issue. > > It might be related to the following issue which we had to deal with > when implementing BIP37: > > * Received pending relevant tx which was thrown away (I guess because of > risk analysis) > * Same transaction was confirmed, partial merkle tree was received but > not the transaction itself again. This is due to how bitcoind works, it > sends only txns it hasn't sent over the same connection. > * Result: Tx was missing from the wallet and never received again > (unless the wallet was replayed). > > Have you looked closely at the logfile? > > > On 06/22/2018 09:43 AM, Anton wrote: > > Yes, seems like it is. Never happens in non-segwit case, consistently > > happens in SegWit case. > > > >> I'm wondering is this actually a regression related to SegWit? Did you > >> try the same in the non-SegWit case? > > > -- You received this message because you are subscribed to the Google Groups "bitcoinj" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoinj+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: BloomFilter issue with segwit addresses
Yes, changes introduced in https://github.com/schildbach/bitcoinj/commit/66d76131a6b453b27d1344a71665b6050c201c43 also fix this issue. Just in case, I see "plus the BLOOM_UPDATE_ALL flag set" in commit comments but no such changes in 66d76131a6b453b27d1344a71665b6050c201c43 itself, am I missing something? суббота, 14 апреля 2018 г., 3:40:52 UTC+3 пользователь Andreas Schildbach написал: > > Investigating more into this... actually the correct way to deal with > bloom filters is inserting UTXO OutPoints into the filter. > > And bitcoinj already has a mechanism for this, which is currently only > used for watched scripts. > > Can you the top commit from my branch? > https://github.com/schildbach/bitcoinj/tree/p2wpkh-bloom-filter > > (Obviously for isRequiringUpdateAllBloomFilter() we need a more > appropriate solution, rather than just returning true. But for a quick > test it should do.) > > I'm looking forward to your findings. > > > On 04/10/2018 06:51 PM, Anton wrote: > > Thanks for the suggestion, > > > > tx.addOutput(Coin.ZERO, new > > ScriptBuilder().op(OP_RETURN).data(pubKeyHash).build()); > > > > does the job. > > > > Of course, I'm interested to know whether a less hacky solution exists. > > > > вторник, 10 апреля 2018 г., 18:42:10 UTC+3 пользователь Andreas > > Schildbach написал: > > > > [+cc Jonas Schnelli, as I discussed this topic with him last year.] > > > > Hmmm, indeed that sounds bad. I totally overlooked this case. Seems > > like > > BIP37 needs to be amended to include the witness in filter matching. > > However, on nodes that prune the witnesses this won't work either. > > > > P2WPKH-P2SH doesn't seem to have this problem, as it contains the > > 20-byte key hash in its scriptSig. > > > > Of course as a workaround you could make sure there is always a > change > > back to you. Maybe to not waste coins/UTXOs create an OP_RETURN > > containing one of the pubkeys? > > > > I wonder if the new client-side filtering (BIP157) has the same > > problem? > > > > > > On 04/09/2018 05:51 PM, Anton wrote: > > > I'm currently experimenting with making a native segwit wallet > here: > > > https://github.com/btcontract/bitcoinj > > <https://github.com/btcontract/bitcoinj> by merging current master > with > > > segwit address support and a > > > https://github.com/bitcoinj/bitcoinj/tree/segwit > > <https://github.com/bitcoinj/bitcoinj/tree/segwit> branch. > > > I've got to a point where wallet can receive and send transactions > > but > > > have stumbled into an issue which may become relevant once > bitcoinj > > > master starts supporting native segwit wallets. > > > > > > The issue is that once a transaction without change (the one which > > > empties a wallet) is sent out it won't be seen by wallet again > after > > > blockchain rescan or restoring from seed. > > > > > > As far as I understand the issue is two-fold: > > > 1. tx scriptSig is empty and does not contain a relevant `pubKey` > > (it's > > > located in a witness). > > > 2. none of tx outputs contain `pubKeyHash`es relevant to a wallet. > > > > > > Any thoughts if my understanding is correct and what can be done > > about it? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "bitcoinj" group. > > > To unsubscribe from this group and stop receiving emails from it, > > send > > > an email to bitcoinj+u...@googlegroups.com > > > <mailto:bitcoinj+u...@googlegroups.com>. > > > For more options, visit https://groups.google.com/d/optout > > <https://groups.google.com/d/optout>. > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "bitcoinj" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to bitcoinj+u...@googlegroups.com > > <mailto:bitcoinj+u...@googlegroups.com >. > > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "bitcoinj" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoinj+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: BloomFilter issue with segwit addresses
Yes, I've tried OR_RETURN case with emptying tx on a testnet and segwit wallet does correctly catch it on blockchain rescans. As far as I understand the best choice for OP_RETURN `pubKeyHash` is one of the signing keys used in tx inputs (in order to not give away more info). OK, I'll definitely put this on mailing list. вторник, 10 апреля 2018 г., 23:43:05 UTC+3 пользователь Andreas Schildbach написал: > > Just for clarification, did you test both cases against real bitcoind? > Or did you just look at the specification, as I did? > > API-wise, you can use ScriptBuilder.createOpReturnScript(). Maybe you > want to randomize which pubkey is used in the OP_RETURN data, in case > there are multiple candidates. > > Maybe you'd like to publish your finding on the bitcoin-dev mailing > list? I'm sure they are interested, because it affects SegWit efficiency > and maybe adoption. > > > On 04/10/2018 06:51 PM, Anton wrote: > > Thanks for the suggestion, > > > > tx.addOutput(Coin.ZERO, new > > ScriptBuilder().op(OP_RETURN).data(pubKeyHash).build()); > > > > does the job. > > > > Of course, I'm interested to know whether a less hacky solution exists. > > > > вторник, 10 апреля 2018 г., 18:42:10 UTC+3 пользователь Andreas > > Schildbach написал: > > > > [+cc Jonas Schnelli, as I discussed this topic with him last year.] > > > > Hmmm, indeed that sounds bad. I totally overlooked this case. Seems > > like > > BIP37 needs to be amended to include the witness in filter matching. > > However, on nodes that prune the witnesses this won't work either. > > > > P2WPKH-P2SH doesn't seem to have this problem, as it contains the > > 20-byte key hash in its scriptSig. > > > > Of course as a workaround you could make sure there is always a > change > > back to you. Maybe to not waste coins/UTXOs create an OP_RETURN > > containing one of the pubkeys? > > > > I wonder if the new client-side filtering (BIP157) has the same > > problem? > > > > > > On 04/09/2018 05:51 PM, Anton wrote: > > > I'm currently experimenting with making a native segwit wallet > here: > > > https://github.com/btcontract/bitcoinj > > <https://github.com/btcontract/bitcoinj> by merging current master > with > > > segwit address support and a > > > https://github.com/bitcoinj/bitcoinj/tree/segwit > > <https://github.com/bitcoinj/bitcoinj/tree/segwit> branch. > > > I've got to a point where wallet can receive and send transactions > > but > > > have stumbled into an issue which may become relevant once > bitcoinj > > > master starts supporting native segwit wallets. > > > > > > The issue is that once a transaction without change (the one which > > > empties a wallet) is sent out it won't be seen by wallet again > after > > > blockchain rescan or restoring from seed. > > > > > > As far as I understand the issue is two-fold: > > > 1. tx scriptSig is empty and does not contain a relevant `pubKey` > > (it's > > > located in a witness). > > > 2. none of tx outputs contain `pubKeyHash`es relevant to a wallet. > > > > > > Any thoughts if my understanding is correct and what can be done > > about it? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "bitcoinj" group. > > > To unsubscribe from this group and stop receiving emails from it, > > send > > > an email to bitcoinj+u...@googlegroups.com > > > <mailto:bitcoinj+u...@googlegroups.com>. > > > For more options, visit https://groups.google.com/d/optout > > <https://groups.google.com/d/optout>. > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "bitcoinj" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to bitcoinj+u...@googlegroups.com > > <mailto:bitcoinj+u...@googlegroups.com >. > > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "bitcoinj" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoinj+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: BloomFilter issue with segwit addresses
Thanks for the suggestion, tx.addOutput(Coin.ZERO, new ScriptBuilder().op(OP_RETURN).data(pubKeyHash).build()); does the job. Of course, I'm interested to know whether a less hacky solution exists. вторник, 10 апреля 2018 г., 18:42:10 UTC+3 пользователь Andreas Schildbach написал: > > [+cc Jonas Schnelli, as I discussed this topic with him last year.] > > Hmmm, indeed that sounds bad. I totally overlooked this case. Seems like > BIP37 needs to be amended to include the witness in filter matching. > However, on nodes that prune the witnesses this won't work either. > > P2WPKH-P2SH doesn't seem to have this problem, as it contains the > 20-byte key hash in its scriptSig. > > Of course as a workaround you could make sure there is always a change > back to you. Maybe to not waste coins/UTXOs create an OP_RETURN > containing one of the pubkeys? > > I wonder if the new client-side filtering (BIP157) has the same problem? > > > On 04/09/2018 05:51 PM, Anton wrote: > > I'm currently experimenting with making a native segwit wallet here: > > https://github.com/btcontract/bitcoinj by merging current master with > > segwit address support and a > > https://github.com/bitcoinj/bitcoinj/tree/segwit branch. > > I've got to a point where wallet can receive and send transactions but > > have stumbled into an issue which may become relevant once bitcoinj > > master starts supporting native segwit wallets. > > > > The issue is that once a transaction without change (the one which > > empties a wallet) is sent out it won't be seen by wallet again after > > blockchain rescan or restoring from seed. > > > > As far as I understand the issue is two-fold: > > 1. tx scriptSig is empty and does not contain a relevant `pubKey` (it's > > located in a witness). > > 2. none of tx outputs contain `pubKeyHash`es relevant to a wallet. > > > > Any thoughts if my understanding is correct and what can be done about > it? > > > > -- > > You received this message because you are subscribed to the Google > > Groups "bitcoinj" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to bitcoinj+u...@googlegroups.com > > <mailto:bitcoinj+u...@googlegroups.com >. > > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "bitcoinj" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoinj+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: bitcoinj-0.14 : wallet backup - proper way ?
Why not just save a BIP39 mnemonic once instead of constant backups? bitcoinj supports that. -- You received this message because you are subscribed to the Google Groups "bitcoinj" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoinj+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Filtering out watched txs
Okay, I think I'll actually take this code out of bitcoinj and make a custom Transaction wrapper in my project which I'll be using for filtering for now. LN transactions indeed don't seem to belong to those "payments" since they're off-chain, no way for bitcoinj to even catch them, except for funding/breach/closing txs which are on-chain. понедельник, 10 июля 2017 г., 23:26:34 UTC+3 пользователь Andreas Schildbach написал: > > There is currently no way to have bitcoinj filter the list of > transactions, you'll have to do that yourself ATM. > > We always envisioned having a second, more usecase-oriented list of > "payments". This would then represent the list shown to the user. > Lightning payments could be visible there too in future (although LN is > a layer on top, so not sure if it's the right thing). So far, work on > this hasn't started. > > > On 07/08/2017 09:07 AM, Anton wrote: > > After an LN payment channel is open there is a constant need to monitor > > if it's P2WSH output has been spent, I use `addWatchedScripts` for that > > and it works great. The only UX level problem with this approach is when > > P2WSH output does get spent a user will see a strange spending tx which > > changes nothing in wallet, this may confuse many. > > > > So what is needed is a way to filter out purely watched transactions > > before displaying a tx list on a UI. Currently I do this as follows: > > > https://github.com/btcontract/bitcoinj/commit/8475c850ef7b5d07ae7b30d605362c2dbea8d0d5, > > > > the code is based on a fact that a purely watched tx does not change > > wallet's balance so it's `getValue` is 0 when watched outputs and inputs > > are excluded so we can use that when deciding if a given tx should be > > displayed. > > > > This works but I don't like an approach because it feels hacky > > (`getValue` is essentially duplicated in `getExcludeWatchedValue`), > > perhaps there are much better ways to do that? > > > > Other simple approaches I was thinking of which won't do: > > > > - Immediately stop watching a P2WSH output script once it is spent so a > > related watched tx won't be shown anymore (won't do because a peer may > > attempt to spend a P2WSH output multiple times using different commit > > txs which will create a race of double spends and there's no telling > > which of them will win, so we need to continue watching a script even > > after the first spend is seen to react accordingly). > > > > - Just filter out all txs which contain a watched P2WSH output in their > > inputs (won't do since that will filter out mutual closing tx which does > > spend a P2WSH output but is legitimate since it sends money back to > > user's wallet) > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google > > Groups "bitcoinj" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to bitcoinj+u...@googlegroups.com > > <mailto:bitcoinj+u...@googlegroups.com >. > > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "bitcoinj" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoinj+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
bitcoinj and Lightning network
Hi everyone, currently I work on a Lightning network implementation which is intended to be used on lite clients such as mobile phones, a repo can be found here: https://github.com/btcontract/lnwallet and also recently I've made a demo if anyone is interested: https://www.youtube.com/watch?v=yNFfUfyL2xE My goals are: - build a core Lightning library which is not tied to any specific wallet or platform and can be reused, related code can be found here: https://github.com/btcontract/lnwallet/tree/master/app/src/main/java/com/lightning/wallet/ln, it's a work in progress and a modification of https://github.com/ACINQ/eclair project where I basically dumb things down (by not using an Akka actor framework which may be too heavy for phones, not assuming a locally accessible Core node API is available, not relaying third party payments for now and so on). It's coming along nicely and I'm confident it will reach a production ready state sooner rather than later. - build an end-user mobile Bitcoin/Lightning wallet on top of an aforementioned Lightning library (for Lightning part) and bitcoinj (for Bitcoin part). This seems like a very convinient and natural solution since users will be able to at once fund Lightning payment channels right from their Bitcoin wallet and also fall back to Bitcoin payments if Lightning is not available for some reason. And this is where some bitcoinj-specific help is really needed. Two things are required for bitcoinj to power a mobile Lightning wallet: segwit support and ability to create segwit outputs. Thanks to Jean-Pierre Rupp's work at https://github.com/bitcoinj/bitcoinj/pull/1334 segwit support is already here, I use his segwit-enabled modification to create a P2WSH output (as required by Lightning funding transaction for a payment channel) and it does work great, as can be seen on a demo video. Sadly, this alone is not enough as not only a funding tx should have a P2WSH output but also all of it's inputs must come from segwit outputs. This is a problem because currently bitcoinj uses base58 addresses which produce non-segwit outputs. A funding tx can be created using such non-segwit outputs but it can be malleated so payment channel can't be established in a trustless manner, basically it's a show stopper. The way I see it, there are two options of getting over this, one requires no big changes to bitcoinj beyond what is already done but is quite horrible from UX point of view, the other one requires some deep changes in bitcoinj: Option 1: instead of creating a funding tx directly, we create an intermediary non-segwit -> P2WPKH tx, then we use it to P2WPKH -> P2WSH (payment channel output). Pros are: no changes to bitcoinj. Cons are: doubled fees, doubled confirmation waiting time, implementation complexities (sending two txs is not atomic, things can happen in the middle). Option 2: add bech32 support to bitcoinj and change it to generate bech32 addresses instead of base58 addresses in order to get native segwit outputs. Pros are: Lightning support out of the box, fee discounts for native segwit txs. Cons are: probably a lot of work and testing, a breaking change (?) which perhaps should live in a separate branch, probably something else I haven't thought of. Despite implementation complexities associated with the sencond option I really like it much more since the first one essentially makes users handle those complexities and I'm not sure they would want to do that, plus sooner or later bech32 support and address generation will have to be done anyway. I'm not sure if I can handle all this myself since my knowledge of bitcoinj if fairly limited and I spend most of my free time working on Lightning but I'm definitely ready to help with programming work, especially if the whole process could be split into more or less clear tasks. Please let me know what you think of all this. -- You received this message because you are subscribed to the Google Groups "bitcoinj" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoinj+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
