Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On 05/25/2015 11:05 PM, Peter Todd wrote: On Mon, May 25, 2015 at 10:29:26PM +0200, Andreas Schildbach wrote: I see this behavior all the time. I am using the latest release, as far as I know. Version 4.30. The same behavior occurs in the Testnet3 variant of the app. Go in there with an empty wallet and receive one payment and wait for it to confirm. Then send a payment and, before it confirms, try to send another one. The wallet won't let you send the second payment. It'll say something like, You need x.xx more bitcoins to make this payment. But if you wait for your first payment to confirm, then you'll be able to make the second payment. If it matters, I configure the app to connect only to my own trusted Bitcoin node, so I only ever have one active connection at most. I notice that outgoing payments never show as Sent until they appear in a block, presumably because the app never sees the transaction come in over any connection. Yes, that's the issue. Because you're connecting only to one node, you don't get any instant confirmations -- due to a Bitcoin protocol limitation you can only get them from nodes you don't post the tx to. Odd, I just tried the above as well - with multiple peers connected - and had the exact same problem. It should work, I'm testing this regularly. Can you report an issue through the app and attach your log when this happens again? -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On 05/25/2015 10:03 PM, Matt Whitlock wrote: On Monday, 25 May 2015, at 8:41 pm, Mike Hearn wrote: some wallets (e.g., Andreas Schildbach's wallet) don't even allow it - you can only spend confirmed UTXOs. I can't tell you how aggravating it is to have to tell a friend, Oh, oops, I can't pay you yet. I have to wait for the last transaction I did to confirm first. All the more aggravating because I know, if I have multiple UTXOs in my wallet, I can make multiple spends within the same block. Andreas' wallet hasn't done that for years. Are you repeating this from some very old memory or do you actually see this issue in reality? The only time you're forced to wait for confirmations is when you have an unconfirmed inbound transaction, and thus the sender is unknown. I see this behavior all the time. I am using the latest release, as far as I know. Version 4.30. The same behavior occurs in the Testnet3 variant of the app. Go in there with an empty wallet and receive one payment and wait for it to confirm. Then send a payment and, before it confirms, try to send another one. The wallet won't let you send the second payment. It'll say something like, You need x.xx more bitcoins to make this payment. But if you wait for your first payment to confirm, then you'll be able to make the second payment. If it matters, I configure the app to connect only to my own trusted Bitcoin node, so I only ever have one active connection at most. I notice that outgoing payments never show as Sent until they appear in a block, presumably because the app never sees the transaction come in over any connection. Yes, that's the issue. Because you're connecting only to one node, you don't get any instant confirmations -- due to a Bitcoin protocol limitation you can only get them from nodes you don't post the tx to. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
some wallets (e.g., Andreas Schildbach's wallet) don't even allow it - you can only spend confirmed UTXOs. I can't tell you how aggravating it is to have to tell a friend, Oh, oops, I can't pay you yet. I have to wait for the last transaction I did to confirm first. All the more aggravating because I know, if I have multiple UTXOs in my wallet, I can make multiple spends within the same block. Andreas' wallet hasn't done that for years. Are you repeating this from some very old memory or do you actually see this issue in reality? The only time you're forced to wait for confirmations is when you have an unconfirmed inbound transaction, and thus the sender is unknown. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
Wallets are incentivised to do a better job with defragmentation already, as if you have lots of tiny UTXOs then your fees end up being huge when trying to make a payment. The reason they largely don't is just one of manpower. Nobody is working on it. As a wallet developer myself, one way I'd like to see this issue be fixed by making free transactions more reliable. Then wallets can submit free transactions to the network to consolidate UTXOs together, e.g. at night when the user is sleeping. They would then fit into whatever space is available in the block during periods of low demand, like on Sunday. If we don't do this then wallets won't automatically defragment, as we'd be unable to explain to the user why their money is slowly leaking out of their wallet without them doing anything. Trying to explain the existing transaction fees is hard enough already (I thought bitcoin doesn't have banks etc). There is another way: as the fee is based on a rounded 1kb calculation, if you go into the next fee band adding some more outputs and making a bigger change output becomes free for another output or two. But wallets don't exploit this today. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On Monday, 25 May 2015, at 8:41 pm, Mike Hearn wrote: some wallets (e.g., Andreas Schildbach's wallet) don't even allow it - you can only spend confirmed UTXOs. I can't tell you how aggravating it is to have to tell a friend, Oh, oops, I can't pay you yet. I have to wait for the last transaction I did to confirm first. All the more aggravating because I know, if I have multiple UTXOs in my wallet, I can make multiple spends within the same block. Andreas' wallet hasn't done that for years. Are you repeating this from some very old memory or do you actually see this issue in reality? The only time you're forced to wait for confirmations is when you have an unconfirmed inbound transaction, and thus the sender is unknown. I see this behavior all the time. I am using the latest release, as far as I know. Version 4.30. The same behavior occurs in the Testnet3 variant of the app. Go in there with an empty wallet and receive one payment and wait for it to confirm. Then send a payment and, before it confirms, try to send another one. The wallet won't let you send the second payment. It'll say something like, You need x.xx more bitcoins to make this payment. But if you wait for your first payment to confirm, then you'll be able to make the second payment. If it matters, I configure the app to connect only to my own trusted Bitcoin node, so I only ever have one active connection at most. I notice that outgoing payments never show as Sent until they appear in a block, presumably because the app never sees the transaction come in over any connection. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On Mon, May 25, 2015 at 10:29:26PM +0200, Andreas Schildbach wrote: I see this behavior all the time. I am using the latest release, as far as I know. Version 4.30. The same behavior occurs in the Testnet3 variant of the app. Go in there with an empty wallet and receive one payment and wait for it to confirm. Then send a payment and, before it confirms, try to send another one. The wallet won't let you send the second payment. It'll say something like, You need x.xx more bitcoins to make this payment. But if you wait for your first payment to confirm, then you'll be able to make the second payment. If it matters, I configure the app to connect only to my own trusted Bitcoin node, so I only ever have one active connection at most. I notice that outgoing payments never show as Sent until they appear in a block, presumably because the app never sees the transaction come in over any connection. Yes, that's the issue. Because you're connecting only to one node, you don't get any instant confirmations -- due to a Bitcoin protocol limitation you can only get them from nodes you don't post the tx to. Odd, I just tried the above as well - with multiple peers connected - and had the exact same problem. -- 'peter'[:-1]@petertodd.org 0e83c311f4244e4eefb54aa845abb181e46f16d126ab21e1 signature.asc Description: Digital signature -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
CPFP also solves it just fine. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On Mon, May 25, 2015 at 08:44:18PM +0200, Mike Hearn wrote: Wallets are incentivised to do a better job with defragmentation already, as if you have lots of tiny UTXOs then your fees end up being huge when trying to make a payment. The reason they largely don't is just one of manpower. Nobody is working on it. As a wallet developer myself, one way I'd like to see this issue be fixed by making free transactions more reliable. Then wallets can submit free transactions to the network to consolidate UTXOs together, e.g. at night when the user is sleeping. They would then fit into whatever space is available in the block during periods of low demand, like on Sunday. This can cause problems as until those transactions confirm, even more of the user's outputs are unavailable for spending, causing confusion as to why they can't send their full balance. It's also inefficient, as in the case where the user does try to send a small payment that could be satisfied by one or more of these small UTXO's, the wallet has to use a larger UTXO. With replace-by-fee however this problem goes away, as you can simply double-spend the pending defragmentation transactions instead if they are still unconfirmed when you need to use them. -- 'peter'[:-1]@petertodd.org 0aa9033c06c10d6131eafa3754c3157d74c2267c1dd2ca35 signature.asc Description: Digital signature -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
Having thousands of utxos floating around for a single address is clearly a bad thing - it creates a lot of memory load on bitcoin nodes. However, having only one utxo for an address is also a bad thing, for concurrent operations. Having several utxos available to spend is good for parallelism, so that 2 or more tasks which are spending from the same address don't have to line up single file waiting for one of the tasks to publish a tx first so that the next task can spend the (unconfirmed) change output of the first. Requiring/Forcing/Having a single output carry the entire balance of an address does not work at scale. (Yes, this presumes that the tasks are coordinated so that they don't attempt to spend the same outputs. Internal coordination is solvable.) In multiple replies, you push for having all utxos of an address spent in one transaction. Why all? If the objective is to reduce the size of the utxo pool, it would be sufficient simply to recommend that wallets and other spenders consume more utxos than they create, on average. I'm ok with consume more utxos than you generate as a good citizen / best practices recommendation, but a requirement that all prior outputs must be spent in one transaction seems excessive and impractical. -Danny On Sat, May 9, 2015 at 10:09 AM, Jim Phillips j...@ergophobia.org wrote: Forgive me if this idea has been suggested before, but I made this suggestion on reddit and I got some feedback recommending I also bring it to this list -- so here goes. I wonder if there isn't perhaps a simpler way of dealing with UTXO growth. What if, rather than deal with the issue at the protocol level, we deal with it at the source of the problem -- the wallets. Right now, the typical wallet selects only the minimum number of unspent outputs when building a transaction. The goal is to keep the transaction size to a minimum so that the fee stays low. Consequently, lots of unspent outputs just don't get used, and are left lying around until some point in the future. What if we started designing wallets to consolidate unspent outputs? When selecting unspent outputs for a transaction, rather than choosing just the minimum number from a particular address, why not select them ALL? Take all of the UTXOs from a particular address or wallet, send however much needs to be spent to the payee, and send the rest back to the same address or a change address as a single output? Through this method, we should wind up shrinking the UTXO database over time rather than growing it with each transaction. Obviously, as Bitcoin gains wider adoption, the UTXO database will grow, simply because there are 7 billion people in the world, and eventually a good percentage of them will have one or more wallets with spendable bitcoin. But this idea could limit the growth at least. The vast majority of users are running one of a handful of different wallet apps: Core, Electrum; Armory; Mycelium; Breadwallet; Coinbase; Circle; Blockchain.info; and maybe a few others. The developers of all these wallets have a vested interest in the continued usefulness of Bitcoin, and so should not be opposed to changing their UTXO selection algorithms to one that reduces the UTXO database instead of growing it. From the miners perspective, even though these types of transactions would be larger, the fee could stay low. Miners actually benefit from them in that it reduces the amount of storage they need to dedicate to holding the UTXO. So miners are incentivized to mine these types of transactions with a higher priority despite a low fee. Relays could also get in on the action and enforce this type of behavior by refusing to relay or deprioritizing the relay of transactions that don't use all of the available UTXOs from the addresses used as inputs. Relays are not only the ones who benefit the most from a reduction of the UTXO database, they're also in the best position to promote good behavior. -- *James G. Phillips IV* https://plus.google.com/u/0/113107039501292625391/posts *Don't bunt. Aim out of the ball park. Aim for the company of immortals. -- David Ogilvy* *This message was created with 100% recycled electrons. Please think twice before printing.* -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- One dashboard for servers and applications
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
I feel your pain. I've had the same thing happen to me in the past. And I agree it's more likely to occur with my proposed scheme but I think with HD wallets there will still be UTXOs left unspent after most transactions since, for privacy sake it's looking for the smallest set of addresses that can be linked. On May 9, 2015 9:11 PM, Matt Whitlock b...@mattwhitlock.name wrote: Minimizing the number of UTXOs in a wallet is sometimes not in the best interests of the user. In fact, quite often I've wished for a configuration option like Try to maintain _[number]_ UTXOs in the wallet. This is because I often want to make multiple spends from my wallet within one block, but spends of unconfirmed inputs are less reliable than spends of confirmed inputs, and some wallets (e.g., Andreas Schildbach's wallet) don't even allow it - you can only spend confirmed UTXOs. I can't tell you how aggravating it is to have to tell a friend, Oh, oops, I can't pay you yet. I have to wait for the last transaction I did to confirm first. All the more aggravating because I know, if I have multiple UTXOs in my wallet, I can make multiple spends within the same block. On Saturday, 9 May 2015, at 12:09 pm, Jim Phillips wrote: Forgive me if this idea has been suggested before, but I made this suggestion on reddit and I got some feedback recommending I also bring it to this list -- so here goes. I wonder if there isn't perhaps a simpler way of dealing with UTXO growth. What if, rather than deal with the issue at the protocol level, we deal with it at the source of the problem -- the wallets. Right now, the typical wallet selects only the minimum number of unspent outputs when building a transaction. The goal is to keep the transaction size to a minimum so that the fee stays low. Consequently, lots of unspent outputs just don't get used, and are left lying around until some point in the future. What if we started designing wallets to consolidate unspent outputs? When selecting unspent outputs for a transaction, rather than choosing just the minimum number from a particular address, why not select them ALL? Take all of the UTXOs from a particular address or wallet, send however much needs to be spent to the payee, and send the rest back to the same address or a change address as a single output? Through this method, we should wind up shrinking the UTXO database over time rather than growing it with each transaction. Obviously, as Bitcoin gains wider adoption, the UTXO database will grow, simply because there are 7 billion people in the world, and eventually a good percentage of them will have one or more wallets with spendable bitcoin. But this idea could limit the growth at least. The vast majority of users are running one of a handful of different wallet apps: Core, Electrum; Armory; Mycelium; Breadwallet; Coinbase; Circle; Blockchain.info; and maybe a few others. The developers of all these wallets have a vested interest in the continued usefulness of Bitcoin, and so should not be opposed to changing their UTXO selection algorithms to one that reduces the UTXO database instead of growing it. From the miners perspective, even though these types of transactions would be larger, the fee could stay low. Miners actually benefit from them in that it reduces the amount of storage they need to dedicate to holding the UTXO. So miners are incentivized to mine these types of transactions with a higher priority despite a low fee. Relays could also get in on the action and enforce this type of behavior by refusing to relay or deprioritizing the relay of transactions that don't use all of the available UTXOs from the addresses used as inputs. Relays are not only the ones who benefit the most from a reduction of the UTXO database, they're also in the best position to promote good behavior. -- *James G. Phillips IV* https://plus.google.com/u/0/113107039501292625391/posts *Don't bunt. Aim out of the ball park. Aim for the company of immortals. -- David Ogilvy* *This message was created with 100% recycled electrons. Please think twice before printing.* -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
That's a lot of work, a lot of extra utxo's, and a lot of blockchain spam, just so I can do a convoluted form of arithmetic on my balance. If a tx contained an explicit miner fee and a change address, but did not compute the change, letting the network compute it (and therefore merge transactions spending the same utxo), could one add some form of ring signature a la Dash to alleviate the worsened privacy implications? Jeff Garzik [jgar...@bitpay.com] wrote: This has been frequently explored on IRC. My general conclusion is dollar bills - pick highly common denominations of bitcoins. Aggregate to obtain these denominations, but do not aggregate further. This permits merge avoidance (privacy++), easy coinjoin where many hide in the noise (privacy++), wallet dust de-fragmentation, while avoiding the over-aggregation problem where you have consolidated down to one output. Thus a wallet would have several consolidation targets. Another strategy is simply doubling outputs. Say you pay 0.1 BTC to Starbucks. Add another 0.1 BTC output to yourself, and a final change output. Who can say which output goes to Starbucks? There are many iterations and trade-offs between fragmentation and privacy. -- Cheers, Bob McElrath The individual has always had to struggle to keep from being overwhelmed by the tribe. If you try it, you will be lonely often, and sometimes frightened. But no price is too high to pay for the privilege of owning yourself. -- Friedrich Nietzsche -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
This is my biggest headache with practical bitcoin usage. I'd love to hear it if anyone has any clever solutions to the wallet/utxo locked problem. Spending unconfirmed outputs really requires a different security model on the part of the receiver than #confirmations, but isn't inherently bad if the receiver has a better security model and knows how to compute the probability that an unconfirmed-spend will get confirmed. Of course the bigger problem is wallet software that refuses to spend unconfirmed outputs. I've thought a bit about a fork/merge design: if the change were computed by the network instead of the submitter, two transactions having the same change address and a common input could be straightforwardly merged or split (in a reorg), where with bitcoin currently it would be considered a double-spend. Of course that has big privacy implications since it directly exposes the change address, and is a hard fork, but is much closer to what people expect of a debit-based account in traditional banking. The fact of the matter is that having numerous sequential debits on an account is an extremely common use case, and bitcoin is obtuse in this respect. On May 9, 2015 1:09:32 PM EDT, Jim Phillips j...@ergophobia.org wrote: Forgive me if this idea has been suggested before, but I made this suggestion on reddit and I got some feedback recommending I also bring it to this list -- so here goes. I wonder if there isn't perhaps a simpler way of dealing with UTXO growth. What if, rather than deal with the issue at the protocol level, we deal with it at the source of the problem -- the wallets. Right now, the typical wallet selects only the minimum number of unspent outputs when building a transaction. The goal is to keep the transaction size to a minimum so that the fee stays low. Consequently, lots of unspent outputs just don't get used, and are left lying around until some point in the future. What if we started designing wallets to consolidate unspent outputs? When selecting unspent outputs for a transaction, rather than choosing just the minimum number from a particular address, why not select them ALL? Take all of the UTXOs from a particular address or wallet, send however much needs to be spent to the payee, and send the rest back to the same address or a change address as a single output? Through this method, we should wind up shrinking the UTXO database over time rather than growing it with each transaction. Obviously, as Bitcoin gains wider adoption, the UTXO database will grow, simply because there are 7 billion people in the world, and eventually a good percentage of them will have one or more wallets with spendable bitcoin. But this idea could limit the growth at least. The vast majority of users are running one of a handful of different wallet apps: Core, Electrum; Armory; Mycelium; Breadwallet; Coinbase; Circle; Blockchain.info; and maybe a few others. The developers of all these wallets have a vested interest in the continued usefulness of Bitcoin, and so should not be opposed to changing their UTXO selection algorithms to one that reduces the UTXO database instead of growing it. From the miners perspective, even though these types of transactions would be larger, the fee could stay low. Miners actually benefit from them in that it reduces the amount of storage they need to dedicate to holding the UTXO. So miners are incentivized to mine these types of transactions with a higher priority despite a low fee. Relays could also get in on the action and enforce this type of behavior by refusing to relay or deprioritizing the relay of transactions that don't use all of the available UTXOs from the addresses used as inputs. Relays are not only the ones who benefit the most from a reduction of the UTXO database, they're also in the best position to promote good behavior. -- *James G. Phillips IV* https://plus.google.com/u/0/113107039501292625391/posts *Don't bunt. Aim out of the ball park. Aim for the company of immortals. -- David Ogilvy* *This message was created with 100% recycled electrons. Please think twice before printing.* !DSPAM:554e4e5450787476022393! -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y !DSPAM:554e4e5450787476022393! ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development !DSPAM:554e4e5450787476022393! -- Sent from my Android
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
This has been frequently explored on IRC. My general conclusion is dollar bills - pick highly common denominations of bitcoins. Aggregate to obtain these denominations, but do not aggregate further. This permits merge avoidance (privacy++), easy coinjoin where many hide in the noise (privacy++), wallet dust de-fragmentation, while avoiding the over-aggregation problem where you have consolidated down to one output. Thus a wallet would have several consolidation targets. Another strategy is simply doubling outputs. Say you pay 0.1 BTC to Starbucks. Add another 0.1 BTC output to yourself, and a final change output. Who can say which output goes to Starbucks? There are many iterations and trade-offs between fragmentation and privacy. On Sun, May 10, 2015 at 9:35 AM, Bob McElrath bob_bitc...@mcelrath.org wrote: This is my biggest headache with practical bitcoin usage. I'd love to hear it if anyone has any clever solutions to the wallet/utxo locked problem. Spending unconfirmed outputs really requires a different security model on the part of the receiver than #confirmations, but isn't inherently bad if the receiver has a better security model and knows how to compute the probability that an unconfirmed-spend will get confirmed. Of course the bigger problem is wallet software that refuses to spend unconfirmed outputs. I've thought a bit about a fork/merge design: if the change were computed by the network instead of the submitter, two transactions having the same change address and a common input could be straightforwardly merged or split (in a reorg), where with bitcoin currently it would be considered a double-spend. Of course that has big privacy implications since it directly exposes the change address, and is a hard fork, but is much closer to what people expect of a debit-based account in traditional banking. The fact of the matter is that having numerous sequential debits on an account is an extremely common use case, and bitcoin is obtuse in this respect. On May 9, 2015 1:09:32 PM EDT, Jim Phillips j...@ergophobia.org wrote: Forgive me if this idea has been suggested before, but I made this suggestion on reddit and I got some feedback recommending I also bring it to this list -- so here goes. I wonder if there isn't perhaps a simpler way of dealing with UTXO growth. What if, rather than deal with the issue at the protocol level, we deal with it at the source of the problem -- the wallets. Right now, the typical wallet selects only the minimum number of unspent outputs when building a transaction. The goal is to keep the transaction size to a minimum so that the fee stays low. Consequently, lots of unspent outputs just don't get used, and are left lying around until some point in the future. What if we started designing wallets to consolidate unspent outputs? When selecting unspent outputs for a transaction, rather than choosing just the minimum number from a particular address, why not select them ALL? Take all of the UTXOs from a particular address or wallet, send however much needs to be spent to the payee, and send the rest back to the same address or a change address as a single output? Through this method, we should wind up shrinking the UTXO database over time rather than growing it with each transaction. Obviously, as Bitcoin gains wider adoption, the UTXO database will grow, simply because there are 7 billion people in the world, and eventually a good percentage of them will have one or more wallets with spendable bitcoin. But this idea could limit the growth at least. The vast majority of users are running one of a handful of different wallet apps: Core, Electrum; Armory; Mycelium; Breadwallet; Coinbase; Circle; Blockchain.info; and maybe a few others. The developers of all these wallets have a vested interest in the continued usefulness of Bitcoin, and so should not be opposed to changing their UTXO selection algorithms to one that reduces the UTXO database instead of growing it. From the miners perspective, even though these types of transactions would be larger, the fee could stay low. Miners actually benefit from them in that it reduces the amount of storage they need to dedicate to holding the UTXO. So miners are incentivized to mine these types of transactions with a higher priority despite a low fee. Relays could also get in on the action and enforce this type of behavior by refusing to relay or deprioritizing the relay of transactions that don't use all of the available UTXOs from the addresses used as inputs. Relays are not only the ones who benefit the most from a reduction of the UTXO database, they're also in the best position to promote good behavior. -- *James G. Phillips IV* https://plus.google.com/u/0/113107039501292625391/posts *Don't bunt. Aim out of the ball park. Aim for the company of immortals. -- David Ogilvy* *This message was created with
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
Makes sense.. So with that said, I'd propose the following criteria for selecting UTXOs: 1. Select the smallest possible set of addresses that can be linked in order to come up with enough BTC to send to the payee. 2. Given multiple possible sets, select the one that has the largest number of UTXOs. 3. Given multiple possible sets, choose the one that contains the largest amount of total BTC. 4. Given multiple possible sets, select the one that destroys the most bitcoin days. 5. If there's still multiple possible sets, just choose one at random. Once the final set of addresses has been identified, use ALL UTXOs from that set, sending appropriate outputs to the recipient(s), a new change address, and a mining fee. Miners should be cognisant of and reward the fact that the user is making an effort to consolidate UTXOs. They can easily spot these transactions by looking at whether all possible UTXOs from each input addresses have been used. Since most miners use Bitcoin Core, and its defaults, this test can be built into Bitcoin Core's logic for determining which transactions to include when mining a block. -- *James G. Phillips IV* https://plus.google.com/u/0/113107039501292625391/posts http://www.linkedin.com/in/ergophobe *Don't bunt. Aim out of the ball park. Aim for the company of immortals. -- David Ogilvy* *This message was created with 100% recycled electrons. Please think twice before printing.* On Sat, May 9, 2015 at 3:38 PM, Pieter Wuille pieter.wui...@gmail.com wrote: Miners do not care about the age of a UTXO entry, apart for two exceptions. It is also economically irrelevant. * There is a free transaction policy, which sets a small portion of block space aside for transactions which do not pay sufficient fee. This is mostly an altruistic way of encouraging Bitcoin adoption. As a DoS prevention mechanism, there is a requirement that these free transactions are of sufficient priority (computed as BTC-days-destroyed per byte), essentially requiring these transactions to consume another scarce resource, even if not money. * Coinbase transaction outputs can, as a consensus rule, only be spent after 100 confirmations. This is to prevent random reorganisations from invalidating transactions that spend young coinbase transactions (which can't move to the new chain). In addition, wallets also select more confirmed outputs first to consume, for the same reason. On May 9, 2015 1:20 PM, Raystonn rayst...@hotmail.com wrote: That policy is included in Bitcoin Core. Miners use it because it is the default. The policy was likely intended to help real transactions get through in the face of spam. But it favors those with more bitcoin, as the priority is determined by amount spent multiplied by age of UTXOs. At the very least the amount spent should be removed as a factor, or fees are unlikely to ever be paid by those who can afford them. We can reassess the role age plays later. One change at a time is better. On 9 May 2015 12:52 pm, Jim Phillips j...@ergophobia.org wrote: On Sat, May 9, 2015 at 2:43 PM, Raystonn rayst...@hotmail.com wrote: How about this as a happy medium default policy: Rather than select UTXOs based solely on age and limiting the size of the transaction, we select as many UTXOs as possible from as few addresses as possible, prioritizing which addresses to use based on the number of UTXOs it contains (more being preferable) and how old those UTXOs are (in order to reduce the fee)? If selecting older UTXOs gives higher priority for a lesser (or at least not greater) fee, that is an incentive for a rational user to use the older UTXOs. Such policy needs to be defended or removed. It doesn't support privacy or a reduction in UTXOs. Before starting this thread, I had completely forgotten that age was even a factor in determining which UTXOs to use. Frankly, I can't think of any reason why miners care how old a particular UTXO is when determining what fees to charge. I'm sure there is one, I just don't know what it is. I just tossed it in there as homage to Andreas who pointed out to me that it was still part of the selection criteria. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
That policy is included in Bitcoin Core. Miners use it because it is the default. The policy was likely intended to help real transactions get through in the face of spam. But it favors those with more bitcoin, as the priority is determined by amount spent multiplied by age of UTXOs. At the very least the amount spent should be removed as a factor, or fees are unlikely to ever be paid by those who can afford them. We can reassess the role age plays later. One change at a time is better. On 9 May 2015 12:52 pm, Jim Phillips j...@ergophobia.org wrote:On Sat, May 9, 2015 at 2:43 PM, Raystonn raystonn@hotmail.com wrote:How about this as a happy medium default policy: Rather than select UTXOs based solely on age and limiting the size of the transaction, we select as many UTXOs as possible from as few addresses as possible, prioritizing which addresses to use based on the number of UTXOs it contains (more being preferable) and how old those UTXOs are (in order to reduce the fee)? If selecting older UTXOs gives higher priority for a lesser (or at least not greater) fee, that is an incentive for a rational user to use the older UTXOs. Such policy needs to be defended or removed. It doesnt support privacy or a reduction in UTXOs.Before starting this thread, I had completely forgotten that age was even a factor in determining which UTXOs to use. Frankly, I cant think of any reason why miners care how old a particular UTXO is when determining what fees to charge. Im sure there is one, I just dont know what it is. I just tossed it in there as homage to Andreas who pointed out to me that it was still part of the selection criteria. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
Minimizing the number of UTXOs in a wallet is sometimes not in the best interests of the user. In fact, quite often I've wished for a configuration option like Try to maintain _[number]_ UTXOs in the wallet. This is because I often want to make multiple spends from my wallet within one block, but spends of unconfirmed inputs are less reliable than spends of confirmed inputs, and some wallets (e.g., Andreas Schildbach's wallet) don't even allow it - you can only spend confirmed UTXOs. I can't tell you how aggravating it is to have to tell a friend, Oh, oops, I can't pay you yet. I have to wait for the last transaction I did to confirm first. All the more aggravating because I know, if I have multiple UTXOs in my wallet, I can make multiple spends within the same block. On Saturday, 9 May 2015, at 12:09 pm, Jim Phillips wrote: Forgive me if this idea has been suggested before, but I made this suggestion on reddit and I got some feedback recommending I also bring it to this list -- so here goes. I wonder if there isn't perhaps a simpler way of dealing with UTXO growth. What if, rather than deal with the issue at the protocol level, we deal with it at the source of the problem -- the wallets. Right now, the typical wallet selects only the minimum number of unspent outputs when building a transaction. The goal is to keep the transaction size to a minimum so that the fee stays low. Consequently, lots of unspent outputs just don't get used, and are left lying around until some point in the future. What if we started designing wallets to consolidate unspent outputs? When selecting unspent outputs for a transaction, rather than choosing just the minimum number from a particular address, why not select them ALL? Take all of the UTXOs from a particular address or wallet, send however much needs to be spent to the payee, and send the rest back to the same address or a change address as a single output? Through this method, we should wind up shrinking the UTXO database over time rather than growing it with each transaction. Obviously, as Bitcoin gains wider adoption, the UTXO database will grow, simply because there are 7 billion people in the world, and eventually a good percentage of them will have one or more wallets with spendable bitcoin. But this idea could limit the growth at least. The vast majority of users are running one of a handful of different wallet apps: Core, Electrum; Armory; Mycelium; Breadwallet; Coinbase; Circle; Blockchain.info; and maybe a few others. The developers of all these wallets have a vested interest in the continued usefulness of Bitcoin, and so should not be opposed to changing their UTXO selection algorithms to one that reduces the UTXO database instead of growing it. From the miners perspective, even though these types of transactions would be larger, the fee could stay low. Miners actually benefit from them in that it reduces the amount of storage they need to dedicate to holding the UTXO. So miners are incentivized to mine these types of transactions with a higher priority despite a low fee. Relays could also get in on the action and enforce this type of behavior by refusing to relay or deprioritizing the relay of transactions that don't use all of the available UTXOs from the addresses used as inputs. Relays are not only the ones who benefit the most from a reduction of the UTXO database, they're also in the best position to promote good behavior. -- *James G. Phillips IV* https://plus.google.com/u/0/113107039501292625391/posts *Don't bunt. Aim out of the ball park. Aim for the company of immortals. -- David Ogilvy* *This message was created with 100% recycled electrons. Please think twice before printing.* -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
Miners do not care about the age of a UTXO entry, apart for two exceptions. It is also economically irrelevant. * There is a free transaction policy, which sets a small portion of block space aside for transactions which do not pay sufficient fee. This is mostly an altruistic way of encouraging Bitcoin adoption. As a DoS prevention mechanism, there is a requirement that these free transactions are of sufficient priority (computed as BTC-days-destroyed per byte), essentially requiring these transactions to consume another scarce resource, even if not money. * Coinbase transaction outputs can, as a consensus rule, only be spent after 100 confirmations. This is to prevent random reorganisations from invalidating transactions that spend young coinbase transactions (which can't move to the new chain). In addition, wallets also select more confirmed outputs first to consume, for the same reason. On May 9, 2015 1:20 PM, Raystonn rayst...@hotmail.com wrote: That policy is included in Bitcoin Core. Miners use it because it is the default. The policy was likely intended to help real transactions get through in the face of spam. But it favors those with more bitcoin, as the priority is determined by amount spent multiplied by age of UTXOs. At the very least the amount spent should be removed as a factor, or fees are unlikely to ever be paid by those who can afford them. We can reassess the role age plays later. One change at a time is better. On 9 May 2015 12:52 pm, Jim Phillips j...@ergophobia.org wrote: On Sat, May 9, 2015 at 2:43 PM, Raystonn rayst...@hotmail.com wrote: How about this as a happy medium default policy: Rather than select UTXOs based solely on age and limiting the size of the transaction, we select as many UTXOs as possible from as few addresses as possible, prioritizing which addresses to use based on the number of UTXOs it contains (more being preferable) and how old those UTXOs are (in order to reduce the fee)? If selecting older UTXOs gives higher priority for a lesser (or at least not greater) fee, that is an incentive for a rational user to use the older UTXOs. Such policy needs to be defended or removed. It doesn't support privacy or a reduction in UTXOs. Before starting this thread, I had completely forgotten that age was even a factor in determining which UTXOs to use. Frankly, I can't think of any reason why miners care how old a particular UTXO is when determining what fees to charge. I'm sure there is one, I just don't know what it is. I just tossed it in there as homage to Andreas who pointed out to me that it was still part of the selection criteria. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
Actually your assumption is wrong. Bitcoin Wallet (and I think most, if not all, other bitcoinj based wallets) picks UTXO by age, in order to maximize priority. So it keeps the number of UTXOs low, though not as low as if it would always pick *all* UTXOs. On 05/09/2015 07:09 PM, Jim Phillips wrote: Forgive me if this idea has been suggested before, but I made this suggestion on reddit and I got some feedback recommending I also bring it to this list -- so here goes. I wonder if there isn't perhaps a simpler way of dealing with UTXO growth. What if, rather than deal with the issue at the protocol level, we deal with it at the source of the problem -- the wallets. Right now, the typical wallet selects only the minimum number of unspent outputs when building a transaction. The goal is to keep the transaction size to a minimum so that the fee stays low. Consequently, lots of unspent outputs just don't get used, and are left lying around until some point in the future. What if we started designing wallets to consolidate unspent outputs? When selecting unspent outputs for a transaction, rather than choosing just the minimum number from a particular address, why not select them ALL? Take all of the UTXOs from a particular address or wallet, send however much needs to be spent to the payee, and send the rest back to the same address or a change address as a single output? Through this method, we should wind up shrinking the UTXO database over time rather than growing it with each transaction. Obviously, as Bitcoin gains wider adoption, the UTXO database will grow, simply because there are 7 billion people in the world, and eventually a good percentage of them will have one or more wallets with spendable bitcoin. But this idea could limit the growth at least. The vast majority of users are running one of a handful of different wallet apps: Core, Electrum; Armory; Mycelium; Breadwallet; Coinbase; Circle; Blockchain.info; and maybe a few others. The developers of all these wallets have a vested interest in the continued usefulness of Bitcoin, and so should not be opposed to changing their UTXO selection algorithms to one that reduces the UTXO database instead of growing it. From the miners perspective, even though these types of transactions would be larger, the fee could stay low. Miners actually benefit from them in that it reduces the amount of storage they need to dedicate to holding the UTXO. So miners are incentivized to mine these types of transactions with a higher priority despite a low fee. Relays could also get in on the action and enforce this type of behavior by refusing to relay or deprioritizing the relay of transactions that don't use all of the available UTXOs from the addresses used as inputs. Relays are not only the ones who benefit the most from a reduction of the UTXO database, they're also in the best position to promote good behavior. -- *James G. Phillips IV* https://plus.google.com/u/0/113107039501292625391/posts /Don't bunt. Aim out of the ball park. Aim for the company of immortals. -- David Ogilvy / /This message was created with 100% recycled electrons. Please think twice before printing./ -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On Sat, May 9, 2015 at 1:45 PM, Peter Todd p...@petertodd.org wrote: On Sat, May 09, 2015 at 12:09:32PM -0500, Jim Phillips wrote: The vast majority of users are running one of a handful of different wallet apps: Core, Electrum; Armory; Mycelium; Breadwallet; Coinbase; Circle; Blockchain.info; and maybe a few others. The developers of all these wallets have a vested interest in the continued usefulness of Bitcoin, and so should not be opposed to changing their UTXO selection algorithms to one that reduces the UTXO database instead of growing it. You can't assume that UTXO growth will be driven by walles at all; the UTXO set's global consensus functionality is incredibly useful and will certainly be used by all manner of applications, many having nothing to do with Bitcoin. You're correct in this point. Future UTXO growth will be coming from all directions. But I'm a believer in the idea that whatever can be done should be done. If we get Bitcoin devs into the mindset now that UTXOs are expensive to those that have to store them, and that they should be good netizens and do what they can to limit them, then hopefully that will ideal will be passed down to future developers. I don't believe consolidating UTXOs in the wallet is the only solution.. I just think it is a fairly easy one to implement, and can only help the problem from getting worse in the future. -- *James G. Phillips IV* https://plus.google.com/u/0/113107039501292625391/posts http://www.linkedin.com/in/ergophobe *Don't bunt. Aim out of the ball park. Aim for the company of immortals. -- David Ogilvy* *This message was created with 100% recycled electrons. Please think twice before printing.* -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
It's a very complex trade-off, which is hard to optimize for all use cases. Using more UTXOs requires larger transactions, and thus more fees in general. In addition, it results in more linkage between coins/addresses used, so lower privacy. The only way you can guarantee an economical reason to keep the UTXO set small is by actually having a consensus rule that punishes increasing its size. On May 9, 2015 12:02 PM, Andreas Schildbach andr...@schildbach.de wrote: Actually your assumption is wrong. Bitcoin Wallet (and I think most, if not all, other bitcoinj based wallets) picks UTXO by age, in order to maximize priority. So it keeps the number of UTXOs low, though not as low as if it would always pick *all* UTXOs. On 05/09/2015 07:09 PM, Jim Phillips wrote: Forgive me if this idea has been suggested before, but I made this suggestion on reddit and I got some feedback recommending I also bring it to this list -- so here goes. I wonder if there isn't perhaps a simpler way of dealing with UTXO growth. What if, rather than deal with the issue at the protocol level, we deal with it at the source of the problem -- the wallets. Right now, the typical wallet selects only the minimum number of unspent outputs when building a transaction. The goal is to keep the transaction size to a minimum so that the fee stays low. Consequently, lots of unspent outputs just don't get used, and are left lying around until some point in the future. What if we started designing wallets to consolidate unspent outputs? When selecting unspent outputs for a transaction, rather than choosing just the minimum number from a particular address, why not select them ALL? Take all of the UTXOs from a particular address or wallet, send however much needs to be spent to the payee, and send the rest back to the same address or a change address as a single output? Through this method, we should wind up shrinking the UTXO database over time rather than growing it with each transaction. Obviously, as Bitcoin gains wider adoption, the UTXO database will grow, simply because there are 7 billion people in the world, and eventually a good percentage of them will have one or more wallets with spendable bitcoin. But this idea could limit the growth at least. The vast majority of users are running one of a handful of different wallet apps: Core, Electrum; Armory; Mycelium; Breadwallet; Coinbase; Circle; Blockchain.info; and maybe a few others. The developers of all these wallets have a vested interest in the continued usefulness of Bitcoin, and so should not be opposed to changing their UTXO selection algorithms to one that reduces the UTXO database instead of growing it. From the miners perspective, even though these types of transactions would be larger, the fee could stay low. Miners actually benefit from them in that it reduces the amount of storage they need to dedicate to holding the UTXO. So miners are incentivized to mine these types of transactions with a higher priority despite a low fee. Relays could also get in on the action and enforce this type of behavior by refusing to relay or deprioritizing the relay of transactions that don't use all of the available UTXOs from the addresses used as inputs. Relays are not only the ones who benefit the most from a reduction of the UTXO database, they're also in the best position to promote good behavior. -- *James G. Phillips IV* https://plus.google.com/u/0/113107039501292625391/posts /Don't bunt. Aim out of the ball park. Aim for the company of immortals. -- David Ogilvy / /This message was created with 100% recycled electrons. Please think twice before printing./ -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On Sat, May 9, 2015 at 2:00 PM, Andreas Schildbach andr...@schildbach.de wrote: Actually your assumption is wrong. Bitcoin Wallet (and I think most, if not all, other bitcoinj based wallets) picks UTXO by age, in order to maximize priority. So it keeps the number of UTXOs low, though not as low as if it would always pick *all* UTXOs. Is it not fair to say though that UTXO database growth is not considered when selecting the UTXOs to use? And that size of transaction is a priority if not the top priority? -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On Sat, May 9, 2015 at 2:06 PM, Pieter Wuille pieter.wui...@gmail.com wrote: It's a very complex trade-off, which is hard to optimize for all use cases. Using more UTXOs requires larger transactions, and thus more fees in general. Unless the miner determines that the reduction in UTXO storage requirements is worth the lower fee. There's no protocol level enforcement of a fee as far as I understand it. It's enforced by the miners and their willingness to include a transaction in a block. In addition, it results in more linkage between coins/addresses used, so lower privacy. Not if you only select all the UTXOs from a single address. A wallet that is geared more towards privacy minded individuals may want to reduce the amount of address linkage, but a wallet geared towards the general masses probably won't have to worry so much about that. The only way you can guarantee an economical reason to keep the UTXO set small is by actually having a consensus rule that punishes increasing its size. There's an economical reason right now to keeping the UTXO set small. The smaller it is, the easier it is for the individual to run a full node. The easier it is to run a full node, the faster Bitcoin will spread to the masses. The faster it spreads to the masses, the more valuable it becomes. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
Lack of privacy is viral. We shouldn't encourage policy in most wallets that discourages privacy. It adversely affects privacy across the entire network. On 9 May 2015 12:17 pm, Jim Phillips j...@ergophobia.org wrote:On Sat, May 9, 2015 at 2:06 PM, Pieter Wuille pieter.wuille@gmail.com wrote:Its a very complex trade-off, which is hard to optimize for all use cases. Using more UTXOs requires larger transactions, and thus more fees in general. Unless the miner determines that the reduction in UTXO storage requirements is worth the lower fee. Theres no protocol level enforcement of a fee as far as I understand it. Its enforced by the miners and their willingness to include a transaction in a block.In addition, it results in more linkage between coins/addresses used, so lower privacy. Not if you only select all the UTXOs from a single address. A wallet that is geared more towards privacy minded individuals may want to reduce the amount of address linkage, but a wallet geared towards the general masses probably wont have to worry so much about that. The only way you can guarantee an economical reason to keep the UTXO set small is by actually having a consensus rule that punishes increasing its size.Theres an economical reason right now to keeping the UTXO set small. The smaller it is, the easier it is for the individual to run a full node. The easier it is to run a full node, the faster Bitcoin will spread to the masses. The faster it spreads to the masses, the more valuable it becomes. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On Sat, May 9, 2015 at 2:12 PM, Patrick Mccorry (PGR) patrick.mcco...@newcastle.ac.uk wrote: Not necessarily. If you want to ensure privacy, you could limit the selection of UTXOs to a single address, and even go so far as to send change back to that same address. This wouldn't be as effective as combining the UTXOs from multiple addresses, but it would help. The key is to do everything that can be done when building a transaction to ensure that as many inputs as possible are consolidated into as few outputs as possible. I would agree if you have multiple utxo for a single address then it makes sense since there is no privacy loss. However sending the change back to the same address would damage privacy (Hive does this) as it is then obvious from looking at the transaction which output is change and which output is sending funds. I tend to agree with you here. But the change output could just as easily be sent to a new change address. Also not everyone is concerned with their own privacy, and I'm not aware of any HD-wallet implementations that won't already combine inputs from multiple addresses within that wallet without user input. For people who do not care for privacy then it would work fine. But adding it into the wallet as default behaviour would deter those who do care for privacy - and making it a customisable option just adds complexity for the users. Wallets do need to combine utxo at times to spend bitcoins which is how people can be tracked today, using the minimum set of utxo tries to reduce the risk. Different wallets are targeted at different demographics. Some are geared towards more mainstream users (for whom the privacy issue is less a concern) and some (such as DarkWallet) are geared more towards the privacy advocates. These wallets may choose to set their defaults at oposite ends of the spectrum as to how they choose to select and link addresses and UTXOs, but they can all improve on their current algorithms and promote some degree of consolidation. Additionally, large wallets that have lots of addresses owned by multiple users like exchanges, blockchain.info, and Coinbase can consolidate UTXOs very effectively when building transactions That's true - I'm not sure how they would feel about it though. I imagine they probably are already to minimise key management. That's what these discussions are for. Hopefully this thread will be seen by developers of these wallets and give them something to consider. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On Sat, May 9, 2015 at 2:25 PM, Raystonn rayst...@hotmail.com wrote: Lack of privacy is viral. We shouldn't encourage policy in most wallets that discourages privacy. It adversely affects privacy across the entire network. How about this as a happy medium default policy: Rather than select UTXOs based solely on age and limiting the size of the transaction, we select as many UTXOs as possible from as few addresses as possible, prioritizing which addresses to use based on the number of UTXOs it contains (more being preferable) and how old those UTXOs are (in order to reduce the fee)? -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
If selecting older UTXOs gives higher priority for a lesser (or at least not greater) fee, that is an incentive for a rational user to use the older UTXOs. Such policy needs to be defended or removed. It doesn't support privacy or a reduction in UTXOs. On 9 May 2015 12:33 pm, Jim Phillips j...@ergophobia.org wrote:On Sat, May 9, 2015 at 2:25 PM, Raystonn raystonn@hotmail.com wrote:Lack of privacy is viral. We shouldnt encourage policy in most wallets that discourages privacy. It adversely affects privacy across the entire network.How about this as a happy medium default policy: Rather than select UTXOs based solely on age and limiting the size of the transaction, we select as many UTXOs as possible from as few addresses as possible, prioritizing which addresses to use based on the number of UTXOs it contains (more being preferable) and how old those UTXOs are (in order to reduce the fee)? -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
I think potential fee subsidies for cleaning up UTXO (and/or penalties for creating more UTXO than you burn) are worth thinking about. As Gavin's post ( gavinandresen.ninja/utxo-uhoh ) indicates, UTXO cost is far higher than block storage, so charging differently for the in/out mismatches should make good economic sense. Ross On 09/05/2015 20:16, Jim Phillips wrote: On Sat, May 9, 2015 at 2:06 PM, Pieter Wuille pieter.wui...@gmail.com mailto:pieter.wui...@gmail.com wrote: It's a very complex trade-off, which is hard to optimize for all use cases. Using more UTXOs requires larger transactions, and thus more fees in general. Unless the miner determines that the reduction in UTXO storage requirements is worth the lower fee. There's no protocol level enforcement of a fee as far as I understand it. It's enforced by the miners and their willingness to include a transaction in a block. In addition, it results in more linkage between coins/addresses used, so lower privacy. Not if you only select all the UTXOs from a single address. A wallet that is geared more towards privacy minded individuals may want to reduce the amount of address linkage, but a wallet geared towards the general masses probably won't have to worry so much about that. The only way you can guarantee an economical reason to keep the UTXO set small is by actually having a consensus rule that punishes increasing its size. There's an economical reason right now to keeping the UTXO set small. The smaller it is, the easier it is for the individual to run a full node. The easier it is to run a full node, the faster Bitcoin will spread to the masses. The faster it spreads to the masses, the more valuable it becomes. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] A suggestion for reducing the size of the UTXO database
On Sat, May 9, 2015 at 2:43 PM, Raystonn rayst...@hotmail.com wrote: How about this as a happy medium default policy: Rather than select UTXOs based solely on age and limiting the size of the transaction, we select as many UTXOs as possible from as few addresses as possible, prioritizing which addresses to use based on the number of UTXOs it contains (more being preferable) and how old those UTXOs are (in order to reduce the fee)? If selecting older UTXOs gives higher priority for a lesser (or at least not greater) fee, that is an incentive for a rational user to use the older UTXOs. Such policy needs to be defended or removed. It doesn't support privacy or a reduction in UTXOs. Before starting this thread, I had completely forgotten that age was even a factor in determining which UTXOs to use. Frankly, I can't think of any reason why miners care how old a particular UTXO is when determining what fees to charge. I'm sure there is one, I just don't know what it is. I just tossed it in there as homage to Andreas who pointed out to me that it was still part of the selection criteria. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
