[Bitcoin-development] Full Clients in the future - Blockchain management
Devs, I have decided to upgrade Armory's blockchain utilities, partly out of necessity due to a poor code decision I made before I even decided I was making a client. In an effort to avoid such mistakes again, I want to do it right this time around, and realize that this is a good discussion for all the devs that will have to deal with this eventually... The part I'm having difficulty with, is the idea that in a few years from now, it just may not be feasible to hold transactions file-/pointers/ in RAM, because even that would overwhelm standard RAM sizes. Without any degree of blockchain compression, I see that the most general, scalable solution is probably a complicated one. On the other hand, where this fails may be where we have already predicted that the network will have to split into super-nodes and lite nodes. In which case, this discussion is still a good one, but just directed more towards the super-nodes. But, there may still be a point at which super-nodes don't have enough RAM to hold this data... (1) As for how small you can get the data: my original idea was that the entire blockchain is stored on disk as blk.dat files. I store all transactions as 10-byte file-references. 10 bytes would be -- X in blkX.dat (2 bytes) -- Tx start byte (4 bytes) -- Tx size bytes (4 bytes) The file-refs would be stored in a multimap indexed by the first 6 bytes of the tx-hash. In this way, when I search the multimap, I potentially get a list of file-refs, and I might have to retrieve a couple of tx from disk before finding the right one, but it would be a good trade-off compared to storing all 32 bytes (that's assuming that multimap nodes don't have too much overhead). But even with this, if there are 1,000,000,000 transactions in the blockchain, each node is probably 48 bytes (16 bytes + map/container overhead), then you're talking about 48 GB to track all the data in RAM. mmap() may help here, but I'm not sure it's the right solution (2) What other ways are there, besides some kind of blockchain compression, to maintain a multi-terabyte blockchain, assuming that storing references to each tx would overwhelm available RAM? Maybe that assumption isn't necessary, but I think it prepares for the worst. Or maybe I'm too narrow in my focus. How do other people envision this will be handled in the future. I've heard so many vague notions of well we could do /this/ or /that/, or it wouldn't be hard to do /that/ but I haven't heard any serious proposals for it. And while I believe that blockchain compression will become ubiquitous in the future, not everyone believes that, and there will undoubtedly be users/devs that /want/ to maintain everything under all circumstances. -Alan -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Fwd: Re: Full Clients in the future - Blockchain management
(response from Doug forwarded below) It's a very good point. I have no experience with database engines. I had assumed that in most cases, data could always be indexed in RAM, and wanted to know where to go when that's not the case. I will look into that solution, further. I am very interested to solve the blockchain compression problem, and think I've got a great way that will not just compress the blockchain, but improve the network for lightweight clients. But the idea is not fully formed yet, so I was holding off... Original Message Subject: Re: [Bitcoin-development] Full Clients in the future - Blockchain management Date: Sat, 2 Jun 2012 12:07:44 -0500 From: Douglas Huff m...@jrbobdobbs.org To: Alan Reiner etothe...@gmail.com I think you're trying to solve something a little out of scope, really. Most of the issues aren't really issues for other clients using established storage mechanisms (bdb,SQLite,etc) and they're using them precisely because this is a problem that people have been working on for decades and a poor candidate for reinvention. If you really look at what you're proposing it's fundamentally how bdb operates except your indexing format is usage domain specific and you're in charge of all the resource management semantics. While at the same time you'll be missing many of the newer features that make working with/recovering/diagnosing issues in the storage layer easier. If you're really wanting to talk about pruning methods to prevent the massive amount of duplicated; but no longer pertinent, data that's a different story and please continue. :) -- Douglas Huff On Jun 2, 2012, at 10:40, Alan Reiner etothe...@gmail.com mailto:etothe...@gmail.com wrote: Devs, I have decided to upgrade Armory's blockchain utilities, partly out of necessity due to a poor code decision I made before I even decided I was making a client. In an effort to avoid such mistakes again, I want to do it right this time around, and realize that this is a good discussion for all the devs that will have to deal with this eventually... The part I'm having difficulty with, is the idea that in a few years from now, it just may not be feasible to hold transactions file-/pointers/ in RAM, because even that would overwhelm standard RAM sizes. Without any degree of blockchain compression, I see that the most general, scalable solution is probably a complicated one. On the other hand, where this fails may be where we have already predicted that the network will have to split into super-nodes and lite nodes. In which case, this discussion is still a good one, but just directed more towards the super-nodes. But, there may still be a point at which super-nodes don't have enough RAM to hold this data... (1) As for how small you can get the data: my original idea was that the entire blockchain is stored on disk as blk.dat files. I store all transactions as 10-byte file-references. 10 bytes would be -- X in blkX.dat (2 bytes) -- Tx start byte (4 bytes) -- Tx size bytes (4 bytes) The file-refs would be stored in a multimap indexed by the first 6 bytes of the tx-hash. In this way, when I search the multimap, I potentially get a list of file-refs, and I might have to retrieve a couple of tx from disk before finding the right one, but it would be a good trade-off compared to storing all 32 bytes (that's assuming that multimap nodes don't have too much overhead). But even with this, if there are 1,000,000,000 transactions in the blockchain, each node is probably 48 bytes (16 bytes + map/container overhead), then you're talking about 48 GB to track all the data in RAM. mmap() may help here, but I'm not sure it's the right solution (2) What other ways are there, besides some kind of blockchain compression, to maintain a multi-terabyte blockchain, assuming that storing references to each tx would overwhelm available RAM? Maybe that assumption isn't necessary, but I think it prepares for the worst. Or maybe I'm too narrow in my focus. How do other people envision this will be handled in the future. I've heard so many vague notions of well we could do /this/ or /that/, or it wouldn't be hard to do /that/ but I haven't heard any serious proposals for it. And while I believe that blockchain compression will become ubiquitous in the future, not everyone believes that, and there will undoubtedly be users/devs that /want/ to maintain everything under all circumstances. -Alan -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
[Bitcoin-development] Defeating the block withholding attack
Analysis, comments, constructive criticism, etc welcome for the following: ==Background== At present, an attacker can harm a pool by intentionally NOT submitting shares that are also valid blocks. All pools are vulnerable to this attack, whether centralized or decentralized and regardless of reward system used. The attack's effectiveness is proportional to ratio of the attacker's hashrate to the rest of the pool. There are obvious solutions that can be used to defeat this attack on centralized pools. For example, including a secret in the coinbase transaction that is accepted by the network as a partial preimage proof-of-work. All these solutions require changes to Bitcoin's proof-of-work acceptance terms, and since centralized pools can be harmful to the network's security, these rule changes are not likely to gain enough acceptance among the greater Bitcoin community. ==Proposed Solution== Please comment on the viability of this new proof-of-work algorithm, which I think should be viable for even decentralized pools: Blocks are accepted at a lower difficulty N (choosable by the pool; eg, the share difficulty) iff they are submitted with a candidate for the next block and SHA256(SHA256(NewBlockHash + NextBlockCandidateHash)) meets difficulty M. The relationship between M and N must be comparable to the normal network difficulty; details on the specifics of this can be figured out later, ideally by someone more qualified than me. M and N must be chosen prior to searching for the block: it should be safe to steal some always-zero bytes from the prevblock header for this. This algorithm should guarantee that every share has an equal chance of being a valid block at the time it is found, and that which ones are actually blocks cannot be known until the subsequent block is found. Thus, attackers have no way to identify which shares to withhold even while they have full knowledge of the shares/blocks themselves. ==Backward Compatibility== Obviously, this change creates a hard-fork in the blockchain. I propose that if it solves the block withholding risk, the gain is sufficient that the community may approve a hard-fork to take place 1-2 years from consensus. Since mining continues to use a double-SHA256 on a fixed 80 byte header, existing miners, FPGAs, etc should work unmodified. Poolservers will need to adapt significantly. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
[Bitcoin-development] Fwd: Defeating the block withholding attack
On Sat, Jun 2, 2012 at 7:52 PM, Luke-Jr l...@dashjr.org wrote: Analysis, comments, constructive criticism, etc welcome for the following: ==Background== At present, an attacker can harm a pool by intentionally NOT submitting shares that are also valid blocks. All pools are vulnerable to this attack, whether centralized or decentralized and regardless of reward system used. The attack's effectiveness is proportional to ratio of the attacker's hashrate to the rest of the pool. This attack has an obvious signature: getting outworked on the same block as the pool was trying to verify, and always by the same person. There are obvious solutions that can be used to defeat this attack on centralized pools. For example, including a secret in the coinbase transaction that is accepted by the network as a partial preimage proof-of-work. All these solutions require changes to Bitcoin's proof-of-work acceptance terms, and since centralized pools can be harmful to the network's security, these rule changes are not likely to gain enough acceptance among the greater Bitcoin community. ==Proposed Solution== Please comment on the viability of this new proof-of-work algorithm, which I think should be viable for even decentralized pools: Blocks are accepted at a lower difficulty N (choosable by the pool; eg, the share difficulty) iff they are submitted with a candidate for the next block and SHA256(SHA256(NewBlockHash + NextBlockCandidateHash)) meets difficulty M. The relationship between M and N must be comparable to the normal network difficulty; details on the specifics of this can be figured out later, ideally by someone more qualified than me. M and N must be chosen prior to searching for the block: it should be safe to steal some always-zero bytes from the prevblock header for this. So the goal is to prevent the attacker double-dipping by submitting cycles to the pool, which if he found a correct answer he could submit himself. I don't see how this does that: if he finds a valid block he finds a valid block. Only if the operator has a secret is this prevented. This algorithm should guarantee that every share has an equal chance of being a valid block at the time it is found, and that which ones are actually blocks cannot be known until the subsequent block is found. Thus, attackers have no way to identify which shares to withhold even while they have full knowledge of the shares/blocks themselves. This further delays the finalization of a transaction. That's not a good thing. ==Backward Compatibility== Obviously, this change creates a hard-fork in the blockchain. I propose that if it solves the block withholding risk, the gain is sufficient that the community may approve a hard-fork to take place 1-2 years from consensus. Since mining continues to use a double-SHA256 on a fixed 80 byte header, existing miners, FPGAs, etc should work unmodified. Poolservers will need to adapt significantly. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Those who would give up Essential Liberty to purchase a little Temporary Safety deserve neither Liberty nor Safety. -- Benjamin Franklin -- Those who would give up Essential Liberty to purchase a little Temporary Safety deserve neither Liberty nor Safety. -- Benjamin Franklin -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
