Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?
I agree that the use protocol buffer and x509 by BIP70 is a poor choice. The choice should have been done to maximize portability, not to maximize efficiency and flexibility. What I ended up doing for having a similar codebase on all plateform is to parse a BIP70 messages with the help of a web service that convert it to JSON. I don't like this solution since it had a trust dependency, and the certificate verification become handled by the web service, not the device. But even if I solved google buffer problem, I would stumble upon having headache to validate the x509 certificate chain on every plateforms. A simple BIP70 using JSON + HTTPS would have make things more easy. I agree that it requires that the merchant own the domain name of the BIP70 endpoint, but I don't consider such a big of a deal, since this is how e-commerce works. -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?
Sure I know that x509 is international standard. And that HTTPS uses TLS. This is not my point, my point is that when we use HTTPS the developer delegates certificates verification to the plateform he is running on, so developer don't have to bother about it, making the implementation safer and easier. On the other hand, if you charge the developer (and not the plateform) to check certificate validity, it means that you have to develop a different codebase for all plateform you are targeting, because each plateform store trusted root certificate in a different manner with different APIs, and also have different types representing a X509 Certificate. So, let's say I want to target IOS + WP + Android + WinRT + desktop win, I need to develop 4 times chain verification and certificate parsing. (Because I can't verify a certificate if it is not in the specific type of the underlying plateform) And since it would take too much time to do that, I end up delegating parsing and trust verification to a third party service. 2015-01-28 14:32 GMT+01:00 Wladimir : > > On Wed, 28 Jan 2015, Nicolas DORIER wrote: > > I agree that the use protocol buffer and x509 by BIP70 is a poor choice. >> > > Well x509 is an international standard in common use, you can't do much > better with regard to portability. Your suggestion about HTTPS makes little > sense, you do know what TLS uses x509 internally as well? > > Re: protocol buffers, I don't know if it's the best possible one, but one > serialization method had to be picked. If it weren't, we could still have > still been discussing which one to use by now. Just like for JSON there are > bindings for many languages. > > Though JSON parsers are much more diverse, which people using Bitcoin > Core's RPC have bumped into e.g. some have some problems handling large > numbers. Something you wouldn't expect using a straightforward binary > format. There's no obvious best choice. > > Wladimir > -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?
My point is not that there is a limitation in BIP70. My point is that you put the burden of certificate verification on developer's shoulder when we can just leverage built in HTTPS support of the platform. This make cross plateform dev a nightmare. Sure I can use a snapshot of moz/apple/msft store. I depends on BouncyCastle, as bitcoinj, so I theorically can use that way. However, if you want to use your plateform's store, then you are toasted, and the code for converting from BC X509 Certificate to one of each plateform is not obvious and is a headache. Thing that could be just left to the HTTPS support of your plateform. Have you tried to do that on windows RT and IOS ? I tried, and I quickly stopped doing that since it is not worth the effort. (Frankly I am not even sure you can on win rt, since the API is a stripped down version of windows) Why have you not heard about the problem ? (until now, because I have this problem because I need to have the same codebase on winrt/win/android/ios/tablets) Because bitcoinj just rely either java's own abstraction of certificate or on BC one. But I highly doubt they are using the plateform store, and even if you theorically can, dealing with X509 is very prone to error... for something that the plateform should just do for you. Also, you bundle mozilla's store in bitcoinj, what happen when the store change and your customer have not intent to use bitcoinj new version ? by leveraging the plateform you benefit from automatic updates. Also, does java stores deals with certificate revocations ? sure you can theorically code that too... or just let the plateform deals with it. BIP70 does not limit to anything but it is a gigantic pain in the ass for easy cross development because of protobuff and embedded certificates. BIP70 is a client side technology, not a performance and storage critical data structure. The only valid point of having embedded certificates is to allow the owner of the website to be different from the merchant. But since merchants often have their own website, a protocol without having to reinvent x509 would have been better suited to current needs. 2015-01-28 16:42 GMT+01:00 Mike Hearn : > On the other hand, if you charge the developer (and not the plateform) to >> check certificate validity, it means that you have to develop a different >> codebase for all plateform you are targeting, because each plateform store >> trusted root certificate in a different manner with different APIs, and >> also have different types representing a X509 Certificate. >> > > That's what cross-platform abstraction libraries are for. Both Java and Qt > provide a key store library that can load from either the OS root store or > a custom one. If your chosen app platform doesn't, OK, then you'll have to > make or find one yourself. Perhaps contribute it upstream or make it a > library. But that's not a limitation of BIP70. > > Just as a reminder, there is no obligation to use the OS root store. You > can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT > etc stores and load it in your app. We do this in bitcoinj by default to > avoid cases where BIP70 requests work on some platforms and not others, > although the developer can easily override this and use the OS root store > instead. > > Of all possible solutions, using a third party service to convert things > to JSON is one of the least obvious and highest effort. I don't know anyone > else who arrived at such a conclusion and respectfully disagree that this > is a problem with the design choices in BIP70. It sounds like a bizarre > hack around lack of features in whatever runtime you're using. > > -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?
For the number of field there is in the spec, I don't consider having a JSON to schama really worthwhile. If you fear it is error prone, then we should provide some testing data for the BIP70. (Which I already did for protobuf, but was rejected, because deemed no useful thanks to the code generator... But such code generator gave me inconsistencies with gavin's implementation for example) Why do you think type support is very useful in our case ? we have 3 types, and dealing only with bytes, int, and string. It cost me more time to find a suitable cross plateform lib for protobuf (in c#, that works in ios and winrt) than I would by just coding the json wrapper classes by hand. (JSON libs are more wildspread and supported than protobuf) 2015-01-28 17:04 GMT+01:00 Jeff Garzik : > Not to mention the tiresome and error-prone task of writing your own > JSON-to-schema marshalling code -- or something equivalent to the protobufs > compiler and libs for JSON. > > protobufs -- and its modern competitors such as msgpack -- natively > provide type support in a way that must be hacked into JSON or XML. > > The protobuf/msgpack design is engineered to avoid bugs routinely found in > JSON parsing code; due to the amount of code & effort involved in JSON > input sanity checking, bugs and inconsistencies inevitable arise. We have > seen this in bitcoind with JSON-RPC. > > > > On Wed, Jan 28, 2015 at 10:42 AM, Mike Hearn wrote: > >> On the other hand, if you charge the developer (and not the plateform) to >>> check certificate validity, it means that you have to develop a different >>> codebase for all plateform you are targeting, because each plateform store >>> trusted root certificate in a different manner with different APIs, and >>> also have different types representing a X509 Certificate. >>> >> >> That's what cross-platform abstraction libraries are for. Both Java and >> Qt provide a key store library that can load from either the OS root store >> or a custom one. If your chosen app platform doesn't, OK, then you'll have >> to make or find one yourself. Perhaps contribute it upstream or make it a >> library. But that's not a limitation of BIP70. >> >> Just as a reminder, there is no obligation to use the OS root store. You >> can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT >> etc stores and load it in your app. We do this in bitcoinj by default to >> avoid cases where BIP70 requests work on some platforms and not others, >> although the developer can easily override this and use the OS root store >> instead. >> >> Of all possible solutions, using a third party service to convert things >> to JSON is one of the least obvious and highest effort. I don't know anyone >> else who arrived at such a conclusion and respectfully disagree that this >> is a problem with the design choices in BIP70. It sounds like a bizarre >> hack around lack of features in whatever runtime you're using. >> >> >> >> -- >> Dive into the World of Parallel Programming. The Go Parallel Website, >> sponsored by Intel and developed in partnership with Slashdot Media, is >> your >> hub for all things parallel software development, from weekly thought >> leadership blogs to news, videos, case studies, tutorials and more. Take a >> look and join the conversation now. http://goparallel.sourceforge.net/ >> ___ >> Bitcoin-development mailing list >> Bitcoin-development@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> >> > > > -- > Jeff Garzik > Bitcoin core developer and open source evangelist > BitPay, Inc. https://bitpay.com/ > -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?
Mike, I am not denying it is impossible to do all of that. Just that it is not a trivial stuff to do to make it works everywhere, and I think that it is not a good thing for a client side technology. BIP70 has its use, and I understand why there is case where it is good to ship the certs in the message and not depends on the transport. But a standard that just use JSON and HTTPS, even if less flexible that BIP70, would make it easier and sufficient for today's use case. On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn wrote: > My point is not that there is a limitation in BIP70. My point is that you >> put the burden of certificate verification on developer's shoulder when we >> can just leverage built in HTTPS support of the platform. >> > > Platforms that support HTTPS but not certificate handling are rare - I > know HTML5 is such a platform but such apps are inherently dependent on the > server anyway and the server can just do the parsing and validation work > itself. If WinRT is such a platform, OK, too bad. > > The embedding of the certificates is not arbitrary or pointless, by the > way. It's there for a very good reason - it makes the signed payment > request verifiable by third parties. Effectively you can store the signed > message and present it later to someone else, it's undeniable. Combined > with the transactions and merkle branches linking them to the block chain, > what you have is a form of digital receipt ... a proof of purchase that can > be automatically verified as legitimate. This has all kinds of use cases. > > Because of how HTTPS works, you can't easily prove to a third party that a > server gave you a piece of data. Doing so requires staggeringly complex > hacks (see tls notary) and when we designed BIP70, those hacks didn't even > exist. So we'd lose the benefit of having a digitally signed request. > > Additionally, doing things this way means BIP70 requests can be signed by > things which are not HTTPS servers. For example you can sign with an email > address cert, an EV certificate i.e. a company, a certificate issued by > some user forum, whatever else we end up wanting. Not every payment > recipient can be identified by a domain name + dynamic session. > > >> However, if you want to use your plateform's store, then you are toasted >> > > That's a bit melodramatic. BitcoinJ is able to use the Android, JRE, > Windows and Mac certificate stores all using the same code or very minor > variants on it (e.g. on Mac you have to specify you want the system store > but it's a one-liner). > > Yes, that's not *every* platform. Some will require custom binding glue > and it depends what abstractions and languages you are using. > > >> Have you tried to do that on windows RT and IOS ? I tried, and I quickly >> stopped doing that since it is not worth the effort. (Frankly I am not even >> sure you can on win rt, since the API is a stripped down version of windows) >> > > There is code to do iOS using the Apple APIs here: > > > https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391 > > >> Why have you not heard about the problem ? (until now, because I have >> this problem because I need to have the same codebase on >> winrt/win/android/ios/tablets) >> > > WinRT is a minority platform in the extreme, and all the other platforms > you mentioned have the necessary APIs. Java abstracts you from them. So I > think you are encountering this problem because you desire to target WinRT > and other platforms with a single codebase. That's an unusual constraint. > > AFAIK the only other people who encountered this are BitPay, because they > want to do everything in Javascript which doesn't really provide any major > APIs. > > >> Also, you bundle mozilla's store in bitcoinj, what happen when the store >> change and your customer have not intent to use bitcoinj new version ? by >> leveraging the plateform you benefit from automatic updates. >> > > Yes, there are pros and cons to bundling a custom root store. > > >> Also, does java stores deals with certificate revocations ? sure you can >> theorically code that too... or just let the plateform deals with it. >> > > It can do OCSP checks, yes, although I believe no wallets currently do so. > A better solution would be to implement an OCSP stapling extension to BIP70 > though. > -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?
Sure, But the mobile targets, it is still easier to use Json + HTTPS, especially when you want one code base for everything. And as you said, developers need to think about fetching mozilla store time to time, and check revocations themselves. This is not obvious thing to do, and hard to test correctly. If your use case was the primary utility of BIP70, then I'd say it fit the bill. But for cross plateform client development an atlernative would be easier. > why not allow both serializations and keep serialization format a parameter, keep everyone happy. It would be another BIP, because if we use JSON with HTTPS, the difference is also in the semantic (no embedded certificates) I will likely provide this option for a product I am developing. I will only use another Content Type. We'll see then how it goes. 2015-01-28 18:14 GMT+01:00 Mike Hearn : > I think we'll just have to agree to disagree on this one. I've implemented > BIP70 a couple of times now and didn't find it to be difficult. I know you > had odd problems with the C# protobuf implementation you were using but > library bugs can happen for any kind of programming. > > I forgot to mention the other reason it's done this way. One of the > driving goals of BIP70 was to support the TREZOR and similar devices. For > hardware wallets, it's critical to keep the amount of code they need to run > as small as possible. Any bugs in the code there can cause security holes > and lead to the device being hacked. > > Doing it the way you suggest would mean the secure code would have to > contain complex and bug-prone text parsing logic as well as a full blown > HTTP and SSL stack, that requires not only X.509 handling but also lots of > other stuff on top. It'd increase cost, complexity and decrease security > quite a bit. > > Whilst I appreciate if your platform provides a scripting-like API and > nothing low level it might seem easier to use JSON+HTTPS, that isn't the > case for one of the primary design targets. > > > > On Wed, Jan 28, 2015 at 6:04 PM, Nicolas Dorier > wrote: > >> Mike, I am not denying it is impossible to do all of that. >> Just that it is not a trivial stuff to do to make it works everywhere, >> and I think that it is not a good thing for a client side technology. >> BIP70 has its use, and I understand why there is case where it is good to >> ship the certs in the message and not depends on the transport. >> >> But a standard that just use JSON and HTTPS, even if less flexible that >> BIP70, would make it easier and sufficient for today's use case. >> >> On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn wrote: >> >>> My point is not that there is a limitation in BIP70. My point is that >>>> you put the burden of certificate verification on developer's shoulder when >>>> we can just leverage built in HTTPS support of the platform. >>>> >>> >>> Platforms that support HTTPS but not certificate handling are rare - I >>> know HTML5 is such a platform but such apps are inherently dependent on the >>> server anyway and the server can just do the parsing and validation work >>> itself. If WinRT is such a platform, OK, too bad. >>> >>> The embedding of the certificates is not arbitrary or pointless, by the >>> way. It's there for a very good reason - it makes the signed payment >>> request verifiable by third parties. Effectively you can store the signed >>> message and present it later to someone else, it's undeniable. Combined >>> with the transactions and merkle branches linking them to the block chain, >>> what you have is a form of digital receipt ... a proof of purchase that can >>> be automatically verified as legitimate. This has all kinds of use cases. >>> >>> Because of how HTTPS works, you can't easily prove to a third party that >>> a server gave you a piece of data. Doing so requires staggeringly complex >>> hacks (see tls notary) and when we designed BIP70, those hacks didn't even >>> exist. So we'd lose the benefit of having a digitally signed request. >>> >>> Additionally, doing things this way means BIP70 requests can be signed >>> by things which are not HTTPS servers. For example you can sign with an >>> email address cert, an EV certificate i.e. a company, a certificate issued >>> by some user forum, whatever else we end up wanting. Not every payment >>> recipient can be identified by a domain name + dynamic session. >>> >>> >>>> However, if you want to use your plateform's store, then you are toasted >>
[Bitcoin-development] Solution for Block Size Increase
Executive Summary: I explain the objectives that we should aim to reach agreement without drama, controversy, and relief the core devs from the central banker role. (As Jeff Garzik pointed out) Knowing the objectives, I propose a solution based on the objectives that can be agreed on tomorrow, would permanently fix the block size problem without controversy and would be immediately applicable. The objectives: There is consensus on the fact that nobody wants the core developers to be seen as central bankers. There is also consensus that more decentralization is better than less. (assuming there is no cost to it) This means you should reject all arguments based on economical, political and ideological principles about what Bitcoin should become. This includes: 1) Whether Bitcoin should be storage of value or suitable for coffee transaction, 2) Whether we need a fee market, block scarcity, and how much of it, 3) Whether we need to periodically increase block size via some voodoo formula which speculate on future bandwidth and cost of storage, Taking decisions based on such reasons is what central bankers do, and you don’t want to be bankers. This follow that decisions should be taken only for technical and decentralization considerations. (more about decentralization after) Scarcity will evolve without you taking any decisions about it, for the only reason that storage and bandwidth is not free, nor a transaction, thanks to increased propagation time. This backed in scarcity will evolve automatically as storage, bandwidth, encoding, evolve without anybody taking any decision, nor making any speculation on the future. Sadly, deciding how much decentralization should be in the system by tweaking the block size limit is also an economic decision that should not have its place between the core devs. This follow : 4) Core devs should not decide about the amount of suitable decentralization by tweaking block size limit, Still, removing the limit altogether is a no-no, what would happen if a block of 100 GB is created? Immediately the network would be decentralized, not only for miners but also for bitcoin service providers. Also, core devs might have technical consideration on bitcoin core which impose a temporary limit until the bug resolved. The solution: So here is a proposal that address all my points, and, I think, would get a reasonable consensus. It can be published tomorrow without any controversy, would be agreed in one year, and can be safely reiterated every year. Developers will also not have to play politics nor central banker. (well, it sounds to good to be true, I waiting for being wrong) The solution is to use block voting. For each block, a miner gives the size of the block he would like to have at the next deadline (for example, 30 may 2015). The rational choice for them is just enough to clear the memory pool, maybe a little less if he believes fee pressure is beneficial for him, maybe a little more if he believes he should leave some room for increased use. At the deadline, we take the median of the votes and implement it as a new block size limit. Reiterate for the next year. Objectives reached: - No central banking decisions on devs shoulder, - Votes can start tomorrow, - Implementation has only to be ready in one year, (no kick-in-the-can) - Will increase as demand is growing, - Will increase as network capacity and storage is growing, - Bitcoin becomes what miners want, not what core devs and politician wants, - Implementation reasonably easy, - Will get miner consensus, no impact on existing bitcoin services, Unknown: - Effect on bitcoin core stability (core devs might have a valid technical reason to impose a limit) - Maybe a better statistical function is possible Additional input for the debate: Some people were debating whether miners are altruist or act rationally. We should always expect them to act rationally, but we should not forget the peculiarity of TCP backoff game: While it is in the best interest of players to NOT reemit TCP packet with a backoff if the ACK is not received, everybody does it. (Because of the fallacy that changing a TCP implementation is costless) Often, when we think a real life situation is a prisoner dilemma problem, it turns out that the incentives where just incorrectly modeled. Core devs, thanks for all your work, but please step out of the banker's role and focus on where you are the best, I speak as an entrepreneur that doesn't want decisions about bitcoin to be taken by who has the biggest. If the decision of the hard limit is taken for other than purely technical decisions, ie, for the maximization of whatever metric, it will clearly put you in banker's shoes. As an entrepreneur, I have other things to speculate than who gets the biggest gun in the core team. Please consider my solution, Nicolas Dorier, ---
