Re: [cryptography] data integrity: secret key vs. non-secret verifier; and: are we winning? (was: “On the limits of the use cases for authenticated encryption”)
On Wed, Apr 25, 2012 at 9:27 PM, Marsh Ray ma...@extendedsubset.com wrote: On 04/25/2012 10:11 PM, Zooko Wilcox-O'Hearn wrote: 1. the secret-oriented way: you make a MAC tag of the chunk (or equivalently you use Authenticated Encryption on it) using a secret key known to the good guy(s) and unknown to the attacker(s). 2. the verifier-oriented way: you make a secure hash of the chunk, and make the resulting hash value known to the good guy(s) in an authenticated way. Is option 2 sort of just pushing the problem around? What's going on under the hood in the term in an authenticated way? How do you do authentication in an automated system without someone somewhere keeping something secret? Is authenticating the hash value fundamentally different from ensuring the integrity of a chunk of data? Those are definitely the right sorts of questions, Marsh. I think that from our bias as crypto engineers familiar with protocols like TLS and SSH, it seems like approach 1 is natural, or easy, or even the only real, complete solution, but I think it is deceptive. I think _both_ approaches are sort of just pushing the problem around, and I suspect that approach 1 actually leaves you with a harder problem left to solve than approach 2 does. Observe that there is an in an authenticated way problem hiding in option 1 as well -- someone has to distribute the secret keys to the legitimate readers in an authenticated way at some point. Basically, as security engineers we tend to assume as a starting point that there is some secret we can use which is known to the good guy(s) and unknown to the attackers. But this isn't really a fair assumption. It implies quite a lot of work that someone else is going to have to do to make that true for us (especially in the multi-party case, but we already don't seem to have solved the problem very well in the traditional 2 party case!), and in practice it seems to often fail. The secrets often turn out to be known to the attackers or unknown to the intended recipients. ... Um, frankly I'm having a hard time understanding exactly why my intuitions about this come out so differently for data-at-rest tools like Tahoe-LAFS and ZFS than for data-in-motion tools like TLS. My intuition is that secret-based integrity-checking is fine for the traditional two-party, time-limited session encryption a la TLS, but that hash-based integrity-checking is more robust and pushes less of the problem around when there are more than two parties and when the data is persistent. The intuitions of the ZFS crypto designers (Darren Moffat and others including Nico Williams) seems to have been that secret-based integrity-checking still had some use even in that scenario. However, I don't remember precisely what ZFS settled on for secure data integrity checking. I think to understand the trade-offs of these two options better we would need an example system in which they could be used. My favorite example is obviously Tahoe-LAFS, but I'm not sure if you would learn from considering that example. There is no use of symmetric MAC (nor Authenticated Encryption) anywhere in the Tahoe-LAFS data formats [*]. All data-integrity mechanisms are in the style of option 2 -- using secure hashes of the data as the verification tag. In some cases (immutable files and directories) that secure-hash-based integrity check is the only integrity check. In others (mutable files and directories), it is combined with a public key (RSA) digital signature, where the message being signed includes the hash value. I can't imagine how one could build a Tahoe-LAFS-like thing using the secret-key integrity-checking approach instead. It seems like it would make it impossible to give someone read-access to a file without also giving them write-access. I.e. they would have to know the secret in order to verify the contents of the file, and knowledge of the secret would empower them to undetectably alter the contents of the file. I guess this would make you want to share files with as few other people as possible, and for as short of a time as possible, thus pushing you back toward the two-party session type of usage. Regards, Zooko [*] There are actually a couple of uses of symmetric MAC in the Tahoe-LAFS system, but not for data integrity checking on the actual file contents nor on the file metadata or directories, so let's ignore those. They have to do with control plane stuff -- network establishment and so on... ___ cryptography mailing list cryptography@randombit.net http://lists.randombit.net/mailman/listinfo/cryptography
Re: [cryptography] data integrity: secret key vs. non-secret verifier; and: are we winning? (was: “On the limits of the use cases for authenticated encryption”)
I think the separate integrity tag is more general, flexible and more secure where the flexibility is needed. Tahoe has more complex requirements and hence needds to make use of a separate integrity tag. I guess in general it is going to be more general, flexible if there are separate keys (including none with keyless self-authenticated URLs) for different properties. Hence there remains a need for separate integrity and encryption even with authenticated encryption modes. And typically AE modes have a cost - several of the standardized encryption modes are actually just standardizing ways to combine separate integrity encryption primitives. The others are mostly patented. They tend to be more fragile through binary reliance on strictly one use nonces, XOR via counter mode and such modes which are I think in implementation terms unforgiving or fragile. Exercise for the reader to list the non-patented, non-trivial (combining an integrity encryption primitive) modes :) Adam ___ cryptography mailing list cryptography@randombit.net http://lists.randombit.net/mailman/listinfo/cryptography
Re: [cryptography] data integrity: secret key vs. non-secret verifier; and: are we winning? (was: “On the limits of the use cases for authenticated encryption”)
On 04/25/2012 10:11 PM, Zooko Wilcox-O'Hearn wrote: It goes like this: suppose you want to ensure the integrity of a chunk of data. There are at least two ways to do this (excluding public key digital signatures): 1. the secret-oriented way: you make a MAC tag of the chunk (or equivalently you use Authenticated Encryption on it) using a secret key known to the good guy(s) and unknown to the attacker(s). 2. the verifier-oriented way: you make a secure hash of the chunk, and make the resulting hash value known to the good guy(s) in an authenticated way. Is option 2 sort of just pushing the problem around? What's going on under the hood in the term in an authenticated way? How do you do authentication in an automated system without someone somewhere keeping something secret? Is authenticating the hash value fundamentally different from ensuring the integrity of a chunk of data? - Marsh ___ cryptography mailing list cryptography@randombit.net http://lists.randombit.net/mailman/listinfo/cryptography
Re: [cryptography] data integrity: secret key vs. non-secret verifier; and: are we winning? (was: “On the limits of the use cases for authenticated encryption”)
You'd have to ask Darren, but IIRC the design he settled on allows for unkeyed integrity verification and repair. I too think that's a critical feature to have even if having it were to mean leaking some information, such as file length in blocks, and number of files, as I look at this from an operations perspective. Nico -- ___ cryptography mailing list cryptography@randombit.net http://lists.randombit.net/mailman/listinfo/cryptography
Re: [cryptography] data integrity: secret key vs. non-secret verifier; and: are we winning? (was: “On the limits of the use cases for authenticated encryption”)
On Wed, Apr 25, 2012 at 10:27 PM, Marsh Ray ma...@extendedsubset.com wrote: On 04/25/2012 10:11 PM, Zooko Wilcox-O'Hearn wrote: 2. the verifier-oriented way: you make a secure hash of the chunk, and make the resulting hash value known to the good guy(s) in an authenticated way. Is option 2 sort of just pushing the problem around? What's going on under the hood in the term in an authenticated way? How do you do authentication in an automated system without someone somewhere keeping something secret? Is authenticating the hash value fundamentally different from ensuring the integrity of a chunk of data? You have two choices for providing AE and (2): a) MAC the root of each file's (or directory's, or dataset's) Merkle hash tree, or b) store a hash and a MAC, thereby forming a Merkle hash tree and a parallel Merkle MAC tree. In terms of additional storage and compute power (a) is clearly superior. I believe the security of (a) is adequate. Nico -- ___ cryptography mailing list cryptography@randombit.net http://lists.randombit.net/mailman/listinfo/cryptography
Re: [cryptography] data integrity: secret key vs. non-secret verifier; and: are we winning? (was: “On the limits of the use cases for authenticated encryption”)
Also, On Wed, Apr 25, 2012 at 10:11 PM, Zooko Wilcox-O'Hearn zo...@zooko.com wrote: Hello Nico Williams. Nice to hear from you. Yes, when David-Sarah Hopwood and I (both Tahoe-LAFS hackers) participated on the zfs-crypto mailing list with you and others, I learned about a lot of similarities between Tahoe-LAFS and ZFS. Yes, I remember that too. It was fun and enlightening. But in traditional network protocols (TLS, SSHv2, ESP, ...) I have to strain to think of reasons to not use AE when you want confidentiality protection (encryption). Yes, I agree with you on that. And OTR ¹, CurveCP ², mosh ³, tcpcrypt ⁴, and ZRTP ⁵. All of these eight protocols we've just named have in common that there are only two parties, that only current data in-flight is protected, and that the protocol has already ensured (more or less -- haha) a shared secret key known to both of the users and not to any attackers. Remember that in ZFS we also speak of end-to-end integrity protection, except in ZFS there's a single end: the system that implements it, and the attackers are presumed to be between that system and its storage devices. It's end-to-end because even though there's only one end, that end is effectively communicating with itself [over untrusted storage media]. The on-disk format is the equivalent of secure transport protocol (with SAS, IB, .. being the equivalent of TCP/IP). Of course, if you access said storage from multiple heads then there will be multiple ends, but since only one can be writing at any given time (and really, even reading)... In ZFS w/ encryption there are no additional ends, and protection against a local privileged agent is not in scope, but protection of data at rest (on the storage devices making up the ZFS volumes) is in scope. Additional protection is available when and for as long as the keys are not loaded on the system running ZFS. I think the distinction between filesystems on the one hand and communication protocols on the other is that in the first case we always have snapshots of data that we can apply Merkle hash trees to, and we always have *all* the data available and subject to use and re-use in random access patterns at any given time, including years later. Whereas in the second case the data is ephemeral, consumed and thrown away or otherwise transformed (outside the scope of the transport protocol) as soon as possible -- there's no need to consider an attack where a block earlier in the octet/message stream gets modified after it's been received and consumed. We could store files as TLS streams using PSK and have those shared keys be the files' keys, but that would be inefficient, particularly if you were to need to write in any fashion other than strictly append-only. This distinction is what I believe drives us to design/apply completely different cryptographic protocols to the two types of protocols. I don't question the usefulness of the Authenticated Encryption abstraction for protocols that fall into that category. Right, me either. I can't even imagine not using AE in that context, whether by generic composition or -much better- via integrated AE ciphers/cipher modes. Nico -- ___ cryptography mailing list cryptography@randombit.net http://lists.randombit.net/mailman/listinfo/cryptography
Re: [cryptography] data integrity: secret key vs. non-secret verifier; and: are we winning? (was: “On the limits of the use cases for authenticated encryption”)
On 2012-04-26 1:11 PM, Zooko Wilcox-O'Hearn wrote: how are we doing? Are we winning? I don't know about you, but I consider myself to be primarily a producer of defense technology. I'd like for every individual on the planet to have confidentiality, data integrity, to be able to share certain access with chosen people while denying it to everyone else, and more. Judging from the news and the chatter, offense is winning. Apparently almost nobody actually enjoys this kind of protection against the modern attacker, even big organizations who pay big money for it, much less the rest of the populace. Obviously this is a solved problem in principle. In principle, we know how to do it. In practice not. So it is a UI problem. Zooko envisaged how to solve the UI problem in principle, but Tahoe does not seem user friendly, nor scalable as an information sharing and transmission solution. It is narrowly targeted at solving the cloud backup problem. ___ cryptography mailing list cryptography@randombit.net http://lists.randombit.net/mailman/listinfo/cryptography
