Re: Switching from SHA1 to a checksum type without known collisions in 1.15 working copy format
Daniel Sahlberg writes: > As far as I understand, the point of multi-hash is to keep the WC format > between versions (so older clients can continue to use the WC). Just as a minor note, the working copies created using the implementation on the `pristine-checksum-salt` branch don't multi-hash the contents, but rather make the [single] used checksum kind configurable and persist it at the moment when a working copy is created or upgraded. > I need some help to understand how that would work in practice. Let's say > that 1.15 adds SHAABC, 1.16 adds SHAXYZ. Then 1.17 drops SHA1. But... > - A 1.17 client will only use SHAABC or SHAXYZ hashes. > - A 1.16 client can use SHA1, SHAABC and SHAXYZ hashes. > - A 1.15 client can only use SHA1 and SHAABC hashes. > > How can these work together? A WC created in 1.17 can't be used by a 1.15 > client and a WC created in 1.15 (with SHA1) can't be used by a 1.17 client. > How is this different from bumping the format? How do we detect this? In the current design available on the `pristine-checksum-salt` branch, the supported checksum kinds are tied to a working copy format, and any supported checksum kind may additionally use a dynamic salt. For example, format 33 supports only SHA-1 (regular or dynamically salted), but a newer format 34 can add support for another checksum kind such as SHA-2 if necessary. When an existing working copy is upgraded to a newer format, its current checksum kind is retained as is (we can't rehash the content in a `--store-pristine=no` case because the pristines are not available). I don't know if we'll find ourselves having to forcefully phase out SHA-1 *even* for such working copies that retain an older checksum kind, i.e., it might be enough to use the new checksum kind only for freshly created working copies. However, there would be a few options to consider: I think that milder options could include warning the user to check out a new working copy (that would use a different checksum kind), and a harsher option could mean adding a new format that doesn't support SHA-1 under any circumstances, and declaring all previously available working copy formats unsupported. Regards, Evgeny Kotkov
Re: Switching from SHA1 to a checksum type without known collisions in 1.15 working copy format
On 18.01.2024 08:43, Daniel Sahlberg wrote: As far as I understand, the point of multi-hash is to keep the WC format between versions (so older clients can continue to use the WC). I need some help to understand how that would work in practice. Let's say that 1.15 adds SHAABC, 1.16 adds SHAXYZ. Then 1.17 drops SHA1. But... - A 1.17 client will only use SHAABC or SHAXYZ hashes. - A 1.16 client can use SHA1, SHAABC and SHAXYZ hashes. - A 1.15 client can only use SHA1 and SHAABC hashes. How can these work together? A WC created in 1.17 can't be used by a 1.15 client and a WC created in 1.15 (with SHA1) can't be used by a 1.17 client. How is this different from bumping the format? How do we detect this? It's just another dimension of changing the format. When you introduce multihash, you have to bump the format number so that clients that don't know about it won't try to use the WC. Clients that _do_ know about it will have to check which hash algorithm(s) are used in any case. At least, we'd need some method of updating the hashes in the database, akin the WC format upgrades in some versions (was it 1.8?). "svn upgrade" is where this would happen. On the multi-wc-format branch (if memory serves), it accepts a target WC version -- which is equivalent to the feature set supported by the WC. There's no reason why it couldn't also grow a "--force-hash=quantum-entangled" option. -- Brane
Re: Getting to first release of pristines-on-demand feature (#525).
Evgeny Kotkov writes: > Merged in https://svn.apache.org/r1905955 > > I'm going to respond on the topic of SHA1 a bit later. For the history: thread [1] proposes the `pristine-checksum-salt` branch that adds the infrastructure to support new pristine checksum kinds in the working copy and makes a switch to the dynamically-salted SHA1. >From the technical standpoint, I think that it would be better to release the first version of the pristines-on-demand feature having this branch merged, because now we rely on the checksum comparison to determine if a file has changed — and currently it's a checksum kind with known collisions. At the same time, having that branch merged probably isn't a formal release blocker for the pristines-on-demand feature. Also, considering that the `pristine-checksum-salt` branch is currently vetoed by danielsh (presumably, for an indefinite period of time), I'd like to note that personally I have no objections to proceeding with a release of the pristines-on-demand feature without this branch. [1] https://lists.apache.org/thread/xmd7x6bx2mrrbw7k5jr1tdmhhrlr9ljc Regards, Evgeny Kotkov
Re: Switching from SHA1 to a checksum type without known collisions in 1.15 working copy format
Daniel Shahaf writes: > Procedurally, the long hiatus is counterproductive. This reminds me that the substantive discussion of your veto ended with my email from 8 Feb 2023 that had four direct questions to you and was left without an answer: `` > That's not how design discussions work. A design discussion doesn't go > "state decision; state pros; implement"; it goes "state problem; discuss > potential solutions, pros, cons; decide; implement" (cf. [4, 5, 6]). Well, I think it may not be as simple as it seems to you. Who decided that we should follow the process you're describing? Is there a thread with a consensus on this topic? Or do you insist on using this specific process because it's the only process that seems obvious to you? What alternatives to it have been considered? As far as I can tell, the process you're suggesting is effectively a waterfall-like process, and there are quite a lot of concerns about its effectiveness, because the decisions have to be made in the conditions of a lack of information. `` It's been more than 11 months since that email, and those questions still don't have an answer. So if we are to resume this discussion, let's do it from the proper point. > You guys are welcome to try to /convince/ me to change my opinion, or to > have the veto invalidated. In either case, you will be more likely to > succeed should your arguments relate not only to the veto's implications > but also to its /sine qua non/ component: its rationale. Just in case, my personal opinion here is that the veto is invalid. Firstly, based on my understanding, the ASF rules prohibit casting a veto without an appropriate technical justification (see [1], which I personally agree with). Secondly, it seems that the process you are imposing hasn't been accepted in this community. As far as I know, this topic was tangentially discussed before (see [2], for example), and it looks like there hasn't been a consensus to change our current Commit-Then-Review process into some sort of Review-Then-Commit. (At the same time I won't even try to /convince/ you, sorry.) [1] https://www.apache.org/foundation/voting.html [2] https://lists.apache.org/thread/ow2x68g2k4lv2ycr81d14p8r8w2jj1xl Regards, Evgeny Kotkov
