On 2020-09-15 9:34 μ.μ., Ryan Sleevi wrote:
On Tue, Sep 15, 2020 at 12:25 PM Dimitris Zacharopoulos (HARICA)
<[email protected] <mailto:[email protected]>> wrote:
I think you have a point about A + B, so we should have
independent review periods for each ballot as we did with version
1.6.4 (we had independent ballots to be reviewed and then a
combined/aggregated new version of the BRs).
I also received some personal messages supporting the "aggregated"
practice as it seems to be much more efficient for a CA's
compliance review to check against one new BR rather than 2 or 3
separate ones within a very short timeframe.
As Chair, could I ask you to encourage these to be shared on the list?
I think it's troubling the increased amount of off-list messages used
to justify Forum behaviour. This feels very much like the mode which
the Forum explicitly tried to move past with our move to open lists
and minuted calls, so that we can more accurately understand
perspectives and ask questions. Certainly, I hope it should be
uncontroversial that the Forum helps us understand the many different
perspectives and constraints, and this sort of "appeal to private
communication, anonymized, without data" actively harms the ability of
the Forum to function and improve. This is part of why we have public
discussion
Sure, I can do that but in any case I forwarded it the argument on the
list. I also support this argument that aggregating new versions of the
documents saves time.
If someone wants to send me something in private to check if I agree or
not, I don't see any harm in doing that. I am sure that direct
communication among Members, although discouraged, is not prohibited.
More answers inline.
On 2020-09-15 5:34 μ.μ., Ryan Sleevi wrote:
On Tue, Sep 15, 2020 at 4:33 AM Dimitris Zacharopoulos (HARICA)
<[email protected] <mailto:[email protected]>> wrote:
On 14/9/2020 8:08 μ.μ., Ryan Sleevi wrote:
Yes, I'm aware of the "what", but it's not clear the "why".
The act of combining ballots is relatively new, as you can
see from
https://cabforum.org/baseline-requirements-documents/ .
Producing multiple versions of the Guidelines, linear based
on when the Ballot concluded, was something our GitHub flow
intentionally was to make easy. While that page stopped
listing dates of adoption around Ballot 189, you can see
previous ballot pairs (e.g. 171+164, 125+118+134+135) that
did that.
It seems worth figuring out the challenges you're facing,
since it was meant to be very easy to create a new version
of the document for each ballot, even ballots that conclude
closely, and to have IP reviews as such.
The administrative overhead of updating public web sites,
sending additional emails, and the fact that we would have
versions of the Guidelines that would be valid for a few
seconds (which seems unreasonable for a public standards
document), are some of the reasons behind aggregated final
guidelines. Version 1.6.4 aggregated 3 ballots, 1.6.7 and
1.7.1. had 2 and now 1.7.2.
This process was discussed with Dean and Wayne back in
February 2019, and we all considered it compliant with our
Bylaws. The results of each IPR review period were sent to
the public lists without receiving any objections or concerns.
Although we have documented a GitHub workflow that supports
the most common case (one ballot, one IPR review, one Final
Maintenance Guideline), it should not prohibit aggregated
ballots to minimize administrative overhead or the production
of Guidelines that have some reasonable validity time.
If there are strong objections to this process, we can revise
it going forward.
I think we should understand the concerns, so we can make sure we
have solutions that work.
The concerns for aggregated ballots is that, from an IP
perspective, it makes multiple ballots share the same fate and
can easily delay adoption. For example, imagine there are Ballots
A and B. A makes a change to one section, B makes a change to
another, and only through their combined implementation does a
Member raise an IP concern.
In the serialized approach, A then B, A can be introduced to the
IP review and not introduce any IP obligations. It can become
effective. When B is introduced and undergoes IP review, and an
exclusion is filed (on the B+A), then B is basically sent to the
PAG to work through what to do, while A remains effective.
In the combined approach, A and B, when A+B trigger the IP review
and result in the IP obligation. An exclusion is filed, and now
A+B are sent to a PAG. The act of understanding whether it is A
introducing the IP obligation or B introducing the IP obligation
is left for the PAG to sort, with neither A nor B being effective
until the PAG reaches recommendations.
This was a concern debated at length throughout our governance
process, and was indeed one of the core motivations for our IPR
policy update (which, you may recall, was motivated by the
removal of the "any other method" introducing obligations on the
other enumerated methods). Our IP policy was revised to try to
make it easier to distinguish whether A or B was introducing the
obligation, and allow modifications like A (and subsequent
modifications) to continue, even while B was addressed within the
PAG.
Beyond that explicit reason, the distinct versioning also helps
better review changes and tie back to ballots. It treats Ballots
as they are - logical units of change - rather than aggregations
of change. The Forum first started with an aggregation-of-change
approach (through submitting amendments and errata) and quickly
confounded the ability to understand the reconciled state. The
same issue applies to unified ballots. Given that we have CAs
asserting that it's difficult to find changes in Ballots they
explicitly voted for, after lengthy reviews, and with policies
incorporated in at least two root programs, I'm incredibly
sensitive to wanting to make sure CAs are set up to succeed in
following the Requirements, by making changes discrete and
digestible.
So that's the context about "why" serialized was intended. I'm
also totally sympathetic to the difficulty involved in producing
ballots, especially as we haven't quite got automation up to
speed, and so I can understand why wanting to avoid that work is
undesirable. What I take from your reply is that there are
several tasks that, as a consequence of our current tooling,
represent undesirable time commitments. It's unclear if that's "5
minutes more" or "5 hours more", and so your help in breaking
down how much time it takes for the following would be greatly
useful, as it will help prioritize what is most important to
resolve. I suspect these can become priorities for the
Infrastructure WG.
1) Merge a single Ballot into GitHub
This greatly depends on whether the ballot proposer has used the
documented and recommended (not yet normative/required) process,
using GitHub to create a redline. Even with GitHub, I've seen
different "flavors" of redlines, for example with and without pull
requests, making it somewhat challenging to find the right steps
to create a cabforum/documents branch that will also create the
artifacts (docx document) from which I can create the redline.
If we are talking about the Baseline Requirements. this process
takes between 15 minutes to 1,5 hours in the more weird cases. For
the other Guidelines, it takes between 30 minutes to 1 hour.
Thanks! What I take from this is that one of the steps to improve can
be to front-load some of this (e.g. the creation of PRs earlier, the
creation of GitHub branches that match balloted text). These are
things that I think any of us can do, and sounds like it would help
somewhat.
2) Produce a Final Guideline and a Redlined Version
Once I have the review period redline ready, it usually takes
between 30 minutes to 1 hour to create the final documents, upload
them to the wiki (the word versions), produce the final PDF
versions and upload them to the public web site.
Wow! I wouldn't have expected this to be more than 5 minutes, so
that's a really surprising amount of time! Do you know where the bulk
of the time is, so we can prioritize? That said, it sounds like your
response here aggregated 2-5 - is that roughly right?
Yes, because to produce a Final Guideline and a redlined version in a
non-GitHub version is quite painful. In the GitHub version, things are
faster but again I need to create the redline manually, compare the two
.docx versions produced by GitHub (before and after the merge to Main),
check everything like track changes mode, make sure the ToC is not
tracked and other minor details. Then, create a PDF without the
formatting comments (otherwise the PDF contains formatting changes in
the redline which doesn't look good). Then make the changes to the web
sites, wiki, etc. I wish I could make that in 5 minutes :-)
This would probably require having our entire public website on GitHub
and automate the process. That would be really something!
3) Draft and send the e-mail for the IP review period announcement
This usually takes between 10 and 20 minutes because I am using
templates on Thunderbird and just replacing the links and attachments.
4) Upload the Final Guideline and Redlined Version to the website
5) Update the Website to link to these
10 to 20 minutes for both 4 and 5.
6) Draft and send the e-mail concluding the IP review period
Again 10 to 20 minutes.
As a number of these tasks are very automatable, figuring out
what we can do so that it represents <30 minutes of additional
work seems... a reasonable target, right? This only comes up when
we have multiple ballots concluding within a short period of
eachother, which although historically rare, certainly was more
common over the summer as stuff got batched to accommodate Covid
disruptions.
The most challenging and time consuming tasks come with ballots
that update more than one Guideline, for which we don't have the
same tools ready to produce docx versions. This requires me to do
the GitHub redlines for the BRs and docx versions (manually) for
EV Guidelines or NSRs.
Even for the BRs, I usually have to remove text after the ToC
because it repeats information that is supposed to be in the cover
page. Some of these minor issues need to be fixed, and I would
really appreciate the help of the Infrastructure Subcommittee.
So what I'm taking from this is that, probably in order of priorities
for optimizing:
- Get the EVGs and NCSSRs fully converted and automated (saves 30 - 60
minutes per ballot)
- Create PRs as Ballots are proposed (saves 30 - 60 minutes per ballot)
For the SCWG, we see:
- 20 ballots to the BRs over the past 2 years
- 7 ballots to the EVGs over the past 2 years
- 3 ballots to the NCSSRs over the past 2 years
Meaning that, in the worst case, at 1.5 hours per ballot/doc, that's
15 ballots a year, or 22.5 hours spent working just on ballots and IP
reviews shepherding as individual docs. By aggregating, you've been
able to shave 7.5 hours off that time. If we automated, and got it
down to 30 minutes per ballot, that could save another 7.5 hours. So a
week or two of work automating would pay for itself within 5 years (or
a day or two of automating within 1 year). More time than that,
however, would be more time spent vs having the chair do it manually,
and might not be worth the time <https://xkcd.com/1205/> to automate.
Certainly, this breakdown has helped highlight where the most
impactful time savings can be for you, or more generally, for the
chair. This is one of those areas where the added transparency, given
all the duties asked of the Chair, helps us improve the tools and
workmode, so that chairing becomes less of a time commitment and thus
something more members can consider in the future.
Would your concerns be addressed if we had a separate/distinct IPR
review for each ballot and once each IPR review is cleared with no
essential claims filed, produce an aggregate document, or you
still prefer separate Final documents with distinct versioning to
be created? What do others think?
The IP review necessitates the production of the Final document, so
that sounds like more work, rather than less, and it seems to run in
the same problem of commingling outputs.
During my entire time as Chair of the Forum and the SCWG I have tried to
perform my duties following the Bylaws to the best of my knowledge and
understanding. Just in case I had misunderstood these requirements, I
shared this process with the Vice Chairs who are native English
speakers. We all agreed that aggregating the final Guidelines was
allowed and not in conflict with our Bylaws.
Can you point me to where you believe this practice is forbidden? I
honestly don't mind the extra work of creating more Guidelines but it
seems pointless to create a document that nobody will ever read because
it will be valid for just a few seconds. Documents with the *same
effective date* would create a lot of confusion to CAs and Relying Parties.
_______________________________________________
Public mailing list
[email protected]
https://lists.cabforum.org/mailman/listinfo/public
