This is an automated email from the ASF dual-hosted git repository.
sbp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-trusted-releases.git
The following commit(s) were added to refs/heads/main by this push:
new da50dbf Move and update the storage interface documentation
da50dbf is described below
commit da50dbf55c91fcc02fc603ecbbfae472ef6924b4
Author: Sean B. Palmer <[email protected]>
AuthorDate: Fri Oct 10 20:18:24 2025 +0100
Move and update the storage interface documentation
---
atr/docs/build-processes.html | 6 +-
atr/docs/build-processes.md | 6 +-
atr/docs/code-conventions.html | 6 +-
atr/docs/code-conventions.md | 6 +-
atr/docs/database.html | 4 +-
atr/docs/database.md | 2 +-
atr/docs/developer-guide.html | 7 +-
atr/docs/developer-guide.md | 7 +-
atr/docs/how-to-contribute.html | 4 +-
atr/docs/how-to-contribute.md | 4 +-
atr/docs/index.html | 7 +-
atr/docs/index.md | 7 +-
atr/docs/overview-of-the-code.html | 2 +-
atr/docs/running-the-server.html | 2 +-
atr/docs/storage-interface.html | 100 ++++++++++++++++++++++++++
atr/docs/storage-interface.md | 140 +++++++++++++++++++++++++++++++++++++
docs/outcome-design-patterns.html | 26 +++++++
docs/outcome-design-patterns.md | 38 ++++++++++
docs/storage-interface.html | 86 -----------------------
docs/storage-interface.md | 125 ---------------------------------
scripts/docs_post_process.py | 2 +-
21 files changed, 342 insertions(+), 245 deletions(-)
diff --git a/atr/docs/build-processes.html b/atr/docs/build-processes.html
index 9c64803..0962705 100644
--- a/atr/docs/build-processes.html
+++ b/atr/docs/build-processes.html
@@ -1,7 +1,7 @@
-<h1 id="4-build-processes">3.4. Build processes</h1>
+<h1 id="build-processes">3.5. Build processes</h1>
<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer
guide</a></p>
-<p><strong>Prev</strong>: <code>3.3.</code> <a href="database">Database</a></p>
-<p><strong>Next</strong>: <code>3.5.</code> <a href="code-conventions">Code
conventions</a></p>
+<p><strong>Prev</strong>: <code>3.4.</code> <a
href="storage-interface">Storage interface</a></p>
+<p><strong>Next</strong>: <code>3.6.</code> <a href="code-conventions">Code
conventions</a></p>
<p><strong>Sections</strong>:</p>
<ul>
<li><a href="#documentation-build-script">Documentation build script</a></li>
diff --git a/atr/docs/build-processes.md b/atr/docs/build-processes.md
index abaec16..ed51416 100644
--- a/atr/docs/build-processes.md
+++ b/atr/docs/build-processes.md
@@ -1,10 +1,10 @@
-# 3.4. Build processes
+# 3.5. Build processes
**Up**: `3.` [Developer guide](developer-guide)
-**Prev**: `3.3.` [Database](database)
+**Prev**: `3.4.` [Storage interface](storage-interface)
-**Next**: `3.5.` [Code conventions](code-conventions)
+**Next**: `3.6.` [Code conventions](code-conventions)
**Sections**:
diff --git a/atr/docs/code-conventions.html b/atr/docs/code-conventions.html
index 4a70206..f959f8a 100644
--- a/atr/docs/code-conventions.html
+++ b/atr/docs/code-conventions.html
@@ -1,7 +1,7 @@
-<h1 id="5-code-conventions">3.5. Code conventions</h1>
+<h1 id="code-conventions">3.6. Code conventions</h1>
<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer
guide</a></p>
-<p><strong>Prev</strong>: <code>3.4.</code> <a href="build-processes">Build
processes</a></p>
-<p><strong>Next</strong>: <code>3.6.</code> <a href="how-to-contribute">How to
contribute</a></p>
+<p><strong>Prev</strong>: <code>3.5.</code> <a href="build-processes">Build
processes</a></p>
+<p><strong>Next</strong>: <code>3.7.</code> <a href="how-to-contribute">How to
contribute</a></p>
<p><strong>Sections</strong>:</p>
<ul>
<li><a href="#python-code">Python code</a></li>
diff --git a/atr/docs/code-conventions.md b/atr/docs/code-conventions.md
index 8457844..03fe859 100644
--- a/atr/docs/code-conventions.md
+++ b/atr/docs/code-conventions.md
@@ -1,10 +1,10 @@
-# 3.5. Code conventions
+# 3.6. Code conventions
**Up**: `3.` [Developer guide](developer-guide)
-**Prev**: `3.4.` [Build processes](build-processes)
+**Prev**: `3.5.` [Build processes](build-processes)
-**Next**: `3.6.` [How to contribute](how-to-contribute)
+**Next**: `3.7.` [How to contribute](how-to-contribute)
**Sections**:
diff --git a/atr/docs/database.html b/atr/docs/database.html
index 656fa73..1db1eb5 100644
--- a/atr/docs/database.html
+++ b/atr/docs/database.html
@@ -1,7 +1,7 @@
-<h1 id="3-database">3.3. Database</h1>
+<h1 id="database">3.3. Database</h1>
<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer
guide</a></p>
<p><strong>Prev</strong>: <code>3.2.</code> <a
href="overview-of-the-code">Overview of the code</a></p>
-<p><strong>Next</strong>: <code>3.4.</code> <a href="build-processes">Build
processes</a></p>
+<p><strong>Next</strong>: <code>3.4.</code> <a
href="storage-interface">Storage interface</a></p>
<p><strong>Sections</strong>:</p>
<ul>
<li><a href="#introduction">Introduction</a></li>
diff --git a/atr/docs/database.md b/atr/docs/database.md
index cd5db57..4c6f74a 100644
--- a/atr/docs/database.md
+++ b/atr/docs/database.md
@@ -4,7 +4,7 @@
**Prev**: `3.2.` [Overview of the code](overview-of-the-code)
-**Next**: `3.4.` [Build processes](build-processes)
+**Next**: `3.4.` [Storage interface](storage-interface)
**Sections**:
diff --git a/atr/docs/developer-guide.html b/atr/docs/developer-guide.html
index d046b56..bb0fc47 100644
--- a/atr/docs/developer-guide.html
+++ b/atr/docs/developer-guide.html
@@ -7,9 +7,10 @@
<li><code>3.1.</code> <a href="running-the-server">Running the server</a></li>
<li><code>3.2.</code> <a href="overview-of-the-code">Overview of the
code</a></li>
<li><code>3.3.</code> <a href="database">Database</a></li>
-<li><code>3.4.</code> <a href="build-processes">Build processes</a></li>
-<li><code>3.5.</code> <a href="code-conventions">Code conventions</a></li>
-<li><code>3.6.</code> <a href="how-to-contribute">How to contribute</a></li>
+<li><code>3.4.</code> <a href="storage-interface">Storage interface</a></li>
+<li><code>3.5.</code> <a href="build-processes">Build processes</a></li>
+<li><code>3.6.</code> <a href="code-conventions">Code conventions</a></li>
+<li><code>3.7.</code> <a href="how-to-contribute">How to contribute</a></li>
</ul>
<p><strong>Sections</strong>:</p>
<ul>
diff --git a/atr/docs/developer-guide.md b/atr/docs/developer-guide.md
index 7ba4881..1880239 100644
--- a/atr/docs/developer-guide.md
+++ b/atr/docs/developer-guide.md
@@ -11,9 +11,10 @@
* `3.1.` [Running the server](running-the-server)
* `3.2.` [Overview of the code](overview-of-the-code)
* `3.3.` [Database](database)
-* `3.4.` [Build processes](build-processes)
-* `3.5.` [Code conventions](code-conventions)
-* `3.6.` [How to contribute](how-to-contribute)
+* `3.4.` [Storage interface](storage-interface)
+* `3.5.` [Build processes](build-processes)
+* `3.6.` [Code conventions](code-conventions)
+* `3.7.` [How to contribute](how-to-contribute)
**Sections**:
diff --git a/atr/docs/how-to-contribute.html b/atr/docs/how-to-contribute.html
index dac0a7d..b80c5f1 100644
--- a/atr/docs/how-to-contribute.html
+++ b/atr/docs/how-to-contribute.html
@@ -1,6 +1,6 @@
-<h1 id="6-how-to-contribute">3.6. How to contribute</h1>
+<h1 id="how-to-contribute">3.7. How to contribute</h1>
<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer
guide</a></p>
-<p><strong>Prev</strong>: <code>3.5.</code> <a href="code-conventions">Code
conventions</a></p>
+<p><strong>Prev</strong>: <code>3.6.</code> <a href="code-conventions">Code
conventions</a></p>
<p><strong>Next</strong>: (none)</p>
<p><strong>Sections</strong>:</p>
<ul>
diff --git a/atr/docs/how-to-contribute.md b/atr/docs/how-to-contribute.md
index ddd2680..d0f0762 100644
--- a/atr/docs/how-to-contribute.md
+++ b/atr/docs/how-to-contribute.md
@@ -1,8 +1,8 @@
-# 3.6. How to contribute
+# 3.7. How to contribute
**Up**: `3.` [Developer guide](developer-guide)
-**Prev**: `3.5.` [Code conventions](code-conventions)
+**Prev**: `3.6.` [Code conventions](code-conventions)
**Next**: (none)
diff --git a/atr/docs/index.html b/atr/docs/index.html
index 540987a..f312ed4 100644
--- a/atr/docs/index.html
+++ b/atr/docs/index.html
@@ -10,9 +10,10 @@
<li><code>3.1.</code> <a href="running-the-server">Running the server</a></li>
<li><code>3.2.</code> <a href="overview-of-the-code">Overview of the
code</a></li>
<li><code>3.3.</code> <a href="database">Database</a></li>
-<li><code>3.4.</code> <a href="build-processes">Build processes</a></li>
-<li><code>3.5.</code> <a href="code-conventions">Code conventions</a></li>
-<li><code>3.6.</code> <a href="how-to-contribute">How to contribute</a></li>
+<li><code>3.4.</code> <a href="storage-interface">Storage interface</a></li>
+<li><code>3.5.</code> <a href="build-processes">Build processes</a></li>
+<li><code>3.6.</code> <a href="code-conventions">Code conventions</a></li>
+<li><code>3.7.</code> <a href="how-to-contribute">How to contribute</a></li>
</ul>
</li>
</ul>
diff --git a/atr/docs/index.md b/atr/docs/index.md
index 07cfbfd..9b801b8 100644
--- a/atr/docs/index.md
+++ b/atr/docs/index.md
@@ -12,6 +12,7 @@ NOTE: This documentation is a work in progress.
* `3.1.` [Running the server](running-the-server)
* `3.2.` [Overview of the code](overview-of-the-code)
* `3.3.` [Database](database)
- * `3.4.` [Build processes](build-processes)
- * `3.5.` [Code conventions](code-conventions)
- * `3.6.` [How to contribute](how-to-contribute)
+ * `3.4.` [Storage interface](storage-interface)
+ * `3.5.` [Build processes](build-processes)
+ * `3.6.` [Code conventions](code-conventions)
+ * `3.7.` [How to contribute](how-to-contribute)
diff --git a/atr/docs/overview-of-the-code.html
b/atr/docs/overview-of-the-code.html
index e9e1f34..4da090b 100644
--- a/atr/docs/overview-of-the-code.html
+++ b/atr/docs/overview-of-the-code.html
@@ -1,4 +1,4 @@
-<h1 id="2-overview-of-the-code">3.2. Overview of the code</h1>
+<h1 id="overview-of-the-code">3.2. Overview of the code</h1>
<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer
guide</a></p>
<p><strong>Prev</strong>: <code>3.1.</code> <a
href="running-the-server">Running the server</a></p>
<p><strong>Next</strong>: <code>3.3.</code> <a href="database">Database</a></p>
diff --git a/atr/docs/running-the-server.html b/atr/docs/running-the-server.html
index fbf31c1..cb53c99 100644
--- a/atr/docs/running-the-server.html
+++ b/atr/docs/running-the-server.html
@@ -1,4 +1,4 @@
-<h1 id="1-running-the-server">3.1. Running the server</h1>
+<h1 id="running-the-server">3.1. Running the server</h1>
<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer
guide</a></p>
<p><strong>Prev</strong>: (none)</p>
<p><strong>Next</strong>: <code>3.2.</code> <a
href="overview-of-the-code">Overview of the code</a></p>
diff --git a/atr/docs/storage-interface.html b/atr/docs/storage-interface.html
new file mode 100644
index 0000000..4170aef
--- /dev/null
+++ b/atr/docs/storage-interface.html
@@ -0,0 +1,100 @@
+<h1 id="storage-interface">3.4. Storage interface</h1>
+<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer
guide</a></p>
+<p><strong>Prev</strong>: <code>3.3.</code> <a href="database">Database</a></p>
+<p><strong>Next</strong>: <code>3.5.</code> <a href="build-processes">Build
processes</a></p>
+<p><strong>Sections</strong>:</p>
+<ul>
+<li><a href="#introduction">Introduction</a></li>
+<li><a href="#how-do-we-read-from-storage">How do we read from
storage?</a></li>
+<li><a href="#how-do-we-write-to-storage">How do we write to storage?</a></li>
+<li><a href="#how-do-we-add-new-storage-functionality">How do we add new
storage functionality?</a></li>
+<li><a href="#how-do-we-use-outcomes">How do we use outcomes?</a></li>
+<li><a href="#what-about-audit-logging">What about audit logging?</a></li>
+</ul>
+<h2 id="introduction">Introduction</h2>
+<p>All database writes, and some reads, in ATR go through the <a
href="/ref/atr/storage/__init__.py"><code>storage</code></a> interface. This
interface <strong>enforces permissions</strong>, <strong>centralizes audit
logging</strong>, and <strong>provides type-safe access</strong> to the
database. In other words, avoid calling <a
href="/ref/atr/db/__init__.py"><code>db</code></a> directly in route handlers
if possible.</p>
+<p>The storage interface recognizes several permission levels: general public
(unauthenticated visitors), foundation committer (any ASF account), committee
participant (committers and PMC members), committee member (PMC members only),
and foundation admin (infrastructure administrators). Each level inherits from
the previous one, so for example committee members can do everything committee
participants can do, plus additional operations.</p>
+<p>The storage interface does not make it impossible to bypass authorization,
because you can always import <code>db</code> directly and write to the
database. But it makes bypassing authorization an explicit choice that requires
deliberate action, and it makes the safer path the easier path. This is a
pragmatic approach to security: we cannot prevent all mistakes, but we can make
it harder to make them accidentally.</p>
+<h2 id="how-do-we-read-from-storage">How do we read from storage?</h2>
+<p>Reading from storage is a work in progress. There are some existing
methods, but most of the functionality is currently in <code>db</code> or
<code>db.interaction</code>, and much work is required to migrate this to the
storage interface. We have given this less priority because reads are generally
safe, with the exception of a few components such as user tokens, which should
be given greater migration priority.</p>
+<h2 id="how-do-we-write-to-storage">How do we write to storage?</h2>
+<p>To write to storage we open a write session, request specific permissions,
use the exposed functionality, and then handle the outcome. Here is an actual
example from <a
href="/ref/atr/routes/start.py"><code>routes/start.py</code></a>:</p>
+<pre><code class="language-python">async with storage.write(session.uid) as
write:
+ wacp = await write.as_project_committee_participant(project_name)
+ new_release, _project = await wacp.release.start(project_name, version)
+</code></pre>
+<p>The <code>wacp</code> object, short for <code>w</code>rite <code>a</code>s
<code>c</code>ommittee <code>p</code>articipant, provides access to
domain-specific writers: <code>announce</code>, <code>checks</code>,
<code>distributions</code>, <code>keys</code>, <code>policy</code>,
<code>project</code>, <code>release</code>, <code>sbom</code>,
<code>ssh</code>, <code>tokens</code>, and <code>vote</code>.</p>
+<p>The write session takes an optional ASF UID, typically
<code>session.uid</code> from the logged-in user. If you omit the UID, the
session determines it automatically from the current request context. The write
object checks LDAP memberships and raises <a
href="/ref/atr/storage/__init__.py:AccessError"><code>storage.AccessError</code></a>
if the user is not authorized for the requested permission level.</p>
+<p>Because projects belong to committees, we provide <a
href="/ref/atr/storage/__init__.py:as_project_committee_member"><code>write.as_project_committee_member(project_name)</code></a>
and <a
href="/ref/atr/storage/__init__.py:as_project_committee_participant"><code>write.as_project_committee_participant(project_name)</code></a>,
which look up the project's committee and authenticate the user as a member or
participant of that committee. This is convenient when, for example, the URL
prov [...]
+<p>Here is a more complete example from <a
href="/ref/atr/blueprints/api/api.py"><code>blueprints/api/api.py</code></a>
that shows the classic three step pattern:</p>
+<pre><code class="language-python">async with storage.write(asf_uid) as write:
+ # 1. Request permissions
+ wafc = write.as_foundation_committer()
+
+ # 2. Use the exposed functionality
+ outcome = await wafc.keys.ensure_stored_one(data.key)
+
+ # 3. Handle the outcome
+ key = outcome.result_or_raise()
+</code></pre>
+<p>In this case we decide to raise as soon as there is any error. We could
also choose to display a warning, ignore the error, collect multiple outcomes
for batch processing, or handle it in any other way appropriate for the
situation.</p>
+<h2 id="how-do-we-add-new-storage-functionality">How do we add new storage
functionality?</h2>
+<p>Add methods to classes in the <a
href="/ref/atr/storage/writers/"><code>storage/writers</code></a> or <a
href="/ref/atr/storage/readers/"><code>storage/readers</code></a> directories.
Code to perform any action associated with public keys that involves writing to
storage, for example, goes in <a
href="/ref/atr/storage/writers/keys.py"><code>storage/writers/keys.py</code></a>.</p>
+<p>Classes in writer and reader modules must be named to match the permission
hierarchy:</p>
+<pre><code class="language-python">class GeneralPublic:
+ def __init__(
+ self,
+ write: storage.Write,
+ write_as: storage.WriteAsGeneralPublic,
+ data: db.Session,
+ ) -> None:
+ self.__write = write
+ self.__write_as = write_as
+ self.__data = data
+
+class FoundationCommitter(GeneralPublic):
+ def __init__(
+ self,
+ write: storage.Write,
+ write_as: storage.WriteAsFoundationCommitter,
+ data: db.Session
+ ) -> None:
+ super().__init__(write, write_as, data)
+ self.__write = write
+ self.__write_as = write_as
+ self.__data = data
+
+class CommitteeParticipant(FoundationCommitter):
+ def __init__(
+ self,
+ write: storage.Write,
+ write_as: storage.WriteAsCommitteeParticipant,
+ data: db.Session,
+ committee_name: str,
+ ) -> None:
+ super().__init__(write, write_as, data)
+ self.__committee_name = committee_name
+
+class CommitteeMember(CommitteeParticipant):
+ ...
+</code></pre>
+<p>This hierarchy that this creates is: <code>GeneralPublic</code> →
<code>FoundationCommitter</code> → <code>CommitteeParticipant</code> →
<code>CommitteeMember</code>. You can add methods at any level. A method on
<code>CommitteeMember</code> is only available to committee members, while a
method on <code>FoundationCommitter</code> is available to everyone who has
logged in.</p>
+<p>Use <code>__private_methods</code> for helper code that is not part of the
public interface. Use <code>public_methods</code> for operations that should be
available to callers at the appropriate permission level. Consider returning <a
href="/ref/atr/storage/outcome.py:Outcome"><code>Outcome</code></a> types to
allow callers flexibility in error handling. Refer to the <a
href="#how-do-we-use-outcomes">section on using outcomes</a> for more
details.</p>
+<p>After adding a new writer module, register it in the appropriate
<code>WriteAs*</code> classes in <a
href="/ref/atr/storage/__init__.py"><code>storage/__init__.py</code></a>. For
example, when adding the <code>distributions</code> writer, it was necessary to
add <code>self.distributions = writers.distributions.CommitteeMember(write,
self, data, committee_name)</code> to the <a
href="/ref/atr/storage/__init__.py:WriteAsCommitteeMember"><code>WriteAsCommitteeMember</code></a>
class.</p>
+<h2 id="how-do-we-use-outcomes">How do we use outcomes?</h2>
+<p>Consider using <strong>outcome types</strong> from <a
href="/ref/atr/storage/outcome.py"><code>storage.outcome</code></a> when
returning results from writer methods. Outcomes let you represent both success
and failure without raising exceptions, which gives callers flexibility in how
they handle errors.</p>
+<p>An <a
href="/ref/atr/storage/outcome.py:Outcome"><code>Outcome[T]</code></a> is
either a <a
href="/ref/atr/storage/outcome.py:Result"><code>Result[T]</code></a> wrapping a
successful value, or an <a
href="/ref/atr/storage/outcome.py:Error"><code>Error[T]</code></a> wrapping an
exception. You can check which it is with the <code>ok</code> property or
pattern matching, extract the value with <code>result_or_raise()</code>, or
extract the error with <code>error_or_raise()</code>.</p>
+<p>Here is an example from <a
href="/ref/atr/routes/keys.py"><code>routes/keys.py</code></a> that processes
multiple keys and collects outcomes:</p>
+<pre><code class="language-python">async with storage.write() as write:
+ wacm = write.as_committee_member(selected_committee)
+ outcomes = await wacm.keys.ensure_associated(keys_text)
+
+success_count = outcomes.result_count
+error_count = outcomes.error_count
+</code></pre>
+<p>The <code>ensure_associated</code> method returns an <a
href="/ref/atr/storage/outcome.py:List"><code>outcome.List</code></a>, which is
a collection of outcomes. Some keys might import successfully, and others might
fail because they are malformed or already exist. The caller can inspect the
list to see how many succeeded and how many failed, and present that
information to the user.</p>
+<p>The <code>outcome.List</code> class provides many useful methods: <a
href="/ref/atr/storage/outcome.py:results"><code>results()</code></a> to get
only the successful values, <a
href="/ref/atr/storage/outcome.py:errors"><code>errors()</code></a> to get only
the exceptions, <a
href="/ref/atr/storage/outcome.py:result_count"><code>result_count</code></a>
and <a
href="/ref/atr/storage/outcome.py:error_count"><code>error_count</code></a> to
count them, and <a href="/ref/atr/storage/outcome [...]
+<p>Use outcomes when an operation might fail for some items but succeed for
others, or when you want to give the caller control over error handling. Do not
use them when failure should always raise an exception, such as authorization
failures or database connection errors. Those should be raised immediately.</p>
+<h2 id="what-about-audit-logging">What about audit logging?</h2>
+<p>Storage write operations can be logged to <a
href="/ref/atr/config.py:STORAGE_AUDIT_LOG_FILE"><code>config.AppConfig.STORAGE_AUDIT_LOG_FILE</code></a>,
which is <code>state/storage-audit.log</code> by default. Each log entry is a
JSON object containing the timestamp, the action name, and relevant parameters.
When you write a storage method that should be audited, call
<code>self.__write_as.append_to_audit_log(**kwargs)</code> with whatever
parameters are relevant to that specific oper [...]
+<p>Audit logging must be done manually because the values to log are often
those computed during method execution, not just those passed as arguments
which could be logged automatically. When deleting a release, for example, we
log <code>asf_uid</code> (instance attribute), <code>project_name</code>
(argument), and <code>version</code> (argument), but when issuing a JWT from a
PAT, we log <code>asf_uid</code> (instance attribute) and <code>pat_hash</code>
(<em>computed</em>). Each operat [...]
diff --git a/atr/docs/storage-interface.md b/atr/docs/storage-interface.md
new file mode 100644
index 0000000..5c47e48
--- /dev/null
+++ b/atr/docs/storage-interface.md
@@ -0,0 +1,140 @@
+# 3.4. Storage interface
+
+**Up**: `3.` [Developer guide](developer-guide)
+
+**Prev**: `3.3.` [Database](database)
+
+**Next**: `3.5.` [Build processes](build-processes)
+
+**Sections**:
+
+* [Introduction](#introduction)
+* [How do we read from storage?](#how-do-we-read-from-storage)
+* [How do we write to storage?](#how-do-we-write-to-storage)
+* [How do we add new storage
functionality?](#how-do-we-add-new-storage-functionality)
+* [How do we use outcomes?](#how-do-we-use-outcomes)
+* [What about audit logging?](#what-about-audit-logging)
+
+## Introduction
+
+All database writes, and some reads, in ATR go through the
[`storage`](/ref/atr/storage/__init__.py) interface. This interface **enforces
permissions**, **centralizes audit logging**, and **provides type-safe access**
to the database. In other words, avoid calling [`db`](/ref/atr/db/__init__.py)
directly in route handlers if possible.
+
+The storage interface recognizes several permission levels: general public
(unauthenticated visitors), foundation committer (any ASF account), committee
participant (committers and PMC members), committee member (PMC members only),
and foundation admin (infrastructure administrators). Each level inherits from
the previous one, so for example committee members can do everything committee
participants can do, plus additional operations.
+
+The storage interface does not make it impossible to bypass authorization,
because you can always import `db` directly and write to the database. But it
makes bypassing authorization an explicit choice that requires deliberate
action, and it makes the safer path the easier path. This is a pragmatic
approach to security: we cannot prevent all mistakes, but we can make it harder
to make them accidentally.
+
+## How do we read from storage?
+
+Reading from storage is a work in progress. There are some existing methods,
but most of the functionality is currently in `db` or `db.interaction`, and
much work is required to migrate this to the storage interface. We have given
this less priority because reads are generally safe, with the exception of a
few components such as user tokens, which should be given greater migration
priority.
+
+## How do we write to storage?
+
+To write to storage we open a write session, request specific permissions, use
the exposed functionality, and then handle the outcome. Here is an actual
example from [`routes/start.py`](/ref/atr/routes/start.py):
+
+```python
+async with storage.write(session.uid) as write:
+ wacp = await write.as_project_committee_participant(project_name)
+ new_release, _project = await wacp.release.start(project_name, version)
+```
+
+The `wacp` object, short for `w`rite `a`s `c`ommittee `p`articipant, provides
access to domain-specific writers: `announce`, `checks`, `distributions`,
`keys`, `policy`, `project`, `release`, `sbom`, `ssh`, `tokens`, and `vote`.
+
+The write session takes an optional ASF UID, typically `session.uid` from the
logged-in user. If you omit the UID, the session determines it automatically
from the current request context. The write object checks LDAP memberships and
raises [`storage.AccessError`](/ref/atr/storage/__init__.py:AccessError) if the
user is not authorized for the requested permission level.
+
+Because projects belong to committees, we provide
[`write.as_project_committee_member(project_name)`](/ref/atr/storage/__init__.py:as_project_committee_member)
and
[`write.as_project_committee_participant(project_name)`](/ref/atr/storage/__init__.py:as_project_committee_participant),
which look up the project's committee and authenticate the user as a member or
participant of that committee. This is convenient when, for example, the URL
provides a project name.
+
+Here is a more complete example from
[`blueprints/api/api.py`](/ref/atr/blueprints/api/api.py) that shows the
classic three step pattern:
+
+```python
+async with storage.write(asf_uid) as write:
+ # 1. Request permissions
+ wafc = write.as_foundation_committer()
+
+ # 2. Use the exposed functionality
+ outcome = await wafc.keys.ensure_stored_one(data.key)
+
+ # 3. Handle the outcome
+ key = outcome.result_or_raise()
+```
+
+In this case we decide to raise as soon as there is any error. We could also
choose to display a warning, ignore the error, collect multiple outcomes for
batch processing, or handle it in any other way appropriate for the situation.
+
+## How do we add new storage functionality?
+
+Add methods to classes in the [`storage/writers`](/ref/atr/storage/writers/)
or [`storage/readers`](/ref/atr/storage/readers/) directories. Code to perform
any action associated with public keys that involves writing to storage, for
example, goes in [`storage/writers/keys.py`](/ref/atr/storage/writers/keys.py).
+
+Classes in writer and reader modules must be named to match the permission
hierarchy:
+
+```python
+class GeneralPublic:
+ def __init__(
+ self,
+ write: storage.Write,
+ write_as: storage.WriteAsGeneralPublic,
+ data: db.Session,
+ ) -> None:
+ self.__write = write
+ self.__write_as = write_as
+ self.__data = data
+
+class FoundationCommitter(GeneralPublic):
+ def __init__(
+ self,
+ write: storage.Write,
+ write_as: storage.WriteAsFoundationCommitter,
+ data: db.Session
+ ) -> None:
+ super().__init__(write, write_as, data)
+ self.__write = write
+ self.__write_as = write_as
+ self.__data = data
+
+class CommitteeParticipant(FoundationCommitter):
+ def __init__(
+ self,
+ write: storage.Write,
+ write_as: storage.WriteAsCommitteeParticipant,
+ data: db.Session,
+ committee_name: str,
+ ) -> None:
+ super().__init__(write, write_as, data)
+ self.__committee_name = committee_name
+
+class CommitteeMember(CommitteeParticipant):
+ ...
+```
+
+This hierarchy that this creates is: `GeneralPublic` → `FoundationCommitter` →
`CommitteeParticipant` → `CommitteeMember`. You can add methods at any level. A
method on `CommitteeMember` is only available to committee members, while a
method on `FoundationCommitter` is available to everyone who has logged in.
+
+Use `__private_methods` for helper code that is not part of the public
interface. Use `public_methods` for operations that should be available to
callers at the appropriate permission level. Consider returning
[`Outcome`](/ref/atr/storage/outcome.py:Outcome) types to allow callers
flexibility in error handling. Refer to the [section on using
outcomes](#how-do-we-use-outcomes) for more details.
+
+After adding a new writer module, register it in the appropriate `WriteAs*`
classes in [`storage/__init__.py`](/ref/atr/storage/__init__.py). For example,
when adding the `distributions` writer, it was necessary to add
`self.distributions = writers.distributions.CommitteeMember(write, self, data,
committee_name)` to the
[`WriteAsCommitteeMember`](/ref/atr/storage/__init__.py:WriteAsCommitteeMember)
class.
+
+## How do we use outcomes?
+
+Consider using **outcome types** from
[`storage.outcome`](/ref/atr/storage/outcome.py) when returning results from
writer methods. Outcomes let you represent both success and failure without
raising exceptions, which gives callers flexibility in how they handle errors.
+
+An [`Outcome[T]`](/ref/atr/storage/outcome.py:Outcome) is either a
[`Result[T]`](/ref/atr/storage/outcome.py:Result) wrapping a successful value,
or an [`Error[T]`](/ref/atr/storage/outcome.py:Error) wrapping an exception.
You can check which it is with the `ok` property or pattern matching, extract
the value with `result_or_raise()`, or extract the error with
`error_or_raise()`.
+
+Here is an example from [`routes/keys.py`](/ref/atr/routes/keys.py) that
processes multiple keys and collects outcomes:
+
+```python
+async with storage.write() as write:
+ wacm = write.as_committee_member(selected_committee)
+ outcomes = await wacm.keys.ensure_associated(keys_text)
+
+success_count = outcomes.result_count
+error_count = outcomes.error_count
+```
+
+The `ensure_associated` method returns an
[`outcome.List`](/ref/atr/storage/outcome.py:List), which is a collection of
outcomes. Some keys might import successfully, and others might fail because
they are malformed or already exist. The caller can inspect the list to see how
many succeeded and how many failed, and present that information to the user.
+
+The `outcome.List` class provides many useful methods:
[`results()`](/ref/atr/storage/outcome.py:results) to get only the successful
values, [`errors()`](/ref/atr/storage/outcome.py:errors) to get only the
exceptions, [`result_count`](/ref/atr/storage/outcome.py:result_count) and
[`error_count`](/ref/atr/storage/outcome.py:error_count) to count them, and
[`results_or_raise()`](/ref/atr/storage/outcome.py:results_or_raise) to extract
all values or raise on the first error.
+
+Use outcomes when an operation might fail for some items but succeed for
others, or when you want to give the caller control over error handling. Do not
use them when failure should always raise an exception, such as authorization
failures or database connection errors. Those should be raised immediately.
+
+## What about audit logging?
+
+Storage write operations can be logged to
[`config.AppConfig.STORAGE_AUDIT_LOG_FILE`](/ref/atr/config.py:STORAGE_AUDIT_LOG_FILE),
which is `state/storage-audit.log` by default. Each log entry is a JSON object
containing the timestamp, the action name, and relevant parameters. When you
write a storage method that should be audited, call
`self.__write_as.append_to_audit_log(**kwargs)` with whatever parameters are
relevant to that specific operation. The action name is extracted automatical
[...]
+
+Audit logging must be done manually because the values to log are often those
computed during method execution, not just those passed as arguments which
could be logged automatically. When deleting a release, for example, we log
`asf_uid` (instance attribute), `project_name` (argument), and `version`
(argument), but when issuing a JWT from a PAT, we log `asf_uid` (instance
attribute) and `pat_hash` (_computed_). Each operation logs what makes sense
for that operation.
diff --git a/docs/outcome-design-patterns.html
b/docs/outcome-design-patterns.html
new file mode 100644
index 0000000..e52a676
--- /dev/null
+++ b/docs/outcome-design-patterns.html
@@ -0,0 +1,26 @@
+<h1 id="outcome-design-patterns">Outcome design patterns</h1>
+<p>One common pattern when designing outcome types is about how to handle an
<strong>exception after a success</strong>, and how to handle a <strong>warning
during success</strong>:</p>
+<ul>
+<li>An <strong>exception after a success</strong> is when an object is
processed in multiple stages, and the first few stages succeed but then
subsequently there is an exception.</li>
+<li>A <strong>warning during success</strong> is when an object is processed
in multiple stages, an exception is raised, but we determine that we can
proceed to subsequent stages as long as we keep a note of the exception.</li>
+</ul>
+<p>Both of these workflows appear incompatible with outcomes. In outcomes, we
can record <em>either</em> a successful result, <em>or</em> an exception. But
in exception after success we want to record the successes up to the exception;
and in a warning during a success we want to record the exception even though
we return a success result.</p>
+<p>The solution is similar in both cases: create a wrapper of the <em>primary
type</em> which can hold an instance of the <em>secondary type</em>.</p>
+<p>In <em>exception after a success</em> the primary type is an exception, and
the secondary type is the result which was obtained up to that exception. The
type will look like this:</p>
+<pre><code class="language-python">class AfterSuccessError(Exception):
+ def __init__(self, result_before_error: Result):
+ self.result_before_error = result_before_error
+</code></pre>
+<p>In <em>warning during success</em>, the primary type is the result, and the
secondary type is the exception raised during successful processing which we
consider a warning. This is the inverse of the above, and the types are
therefore inverted too.</p>
+<pre><code class="language-python">@dataclasses.dataclass
+class Result:
+ value: Value
+ warning: Exception | None
+</code></pre>
+<p>This could just as easily be a Pydantic class or whatever is appropriate in
the situation, as long as it can hold the warning. If the warning is generated
during an additional or side task, we can use <code>Outcome[SideValue]</code>
instead. We do this, for example, in the type representing a linked
committee:</p>
+<pre><code class="language-python">@dataclasses.dataclass
+class LinkedCommittee:
+ name: str
+ autogenerated_keys_file: Outcome[str]
+</code></pre>
+<p>In this case, if the autogenerated keys file call succeeded without an
error, the <code>Outcome</code> will be an <code>OutcomeResult[str]</code>
where the <code>str</code> represents the full path to the autogenerated
file.</p>
diff --git a/docs/outcome-design-patterns.md b/docs/outcome-design-patterns.md
new file mode 100644
index 0000000..d5430cd
--- /dev/null
+++ b/docs/outcome-design-patterns.md
@@ -0,0 +1,38 @@
+# Outcome design patterns
+
+One common pattern when designing outcome types is about how to handle an
**exception after a success**, and how to handle a **warning during success**:
+
+* An **exception after a success** is when an object is processed in multiple
stages, and the first few stages succeed but then subsequently there is an
exception.
+* A **warning during success** is when an object is processed in multiple
stages, an exception is raised, but we determine that we can proceed to
subsequent stages as long as we keep a note of the exception.
+
+Both of these workflows appear incompatible with outcomes. In outcomes, we can
record _either_ a successful result, _or_ an exception. But in exception after
success we want to record the successes up to the exception; and in a warning
during a success we want to record the exception even though we return a
success result.
+
+The solution is similar in both cases: create a wrapper of the _primary type_
which can hold an instance of the _secondary type_.
+
+In _exception after a success_ the primary type is an exception, and the
secondary type is the result which was obtained up to that exception. The type
will look like this:
+
+```python
+class AfterSuccessError(Exception):
+ def __init__(self, result_before_error: Result):
+ self.result_before_error = result_before_error
+```
+
+In _warning during success_, the primary type is the result, and the secondary
type is the exception raised during successful processing which we consider a
warning. This is the inverse of the above, and the types are therefore inverted
too.
+
+```python
[email protected]
+class Result:
+ value: Value
+ warning: Exception | None
+```
+
+This could just as easily be a Pydantic class or whatever is appropriate in
the situation, as long as it can hold the warning. If the warning is generated
during an additional or side task, we can use `Outcome[SideValue]` instead. We
do this, for example, in the type representing a linked committee:
+
+```python
[email protected]
+class LinkedCommittee:
+ name: str
+ autogenerated_keys_file: Outcome[str]
+```
+
+In this case, if the autogenerated keys file call succeeded without an error,
the `Outcome` will be an `OutcomeResult[str]` where the `str` represents the
full path to the autogenerated file.
diff --git a/docs/storage-interface.html b/docs/storage-interface.html
deleted file mode 100644
index 01feacf..0000000
--- a/docs/storage-interface.html
+++ /dev/null
@@ -1,86 +0,0 @@
-<h1 id="storage-interface">Storage interface</h1>
-<p>All writes to the database and filesystem are to be mediated through the
storage interface in <code>atr.storage</code>. The storage interface
<strong>enforces permissions</strong>, <strong>centralises audit
logging</strong>, and <strong>exposes misuse resistant methods</strong>.</p>
-<h2 id="how-do-we-use-the-storage-interface">How do we use the storage
interface?</h2>
-<p>Open a storage interface session with a context manager. Then:</p>
-<ol>
-<li>Request permissions from the session depending on the role of the
user.</li>
-<li>Use the exposed functionality.</li>
-<li>Handle the outcome or outcomes.</li>
-</ol>
-<p>Here is an actual example from our API code:</p>
-<pre><code class="language-python">async with storage.write(asf_uid) as write:
- wafc = write.as_foundation_committer()
- ocr: types.Outcome[types.Key] = await wafc.keys.ensure_stored_one(data.key)
- key = ocr.result_or_raise()
-
- for selected_committee_name in selected_committee_names:
- wacm = write.as_committee_member(selected_committee_name)
- oc: types.Outcome[types.LinkedCommittee] = await
wacm.keys.associate_fingerprint(
- key.key_model.fingerprint
- )
- oc.result_or_raise()
-</code></pre>
-<p>The <code>wafm</code> (<strong>w</strong>rite <strong>a</strong>s
<strong>f</strong>oundation <strong>m</strong>ember) object exposes
functionality which is only available to foundation members. The
<code>wafm.keys.ensure_stored_one</code> method is an example of such
functionality. The <code>wacm</code> object goes further and exposes
functionality only available to committee members.</p>
-<p>In this case we decide to raise as soon as there is any error. We could
also choose instead to display a warning, ignore the error, etc.</p>
-<p>The first few lines in the context session show the classic three step
approach. Here they are again with comments:</p>
-<pre><code class="language-python"> # 1. Request permissions
- wafc = write.as_foundation_committer()
-
- # 2. Use the exposed functionality
- ocr: types.Outcome[types.Key] = await wafc.keys.ensure_stored_one(data.key)
-
- # 3. Handle the outcome
- key = ocr.result_or_raise()
-</code></pre>
-<h2 id="how-do-we-add-functionality-to-the-storage-interface">How do we add
functionality to the storage interface?</h2>
-<p>Add all the functionality to classes in modules in the
<code>atr/storage/writers</code> directory. Code to write public keys to
storage, for example, goes in <code>atr/storage/writers/keys.py</code>.</p>
-<p>Classes in modules in the <code>atr/storage/writers</code> directory must
be named as follows:</p>
-<pre><code class="language-python">class GeneralPublic:
- ...
-
-class FoundationCommitter(GeneralPublic):
- ...
-
-class CommitteeParticipant(FoundationCommitter):
- ...
-
-class CommitteeMember(CommitteeParticipant):
- ...
-</code></pre>
-<p>This creates a hierarchy, <code>GeneralPublic</code> →
<code>FoundationCommitter</code> → <code>CommitteeParticipant</code> →
<code>CommitteeMember</code>. We can add other permissions levels if
necessary.</p>
-<p>Use <code>__private_methods</code> for code specific to one permission
level which is not exposed in the interface, e.g. helpers. Use
<code>public_methods</code> for code appropriate to expose when users meet the
appropriate permission level. Consider returning outcomes, as explained in the
next section.</p>
-<h2 id="returning-outcomes">Returning outcomes</h2>
-<p>Consider using the <strong>outcome types</strong> in
<code>atr.storage.types</code> when returning results from writer module
methods. The outcome types <em>solve many problems</em>, but here is an
example:</p>
-<p>Imagine the user is submitting a <code>KEYS</code> file containing several
keys. Some of the keys are already in the database, some are not in the
database, and some are broken keys that do not parse. After processing, each
key is associated with a different state: the key was parsed but not added, the
key was parsed and added, or the key wasn't even parsed. We consider some of
these success states, some warning states, and others error states.</p>
-<p>How do we represent this?</p>
-<p>Outcomes are one possibility. For each key we can return
<code>OutcomeResult</code> for a success, and <code>OutcomeException</code>
when there was a Python error. The caller can then decide what to do with this
information. It might ignore the exception, raise it, or print an error message
to the user. Better yet, we can aggregate these into an <code>Outcomes</code>
list, which provides many useful methods for processing all of the outcomes
together. It can count how many exceptions [...]
-<p>We do not have to return outcomes from public storage interface methods,
but these classes were designed to make the storage interface easy to use.</p>
-<h3 id="outcome-design-patterns">Outcome design patterns</h3>
-<p>One common pattern when designing outcome types is about how to handle an
<strong>exception after a success</strong>, and how to handle a <strong>warning
during success</strong>:</p>
-<ul>
-<li>An <strong>exception after a success</strong> is when an object is
processed in multiple stages, and the first few stages succeed but then
subsequently there is an exception.</li>
-<li>A <strong>warning during success</strong> is when an object is processed
in multiple stages, an exception is raised, but we determine that we can
proceed to subsequent stages as long as we keep a note of the exception.</li>
-</ul>
-<p>Both of these workflows appear incompatible with outcomes. In outcomes, we
can record <em>either</em> a successful result, <em>or</em> an exception. But
in exception after success we want to record the successes up to the exception;
and in a warning during a success we want to record the exception even though
we return a success result.</p>
-<p>The solution is similar in both cases: create a wrapper of the <em>primary
type</em> which can hold an instance of the <em>secondary type</em>.</p>
-<p>In <em>exception after a success</em> the primary type is an exception, and
the secondary type is the result which was obtained up to that exception. The
type will look like this:</p>
-<pre><code class="language-python">class AfterSuccessError(Exception):
- def __init__(self, result_before_error: Result):
- self.result_before_error = result_before_error
-</code></pre>
-<p>In <em>warning during success</em>, the primary type is the result, and the
secondary type is the exception raised during successful processing which we
consider a warning. This is the inverse of the above, and the types are
therefore inverted too.</p>
-<pre><code class="language-python">@dataclasses.dataclass
-class Result:
- value: Value
- warning: Exception | None
-</code></pre>
-<p>This could just as easily be a Pydantic class or whatever is appropriate in
the situation, as long as it can hold the warning. If the warning is generated
during an additional or side task, we can use <code>Outcome[SideValue]</code>
instead. We do this, for example, in the type representing a linked
committee:</p>
-<pre><code class="language-python">@dataclasses.dataclass
-class LinkedCommittee:
- name: str
- autogenerated_keys_file: Outcome[str]
-</code></pre>
-<p>In this case, if the autogenerated keys file call succeeded without an
error, the <code>Outcome</code> will be an <code>OutcomeResult[str]</code>
where the <code>str</code> represents the full path to the autogenerated
file.</p>
-<h2 id="what-makes-this-safe">What makes this safe?</h2>
-<p>We can always open a database session or write to the filesystem, so there
is no way to make storage access truly safe. But abstracting these operations
to a well known interface makes it more likely that we use only this way of
doing things, which we can then concentrate on getting right. This is in
contrast to writing storage access in <em>ad hoc</em> ways, some of which may
be correct and some of which may not.</p>
-<p>Code relative to a permissions level is only ever exposed in the storage
interface when it is proven, at the type level and during runtime, that the
user has credentials for those permissions. Helper code remains private due to
the use of <code>__private_methods</code>, which undergo name mangling in
Python. As mentioned in the introduction, the storage interface is also the
suitable place to add audit logging, currently planned and not yet
implemented.</p>
diff --git a/docs/storage-interface.md b/docs/storage-interface.md
deleted file mode 100644
index 5d64e85..0000000
--- a/docs/storage-interface.md
+++ /dev/null
@@ -1,125 +0,0 @@
-# Storage interface
-
-All writes to the database and filesystem are to be mediated through the
storage interface in `atr.storage`. The storage interface **enforces
permissions**, **centralises audit logging**, and **exposes misuse resistant
methods**.
-
-## How do we use the storage interface?
-
-Open a storage interface session with a context manager. Then:
-
-1. Request permissions from the session depending on the role of the user.
-2. Use the exposed functionality.
-3. Handle the outcome or outcomes.
-
-Here is an actual example from our API code:
-
-```python
-async with storage.write(asf_uid) as write:
- wafc = write.as_foundation_committer()
- ocr: types.Outcome[types.Key] = await wafc.keys.ensure_stored_one(data.key)
- key = ocr.result_or_raise()
-
- for selected_committee_name in selected_committee_names:
- wacm = write.as_committee_member(selected_committee_name)
- oc: types.Outcome[types.LinkedCommittee] = await
wacm.keys.associate_fingerprint(
- key.key_model.fingerprint
- )
- oc.result_or_raise()
-```
-
-The `wafm` (**w**rite **a**s **f**oundation **m**ember) object exposes
functionality which is only available to foundation members. The
`wafm.keys.ensure_stored_one` method is an example of such functionality. The
`wacm` object goes further and exposes functionality only available to
committee members.
-
-In this case we decide to raise as soon as there is any error. We could also
choose instead to display a warning, ignore the error, etc.
-
-The first few lines in the context session show the classic three step
approach. Here they are again with comments:
-
-```python
- # 1. Request permissions
- wafc = write.as_foundation_committer()
-
- # 2. Use the exposed functionality
- ocr: types.Outcome[types.Key] = await wafc.keys.ensure_stored_one(data.key)
-
- # 3. Handle the outcome
- key = ocr.result_or_raise()
-```
-
-## How do we add functionality to the storage interface?
-
-Add all the functionality to classes in modules in the `atr/storage/writers`
directory. Code to write public keys to storage, for example, goes in
`atr/storage/writers/keys.py`.
-
-Classes in modules in the `atr/storage/writers` directory must be named as
follows:
-
-```python
-class GeneralPublic:
- ...
-
-class FoundationCommitter(GeneralPublic):
- ...
-
-class CommitteeParticipant(FoundationCommitter):
- ...
-
-class CommitteeMember(CommitteeParticipant):
- ...
-```
-
-This creates a hierarchy, `GeneralPublic` → `FoundationCommitter` →
`CommitteeParticipant` → `CommitteeMember`. We can add other permissions levels
if necessary.
-
-Use `__private_methods` for code specific to one permission level which is not
exposed in the interface, e.g. helpers. Use `public_methods` for code
appropriate to expose when users meet the appropriate permission level.
Consider returning outcomes, as explained in the next section.
-
-## Returning outcomes
-
-Consider using the **outcome types** in `atr.storage.types` when returning
results from writer module methods. The outcome types _solve many problems_,
but here is an example:
-
-Imagine the user is submitting a `KEYS` file containing several keys. Some of
the keys are already in the database, some are not in the database, and some
are broken keys that do not parse. After processing, each key is associated
with a different state: the key was parsed but not added, the key was parsed
and added, or the key wasn't even parsed. We consider some of these success
states, some warning states, and others error states.
-
-How do we represent this?
-
-Outcomes are one possibility. For each key we can return `OutcomeResult` for a
success, and `OutcomeException` when there was a Python error. The caller can
then decide what to do with this information. It might ignore the exception,
raise it, or print an error message to the user. Better yet, we can aggregate
these into an `Outcomes` list, which provides many useful methods for
processing all of the outcomes together. It can count how many exceptions there
were, for example, or apply a [...]
-
-We do not have to return outcomes from public storage interface methods, but
these classes were designed to make the storage interface easy to use.
-
-### Outcome design patterns
-
-One common pattern when designing outcome types is about how to handle an
**exception after a success**, and how to handle a **warning during success**:
-
-* An **exception after a success** is when an object is processed in multiple
stages, and the first few stages succeed but then subsequently there is an
exception.
-* A **warning during success** is when an object is processed in multiple
stages, an exception is raised, but we determine that we can proceed to
subsequent stages as long as we keep a note of the exception.
-
-Both of these workflows appear incompatible with outcomes. In outcomes, we can
record _either_ a successful result, _or_ an exception. But in exception after
success we want to record the successes up to the exception; and in a warning
during a success we want to record the exception even though we return a
success result.
-
-The solution is similar in both cases: create a wrapper of the _primary type_
which can hold an instance of the _secondary type_.
-
-In _exception after a success_ the primary type is an exception, and the
secondary type is the result which was obtained up to that exception. The type
will look like this:
-
-```python
-class AfterSuccessError(Exception):
- def __init__(self, result_before_error: Result):
- self.result_before_error = result_before_error
-```
-
-In _warning during success_, the primary type is the result, and the secondary
type is the exception raised during successful processing which we consider a
warning. This is the inverse of the above, and the types are therefore inverted
too.
-
-```python
[email protected]
-class Result:
- value: Value
- warning: Exception | None
-```
-
-This could just as easily be a Pydantic class or whatever is appropriate in
the situation, as long as it can hold the warning. If the warning is generated
during an additional or side task, we can use `Outcome[SideValue]` instead. We
do this, for example, in the type representing a linked committee:
-
-```python
[email protected]
-class LinkedCommittee:
- name: str
- autogenerated_keys_file: Outcome[str]
-```
-
-In this case, if the autogenerated keys file call succeeded without an error,
the `Outcome` will be an `OutcomeResult[str]` where the `str` represents the
full path to the autogenerated file.
-
-## What makes this safe?
-
-We can always open a database session or write to the filesystem, so there is
no way to make storage access truly safe. But abstracting these operations to a
well known interface makes it more likely that we use only this way of doing
things, which we can then concentrate on getting right. This is in contrast to
writing storage access in _ad hoc_ ways, some of which may be correct and some
of which may not.
-
-Code relative to a permissions level is only ever exposed in the storage
interface when it is proven, at the type level and during runtime, that the
user has credentials for those permissions. Helper code remains private due to
the use of `__private_methods`, which undergo name mangling in Python. As
mentioned in the introduction, the storage interface is also the suitable place
to add audit logging, currently planned and not yet implemented.
diff --git a/scripts/docs_post_process.py b/scripts/docs_post_process.py
index caa0859..bb165f4 100644
--- a/scripts/docs_post_process.py
+++ b/scripts/docs_post_process.py
@@ -23,7 +23,7 @@ import sys
def generate_heading_id(text: str) -> str:
- text = re.sub(r"^\d+\.\s*", "", text)
+ text = re.sub(r"^[\d.]+\s*", "", text)
text = text.lower()
text = re.sub(r"[^\w\s-]", "", text)
text = re.sub(r"[\s_]+", "-", text)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
