kevinjqliu commented on code in PR #3969:
URL: https://github.com/apache/iceberg-python/pull/3969#discussion_r4341803427
##########
tests/encryption/ags1/GenerateAgs1Fixtures.java:
##########
Review Comment:
im a bit concerned about checking in a java file thats not used at all. i
think its just mentioned in the README. Perhaps we can just embed the file in
the README itself... as collapsed details, like so:
<details>
what do you think?
##########
pyiceberg/encryption/stream.py:
##########
@@ -0,0 +1,149 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Format primitives for the AGS1 stream, used to encrypt manifests and
manifest lists.
+
+An AGS1 stream is an 8 byte header followed by a sequence of AES-GCM blocks::
+
+ "AGS1" || plain_block_size (4 bytes, little endian)
+ nonce || ciphertext || tag (block 0, up to plain_block_size of
plaintext)
+ nonce || ciphertext || tag (block 1..n, the last of which may be
shorter)
+
+Each block authenticates `aad_prefix || block_index` as additional data, so
blocks cannot
+be reordered or moved between files.
+
+The spec gives the last block a non-zero length, which makes a bare header its
encoding of an
+empty file. Java, iceberg-rust and PyIceberg all require at least one block
instead, so an
+empty file is a header followed by a single empty block. apache/iceberg#18219
tracks which of
+the two forms writers should produce.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from pyiceberg.encryption.ciphers import AesGcmCipher
+
+_GCM_STREAM_MAGIC = b"AGS1"
+_PLAIN_BLOCK_SIZE = 1024 * 1024
+_GCM_STREAM_HEADER_LENGTH = len(_GCM_STREAM_MAGIC) + 4
+_BLOCK_OVERHEAD = AesGcmCipher.NONCE_LENGTH + AesGcmCipher.TAG_LENGTH
+_CIPHER_BLOCK_SIZE = _PLAIN_BLOCK_SIZE + _BLOCK_OVERHEAD
+_BLOCK_INDEX_LENGTH = 4
+_MAX_BLOCKS = 2 ** (8 * _BLOCK_INDEX_LENGTH) - 1
+_MIN_STREAM_LENGTH = _GCM_STREAM_HEADER_LENGTH + _BLOCK_OVERHEAD
+
+
+def _stream_block_aad(aad_prefix: bytes | None, block_index: int) -> bytes:
+ """Return the additional authenticated data for the block at `block_index`.
+
+ Args:
+ aad_prefix (bytes | None): The file's AAD prefix, from its key
metadata.
+ block_index (int): The zero-based index of the block within the stream.
+ """
+ return (aad_prefix or b"") + block_index.to_bytes(_BLOCK_INDEX_LENGTH,
"little")
+
+
+def _encode_stream_header() -> bytes:
+ """Encode the AGS1 header that precedes the first block."""
+ return _GCM_STREAM_MAGIC + _PLAIN_BLOCK_SIZE.to_bytes(4, "little")
+
+
+def _decode_stream_header(header: bytes) -> int:
+ """Decode an AGS1 header, returning the plaintext block size it declares.
+
+ Args:
+ header (bytes): At least `_GCM_STREAM_HEADER_LENGTH` bytes from the
start of the stream.
+ """
+ if len(header) < _GCM_STREAM_HEADER_LENGTH:
+ raise ValueError(f"Invalid AGS1 header: expected
{_GCM_STREAM_HEADER_LENGTH} bytes, got {len(header)}")
+
+ if (magic := header[: len(_GCM_STREAM_MAGIC)]) != _GCM_STREAM_MAGIC:
+ raise ValueError(f"Invalid AGS1 header: magic {magic!r} does not match
{_GCM_STREAM_MAGIC!r}")
+
+ plain_block_size = int.from_bytes(header[len(_GCM_STREAM_MAGIC) :
_GCM_STREAM_HEADER_LENGTH], "little")
+ if plain_block_size != _PLAIN_BLOCK_SIZE:
+ raise ValueError(f"Unsupported AGS1 block size: {plain_block_size}
(expected {_PLAIN_BLOCK_SIZE})")
+
+ return plain_block_size
+
+
+@dataclass(frozen=True)
+class _Ags1Layout:
Review Comment:
thanks for adhering to python conventions 😄
##########
tests/encryption/ags1/README.md:
##########
@@ -0,0 +1,74 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# AGS1 cross-client test fixtures
+
+These files were written by Java's `AesGcmOutputStream` (Apache Iceberg
1.11.0), not
+by PyIceberg. They exist so PyIceberg's AGS1 support is checked against another
+implementation's bytes rather than only against its own round trip. All four
also
+decrypt with iceberg-rust 0.10.1, through its `EncryptedInputFile`.
+
+## Fixtures
+
+| File | Encrypted size | Plaintext | Pins |
+| --- | --- | --- | --- |
+| `empty.ags1` | 36 B | 0 B | Java writes an 8 byte header **plus one empty
block** for an empty file, not a bare header |
+| `partial-block.ags1` | 136 B | 100 B | Header, nonce/tag layout, and a
single short block |
+| `partial-block-no-aad.ags1` | 136 B | 100 B | The same stream with a null
AAD prefix, so the block index alone is the AAD |
+| `aligned-multi-block.ags1` | 2097216 B | 2 MiB | Two full blocks: the
little-endian block index in each block's AAD, and that a block-aligned write
appends **no** trailing empty block |
Review Comment:
oh thanks for flagging this. im not a fan of checking in fixtures into the
repo. and if we must, we should exclude them from the packaged artifacts.
are all of these `.ags1` files necessary? I dont see them in the
iceberg-rust repo. Can we just generate them in the tests? Or perhaps inline
the specific content in the tests?
##########
pyiceberg/encryption/stream.py:
##########
Review Comment:
these are all "private" functions and classes, which is fine to merge.
I think we should start thinking about defining the correct api surface and
interface so that we can easily take the pyiceberg-core encryption features
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
