hectar-glitches commented on issue #1289: URL: https://github.com/apache/iceberg-go/issues/1289#issuecomment-4819799544
@laskoviymishka I did some scavenging around the repo. I saw that encryption.go has the EncryptionKey struct (v3-gated), manifest.go has KeyMetadata() []byte on both ManifestFile and DataFile, statistics.go has KeyMetadata *string on StatisticsFile, and the Avro schemas already wire key_metadata fields (IDs 519 and 131). The core design question is how key_metadata is delivered to the IO layer. We can extend with an optional interface mirroring how ReadFileIO/WriteFileIO/ListableIO extend today, with an EncryptedIO that includes OpenEncrypted and CreateEncrypted methods that accept keyMetadata []byte. Existing io.IO users are unaffected; the scan and write paths already have DataFile.KeyMetadata() and pass it through when the underlying IO implements EncryptedIO. Parquet native encryption is already supported. arrow-go/v18 already exposes FileEncryptionProperties/FileDecryptionProperties publicly. We can then just wire them up. I don't think pqarrow changes are needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
