ggershinsky commented on PR #41821: URL: https://github.com/apache/arrow/pull/41821#issuecomment-2145661076
> > mainly because there is a limit on number of crypto operations with one key > > What is the theoretical limit, assuming a 256-bit AES key? ~ a billion crypto operations (meaning a billion parquet pages) > Also, if column key encryption is used, wouldn't the limit basically become irrelevant? Yes, if columns are encrypted with column-specific keys. If a column is encrypted in a "footer key" mode, the counter applies. The cleanest general solution would be to create and manage a separate "summary file" key (eg keeping it - wrapped by a master key in KMS - in the summary file itself). Lacking that, a workaround that somehow applies the same footer key across all nodes / workers, would be safe enough under certain conditions. This workaround won't be compatible with the higher-level Parquet/Arrow/etc APIs that generate a new footer key for each data file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
