Hi, The upcoming cross EU law GDPR requires companies to remove data collected from consumers as requested. I'm exploring the options concerning our Parquet tables.
I don't see any support for mutating parquet files, if it's not there is it possible to add that? I wonder if anyone has any knowledge of how a deletion could be processed in the parquet world. Of course there is the option to sift through billions of records and recreate all our tables for each deletion request but I'm hoping for a more efficient method. Perhaps a delete flag could be added to the format or is there a way to zero out existing data? At some point all companies storing data of EU citizens will need to have an answer to this. Simply locking the data behind more restrictions is not an option, data should be erased. Companies are already looking into ways to delete data from tape backups, the law is that far reaching.
