Thanks for sending the spec. I will take a look once I am back from vacation on Sunday.
I will be happy to collaborate on this and lead the delete markers stuff On 8 Dec 2017 00:36, "Ryan Blue" <[email protected]> wrote: > I sent an intro to the Iceberg table format that we're building yesterday. > I think the table format is the right place to implement GDPR requirements > because it isn't really feasible to modify Parquet files without rewriting > entirely. > > Delete markers should be tracked in separate files that support appends > (like Avro). The table format must also support the compaction jobs that > rewrite Parquet files to completely delete the data, which can't be done > atomically in the current Hive table layout (you can't atomically swap one > file for another). And, getting this into the table format early is also a > good opportunity to ensure readers always apply the deletes. > > I'm interested in building this capability into the table format we're > working on and would be happy to collaborate on a spec for this if anyone > would like to lead that effort. > > rb > > On Tue, Dec 5, 2017 at 8:54 AM, Eric Owhadi <[email protected]> wrote: > > > One thing to account for is the row count stats spread all over the > > various level of stats. If a record is logical deleted, then rowcount = > > rowcount -1. > > So when using any level of stats to compute row count, how do we account > > for logical deletes? > > Eric > > > > -----Original Message----- > > From: Atri Sharma [mailto:[email protected]] > > Sent: Tuesday, December 5, 2017 10:48 AM > > To: [email protected] > > Subject: Re: Regarding PARQUET-1155 > > > > Thanks. > > > > I think a configurable purger which can replace pages (like HBase > > Compaction, as mentioned above) should suffice and the frequency of > > compaction can be defined. > > > > Do we do the full page replacement technique for replacing records today > > in any scenario? > > > > Regards, > > > > Atri > > > > On Tue, Dec 5, 2017 at 9:17 PM, lukas nalezenec <[email protected]> > wrote: > > > Hi, > > > I think that delete marker is good idea. > > > I was in basic GDPR training and i think that it meets EU law > > > requirements > > > > > > Lukas > > > > > > 2017-12-05 11:37 GMT+01:00 Atri Sharma <[email protected]>: > > > > > >> Agreed. > > >> > > >> I have come up with a patch to add metadata to the page header > > >> marking the tuples deleted. The visibility checks will need to > > >> consult page header before returning the read results back. > > >> > > >> The pruning still needs to be implemented. > > >> > > >> On Tue, Dec 5, 2017 at 3:20 AM, Eric Owhadi <[email protected]> > > wrote: > > >> > May be the EU requirement provide a deadline for the delete. So one > > >> > can > > >> imagine to implement a "logical delete", and on a monthly basis > > >> (assuming that is the EU deadline to be compliant), perform a > > >> physical delete by reloading the data without the logical deletes? It > > >> is like HBase major compaction concept? > > >> > Eric > > >> > > > >> > > > >> > -----Original Message----- > > >> > From: Wes McKinney [mailto:[email protected]] > > >> > Sent: Monday, December 4, 2017 3:38 PM > > >> > To: [email protected] > > >> > Subject: Re: Regarding PARQUET-1155 > > >> > > > >> > hi Atri -- even if we could, I am not sure this would meet the > > >> requirements of the EU law, since the "deleted" data could still be > > >> read by an adversary even if a Parquet implementation like parquet-mr > > >> did not permit it > > >> > > > >> > - Wes > > >> > > > >> > On Mon, Dec 4, 2017 at 11:55 AM, Atri Sharma <[email protected]> > > >> wrote: > > >> >> I see, thanks. > > >> >> > > >> >> Could we not introduce the concept of a delete marker, where we > > >> >> mark the deleted records in the page header? > > >> >> > > >> >> On Mon, Dec 4, 2017 at 10:23 PM, Wes McKinney > > >> >> <[email protected]> > > >> wrote: > > >> >>> I don't think this is possible due to the encoding and > > >> >>> compression > > >> schemes. > > >> >>> > > >> >>> For example, suppose that you had the following data > > >> >>> > > >> >>> 1 > > >> >>> 1 > > >> >>> 1 > > >> >>> 4 > > >> >>> 4 > > >> >>> 4 > > >> >>> 4 > > >> >>> > > >> >>> This would be dictionary-encoded and compressed to semantically > > >> >>> look like > > >> >>> > > >> >>> dictionary: 1, 4 > > >> >>> data page: (3, 0) (4, 1) > > >> >>> > > >> >>> The encoded data page (using the hybrid bit-packing / RLE > > >> >>> encoding > > >> >>> scheme) would furthermore be compressed. Editing records in > > >> >>> general would change the size of the compressed and encoded data > > >> >>> stream, so you could not edit the page without rewriting the file. > > >> >>> > > >> >>> - Wes > > >> >>> > > >> >>> On Mon, Dec 4, 2017 at 11:46 AM, Atri Sharma > > >> >>> <[email protected]> > > >> wrote: > > >> >>>> Hi Wes, > > >> >>>> > > >> >>>> Thanks for your response. > > >> >>>> > > >> >>>> My main use case is that I want to introduce updatability to > > >> >>>> Parquet records without going the route of replacing the entire > > page. > > >> >>>> > > >> >>>> Is that something that has already been discussed please? > > >> >>>> > > >> >>>> Regards, > > >> >>>> > > >> >>>> Atri > > >> >>>> > > >> >>>> On Mon, Dec 4, 2017 at 10:10 PM, Wes McKinney > > >> >>>> <[email protected]> > > >> wrote: > > >> >>>>> hi Atri, > > >> >>>>> > > >> >>>>> From a prior discussion on the mailing list, it is not clear > > >> >>>>> that this is a problem that concerns either Parquet format or > > >> >>>>> the implementations in the Apache Parquet project. If data must > > >> >>>>> be edited or deleted, then the point-of-truth Parquet files > > >> >>>>> must be scanned and overwritten with the offending records > > deleted. > > >> >>>>> Modifying files in place is not feasible due to the compression > > >> >>>>> and encoding schemes (dictionary, run-length encoding) used in > > >> >>>>> the Parquet format. Let me know if I am misunderstanding the use > > case. > > >> >>>>> > > >> >>>>> Thanks > > >> >>>>> Wes > > >> >>>>> > > >> >>>>> On Mon, Dec 4, 2017 at 11:30 AM, Atri Sharma > > >> >>>>> <[email protected]> > > >> wrote: > > >> >>>>>> Hi Folks, > > >> >>>>>> > > >> >>>>>> Any update? > > >> >>>>>> > > >> >>>>>> On Fri, Dec 1, 2017 at 9:23 AM, Atri Sharma > > >> >>>>>> <[email protected]> > > >> wrote: > > >> >>>>>>> https://issues.apache.org/jira/browse/PARQUET-1155 > > >> >>>>>>> > > >> >>>>>>> Anybody working on it? Can I take it up? > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> -- > > >> >>>>>> Regards, > > >> >>>>>> > > >> >>>>>> Atri > > >> >>>>>> l'apprenant > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> -- > > >> >>>> Regards, > > >> >>>> > > >> >>>> Atri > > >> >>>> l'apprenant > > >> >> > > >> >> > > >> >> > > >> >> -- > > >> >> Regards, > > >> >> > > >> >> Atri > > >> >> l'apprenant > > >> > > >> > > >> > > >> -- > > >> Regards, > > >> > > >> Atri > > >> l'apprenant > > >> > > > > > > > > -- > > Regards, > > > > Atri > > l'apprenant > > > > > > -- > Ryan Blue > Software Engineer > Netflix >
