Hi, I think that delete marker is good idea. I was in basic GDPR training and i think that it meets EU law requirements
Lukas 2017-12-05 11:37 GMT+01:00 Atri Sharma <[email protected]>: > Agreed. > > I have come up with a patch to add metadata to the page header marking > the tuples deleted. The visibility checks will need to consult page > header before returning the read results back. > > The pruning still needs to be implemented. > > On Tue, Dec 5, 2017 at 3:20 AM, Eric Owhadi <[email protected]> wrote: > > May be the EU requirement provide a deadline for the delete. So one can > imagine to implement a "logical delete", and on a monthly basis (assuming > that is the EU deadline to be compliant), perform a physical delete by > reloading the data without the logical deletes? It is like HBase major > compaction concept? > > Eric > > > > > > -----Original Message----- > > From: Wes McKinney [mailto:[email protected]] > > Sent: Monday, December 4, 2017 3:38 PM > > To: [email protected] > > Subject: Re: Regarding PARQUET-1155 > > > > hi Atri -- even if we could, I am not sure this would meet the > requirements of the EU law, since the "deleted" data could still be read by > an adversary even if a Parquet implementation like parquet-mr did not > permit it > > > > - Wes > > > > On Mon, Dec 4, 2017 at 11:55 AM, Atri Sharma <[email protected]> > wrote: > >> I see, thanks. > >> > >> Could we not introduce the concept of a delete marker, where we mark > >> the deleted records in the page header? > >> > >> On Mon, Dec 4, 2017 at 10:23 PM, Wes McKinney <[email protected]> > wrote: > >>> I don't think this is possible due to the encoding and compression > schemes. > >>> > >>> For example, suppose that you had the following data > >>> > >>> 1 > >>> 1 > >>> 1 > >>> 4 > >>> 4 > >>> 4 > >>> 4 > >>> > >>> This would be dictionary-encoded and compressed to semantically look > >>> like > >>> > >>> dictionary: 1, 4 > >>> data page: (3, 0) (4, 1) > >>> > >>> The encoded data page (using the hybrid bit-packing / RLE encoding > >>> scheme) would furthermore be compressed. Editing records in general > >>> would change the size of the compressed and encoded data stream, so > >>> you could not edit the page without rewriting the file. > >>> > >>> - Wes > >>> > >>> On Mon, Dec 4, 2017 at 11:46 AM, Atri Sharma <[email protected]> > wrote: > >>>> Hi Wes, > >>>> > >>>> Thanks for your response. > >>>> > >>>> My main use case is that I want to introduce updatability to Parquet > >>>> records without going the route of replacing the entire page. > >>>> > >>>> Is that something that has already been discussed please? > >>>> > >>>> Regards, > >>>> > >>>> Atri > >>>> > >>>> On Mon, Dec 4, 2017 at 10:10 PM, Wes McKinney <[email protected]> > wrote: > >>>>> hi Atri, > >>>>> > >>>>> From a prior discussion on the mailing list, it is not clear that > >>>>> this is a problem that concerns either Parquet format or the > >>>>> implementations in the Apache Parquet project. If data must be > >>>>> edited or deleted, then the point-of-truth Parquet files must be > >>>>> scanned and overwritten with the offending records deleted. > >>>>> Modifying files in place is not feasible due to the compression and > >>>>> encoding schemes (dictionary, run-length encoding) used in the > >>>>> Parquet format. Let me know if I am misunderstanding the use case. > >>>>> > >>>>> Thanks > >>>>> Wes > >>>>> > >>>>> On Mon, Dec 4, 2017 at 11:30 AM, Atri Sharma <[email protected]> > wrote: > >>>>>> Hi Folks, > >>>>>> > >>>>>> Any update? > >>>>>> > >>>>>> On Fri, Dec 1, 2017 at 9:23 AM, Atri Sharma <[email protected]> > wrote: > >>>>>>> https://issues.apache.org/jira/browse/PARQUET-1155 > >>>>>>> > >>>>>>> Anybody working on it? Can I take it up? > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Regards, > >>>>>> > >>>>>> Atri > >>>>>> l'apprenant > >>>> > >>>> > >>>> > >>>> -- > >>>> Regards, > >>>> > >>>> Atri > >>>> l'apprenant > >> > >> > >> > >> -- > >> Regards, > >> > >> Atri > >> l'apprenant > > > > -- > Regards, > > Atri > l'apprenant >
