May be the EU requirement provide a deadline for the delete. So one can imagine to implement a "logical delete", and on a monthly basis (assuming that is the EU deadline to be compliant), perform a physical delete by reloading the data without the logical deletes? It is like HBase major compaction concept? Eric
-----Original Message----- From: Wes McKinney [mailto:[email protected]] Sent: Monday, December 4, 2017 3:38 PM To: [email protected] Subject: Re: Regarding PARQUET-1155 hi Atri -- even if we could, I am not sure this would meet the requirements of the EU law, since the "deleted" data could still be read by an adversary even if a Parquet implementation like parquet-mr did not permit it - Wes On Mon, Dec 4, 2017 at 11:55 AM, Atri Sharma <[email protected]> wrote: > I see, thanks. > > Could we not introduce the concept of a delete marker, where we mark > the deleted records in the page header? > > On Mon, Dec 4, 2017 at 10:23 PM, Wes McKinney <[email protected]> wrote: >> I don't think this is possible due to the encoding and compression schemes. >> >> For example, suppose that you had the following data >> >> 1 >> 1 >> 1 >> 4 >> 4 >> 4 >> 4 >> >> This would be dictionary-encoded and compressed to semantically look >> like >> >> dictionary: 1, 4 >> data page: (3, 0) (4, 1) >> >> The encoded data page (using the hybrid bit-packing / RLE encoding >> scheme) would furthermore be compressed. Editing records in general >> would change the size of the compressed and encoded data stream, so >> you could not edit the page without rewriting the file. >> >> - Wes >> >> On Mon, Dec 4, 2017 at 11:46 AM, Atri Sharma <[email protected]> wrote: >>> Hi Wes, >>> >>> Thanks for your response. >>> >>> My main use case is that I want to introduce updatability to Parquet >>> records without going the route of replacing the entire page. >>> >>> Is that something that has already been discussed please? >>> >>> Regards, >>> >>> Atri >>> >>> On Mon, Dec 4, 2017 at 10:10 PM, Wes McKinney <[email protected]> wrote: >>>> hi Atri, >>>> >>>> From a prior discussion on the mailing list, it is not clear that >>>> this is a problem that concerns either Parquet format or the >>>> implementations in the Apache Parquet project. If data must be >>>> edited or deleted, then the point-of-truth Parquet files must be >>>> scanned and overwritten with the offending records deleted. >>>> Modifying files in place is not feasible due to the compression and >>>> encoding schemes (dictionary, run-length encoding) used in the >>>> Parquet format. Let me know if I am misunderstanding the use case. >>>> >>>> Thanks >>>> Wes >>>> >>>> On Mon, Dec 4, 2017 at 11:30 AM, Atri Sharma <[email protected]> wrote: >>>>> Hi Folks, >>>>> >>>>> Any update? >>>>> >>>>> On Fri, Dec 1, 2017 at 9:23 AM, Atri Sharma <[email protected]> wrote: >>>>>> https://issues.apache.org/jira/browse/PARQUET-1155 >>>>>> >>>>>> Anybody working on it? Can I take it up? >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Atri >>>>> l'apprenant >>> >>> >>> >>> -- >>> Regards, >>> >>> Atri >>> l'apprenant > > > > -- > Regards, > > Atri > l'apprenant
