I see, thanks. Could we not introduce the concept of a delete marker, where we mark the deleted records in the page header?
On Mon, Dec 4, 2017 at 10:23 PM, Wes McKinney <[email protected]> wrote: > I don't think this is possible due to the encoding and compression schemes. > > For example, suppose that you had the following data > > 1 > 1 > 1 > 4 > 4 > 4 > 4 > > This would be dictionary-encoded and compressed to semantically look like > > dictionary: 1, 4 > data page: (3, 0) (4, 1) > > The encoded data page (using the hybrid bit-packing / RLE encoding > scheme) would furthermore be compressed. Editing records in general > would change the size of the compressed and encoded data stream, so > you could not edit the page without rewriting the file. > > - Wes > > On Mon, Dec 4, 2017 at 11:46 AM, Atri Sharma <[email protected]> wrote: >> Hi Wes, >> >> Thanks for your response. >> >> My main use case is that I want to introduce updatability to Parquet >> records without going the route of replacing the entire page. >> >> Is that something that has already been discussed please? >> >> Regards, >> >> Atri >> >> On Mon, Dec 4, 2017 at 10:10 PM, Wes McKinney <[email protected]> wrote: >>> hi Atri, >>> >>> From a prior discussion on the mailing list, it is not clear that this >>> is a problem that concerns either Parquet format or the >>> implementations in the Apache Parquet project. If data must be edited >>> or deleted, then the point-of-truth Parquet files must be scanned and >>> overwritten with the offending records deleted. Modifying files in >>> place is not feasible due to the compression and encoding schemes >>> (dictionary, run-length encoding) used in the Parquet format. Let me >>> know if I am misunderstanding the use case. >>> >>> Thanks >>> Wes >>> >>> On Mon, Dec 4, 2017 at 11:30 AM, Atri Sharma <[email protected]> wrote: >>>> Hi Folks, >>>> >>>> Any update? >>>> >>>> On Fri, Dec 1, 2017 at 9:23 AM, Atri Sharma <[email protected]> wrote: >>>>> https://issues.apache.org/jira/browse/PARQUET-1155 >>>>> >>>>> Anybody working on it? Can I take it up? >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Atri >>>> l'apprenant >> >> >> >> -- >> Regards, >> >> Atri >> l'apprenant -- Regards, Atri l'apprenant
