prtkgaur commented on code in PR #617:
URL: https://github.com/apache/parquet-format/pull/617#discussion_r3936638490
##########
Encodings.md:
##########
@@ -420,3 +420,636 @@ enabling random access to individual values and parallel
encoding/decoding.
The detailed specification of the ALP encoding, including the page layout and
the encoding and decoding procedures, is in [AlpEncoding.md](AlpEncoding.md).
+<a name="PFOR"></a>
+### Patched Frame of Reference (PFOR = 11)
+
+Supported Types: INT32, INT64
+
+PFOR (Patched Frame of Reference) compresses integer columns by subtracting the
+minimum value (Frame of Reference), then bit-packing the residuals at an
optimal
+bit width selected by a cost model. Values that do not fit in the chosen bit
width
+are stored as exceptions ("patches"). The cost model trades off narrower
bit-packing
+against the overhead of storing exceptions, achieving better compression than
plain
+FOR when a few outlier values would otherwise inflate the bit width.
+
+A writer MAY instead pack the differences between successive values. That
choice
+is the **delta mode**, it is recorded per vector in bit 7 of `bit_width`, and a
+vector in that mode carries its own first value so that it still decodes
without
+reading the vector before it. Everything else -- the frame of reference, the
bit
+width, the exceptions -- is then chosen over the differences exactly as it
would
+be over the values, and a reader runs one extra step, a prefix sum, to reverse
+the differencing.
+
+The values that reach the packed stream are called *residuals* throughout this
+section, in both modes: a residual is what remains after the frame of reference
+is subtracted, whether the frame was subtracted from a value or from a
+difference.
+
+#### Overview
Review Comment:
Will be moved to a separate file (just like ALP)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
