pitrou commented on code in PR #1291:
URL: https://github.com/apache/parquet-mr/pull/1291#discussion_r1516433630
##########
parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV2ValuesWriterFactory.java:
##########
@@ -84,10 +92,25 @@ private ValuesWriter getBooleanValuesWriter() {
}
private ValuesWriter getFixedLenByteArrayValuesWriter(ColumnDescriptor path)
{
- ValuesWriter fallbackWriter = new DeltaByteArrayWriter(
- parquetProperties.getInitialSlabSize(),
- parquetProperties.getPageSizeThreshold(),
- parquetProperties.getAllocator());
+ final ValuesWriter fallbackWriter;
+ // Heuristic: enable BYTE_STREAM_SPLIT for DECIMAL and FLOAT16 columns.
+ final boolean useByteStreamSplit =
parquetProperties.isExtendedByteStreamSplitEnabled()
Review Comment:
* Footprint-wise:
1. `DELTA_BYTE_ARRAY` is, at least conceptually, suboptimal for FLBA as it
will store both prefix and suffix lengths, even though one can be trivially
deduced from the other.
2. `DELTA_BYTE_ARRAY` is most certainly a bad fit for very short data such
as `FLOAT16`.
3. While https://issues.apache.org/jira/browse/PARQUET-2414 did exercise
`BYTE_STREAM_SPLIT` on `DECIMAL` and `FLOAT16` values, we don't know if it can
be beneficial or detrimental for other kinds of `FIXED_LEN_BYTE_ARRAY` column.
* CPU-wise, `BYTE_STREAM_SPLIT` should be much faster than
`DELTA_BYTE_ARRAY`, though concrete benchmarks would be needed for parquet-mr
(we do have benchmarks for Parquet C++).
That said, perhaps we should indeed not add any heuristics and instead let
the user configure things column-by-column (combining heuristics + column
selection would be a bit delicate, though).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
