pitrou commented on code in PR #55453:
URL: https://github.com/apache/arrow/pull/55453#discussion_r4583912742
##########
cpp/src/parquet/file_serialize_test.cc:
##########
@@ -148,8 +148,15 @@ class TestSerialize : public PrimitiveTypedTest<TestType> {
const int64_t total_compressed_size =
rg_metadata->total_compressed_size();
if (expected_codec_type == Compression::UNCOMPRESSED) {
ASSERT_EQ(total_byte_size, total_compressed_size);
- } else {
+ } else if (writer_properties->data_page_version() ==
ParquetDataPageVersion::V1) {
ASSERT_NE(total_byte_size, total_compressed_size);
+ } else {
+ // In Parquet V2, selective compression may keep small or
incompressible data
+ // pages uncompressed (total_compressed_size == total_byte_size), while
+ // compressed dictionary pages or pages with savings may be larger or
smaller
Review Comment:
They shouldn't be "larger or smaller", they should be smaller, precisely
because of selective compression.
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -2024,7 +2024,8 @@ Status
TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary(
::arrow::compute::Unique(*chunk_indices,
&exec_ctx));
// On first run, we might be able to re-use the existing dictionary
Review Comment:
Let's update this comment? This isn't about a first run but whether all dict
values are referenced.
```suggestion
// Use the original dictionary if we can detect that all indices are
referenced,
// otherwise call Take to find referenced dictionary values.
```
##########
python/pyarrow/parquet/core.py:
##########
@@ -999,8 +999,8 @@ def _sanitize_table(table, new_schema, flavor):
Use a value from 0.0 to 1.0. For example, 0.1 requires at least 10%
savings.
Savings are measured on encoded values, excluding repetition and definition
levels. The default is 0.01, so compression must save at least 1%.
- Values above this default require ``data_page_version="2.0"``. Using
- a larger value with V1 or a number outside the allowed range raises
``ValueError``.
+ Using a larger value with V1 or a number outside the allowed range
+ raises ``ValueError``.
Review Comment:
```suggestion
Using a larger value with V1 data pages or a number outside
the allowed range raises ``ValueError``.
```
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -2024,7 +2024,8 @@ Status
TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary(
::arrow::compute::Unique(*chunk_indices,
&exec_ctx));
// On first run, we might be able to re-use the existing dictionary
- if (referenced_indices.length() == dictionary->length()) {
+ if (chunk_indices->null_count() == 0 &&
+ referenced_indices.length() == dictionary->length()) {
Review Comment:
Would it make sense to refine the shortcut condition? Something like:
```suggestion
const bool all_indices_referenced =
(referenced_indices.null_count() == 0)
? referenced_indices.length() == dictionary->length()
: referenced_indices.length() == dictionary->length() + 1;
if (all_indices_referenced) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]