wgtmac commented on code in PR #51357:
URL: https://github.com/apache/arrow/pull/51357#discussion_r4056000436
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1394,21 +1394,22 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
MaybeCalculateValidityBits(AddIfNotNull(def_levels, offset), batch_size,
&batch_num_values, &batch_num_spaced_values,
&null_count);
+ const int64_t total_null_count = batch_size - batch_num_values;
Review Comment:
Can we rename `total_null_count` to `parquet_null_count` because it means
number of nulls in the parquet leaf column? `null_count` is for the Arrow leaf
array but renaming it would result in a larger change.
##########
cpp/src/parquet/arrow/arrow_statistics_test.cc:
##########
@@ -160,6 +160,51 @@ INSTANTIATE_TEST_SUITE_P(
/*expected_min=*/"z",
/*expected_max=*/"z"}));
+TEST(StatisticsTest, FixedWidthLeafUnderListStructNullCount) {
+ // Null counts for fixed-width leaves under list<struct<...>>
+ // must include null and empty list entries from the repeated ancestor.
+ auto schema = ::arrow::schema({::arrow::field(
+ "col", ::arrow::list(::arrow::struct_(
+ {::arrow::field("s", ::arrow::utf8()),
+ ::arrow::field("i32", ::arrow::int32())})))});
+
+ auto table = ::arrow::Table::Make(
+ schema,
+ {::arrow::ArrayFromJSON(
+ ::arrow::list(::arrow::struct_(
+ {::arrow::field("s", ::arrow::utf8()),
+ ::arrow::field("i32", ::arrow::int32())})),
+
R"([[{"s":"a","i32":1}],null,[],[{"s":null,"i32":null},{"s":"b","i32":2}]])")});
+
+ std::shared_ptr<::arrow::ResizableBuffer> serialized_data = AllocateBuffer();
+ auto out_stream =
+ std::make_shared<::arrow::io::BufferOutputStream>(serialized_data);
+
+ ASSERT_OK_AND_ASSIGN(
+ std::unique_ptr<FileWriter> writer,
+ FileWriter::Open(*schema, default_memory_pool(), out_stream,
+ default_writer_properties(),
+ default_arrow_writer_properties()));
+ ASSERT_OK(writer->WriteTable(*table));
+ ASSERT_OK(writer->Close());
+ ASSERT_OK(out_stream->Close());
+
+ auto buffer_reader =
std::make_shared<::arrow::io::BufferReader>(serialized_data);
+ auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+ auto metadata = parquet_reader->metadata();
+ auto row_group = metadata->RowGroup(0);
+
+ ASSERT_EQ(row_group->num_columns(), 2);
+
+ auto int32_stats = row_group->ColumnChunk(1)->statistics();
Review Comment:
Can we run this test for both data page V1 and V2 and then check both leaf
columns? The old change fixed `i32` but broke the string leaf, so the current
checks would not catch that error.
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -2038,6 +2040,7 @@ Status
TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary(
// had so we need to recompute it from def levels.
MaybeCalculateValidityBits(AddIfNotNull(def_levels, offset), batch_size,
&batch_num_values, &batch_num_spaced_values,
&null_count);
+ const int64_t total_null_count = batch_size - batch_num_values;
Review Comment:
Please add the same nested test case with a dictionary leaf. Run it for data
page V1 and V2 by checking the statistics and reading the values back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
