Copilot commented on code in PR #47498:
URL: https://github.com/apache/arrow/pull/47498#discussion_r4027334569
##########
python/pyarrow/tests/test_dataset.py:
##########
@@ -5936,6 +5936,19 @@ def
test_checksum_write_dataset_read_dataset_to_table(tempdir):
).to_table()
[email protected]
[email protected]("cdc", [
+ True,
+ {"min_chunk_size": 32 * 1024, "max_chunk_size": 64 * 1024},
+])
+def test_write_dataset_content_defined_chunking(tempdir, cdc):
+ expected_table = pa.table({'a': [1, 2, 3]})
+ fmt = ds.ParquetFileFormat()
+ opts = fmt.make_write_options(use_content_defined_chunking=cdc)
+ ds.write_dataset(expected_table, tempdir, format=fmt, file_options=opts)
+ assert ds.dataset(tempdir, format=fmt).to_table().equals(expected_table)
Review Comment:
This only round-trips three rows, so neither the default nor the 32–64 KiB
configuration can create a content-defined boundary. As a result, the test
still passes if `_set_properties` drops the new option at line 663; use a
sufficiently large table and compare the output against a CDC-disabled control
(including page/row-group size or another observable CDC-specific property) to
verify that the dataset writer actually applies the setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
