jorisvandenbossche commented on code in PR #12811:
URL: https://github.com/apache/arrow/pull/12811#discussion_r851234914
##########
python/pyarrow/parquet/__init__.py:
##########
@@ -2962,11 +2965,47 @@ def write_to_dataset(table, root_path,
partition_cols=None,
and allow you to override the partition filename. If nothing is
passed, the filename will consist of a uuid.
use_legacy_dataset : bool
- Default is True unless a ``pyarrow.fs`` filesystem is passed.
- Set to False to enable the new code path (experimental, using the
- new Arrow Dataset API). This is more efficient when using partition
- columns, but does not (yet) support `partition_filename_cb` and
- `metadata_collector` keywords.
+ Default is False. Set to True to use the the legacy behaviour
+ (this option is deprecated, and the legacy implementation will be
+ removed in a future version). The legacy implementation still
+ supports `partition_filename_cb` and `metadata_collector` keywords
+ but is less efficient when using partition columns.
+ file_options : pyarrow.dataset.FileWriteOptions, optional
Review Comment:
I think `file_options` is similar as `format`? (only used to have it raise
for the legacy version, and so can also be removed)
##########
python/pyarrow/parquet/__init__.py:
##########
@@ -2962,11 +2965,47 @@ def write_to_dataset(table, root_path,
partition_cols=None,
and allow you to override the partition filename. If nothing is
passed, the filename will consist of a uuid.
use_legacy_dataset : bool
- Default is True unless a ``pyarrow.fs`` filesystem is passed.
- Set to False to enable the new code path (experimental, using the
- new Arrow Dataset API). This is more efficient when using partition
- columns, but does not (yet) support `partition_filename_cb` and
- `metadata_collector` keywords.
+ Default is False. Set to True to use the the legacy behaviour
+ (this option is deprecated, and the legacy implementation will be
+ removed in a future version). The legacy implementation still
+ supports `partition_filename_cb` and `metadata_collector` keywords
+ but is less efficient when using partition columns.
+ file_options : pyarrow.dataset.FileWriteOptions, optional
+ FileFormat specific write options, created using the
+ ``FileFormat.make_write_options()`` function.
+ use_threads : bool, default True
+ Write files in parallel. If enabled, then maximum parallelism will be
+ used determined by the number of available CPU cores.
+ schema : Schema, optional
+ partitioning : Partitioning or list[str], optional
+ The partitioning scheme specified with the ``partitioning()``
+ function or a list of field names. When providing a list of
+ field names, you can use ``partitioning_flavor`` to drive which
+ partitioning type should be used.
+ basename_template : str, optional
+ A template string used to generate basenames of written data files.
+ The token '{i}' will be replaced with an automatically incremented
+ integer. If not specified, it defaults to
+ "part-{i}." + format.default_extname
Review Comment:
```suggestion
integer. If not specified, it defaults to "guid-{i}.parquet"
```
##########
python/pyarrow/parquet/__init__.py:
##########
@@ -3042,13 +3079,50 @@ def file_visitor(written_file):
part_schema = table.select(partition_cols).schema
partitioning = ds.partitioning(part_schema, flavor="hive")
+ if basename_template is None:
+ basename_template = guid() + '{i}.parquet'
Review Comment:
```suggestion
basename_template = guid() + '-{i}.parquet'
```
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -1531,8 +1531,8 @@ def test_dataset_read_dictionary(tempdir,
use_legacy_dataset):
t1 = pa.table([[util.rands(10) for i in range(5)] * 10], names=['f0'])
t2 = pa.table([[util.rands(10) for i in range(5)] * 10], names=['f0'])
# TODO pass use_legacy_dataset (need to fix unique names)
- pq.write_to_dataset(t1, root_path=str(path))
- pq.write_to_dataset(t2, root_path=str(path))
+ pq.write_to_dataset(t1, root_path=str(path), use_legacy_dataset=True)
+ pq.write_to_dataset(t2, root_path=str(path), use_legacy_dataset=True)
Review Comment:
Same here, this test can now be updated?
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -1290,7 +1290,7 @@ def _test_write_to_dataset_no_partitions(base_path,
# Without partitions, append files to root_path
n = 5
for i in range(n):
- pq.write_to_dataset(output_table, base_path,
+ pq.write_to_dataset(output_table, base_path, use_legacy_dataset=True,
Review Comment:
With the latest changes, it should be possible to now change the hardcoded
`use_legacy_dataset=True` to `use_legacy_dataset=use_legacy_dataset`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
