[
https://issues.apache.org/jira/browse/ARROW-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320167#comment-17320167
]
Lance Dacey commented on ARROW-10695:
-------------------------------------
I have been creating my own basename_template with either a uuid or a name with
the task+timestamp of when the data was processed and it has worked well. I
like that approach better than the uuid filename actually.
I think the remaining issue with the default part-{i} template is that it can
also be a bit inconsistent when writing data in loops. Say I am processing a
directory of files one by one in a loop and I partition the data on the "date"
column. A lot of the files will just overwrite the part-0.parquet file, but you
might also see part-11.parquet or another random filename. I suppose the
surprising part is that write_dataset() does not always append new random files
nor does it *always* overwrite what is there. This does not impact me now that
I customize the basename_template though, but I think an "append" or "replace"
flag would make a lot of sense
I'll open another issue with my use case for metadata_collector and
partition_filename_cb which I am using heavily
> [C++][Dataset] Allow to use a UUID in the basename_template when writing a
> dataset
> ----------------------------------------------------------------------------------
>
> Key: ARROW-10695
> URL: https://issues.apache.org/jira/browse/ARROW-10695
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Joris Van den Bossche
> Priority: Minor
> Labels: dataset, dataset-parquet-write
> Fix For: 5.0.0
>
>
> Currently we allow the user to specify a {{basename_template}}, and this can
> include a {{"\{i\}"}} part to replace it with an automatically incremented
> integer (so each generated file written to a single partition is unique):
> https://github.com/apache/arrow/blob/master/python/pyarrow/dataset.py#L713-L717
> It _might_ be useful to also have the ability to use a UUID, to ensure the
> file is unique in general (not only for a single write) and to mimic the
> behaviour of the old {{write_to_dataset}} implementation.
> For example, we could look for a {{"\{uuid\}"}} in the template string, and
> if present replace it for each file with a new UUID.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
