[
https://issues.apache.org/jira/browse/ARROW-13074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375901#comment-17375901
]
Lance Dacey commented on ARROW-13074:
-------------------------------------
I have run into a few issues with basename_template:
1) If I run tasks in parallel (for example, Airflow downloads data from various
SQL servers and writes to the same partitions), then there is a chance to
overwrite existing data (part-0.parquet)
2) If I make the basename_template unique, then I can end up with duplicate
data inside of my partitions because I am not overwriting what is already there.
The way I have been organizing this so far is to have use two datasets:
*Dataset A*:
* UUID filenames, so everything is unique. This most likely has duplicate
values, and most certainly will have old versions of rows (based on an
updated_at timestamp)
* This normally has a lot of files per partition since I download data every
30 minutes - 1 hour in many cases
*Dataset B:*
* Reads from Dataset A, sorts, drop duplicates, and then resave using a
partition_filename_cb
{code:java}
use_legacy_dataset=True,
partition_filename_cb=lambda x: str(x[-1]) + ".parquet",{code}
* I normally partition by date_id, so each partition is something like
{code:java}
path/date_id=20210706/20210706.parquet{code}
* This allows me to have a single file per partition which has the final
version of the each row with no duplicates. Our visualization tool connects to
these fragments directly (Power BI in this case)
I think that I might be able to use basename_template if I was careful and made
sure that I did not write data in parallel, so the part-0.parquet file would be
overwritten each time. Or perhaps I could list the files in that partition and
delete them before saving new data (risky if another process might be using
those files at that time).
> [Python] Start with deprecating ParquetDataset custom attributes
> ----------------------------------------------------------------
>
> Key: ARROW-13074
> URL: https://issues.apache.org/jira/browse/ARROW-13074
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Joris Van den Bossche
> Assignee: Joris Van den Bossche
> Priority: Major
> Labels: pull-request-available
> Fix For: 5.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> As a first step for ARROW-9720, we should start with deprecating
> attributes/methods of {{pq.ParquetDataset}} that we would definitely not keep
> / are conflicting with the "dataset API".
> I am thinking of the {{pieces}} attribute (and the {{ParquetDatasetPiece}}
> class), the {{partitions}} attribute (and the {{ParquetPartitions}} class).
> In addition, some of the keywords are also exposed as properties (memory_map,
> read_dictionary, buffer_size, fs), and could be deprecated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
