[
https://issues.apache.org/jira/browse/ARROW-16506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534261#comment-17534261
]
Daniel Friar commented on ARROW-16506:
--------------------------------------
Thanks [~westonpace], that makes sense. I wonder if it's worth adding to the
docs just to make it clear that this is the case and ordered writes should not
be expected/relied upon?
I do think there are cases where an ordered write is important, particularly
for e.g. time series data where a particular ordering is necessary for
downstream tasks and additional in-memory sorts once the data is loaded may be
expensive.
More of an opinion but generally I think it can be a little surprising to write
from a table/dataframe and read back to discover the order has changed. Adding
sequencing to achieve an ordered write would be an improvement IMO!
> Pyarrow 8.0.0 write_dataset writes data in different order with
> use_threads=True
> --------------------------------------------------------------------------------
>
> Key: ARROW-16506
> URL: https://issues.apache.org/jira/browse/ARROW-16506
> Project: Apache Arrow
> Issue Type: Bug
> Reporter: Daniel Friar
> Priority: Major
> Labels: dataset, parquet, pyarrow
>
> In the latest (8.0.0) release the following code snippet seems to write out
> data in a different order for each of the partitions when
> {{use_threads=True}} vs when {{{}use_threads=False{}}}.
> Testing the same snippet with pyarrow 7.0.0 gives the same order regardless
> of whether {{use_threads}} is set to True when the data is written.
>
> {code:java}
> import itertools
> import numpy as np
> import pyarrow.dataset as ds
> import pyarrow as pa
> n_rows, n_cols = 100_000, 20
> def create_dataframe(color, year):
> arr = np.random.randn(n_rows, n_cols)
> df = pd.DataFrame(data=arr, columns=[f"column_{i}" for i in
> range(n_cols)])
> df["color"] = color
> df["year"] = year
> df["id"] = np.arange(len(df))
> return df
> partitions = ["red", "green", "blue"]
> years = [2011, 2012, 2013]
> dataframes = [create_dataframe(p, y) for p, y in
> itertools.product(partitions, years)]
> df = pd.concat(dataframes)
> table = pa.Table.from_pandas(df=df)
> ds.write_dataset(
> table,
> "./test",
> format="parquet",
> max_rows_per_group=1_000_000,
> min_rows_per_group=1_000_000,
> existing_data_behavior="overwrite_or_ignore",
> partitioning=ds.partitioning(pa.schema([
> ("color", pa.string()),
> ("year", pa.int64())
> ]), flavor="hive"),
> use_threads=True,
> )
> df_read = pd.read_parquet("./test/color=blue/year=2012")
> df_read.head()[["id"]]
> {code}
>
> Tested on Ubuntu 20.04 with Python 3.8 and arrow versions 8.0.0 and 7.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
