[
https://issues.apache.org/jira/browse/ARROW-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320759#comment-16320759
]
ASF GitHub Bot commented on ARROW-1980:
---------------------------------------
wesm commented on a change in pull request #1468: ARROW-1980: [Python] Fix race
condition in write_to_dataset
URL: https://github.com/apache/arrow/pull/1468#discussion_r160757113
##########
File path: python/pyarrow/parquet.py
##########
@@ -966,6 +966,14 @@ def write_table(table, where, row_group_size=None,
version='1.0',
""".format(_parquet_writer_arg_docs)
+def _mkdir_if_not_exists(fs, path):
+ if fs._isfilestore() and not fs.exists(path):
+ try:
+ fs.mkdir(path)
+ except OSError:
+ assert fs.exists(path)
Review comment:
Maybe worth opening a JIRA about this in either case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> [Python] Race condition in `write_to_dataset`
> ---------------------------------------------
>
> Key: ARROW-1980
> URL: https://issues.apache.org/jira/browse/ARROW-1980
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Jim Crist
> Labels: pull-request-available
>
> One race condition when creating directories was fixed in #1902, but a race
> condition still exists when using `partition_cols`.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
