Ok I figured out..
You have to create a pyarrow.dataset.CsvFileFormat object first and generate a
csv_file_options=CsvFileFormat.make_write_options(**{include_header: True})
first..
Then pass file_options = csv_file_options in write_dataset()..
The only issue I’ve seen is that when using quoting_style=”none”.. The rows are
no longer quoted, but the header row still have quotes in it..
From: Lee, David (PAG)
Sent: Tuesday, April 16, 2024 12:56 PM
To: [email protected]; [email protected]
Subject: [Python] pyarrow dataset writing CSV with or without headers and
quoting style
How do you pass a csv.WriteOptions() class to pyarrow.dataset.write_dataset() ??
I tried pass in file_options = pa.csv.WriteOptions(include_header=True) and
file_options = {“include_header”: True}
Both attempts came back with an error: object has no attribute 'format'
CSV cookbook example:
pa.csv.write_csv(table, "table.csv",
write_options=pa.csv.WriteOptions(include_header=True))
Dataset supports a file_options arg
https://arrow.apache.org/docs/python/generated/pyarrow.dataset.write_dataset.html
file_options pyarrow.dataset.FileWriteOptions, optional
FileFormat specific write options, created using the
FileFormat.make_write_options() function.
Within Dataset I see references to csv.WriteOptions..
https://github.com/apache/arrow/blob/56186b994db3eab8b2684fde9e1726f0b0658ef6/python/pyarrow/_dataset.pyx#L1344
def make_write_options(self, **kwargs):
"""
Parameters
----------
**kwargs : dict
Returns
-------
pyarrow.csv.WriteOptions
"""
cdef CsvFileWriteOptions opts = \
<CsvFileWriteOptions> FileFormat.make_write_options(self)
opts.write_options = WriteOptions(**kwargs)
return opts
This message may contain information that is confidential or privileged. If you
are not the intended recipient, please advise the sender immediately and delete
this message. See
http://www.blackrock.com/corporate/compliance/email-disclaimers for further
information. Please refer to
http://www.blackrock.com/corporate/compliance/privacy-policy for more
information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2024 BlackRock, Inc. All rights reserved.
