How do you pass a csv.WriteOptions() class to pyarrow.dataset.write_dataset() ??
I tried pass in file_options = pa.csv.WriteOptions(include_header=True) and
file_options = {“include_header”: True}
Both attempts came back with an error: object has no attribute 'format'
CSV cookbook example:
pa.csv.write_csv(table, "table.csv",
write_options=pa.csv.WriteOptions(include_header=True))
Dataset supports a file_options arg
https://arrow.apache.org/docs/python/generated/pyarrow.dataset.write_dataset.html
file_options pyarrow.dataset.FileWriteOptions, optional
FileFormat specific write options, created using the
FileFormat.make_write_options() function.
Within Dataset I see references to csv.WriteOptions..
https://github.com/apache/arrow/blob/56186b994db3eab8b2684fde9e1726f0b0658ef6/python/pyarrow/_dataset.pyx#L1344
def make_write_options(self, **kwargs):
"""
Parameters
----------
**kwargs : dict
Returns
-------
pyarrow.csv.WriteOptions
"""
cdef CsvFileWriteOptions opts = \
<CsvFileWriteOptions> FileFormat.make_write_options(self)
opts.write_options = WriteOptions(**kwargs)
return opts
This message may contain information that is confidential or privileged. If you
are not the intended recipient, please advise the sender immediately and delete
this message. See
http://www.blackrock.com/corporate/compliance/email-disclaimers for further
information. Please refer to
http://www.blackrock.com/corporate/compliance/privacy-policy for more
information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2024 BlackRock, Inc. All rights reserved.
