I couldn't find anything in the docs, but you could just append a duplicate
set of column values to the table before calling write_to_dataset().
>>> for column in partitionCols:
... table = table.append_column(column + '_new', table[column])
...
>>> table
pyarrow.Table
animals: string
animals_new: string
----
animals: [["Flamingo"]]
animals_new: [["Flamingo"]]
-----Original Message-----
From: Mahesha Nayak <micomah...@gmail.com>
Sent: Wednesday, October 26, 2022 1:00 PM
To: dev@arrow.apache.org
Subject: pyarrow.parquet.write_to_dataset - Seeking an help in addressing some
customer request
External Email: Use caution with links and attachments
Hello Team,
I'm glad that I would be connecting with people who are helping in 'pyarrow'
usages.
my question is below snippet creating datafiles in s3 with partition keys
however partition columns are not part of datafiles hence is there any way we
can add the partition column in the datafiles so that customer query the
datafile will see the data for all columns + partition column.
kindly help me. onthis.
"targetKey = self.s3BucketName + '/' + outputDirectory + targetDirectory[:-1]
log.debug("Single or Multiple Partition columns being passed.",partitionCols
,[partitionCols], targetKey) pq.write_to_dataset(table=table,
root_path=targetKey, row_group_size=self.
chunkSizeLimit, partition_cols=[partitionCols], filesystem=s3, compression=
'snappy',partition_filename_cb=lambda x:'-'.join(str(x))+'.parquet')
--
Regards,
Mahesha S
Cell:8015127140
This message may contain information that is confidential or privileged. If you
are not the intended recipient, please advise the sender immediately and delete
this message. See
http://www.blackrock.com/corporate/compliance/email-disclaimers for further
information. Please refer to
http://www.blackrock.com/corporate/compliance/privacy-policy for more
information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2022 BlackRock, Inc. All rights reserved.
