[
https://issues.apache.org/jira/browse/ARROW-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845908#comment-16845908
]
Rick Zamora edited comment on ARROW-5349 at 5/22/19 2:17 PM:
-------------------------------------------------------------
Okay - the file path should not be set in the footer metadata (only in
_metadata). Does this mean that a mechanism for setting the file_path in C++
is completely unnecessary? My understanding is that the motivation for this
issue was to populate the file_path for the following step of writing the
metadata file. Is it sufficient to add a python-only mechanism to set the
path? Or should we leave it up to the user to modify the metadata object
themselves?
was (Author: rjzamora):
Okay - the file path should not be set in the footer metadata (only in
_metadata). Does this mean that a mechanism for setting the file_path in C++
is completely unnecessary? My understanding is that the motivation for this
issue was to populate the file_path for the following step of writing the
metadata file. Is it sufficient to add a python-only mechanism to set the path?
> [Python/C++] Provide a way to specify the file path in parquet
> ColumnChunkMetaData
> ----------------------------------------------------------------------------------
>
> Key: ARROW-5349
> URL: https://issues.apache.org/jira/browse/ARROW-5349
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Python
> Reporter: Joris Van den Bossche
> Priority: Major
> Labels: parquet, pull-request-available
> Fix For: 0.14.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> After ARROW-5258 / https://github.com/apache/arrow/pull/4236 it is now
> possible to collect the file metadata while writing different files (then how
> to write those metadata was not yet addressed -> original issue ARROW-1983).
> However, currently, the {{file_path}} information in the ColumnChunkMetaData
> object is not set. This is, I think, expected / correct for the metadata as
> included within the single file; but for using the metadata in the combined
> dataset `_metadata`, it needs a file path set.
> So if you want to use this metadata for a partitioned dataset, there needs to
> be a way to specify this file path.
> Ideas I am thinking of currently: either, we could specify a file path to be
> used when writing, or expose the `set_file_path` method on the Python side so
> you can create an updated version of the metadata after collecting it.
> cc [~pearu] [~mdurant]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
