[
https://issues.apache.org/jira/browse/ARROW-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217917#comment-17217917
]
Joris Van den Bossche edited comment on ARROW-10344 at 10/20/20, 8:38 PM:
--------------------------------------------------------------------------
No, there is no way to concat horizontally with the datasets API.
>From your descriptions and the type of data, I am wondering if Feather is
>actually a suited file format. It seems you have a big array like data (and
>numeric data, but with labeled dimensions), which is potentially very wide,
>and which you would ideally chunk both on the rows as the columns. For
>something like that, it might be interesting to look at the zarr format
>(https://zarr.readthedocs.io/en/stable/index.html, it was originally started
>by someone working on genome data, but is now also becoming popular in
>geoscience/climate data that historically uses a lot of NetCDF)
was (Author: jorisvandenbossche):
No, there is no way to concat horizontally with the datasets API.
>From your descriptions and the type of data, I am wondering if Feather is
>actually a suited file format. It seems you have a big array like data (and
>numeric data, but with labeled dimensions), which is potentially very wide,
>and which you would ideally chunk both on the rows as the columns. For
>something like that, it might be interesting to look at the zarr format
>(https://zarr.readthedocs.io/en/stable/index.html)
> [Python] Get all columns names (or schema) from Feather file, before loading
> whole Feather file
> ------------------------------------------------------------------------------------------------
>
> Key: ARROW-10344
> URL: https://issues.apache.org/jira/browse/ARROW-10344
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Python
> Affects Versions: 1.0.1
> Reporter: Gert Hulselmans
> Priority: Major
>
> Is there a way to get all column names (or schema) from a Feather file before
> loading the full Feather file?
> My Feather files are big (like 100GB) and the names of the columns are
> different per analysis and can't be hard coded.
> {code:python}
> import pyarrow.feather as feather
> # Code here to check which columns are in the feather file.
> ...
> my_columns = ...
> # Result is pandas.DataFrame
> read_df = feather.read_feather('/path/to/file', columns=my_columns)
> # Result is pyarrow.Table
> read_arrow = feather.read_table('/path/to/file', columns=my_columns)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
