[
https://issues.apache.org/jira/browse/ARROW-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joris Van den Bossche updated ARROW-3866:
-----------------------------------------
Description:
Hello everyone,
transferring this from Github for Pyarrow. While working with pyarrow I noticed
that field metadata does not get carried foreward when creating a table out of
several columns. Is this intended behaviour or is there a way to add column
metadata later on? The last command in my example does not return anything.
I also could not verify whether this data would be written to parquet later
on, because I could not find a way to add field metadata directly to a table.
{code:java}
>>> import pyarrow as pa
>>> import pyarrow.parquet as pq
>>> arr1 = pa.array([1,2])
>>> arr2 = pa.array([3,4])
>>> field1 = pa.field('field1', pa.int64())
>>> field2 = pa.field('field2', pa.int64())
>>> field1 = field1.add_metadata({'foo1': 'bar1'})
>>> field2 = field2.add_metadata({'foo2': 'bar2'})
>>> field1.metadata
{b'foo1': b'bar1'}
>>> field2.metadata
{b'foo2': b'bar2'}
>>> col1 = pa.column(field1, arr1)
>>> col2 = pa.column(field2, arr2)
>>> col1.field.metadata
{b'foo1': b'bar1'}
>>> tab = pa.Table.from_arrays([col1, col2])
>>> tab
pyarrow.Table field1: int64 field2: int64
>>> tab.column(0).field.metadata
{code}
was:
Hello everyone,
transferring this from Github for Pyarrow. While working with pyarrow I noticed
that field metadata does not get carried foreward when creating a table out of
several columns. Is this intended behaviour or is there a way to add column
metadata later on? The last command in my example does not return anything.
I also could not verify whether this data would be written to parquet later
on, because I could not find a way to add field metadata directly to a table.
{code:java}
>>>import pyarrow as pa
>>>import pyarrow.parquet as pq
>>>arr1 = pa.array([1,2])
>>>arr2 = pa.array([3,4])
>>>field1 = pa.field('field1', pa.int64())
>>>field2 = pa.field('field2', pa.int64())
>>>field1 = field1.add_metadata({'foo1': 'bar1'})
>>>field2 = field2.add_metadata({'foo2': 'bar2'})
>>>field1.metadata {b'foo1': b'bar1'}
>>>field2.metadata {b'foo2': b'bar2'}
>>>col1 = pa.column(field1, arr1)
>>>col2 = pa.column(field2, arr2)
>>>col1.field.metadata {b'foo1': b'bar1'}
>>>tab = pa.Table.from_arrays([col1, col2])
>>>tab pyarrow.Table field1: int64 field2: int64
>>>tab.column(0).field.metadata
{code}
> [Python] Column metadata is not transferred to tables in pyarrow
> ----------------------------------------------------------------
>
> Key: ARROW-3866
> URL: https://issues.apache.org/jira/browse/ARROW-3866
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Seb Fru
> Assignee: Tanya Schlusser
> Priority: Major
> Labels: features, newbie, pull-request-available
> Fix For: 0.12.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Hello everyone,
> transferring this from Github for Pyarrow. While working with pyarrow I
> noticed that field metadata does not get carried foreward when creating a
> table out of several columns. Is this intended behaviour or is there a way to
> add column metadata later on? The last command in my example does not return
> anything.
> I also could not verify whether this data would be written to parquet later
> on, because I could not find a way to add field metadata directly to a table.
>
> {code:java}
> >>> import pyarrow as pa
> >>> import pyarrow.parquet as pq
> >>> arr1 = pa.array([1,2])
> >>> arr2 = pa.array([3,4])
> >>> field1 = pa.field('field1', pa.int64())
> >>> field2 = pa.field('field2', pa.int64())
> >>> field1 = field1.add_metadata({'foo1': 'bar1'})
> >>> field2 = field2.add_metadata({'foo2': 'bar2'})
> >>> field1.metadata
> {b'foo1': b'bar1'}
> >>> field2.metadata
> {b'foo2': b'bar2'}
> >>> col1 = pa.column(field1, arr1)
> >>> col2 = pa.column(field2, arr2)
> >>> col1.field.metadata
> {b'foo1': b'bar1'}
> >>> tab = pa.Table.from_arrays([col1, col2])
> >>> tab
> pyarrow.Table field1: int64 field2: int64
> >>> tab.column(0).field.metadata
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
