[
https://issues.apache.org/jira/browse/ARROW-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706664#comment-16706664
]
David Lee edited comment on ARROW-3918 at 12/3/18 5:17 AM:
-----------------------------------------------------------
Passed them into ParquetWriter and it still gives the same error..
File "../python3.6/site-packages/pyarrow/parquet.py", line 374, in write_table
raise ValueError(msg)
ValueError: Table schema does not match schema used to create file:
table:
Id: string
modified: timestamp[ms]
converter: string
records: int32
metadata
--------
{b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": [
{"name":' b' "Id", "field_name": "Id", "' b'pandas_type": "unicode",
"numpy_type": "object", "metadata": nul' b'l}
,
{"name": "modified", "field_name": "modified", "pandas_type"' b': "datetime",
"numpy_type": "datetime64[ns]", "metadata": null}
,'
b'
{"name": "converter", "field_name": "converter", "pandas_type":' b' "unicode",
"numpy_type": "object", "metadata": null}
,
{"name": ' b'"records", "field_name": "records", "pandas_type": "int32", "num'
b'py_type": "int64", "metadata": null}
], "pandas_version": "0.23.4'
b'"}'} vs.
file:
Id: string
modified: timestamp[ms]
converter: string
records: int32
Code:
{code:java}
processed_schema = pa.schema([
pa.field('Id', pa.string()),
pa.field('modified', pa.timestamp('ms')),
pa.field('converter', pa.string()),
pa.field('records', pa.int32())
])
.....
arrow_tables.append(pa.Table.from_pandas(df, schema=processed_schema,
preserve_index=False, safe=False))
.....
if len(arrow_tables) > 0:
writer = pq.ParquetWriter(os.path.join(self.conf['work_dir'],
processed_file), schema=processed_schema, use_dictionary=True,
compression='snappy', coerce_timestamps='ms', allow_truncated_timestamps=True)
for v in arrow_tables:
writer.write_table(v)
writer.close()
{code}
was (Author: [email protected]):
Passed them into ParquetWriter and it still gives the same error..
File "../python3.6/site-packages/pyarrow/parquet.py", line 374, in write_table
raise ValueError(msg)
ValueError: Table schema does not match schema used to create file:
table:
Id: string
modified: timestamp[ms]
converter: string
records: int32
metadata
--------
{b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": [{"name":'
b' "Id", "field_name": "Id", "'
b'pandas_type": "unicode", "numpy_type": "object", "metadata": nul'
b'l}, {"name": "modified", "field_name": "modified", "pandas_type"'
b': "datetime", "numpy_type": "datetime64[ns]", "metadata": null},'
b' {"name": "converter", "field_name": "converter", "pandas_type":'
b' "unicode", "numpy_type": "object", "metadata": null}, {"name": '
b'"records", "field_name": "records", "pandas_type": "int32", "num'
b'py_type": "int64", "metadata": null}], "pandas_version": "0.23.4'
b'"}'} vs.
file:
Id: string
modified: timestamp[ms]
converter: string
records: int32
Code:
{code:java}
processed_schema = pa.schema([
pa.field('Id', pa.string()),
pa.field('modified', pa.timestamp('ms')),
pa.field('converter', pa.string()),
pa.field('records', pa.int32())
])
if len(arrow_tables) > 0:
writer = pq.ParquetWriter(os.path.join(self.conf['work_dir'],
processed_file), schema=processed_schema, use_dictionary=True,
compression='snappy', coerce_timestamps='ms', allow_truncated_timestamps=True)
for v in arrow_tables:
writer.write_table(v)
writer.close()
{code}
> [Python] ParquetWriter.write_table doesn't support coerce_timestamps or
> allow_truncated_timestamps
> --------------------------------------------------------------------------------------------------
>
> Key: ARROW-3918
> URL: https://issues.apache.org/jira/browse/ARROW-3918
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.11.1
> Reporter: David Lee
> Priority: Major
>
> Error: Table Schema does not match schema used to create file.
> The 0.11.1 release added these parameters to pyarrow.parquet.write_table(),
> but they are missing from pyarrow.parquet.ParquetWriter.write_table().. I'm
> seeing mismatches between the table schema and the file schema, but they are
> identical in the error message with modified: timestamp[ms] column types in
> both schemas. The only thing which looks odd is the Pandas metadata that has
> a modified column with a panda datatype of datetime and a numpy datatype of
> datetime64[ns]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
