[
https://issues.apache.org/jira/browse/ARROW-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Dipale updated ARROW-11803:
-----------------------------------
Description:
We currently do not read nor write the version 2 logical types. This is mainly
because we do not have a mapping for it from parquet-format-rs.
To implement this, we can:
- convert "parquet::basic::LogicalType" to "parquet::basic::ConvertedType"
- implement "parquet::basic::LogicalType" which mirrors
"parquet_format::LogicalType"
- create a mapping between ConvertedType and LogicalType
- write LogicalType to "parquet_format::SchemaElement" if v2 of the writer is
used
This would be a good starting point for implementing 2.6 types (UUID, NANOS
precision time & timestamp).
Follow-up work would be:
- parsing v2 of the schema [ARROW-11365]
- Using v2 in the Arrow writer (mostly schema conversion)
- Supporting nanosecond precision time & timestamp
was:
We currently do not read nor write the version 2 logical types. This is mainly
because we do not have a mapping for it from parquet-format-rs.
To implement this, we can:
- convert "parquet::basic::LogicalType" to "parquet::basic::ConvertedType"
- implement "parquet::basic::LogicalType" which mirrors
"parquet_format::LogicalType"
- create a mapping between ConvertedType and LogicalType
- write LogicalType to "parquet_format::SchemaElement" if v2 of the writer is
used
This would be a good starting point for implementing 2.6 types (UUID, NANOS
precision time & timestamp).
Follow-up work would be:
- parsing v2 of the schema
- Using v2 in the Arrow writer (mostly schema conversion)
- Supporting nanosecond precision time & timestamp
> [Rust] Parquet] Support v2 LogicalType
> --------------------------------------
>
> Key: ARROW-11803
> URL: https://issues.apache.org/jira/browse/ARROW-11803
> Project: Apache Arrow
> Issue Type: Sub-task
> Reporter: Neville Dipale
> Assignee: Neville Dipale
> Priority: Major
>
> We currently do not read nor write the version 2 logical types. This is
> mainly because we do not have a mapping for it from parquet-format-rs.
> To implement this, we can:
> - convert "parquet::basic::LogicalType" to "parquet::basic::ConvertedType"
> - implement "parquet::basic::LogicalType" which mirrors
> "parquet_format::LogicalType"
> - create a mapping between ConvertedType and LogicalType
> - write LogicalType to "parquet_format::SchemaElement" if v2 of the writer is
> used
> This would be a good starting point for implementing 2.6 types (UUID, NANOS
> precision time & timestamp).
> Follow-up work would be:
> - parsing v2 of the schema [ARROW-11365]
> - Using v2 in the Arrow writer (mostly schema conversion)
> - Supporting nanosecond precision time & timestamp
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
