[
https://issues.apache.org/jira/browse/IMPALA-14593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer updated IMPALA-14593:
-------------------------------------
Description:
Add support for Iceberg/Parquet GEOMETRY type. This are WKB byte arrays, so
this change involves switching to WKB as internal serialization format from the
current one used by Hive ESRI functions.
The old mode is not removed at this point, but allow switching with flag
geospatail_libary (current: ESRI_HIVE new: WKB_EXPERIMENTAL).
The main goal of keeping the old mode is to be able to help testing by checking
if both modes return the same result.
The plan is to do this in several steps:
1. allow Java functions to use WKB and switch to this with WKB_EXPERIMENTAL
(only java, even where c++ implementation exists for ESRI_HIVE mode)
2. and GEOMETRY type that is used in WKB_EXPERIMENTAL mode, and switch Java
functions to use it instead of BINARY
3. add read/write support for Iceberg/Parquet GEOMETRY columns (no bounding
rect statistics yet)
4. add bounding rect support for Parquet (read/write) and Iceberg (only write
due to gap in Icebeg lib)
5. implement critical functions in c++ - until this step the performance in
worse in WKB_EXPERIMENTAL mode due to having 0 c++ functions and the extra
conversion in Java)
The c++ changes are based on https://gerrit.cloudera.org/#/c/20602/ , but use
WKB.
4 needs adding basic c++ support for WKB parsing, otherwise would need to call
up to Java per row during writing, so some parts of c++ geometry handling would
be implemented earlier
was:
Add support for Iceberg/Parquet GEOMETRY type. This are WKB byte arrays, so
this change involves switching to WKB as internal serialization format from the
current one used by Hive ESRI functions.
The old mode is not removed at this point, but allow switching with flag
geospatail_libary (current: ESRI_HIVE new: WKB_EXPERIMENTAL).
The main goal of keeping the old mode is to be able to help testing by checking
if both modes return the same result.
The plan is to do this in several steps:
1. allow Java functions to use WKB and switch to this with WKB_EXPERIMENTAL
(only java, even where c++ implementation exists for ESRI_HIVE mode)
2. and GEOMETRY type that is used in WKB_EXPERIMENTAL mode, and switch Java
functions to use it instead of BINARY
3. add read/write support for Iceberg/Parquet GEOMETRY columns (no bounding
rect statistics yet)
4. add bounding rect support for Parquet (read/write) and Iceberg (only write
due to gap in Icebeg lib)
- this needs adding basic c++ support for WKB parsing, otherwise would need
to call up to Java per row during writing
5. implement critical functions in c++ - until this step the performance in
worse in WKB_EXPERIMENTAL mode due to having 0 c++ functions and the extra
conversion in Java)
The c++ changes are based on https://gerrit.cloudera.org/#/c/20602/ , but use
WKB.
> Iceberg V3 Geometry support
> ---------------------------
>
> Key: IMPALA-14593
> URL: https://issues.apache.org/jira/browse/IMPALA-14593
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Csaba Ringhofer
> Priority: Major
> Labels: impala-iceberg
>
> Add support for Iceberg/Parquet GEOMETRY type. This are WKB byte arrays, so
> this change involves switching to WKB as internal serialization format from
> the current one used by Hive ESRI functions.
> The old mode is not removed at this point, but allow switching with flag
> geospatail_libary (current: ESRI_HIVE new: WKB_EXPERIMENTAL).
> The main goal of keeping the old mode is to be able to help testing by
> checking if both modes return the same result.
> The plan is to do this in several steps:
> 1. allow Java functions to use WKB and switch to this with WKB_EXPERIMENTAL
> (only java, even where c++ implementation exists for ESRI_HIVE mode)
> 2. and GEOMETRY type that is used in WKB_EXPERIMENTAL mode, and switch Java
> functions to use it instead of BINARY
> 3. add read/write support for Iceberg/Parquet GEOMETRY columns (no bounding
> rect statistics yet)
> 4. add bounding rect support for Parquet (read/write) and Iceberg (only write
> due to gap in Icebeg lib)
> 5. implement critical functions in c++ - until this step the performance in
> worse in WKB_EXPERIMENTAL mode due to having 0 c++ functions and the extra
> conversion in Java)
> The c++ changes are based on https://gerrit.cloudera.org/#/c/20602/ , but use
> WKB.
> 4 needs adding basic c++ support for WKB parsing, otherwise would need to
> call up to Java per row during writing, so some parts of c++ geometry
> handling would be implemented earlier
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
