[
https://issues.apache.org/jira/browse/ARROW-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17544801#comment-17544801
]
Joris Van den Bossche commented on ARROW-16688:
-----------------------------------------------
bq. The interaction that breaks GDAL is that the type is stripped if the
extension type is not registered (the most common case). We can PR to fix this
in GDAL by checking the storage type if it is an extension array or by
registering the extension type there. I could also stop writing files with
extension type columns (although then we don't get query engine magic without a
custom dataset class).
For this specific aspect (which is not directly related to registering the same
extension type from two bindings), I opened an issue upstream:
https://github.com/OSGeo/gdal/issues/5834
> [R][Python] Extension types cannot be registered in both R and Python
> ---------------------------------------------------------------------
>
> Key: ARROW-16688
> URL: https://issues.apache.org/jira/browse/ARROW-16688
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python, R
> Reporter: Dewey Dunnington
> Priority: Major
>
> When registering extension types as is now possible in the R bindings, it
> looks as though we cannot register an extension type in R and Python at the
> same time:
> {code:R}
> # apache/arrow@master
> library(arrow, warn.conflicts = FALSE)
> library(reticulate)
> # this is a virtualenv with pyarrow installed against the same commit
> use_virtualenv(
> "/Users/deweydunnington/Desktop/rscratch/pyarrow-dev",
> required = TRUE
> )
> pa <- import("pyarrow")
> pa[["__version__"]]
> #> [1] "9.0.0.dev131+g8a36f0f6c"
> py_run_string("
> import pyarrow as pa
> class TestExtensionType(pa.ExtensionType):
>
> def __init__(self):
> super().__init__(pa.int32(), 'arrow.test_type')
>
> def __arrow_ext_serialize__(self):
> return b''
> @classmethod
> def __arrow_ext_deserialize__(cls, storage_type, serialized):
> return cls()
> pa.register_extension_type(TestExtensionType())
> ")
> arrow::register_extension_type(
> arrow::new_extension_type(int32(), "arrow.test_type")
> )
> #> Error: Key error: A type extension with name arrow.test_type already
> defined
> {code}
> I also get a segfault if I try to surface a Python type into R (probably
> because the R bindings mistakenly assume that if {{type.id() ==
> Type::EXTENSION}} then it is safe to cast to our own {{ExtensionType}} C++
> subclass that implements R-specific things.
> This came about because the 'geoarrow' Python and 'geoarrow' R packages both
> register a number of extension type definitions.
> - geoarrow's Python registration:
> https://github.com/jorisvandenbossche/python-geoarrow/blob/main/src/geoarrow/extension_types.py#L108-L117
> - geoarrow's R registration:
> https://github.com/paleolimbot/geoarrow/blob/master/R/pkg-arrow.R#L208-L223
> I can also force an interaction if I build GDAL against the same Arrow that
> the arrow R package is linked against and attempt to load a Feather file
> saved with an extension type using the sf package. I will attempt to recreate
> that interaction as well in both R and Python.
> I don't know enough about linking to know to what extent this is linked to my
> own development setup/build of the R package, although I think there are at
> least some environments where a shared library is picked up first by the R
> config script (fedora36, for example). It does look like my own R package
> build is dynamically linking to libarrow.dylib.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
