Hello,
I have roughly the same position as Weston here.
Regards
Antoine.
Le 09/11/2026 à 06:04, Weston Pace a écrit :
Are you saying there should be super-project canonical extension types? :)
No. I am suggesting the Parquet PMC publish an extension-types.md document
in [1] that describes what the extension type definitions are. This would
be exactly like [2]. This lets the Parquet team bless whatever
unambiguous definition they like. Anyone wanting to consume custom Parquet
types must use those definitions. The Parquet team is then in complete
control and doesn't need to seek the Arrow team's blessing. The Arrow team
can then create a link back to that document in [3] in much the same way we
do for GeoArrow.
For example, the Substrait type system does not name all Arrow types. in
Arrow, we publish [4] which describes Substrait extension types to map to
the existing Arrow types.
Or, as another example, Lance's blob type is an Arrow extension type
defined in [5]. Consumers of Lance custom types consult the Lance spec for
those types' definitions.
Similarily, I was thinking of adding more canonical Parquet extension
types like GEOMETRY and GEOGRAPHY because I found it really tricky
when implementing Iceberg V3 types in iceberg-cpp where GeoArrow type
were not an option.
If Iceberg has its own set of types (separate from Parquet and GeoArrow)
then Iceberg can also maintain an extension-types.md document right?
[1] https://github.com/apache/parquet-format/
[2] https://github.com/geoarrow/geoarrow/blob/main/extension-types.md
[3]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#community-extension-types
[4]
https://github.com/apache/arrow/blob/main/format/substrait/extension_types.yaml
[5]
https://github.com/lance-format/lance/blob/main/docs/src/format/table/blob.md#table-schema
On Sat, Nov 7, 2026 at 7:00 AM Rok Mihevc <[email protected]> wrote:
Thanks for the discussion Weston and Gang!
I also somewhat disagree with this characterization. As long as the
producer and consumer agree on the type then there is no need for every
middleman (or even Arrow itself) to be aware of it. That's the entire
goal
of extension types. IPC streams, the C data interface, Arrow flight, and
language bindings should all respect existing metadata and should _not_
strip it out, even if they don't recognize it.
This is a great point and it's ideally how things should work. Now how do
we get there? Should Parquet readers that output Arrow map Parquet logical
types into "parquet.<logical_type>" extension types? If this requires
non-trivial logic, to make the mapping unambiguous, are we implicitly
specifying extension types at the conversion layer?
and so I will turn my vote into a +1 if needed for passage. Otherwise
I'll
leave it as a half vote to discourage Arrow taking on the mantle of
blessing another format's types.
Are you saying there should be super-project canonical extension types? :)
Best,
Rok
On Sat, Nov 7, 2026 at 2:57 AM Gang Wu <[email protected]> wrote:
Thanks Weston for raising this! I have a different opinion.
Canonical extension type for Parquet types like VARIANT、FILE and future
VECTOR brings better interoperability to different Parquet bindings in
Arrow. AFAIK, it would be a headache if different bindings define their
own custom extensions then we end up with different interpretations of
ARROW:schema serialization across bindings.
Similarily, I was thinking of adding more canonical Parquet extension
types like GEOMETRY and GEOGRAPHY because I found it really tricky
when implementing Iceberg V3 types in iceberg-cpp where GeoArrow type
were not an option.
On Sat, Nov 7, 2026 at 6:38 AM Weston Pace <[email protected]>
wrote:
+0.5
I guess I'm a bit late to the party since we already have
arrow.parquet.variant but it feels like parquet-specific types would
just
fit the definition for community extension types[1]. In other words,
Parquet is free to define whatever extension types they want and there
is
no need for Arrow PMC approval in the same way that GeoArrow is free to
define whatever extension types they want.
To ensure full interoperability, Arrow requires a matching canonical
in-memory extension type so that file fields do not lose their
semantic metadata during serialization across IPC streams, the C
Data Interface, Arrow Flight, and language bindings.
I also somewhat disagree with this characterization. As long as the
producer and consumer agree on the type then there is no need for every
middleman (or even Arrow itself) to be aware of it. That's the entire
goal
of extension types. IPC streams, the C data interface, Arrow flight,
and
language bindings should all respect existing metadata and should _not_
strip it out, even if they don't recognize it.
That being said, I am happy for someone (even if it is Arrow and not
Parquet) to bless a canonical extension type for any Parquet-specific
types
and so I will turn my vote into a +1 if needed for passage. Otherwise
I'll
leave it as a half vote to discourage Arrow taking on the mantle of
blessing another format's types.
[1]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#community-extension-types
On Fri, Nov 6, 2026 at 11:36 AM Mandukhai Alimaa <
[email protected]>
wrote:
Hi everyone,
The FILE column type is being adopted across Parquet, Delta Lake,
Apache Iceberg, and Apache Spark so the broader ecosystem can
natively
leverage structured file-level references. To ensure full
interoperability, Arrow requires a matching canonical in-memory
extension type so that file fields do not lose their semantic
metadata
during serialization across IPC streams, the C Data Interface, Arrow
Flight, and language bindings.
Following up on the discussion thread [1] and the subsequent format
change PR [2], we (@rok and I) would like to propose a formal vote to
add arrow.parquet.file as a canonical extension type to the Apache
Arrow format specification. This proposal already had reviews from
four different committers on GitHub and closely matches the Parquet
file logical type.
[1]
https://lists.apache.org/thread.html/5qw2k7rwstk0ol9ymds5qym81myznmpl
[2] https://github.com/apache/arrow/pull/51248
[3] GitHub Issue: https://github.com/apache/arrow/issues/51143
The vote will be open for at least 72 hours.
Please cast your vote:
[ ] +1 Accept this proposal
[ ] +0
[ ] -1 Do not accept this proposal because...
Thanks
