To practically implement inline JSON for extension types, though, would we be
changing (breaking?) all the extension APIs in Arrow implementations to somehow
flag that they use JSON? And in any case, if some Arrow implementation doesn't
handle this, or the extension isn't registered, etc. you could still end up
having a string instead of JSON and would need to handle it right? I don't
think implementations will guarantee that JSON metadata always gets serialized
as JSON.
Hence while it seems convenient I'd like to avoid trying to special-case
extension types and JSON.
I could be convinced that extension types should be lifted to the type and not
the metadata, but only if implementations are required to do this (so no
extension type metadata in the metadata, ever) again because the inconsistency
would otherwise make things annoying (I already run into this with Arrow
implementations where only registered extension types get promoted to actual
types, leading to duplicate logic/checking all over the place).
(FWIW: Java does actually enumerate bitwidth and signedness explicitly; it also
puts child fields on the field and not the type, both choices matching
Flatbuffers exactly. But I think "simple" string types is reasonable for
unparameterized types.)
On Sat, Jul 11, 2026, at 00:41, Dewey Dunnington wrote:
> For at least the "uint32" abbreviation, this is how most if not all
> implementations actually enumerate the type (not integer + signedness +
> bitwidth), so I think that particular change makes the parsers simpler in
> addition to being more compact.
>
> For extension types that use JSON, this requires a very awkward path to
> inspecting the type, perhaps before an Arrow implementation is involved.
> Maybe
>
> "type": {"name": "extension", "extension:name": "geoarrow.wkb",
> "extension:metadata": {...}}
>
> ...would make API responses that contain extension types sufficiently
> compact that a flat list of named key/value objects doesn't matter. With
> the metadata as an object instead of a list and the value field as any JSON
> value I think it's still sufficiently compact to pluck an extension
> parameter. The alternative is sufficiently gross (loop through named key
> and pick the first? or last? and re-parse escaped JSON?) that I am not sure
> I would actually use it on the day I need to return a schema with GeoArrow
> types in a JSON API response (also: this day may never come :))
>
> -dewey
>
> On Wed, Jul 8, 2026 at 9:16 PM David Li <[email protected]> wrote:
>
>> I think there's value, though, in having a representation that is (1)
>> unambiguous and consistent (2) reasonably human-friendly and
>> machine-friendly, even if it's not the most compact. This would be useful
>> for (REST) APIs and ADBC, and I don't think the compactness of the
>> representation is as important there. What kinds of use cases are you
>> envisioning?
>>
>> FWIW, we debated the JSON metadata question - I kind of wanted this too,
>> but it raises too many questions about what happens to the canonical
>> representation/equality/serialization/round-tripping. (For example: we need
>> to stringify the JSON to pass it to the type parser, and a round-trip would
>> possibly present semantically equivalent JSON but different JSON strings.
>> JSON doesn't _disallow_ duplicate keys, so what if we get different
>> behavior? etc.)
>>
>> Ultimately the motivation (for me) is to use this in ADBC where we want to
>> represent Arrow schemas inside Arrow data (where serialized IPC schemas
>> seem to be unpalatable), and so being well-defined is most important (I
>> don't really want to handroll something that is only used in ADBC).
>>
>> -David
>>
>> On Wed, Jul 8, 2026, at 04:36, Dewey Dunnington wrote:
>> > Thank you for drafting!
>> >
>> > I gave a few comments in the community call last Wednesday, but I'll
>> > echo them here for discussion. None of these are blocking concerns,
>> > just thoughts having seen this particular concept be reinvented in a
>> > lot of places.
>> >
>> > While this does a great job of losslessly translating flatbuffer
>> > schemas into JSON and back again, flatbuffer schemas are rather
>> > awkwardly laid out and the human typability/readability is not great.
>> > I think there is an opportunity for this JSON spec to replace a lot of
>> > bespoke data type / field / schema representations, but I worry that
>> > it's a little verbose in its current form to fill that niche.
>> >
>> > Just as an example, a schema with a single uint32 column with this
>> proposal is:
>> >
>> > {"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable": true,
>> > "type": {"name": "int", "bitWidth": 32, "isSigned": false}}]}
>> >
>> > ...whereas with a bit of language (apply defaults for nullability,
>> > optional/independent versioning, special case unparameterized types)
>> > this could be more like:
>> >
>> > [{"name": "col0", "type": "uint32"}]
>> >
>> > ...drawing from BigQuery [1] here), although that does open up a
>> > significantly more surface area for bikeshedding on which strings map
>> > to which type (unless we want to reuse the C data interface ones,
>> > which from the proposal it seems like maybe we don't).
>> >
>> > Another opportunity to reduce verbosity is metadata, which for an
>> > extension type with JSON metadata would be
>> >
>> > [{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key":
>> > "ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}]
>> >
>> > ...and could be more readable:
>> >
>> > {"ARROW:extension:name": "geoarrow.wkb", "ARROW:extension:metadata":
>> > {"crs": "OGC:CRS84"}}
>> >
>> > Basically, if the goal is human readability, I think it's worth the
>> > effort to do the bikeshedding on how to represent these concepts that
>> > mirror how they are printed/enumerated in implementations (not
>> > necessarily the .fbs files). If the goal is machine readability within
>> > existing arrow implementations, perfectly mirroring flatbuffers JSON
>> > or the C data interface would be less work (more opportunity for
>> > reusing existing functions).
>> >
>> > Cheers,
>> >
>> > -dewey
>> >
>> > [1]
>> >
>> https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_
>> >
>> > On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <[email protected]> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I'd like to raise a topic for discussion that has surfaced a few
>> >> separate times over the years in the Arrow community, which is the
>> >> lack of a canonical human-readable representation of Arrow schemas.
>> >>
>> >> Arrow schemas today only canonically serialize as IPC binary, which is
>> >> a friction point for application-level tasks where binary blobs are
>> >> not ergonomic, such as JSON API contracts, hand authoring or reading,
>> >> and persistence use cases, to name a few.
>> >>
>> >> The motivating pain point right now is at the ADBC metadata boundary.
>> >> The 1.2 milestone includes proposals for new metadata APIs, and
>> >> several contributors have noted that returning Arrow schemas via IPC
>> >> is unsatisfactory for this use case.
>> >>
>> >> ADBC 1.2 Milestone
>> >> - https://github.com/apache/arrow-adbc/milestone/9
>> >>
>> >> Discussion within various issues:
>> >> - https://github.com/apache/arrow-adbc/issues/4400
>> >> - https://github.com/apache/arrow-adbc/issues/1514
>> >> - https://github.com/apache/arrow-adbc/issues/1704
>> >> - https://github.com/apache/arrow-adbc/pull/4031
>> >>
>> >> As a starting point, I've put together a proposal for how a JSON
>> >> representation might be structured.
>> >>
>> >> Rather than dump the whole thing here, I've put it in this google doc
>> >> which is open for comments:
>> >>
>> >>
>> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
>> >>
>> >> It's early and open to change, so I'd welcome feedback of any kind.
>> >>
>> >> Looking forward to hearing your thoughts
>>
