Hi all, I'd like to raise a topic for discussion that has surfaced a few separate times over the years in the Arrow community, which is the lack of a canonical human-readable representation of Arrow schemas.
Arrow schemas today only canonically serialize as IPC binary, which is a friction point for application-level tasks where binary blobs are not ergonomic, such as JSON API contracts, hand authoring or reading, and persistence use cases, to name a few. The motivating pain point right now is at the ADBC metadata boundary. The 1.2 milestone includes proposals for new metadata APIs, and several contributors have noted that returning Arrow schemas via IPC is unsatisfactory for this use case. ADBC 1.2 Milestone - https://github.com/apache/arrow-adbc/milestone/9 Discussion within various issues: - https://github.com/apache/arrow-adbc/issues/4400 - https://github.com/apache/arrow-adbc/issues/1514 - https://github.com/apache/arrow-adbc/issues/1704 - https://github.com/apache/arrow-adbc/pull/4031 As a starting point, I've put together a proposal for how a JSON representation might be structured. Rather than dump the whole thing here, I've put it in this google doc which is open for comments: https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing It's early and open to change, so I'd welcome feedback of any kind. Looking forward to hearing your thoughts
