This is an automated email from the ASF dual-hosted git repository.
alenka pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new fa6a003885 GH-41673: [Format][Docs] Add arrow format introductory page
(#41593)
fa6a003885 is described below
commit fa6a0038851ec4a69d2c8f568acc74130c076a14
Author: Alenka Frim <[email protected]>
AuthorDate: Tue Oct 1 06:31:16 2024 +0200
GH-41673: [Format][Docs] Add arrow format introductory page (#41593)
### Rationale for this change
The documentation for Arrow Format could be improved:
- all types are not listed
- all layouts are not explained
### What changes are included in this PR?
This PR includes:
- motivation behind the columnar format
- different physical layouts explained together with diagrams of example
type in comparison to the physical layout
- Arrow terminology
- Extension types and sharing of Arrow data
in a separate "introduction" page with no technical details. Specifications
index page is also restructured to include captions and make the left sidebar
menu better organised.
**Note:** a table with all types listed together with their physical layout
will be added in a separate PR to existing _Columnar.rst_ page:
https://github.com/apache/arrow/issues/14752
### Are these changes tested?
No, this is a docs change.
### Are there any user-facing changes?
No.
* GitHub Issue: #41673
Lead-authored-by: AlenkaF <[email protected]>
Co-authored-by: Alenka Frim <[email protected]>
Co-authored-by: Alenka Frim <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Felipe Oliveira Carvalho <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Bryce Mecum <[email protected]>
Signed-off-by: AlenkaF <[email protected]>
---
docs/source/format/Columnar.rst | 2 +
docs/source/format/Glossary.rst | 2 +
docs/source/format/Intro.rst | 512 +++++++++++++++++++++
docs/source/format/images/all-diagrams.svg | 17 +
docs/source/format/images/bool-diagram.svg | 10 +
docs/source/format/images/columnar-diagram_1.svg | 21 +
docs/source/format/images/columnar-diagram_2.svg | 21 +
docs/source/format/images/columnar-diagram_3.svg | 21 +
docs/source/format/images/dense-union-diagram.svg | 21 +
docs/source/format/images/dictionary-diagram.svg | 21 +
docs/source/format/images/fixed-list-diagram.svg | 21 +
docs/source/format/images/fixed-string-diagram.svg | 21 +
docs/source/format/images/map-diagram.svg | 21 +
docs/source/format/images/primitive-diagram.svg | 21 +
docs/source/format/images/ree-diagram.svg | 13 +
docs/source/format/images/sparse-union-diagram.svg | 21 +
docs/source/format/images/struct-diagram.svg | 21 +
docs/source/format/images/var-list-diagram.svg | 21 +
.../source/format/images/var-list-view-diagram.svg | 21 +
docs/source/format/images/var-string-diagram.svg | 21 +
.../format/images/var-string-view-diagram.svg | 21 +
docs/source/format/index.rst | 5 +-
22 files changed, 874 insertions(+), 2 deletions(-)
diff --git a/docs/source/format/Columnar.rst b/docs/source/format/Columnar.rst
index b144f1cc98..33c937ea34 100644
--- a/docs/source/format/Columnar.rst
+++ b/docs/source/format/Columnar.rst
@@ -89,6 +89,8 @@ concepts, here is a small glossary to help disambiguate.
are parametric by construction. A timestamp is also parametric as it needs
a unit (such as microseconds) and a timezone.
+.. _data_types:
+
Data Types
==========
diff --git a/docs/source/format/Glossary.rst b/docs/source/format/Glossary.rst
index 11c19c5fa7..7c6a23b8ce 100644
--- a/docs/source/format/Glossary.rst
+++ b/docs/source/format/Glossary.rst
@@ -15,6 +15,8 @@
.. specific language governing permissions and limitations
.. under the License.
+.. _glossary:
+
========
Glossary
========
diff --git a/docs/source/format/Intro.rst b/docs/source/format/Intro.rst
new file mode 100644
index 0000000000..c230be724d
--- /dev/null
+++ b/docs/source/format/Intro.rst
@@ -0,0 +1,512 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+************
+Introduction
+************
+
+Apache Arrow was born from the need for a set of standards around
+tabular data representation and interchange between systems.
+The adoption of these standards reduces computing costs of data
+serialization/deserialization and implementation costs across
+systems implemented in different programming languages.
+
+The Apache Arrow specification can be implemented in any programming
+language but official implementations for many languages are available.
+An implementation consists of format definitions using the constructs
+offered by the language and common in-memory data processing algorithms
+(e.g. slicing and concatenating). Users can extend and use the utilities
+provided by the Apache Arrow implementation in their programming
+language of choice. Some implementations are further ahead and feature a
+vast set of algorithms for in-memory analytical data processing. More detail
+about how implementations differ can be found on the :ref:`status` page.
+
+Apart from this initial vision, Arrow has grown to also develop a
+multi-language collection of libraries for solving problems related to
+in-memory analytical data processing. This covers topics like:
+
+* Zero-copy shared memory and RPC-based data movement
+* Reading and writing file formats (like CSV, `Apache ORC`_, and `Apache
Parquet`_)
+* In-memory analytics and query processing
+
+.. _Apache ORC: https://orc.apache.org/
+.. _Apache Parquet: https://parquet.apache.org/
+
+Arrow Columnar Format
+=====================
+
+Apache Arrow focuses on tabular data. For an example, let's consider
+we have data that can be organized into a table:
+
+.. figure:: ./images/columnar-diagram_1.svg
+ :scale: 70%
+ :alt: Diagram with tabular data of 4 rows and columns.
+
+ Diagram of a tabular data structure.
+
+Tabular data can be represented in memory using a row-based format or a
+column-based format. The row-based format stores data row-by-row, meaning the
rows
+are adjacent in the computer memory:
+
+.. figure:: ./images/columnar-diagram_2.svg
+ :alt: Tabular data being structured row by row in computer memory.
+
+ Tabular data being saved in memory row by row.
+
+In a columnar format, the data is organized column-by-column instead.
+This organization makes analytical operations like filtering, grouping,
+aggregations and others, more efficient thanks to memory locality.
+When processing the data, the memory locations accessed by the CPU tend
+be near one another. By keeping the data contiguous in memory, it also
+enables vectorization of the computations. Most modern CPUs have
+`SIMD instructions`_ (a single instruction that operates on multiple values at
+once) enabling parallel processing and execution of operations on vector data
+using a single CPU instruction.
+
+.. _SIMD instructions:
https://en.wikipedia.org/wiki/Single_instruction,_multiple_data
+
+Apache Arrow is solving this exact problem. It is the specification that
+uses the columnar layout.
+
+.. figure:: ./images/columnar-diagram_3.svg
+ :alt: Tabular data being structured column by column in computer memory.
+
+ The same tabular data being saved in memory column by column.
+
+Each column is called an **Array** in Arrow terminology. Arrays can be of
+different data types and the way their values are stored in memory varies among
+the data types. The specification of how these values are arranged in memory is
+what we call a **physical memory layout**. One contiguous region of memory that
+stores data for arrays is called a **Buffer**. An array consists of one or more
+buffers.
+
+Next sections give an introduction to Arrow Columnar Format explaining the
+different physical layouts. The full specification of the format can be found
+at :ref:`format_columnar`.
+
+Support for Null Values
+=======================
+
+Arrow supports missing values or "nulls" for all data types: any value
+in an array may be semantically null, whether primitive or nested data type.
+
+In Arrow, a dedicated buffer, known as the validity (or "null") bitmap,
+is used alongside the data indicating whether each value in the array is
+null or not: a value of 1 means that the value is not-null ("valid"), whereas
+a value of 0 indicates that the value is null.
+
+This validity bitmap is optional: if there are no missing values in
+the array the buffer does not need to be allocated (as in the example
+column 1 in the diagram below).
+
+.. note::
+
+ We read validity bitmaps right-to-left within a group of 8 bits due to
+ `least-significant bit numbering
<https://en.wikipedia.org/wiki/Bit_numbering>`_
+ being used.
+
+ This is also how we have represented the validity bitmaps in the
+ diagrams included in this document.
+
+Primitive Layouts
+=================
+
+Fixed Size Primitive Layout
+---------------------------
+
+A primitive column represents an array of values where each value
+has the same physical size measured in bytes. Data types that use the
+fixed size primitive layout are, for example, signed and unsigned
+integer data types, floating point numbers, boolean, decimal and temporal
+data types.
+
+.. figure:: ./images/primitive-diagram.svg
+ :alt: Diagram is showing the difference between the primitive data
+ type presented in a Table and the data actually stored in
+ computer memory.
+
+ Physical layout diagram for primitive data types.
+
+.. note::
+ The boolean data type is represented with a primitive layout where the
+ values are encoded in bits instead of bytes. That means the physical
+ layout includes a values bitmap buffer and possibly a validity bitmap
+ buffer.
+
+ .. figure:: ./images/bool-diagram.svg
+ :alt: Diagram is showing the difference between the boolean data
+ type presented in a Table and the data actually stored in
+ computer memory.
+
+ Physical layout diagram for boolean data type.
+
+.. note::
+ Arrow also has a concept of Null data type where all values are null. In
+ this case no buffers are allocated.
+
+Variable length binary and string
+---------------------------------
+
+In contrast to the fixed size primitive layout, the variable length layout
+allows representing an array where each element can have a variable size
+in bytes. This layout is used for binary and string data.
+
+The bytes of all elements in a binary or string column are stored together
+consecutively in a single buffer or region of memory. To know where each
element
+of the column starts and ends, the physical layout also includes integer
offsets.
+The offsets buffer is always one element longer than the array.
+The last two offsets define the start and the end of the last
+binary/string element.
+
+Binary and string data types share the same physical layout. The only
+difference between them is that a string-typed array is assumed to contain
+valid UTF-8 string data.
+
+The difference between binary/string and large binary/string is in the offset
+data type. In the first case that is int32 and in the second it is int64.
+
+The limitation of data types using 32 bit offsets is that they have a maximum
size of
+2GB per array. One can still use the non-large variants for bigger data, but
+then multiple chunks are needed.
+
+.. figure:: ./images/var-string-diagram.svg
+ :alt: Diagram is showing the difference between the variable length
+ string data type presented in a Table and the data actually
+ stored in computer memory.
+
+ Physical layout diagram for variable length string data types.
+
+Variable length binary and string view
+--------------------------------------
+
+.. _UmbraDB: https://umbra-db.com/
+.. _DuckDB: https://duckdb.com
+.. _Velox: https://velox-lib.io/
+
+This layout is an alternative for the variable length binary layout and is
adapted
+from TU Munich's `UmbraDB`_ and is similar to the string layout used in
`DuckDB`_ and
+`Velox`_ (and sometimes also called "German strings").
+
+The main difference to the classical binary and string layout is the views
buffer.
+It includes the length of the string, and then either its characters appearing
+inline (for small strings) or only the first 4 bytes of the string and an
offset into
+one of the potentially several data buffers. Because it uses an offset and
length to refer
+to the data buffer, the bytes of all elements do not need to be stored
+consecutively in a single buffer. This enables out of order writing of
+variable length elements into the array.
+
+These properties are important for efficient string processing. The prefix
+enables a profitable fast path for string comparisons, which are frequently
+determined within the first four bytes. Selecting elements is a simple "gather"
+operation on the fixed-width views buffer and does not need to rewrite the
+values buffers.
+
+.. figure:: ./images/var-string-view-diagram.svg
+ :alt: Diagram is showing the difference between the variable length
+ string view data type presented in a Table and the data actually
+ stored in computer memory.
+
+ Physical layout diagram for variable length string view data type.
+
+Nested Layouts
+==============
+
+Nested data types introduce the concept of parent and child arrays. They
express
+relationships between physical value arrays in a nested data type structure.
+
+Nested data types depend on one or more other child data types. For instance,
List
+is a nested data type (parent) that has one child (the data type of the values
in
+the list).
+
+List
+----
+
+The list data type enables representing an array where each element is a
sequence
+of elements of the same data type. The layout is similar to variable-size
binary
+or string layout as it has an offsets buffer to define where the sequence of
values
+for each element starts and ends, with all the values being stored
consecutively
+in a values child array.
+
+The offsets in the list data type are int32 while in the large list the offsets
+are int64.
+
+.. figure:: ./images/var-list-diagram.svg
+ :alt: Diagram is showing the difference between the variable size
+ list data type presented in a Table and the data actually
+ stored in computer memory.
+
+ Physical layout diagram for variable size list data type.
+
+Fixed Size List
+---------------
+
+Fixed-size list is a special case of variable-size list where each column slot
+contains a fixed size sequence meaning all lists are the same size and so the
+offset buffer is no longer needed.
+
+.. figure:: ./images/fixed-list-diagram.svg
+ :alt: Diagram is showing the difference between the fixed size list data
+ type presented in a Table and the data actually stored in computer
+ memory.
+
+ Physical layout diagram for fixed size list data type.
+
+List View
+---------
+
+In contrast to the list type, list view type also has a size buffer together
+with an offset buffer. The offsets continue to indicate the start of each
+element but size is now saved in a separate size buffer. This allows
+out-of-order offsets as the sizes aren't derived from the consecutive
+offsets anymore.
+
+.. figure:: ./images/var-list-view-diagram.svg
+ :alt: Diagram is showing the difference between the variable size list view
+ data type presented in a Table and the data actually stored in
+ computer memory.
+
+ Physical layout diagram for variable size list view data type.
+
+Struct
+------
+
+A struct is a nested data type parameterized by an ordered sequence of fields
+(a data type and a name).
+
+* There is one child array for each field
+* Child arrays are independent and need not be adjacent to each other in
+ memory. They only need to have the same length.
+
+One can think of an individual struct field as a key-value pair where the
+key is the field name and the child array its values. The field (key) is
+saved in the schema and the values of a specific field (key) are saved in
+the child array.
+
+.. figure:: ./images/struct-diagram.svg
+ :alt: Diagram is showing the difference between the struct data type
+ presented in a Table and the data actually stored in computer
+ memory.
+
+ Physical layout diagram for struct data type.
+
+Map
+---
+
+The Map data type represents nested data where each value is a variable number
of
+key-value pairs. Its physical representation is the same as a list of ``{key,
value}``
+structs.
+
+The difference between the struct and map data types is that a struct holds
the key
+in the schema, requiring keys to be strings, and the values are stored in the
+child arrays,
+one for each field. There can be multiple keys and therefore multiple child
arrays.
+The map, on the other hand, has one child array holding all the different keys
(that
+thus all need to be of the same data type, but not necessarily strings) and a
second
+child array holding all the values. The values need to be of the same data
type; however,
+the data type doesn't have to match that of the keys.
+
+Also, the map stores the struct in a list and needs an offset as the list is
+variable shape.
+
+.. figure:: ./images/map-diagram.svg
+ :alt: Diagram is showing the difference between the map data type
+ presented in a Table and the data actually stored in computer
+ memory.
+
+ Physical layout diagram for map data type.
+
+Union
+-----
+
+The union is a nested data type where each slot in the union has a value with
a data type
+chosen from a subset of possible Arrow data types. That means that a union
array represents
+a mixed-type array. Unlike other data types, unions do not have their own
validity bitmap
+and the nullness is determined by the child arrays.
+
+Arrow defines two distinct union data types, "dense" and "sparse".
+
+Dense Union
+^^^^^^^^^^^
+
+A Dense Union has one child array for each data type present in the mixed-type
array and
+two buffers of its own:
+
+* **Types buffer:** holds data type id for each slot of the array. Data type
id is
+ frequently the index of the child array; however, the relationship between
data type
+ ID and the child index is a parameter of the data type.
+* **Offsets buffer:** holds relative offset into the respective child array
for each
+ array slot.
+
+.. figure:: ./images/dense-union-diagram.svg
+ :alt: Diagram is showing the difference between the dense union data type
+ presented in a Table and the data actually stored in computer
+ memory.
+
+ Physical layout diagram for dense union data type.
+
+Sparse union
+^^^^^^^^^^^^
+
+A sparse union has the same structure as a dense union, with the omission of
the offsets
+buffer. In this case, the child arrays are each equal in length to the length
of the union.
+
+
+.. figure:: ./images/sparse-union-diagram.svg
+ :alt: Diagram is showing the difference between the sparse union data type
+ presented in a Table and the data actually stored in computer
+ memory.
+
+ Physical layout diagram for sparse union data type.
+
+Dictionary Encoded Layout
+=========================
+
+Dictionary encoding can be effective when one has data with many repeated
values.
+The values are represented by integers referencing a dictionary usually
consisting of
+unique values.
+
+.. figure:: ./images/dictionary-diagram.svg
+ :alt: Diagram is showing the difference between the dictionary data type
+ presented in a Table and the data actually stored in computer
+ memory.
+
+ Physical layout diagram for dictionary data type.
+
+Run-End Encoded Layout
+======================
+
+Run-end encoding is well-suited for representing data containing sequences of
the
+same value. These sequences are called runs. A run-end encoded array has no
buffers
+of its own, but has two child arrays:
+
+* **Run ends array:** holds the index in the array where each run ends. The
number
+ of run ends is the same as the length of its parent array.
+* **Values array:** the actual values without repetitions (together with null
values).
+
+Note that nulls of the parent array are strictly represented in the values
array.
+
+.. figure:: ./images/ree-diagram.svg
+ :alt: Diagram is showing the difference between the run-end encoded data
+ type presented in a Table and the data actually stored in computer
+ memory.
+
+ Physical layout diagram for run-end encoded data type.
+
+.. seealso::
+ Table of all Arrow :ref:`data_types`.
+
+Overview of Arrow Terminology
+=============================
+
+**Physical layout**
+A specification for how to represent values of an array in memory.
+
+**Buffer**
+A contiguous region of memory with a given length in bytes. Buffers are used
to store data
+for arrays. Sometimes we use the notion of number of elements in a buffer
which can only be
+used if we know the data type of the array that wraps this specific buffer.
+
+**Array**
+A contiguous, one-dimensional sequence of values with known length where all
values have the
+same data type. An array consists of zero or more buffers.
+
+**Chunked Array**
+A discontiguous, one-dimensional sequence of values with known length where
all values have
+the same data type. Consists of zero or more arrays, the “chunks”.
+
+.. note::
+ Chunked Array is a concept specific to certain implementations such as
Arrow C++ and PyArrow.
+
+**RecordBatch**
+A contiguous, two-dimensional data structure which consists of an ordered
collection of arrays
+of the same length.
+
+**Schema**
+An ordered collection of fields that communicates all the data types of an
object
+like a RecordBatch or Table. Schemas can contain optional key/value metadata.
+
+A Field includes a field name, a data type, a nullability flag and optional
key-value metadata
+for a specific column in a RecordBatch.
+
+**Table**
+A discontiguous, two-dimensional chunk of data consisting of an ordered
collection of Chunked
+Arrays. All Chunked Arrays have the same length, but may have different types.
Different columns
+may be chunked differently.
+
+.. note::
+ Table is a concept specific to certain implementations such as Arrow C++
and PyArrow. In Java
+ implementation, for example, a Table is not a collection of Chunked Arrays
but a collection of
+ RecordBatches.
+
+.. image:: ../cpp/tables-versus-record-batches.svg
+ :alt: A graphical representation of an Arrow Table and a
+ Record Batch, with structure as described in text above.
+
+.. seealso::
+ The :ref:`glossary` for more terms.
+
+Extension Types
+===============
+
+In case the system or application needs to extend standard Arrow data types
with
+custom semantics, this is enabled by defining extension types.
+
+Examples of an extension type are :ref:`uuid_extension` or
+:ref:`fixed_shape_tensor_extension` extension type.
+
+Extension types can be defined by annotating any of the built-in Arrow data
types
+(the “storage type”) with a custom type name and optional serialized
representation
+(``'ARROW:extension:name'`` and ``'ARROW:extension:metadata'`` keys in the
Field
+metadata structure).
+
+.. seealso::
+ The :ref:`format_metadata_extension_types` documentation.
+
+Canonical Extension Types
+-------------------------
+
+It is beneficial to share the definitions of well-known extension types so as
to
+improve interoperability between different systems integrating Arrow columnar
data.
+For this reason canonical extension types are defined in Arrow itself.
+
+.. seealso::
+ The :ref:`format_canonical_extensions` documentation.
+
+Community Extension Types
+-------------------------
+These are Arrow extension types that have been established as standards within
specific
+domain areas.
+
+Example:
+
+* `GeoArrow`_: A collection of Arrow extension types for representing vector
geometries
+
+.. _GeoArrow: https://geoarrow.org
+
+Sharing Arrow data
+==================
+
+Arrow memory layout is meant to be a universal standard for representing
tabular data in memory,
+not tied to a specific implementation. The Arrow standard defines two
protocols for
+well-defined and unambiguous communication of Arrow data between applications:
+
+* Protocol to share Arrow data between processes or over the network is called
:ref:`format-ipc`.
+ The specification for sharing data is called IPC message format which
defines how Arrow
+ array or record batch buffers are stacked together to be serialized and
deserialized.
+
+* To share Arrow data in the same process :ref:`c-data-interface` is used,
meant for sharing
+ the same buffer zero-copy in memory between different libraries within the
same process.
diff --git a/docs/source/format/images/all-diagrams.svg
b/docs/source/format/images/all-diagrams.svg
new file mode 100644
index 0000000000..0238fb84bf
--- /dev/null
+++ b/docs/source/format/images/all-diagrams.svg
@@ -0,0 +1,17 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
3927.9427907255213 7991.21493509393" width="7855.8855814510425"
height="15982.42987018786">
+ <!-- svg-source:excalidraw -->
+ <!-- payload-type:application/vnd.excalidraw+json --><!-- payload-version:2
--><!-- payload-start
-->eyJ2ZXJzaW9uIjoiMSIsImVuY29kaW5nIjoiYnN0cmluZyIsImNvbXByZXNzZWQiOnRydWUsImVuY29kZWQiOiJ4nOx9WVdcIlmU7nv9XG5W3ZfutYq4Z1x1MDAxZfpcclx1MDAwN1BcdTAwMTFBnNBbvVxczFwio4CI9ur/fvcmMyVcdTAwMThOMFx0RqaaVWlcbsg5RJxvz/vb//NXJPJ3/7VT/vu/XCJ/l4fFfKNW6uZf/v5cdTAwMDdcdTAwMWZcdTAwMWaUu71au1x1MDAwNU+x0c+99nO3OHrlQ7/f6f3X//2/49/wiu3mj98qN8rNcqvfg9f9P/g5XHUwMDEy+Z/R3751uuViP9+qNsqjX1x1MDAxOD01XopLOv3oWbs1W
[...]
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: Virgil;
+ src:
url(data:font/woff2;base64,d09GMgABAAAAACf0AAsAAAAAPzAAACekAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAABmAAgRwRCArzHNR9C4EKAAE2AiQDghAEIAWDHAcgG8YuUZTQXhDZFwfmOdGdFSO+Zy83oazq1LXcqScjJJkdojlrZjcbVYLFIIglBLFCnBAsSBANEkyKVyk1akrrUL8q1vpfVanJfU1OKn4CqmN74crq/F86/R9tZ7Z3ZgTCTkMCUjqxE6d059UVmPMOv5fFNmydrSQkqosKxZCGwTZ2SrNf6SPNLwattc+39VHzEUx6J1IabYXKT27+97DbQUvytNv1zuyblavp2pl2Tmsw5IYrA8RunkWu23fk5+dSm5QVgWdZYWtdLofvp/Dvl3NXhqSYuOLATpkJQ8no52f4M/y9UW4sr7JKTzhROaPkNGyzzubJxA9qR597mEux5+jccmL0
[...]
+ }
+@font-face {
+ font-family: Cascadia;
+ src:
url(data:font/woff2;base64,d09GMgABAAAAACw8ABEAAAAAW8wAACvcAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGiIbHhycXAZgAIE4CIIWCZ8DERAK5XDVOwuCOAABNgIkA4I4BCAFgxwHIAyFMxs4USVj3A0PGwcAZL5dSRT1YZDqzf7/c3IyhoM6bE1NH5dN0ypnguupwnJGF1SM7oy5knQoJHxjikpEKBQfxic21B0JfVYYO7nFkzqa6pYN+i/YKBSmQ7o+iW5bvSF3+HesK4t94xw0WJiS+VD+xNKIga/P1/jOhznHKqKvMKHyossfOUJjn+QOz2/z/5AiIqIXFCzCmG4GxpzTiRiJgY2FhYLNsDBmTMyhvfI5n7XSTZkumToXz7nybca+38faxetVPu+b66uqkwxUdU92WbKQ+Py+desahj5CFuiStJm544/CHIEFkqeAVZIFIPj8hrQ/2Umy
[...]
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(1787.2465290831112
728.9571871401088) rotate(0 733.5155650629545 29.721659908090714)"><path
d="M3.27 -3.31 L1465.95 -3.25 L1463.97 60.74 L-1.33 62.79" stroke="none"
stroke-width="0" fill="#b2f2bb"></path><path d="M1.37 -1.55 C522.46 0.72,
1043.12 -1.32, 1465.99 -0.74 M-0.64 -0.54 C490.84 -2.79, 979.48 -2.87, 1467.69
-0.23 M1463.27 -0.24 C1464.77 25.86, 1466.81 47.1, 1464.18 56.8 M1467.96 -0.01
C1469.43 18.23, 1466.74 36.38, 1465.99 59.47 M [...]
\ No newline at end of file
diff --git a/docs/source/format/images/bool-diagram.svg
b/docs/source/format/images/bool-diagram.svg
new file mode 100644
index 0000000000..76ac8af55e
--- /dev/null
+++ b/docs/source/format/images/bool-diagram.svg
@@ -0,0 +1,10 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1487.0311301259087 536.5291133038454" width="1487.0311301259087"
height="536.5291133038454">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 10) rotate(0
733.5155650629544 29.721659908090714)"><path d="M2.63 2.06 L1470.64 -1.76
L1463.73 60.65 L0.38 58.03" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M-0.23 -0.8 C433.89 2.22, 866.6 2.94, 1467.65
-1.13 M0.7 -0.62 C405.89 -9.46, 811.06 -10.58, 1467.24 -0.64 M1465.4 -2.69
C1469.53 19.18, 1464.45 30.13, 1465.49 59.93 M1465.68 -1.24 C1466.5 12.04,
1466.58 29.18, 1468.78 58.88 M1467.32 58.76 C890.31 55.97, 316.71 5 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/columnar-diagram_1.svg
b/docs/source/format/images/columnar-diagram_1.svg
new file mode 100644
index 0000000000..5cc05eaf4d
--- /dev/null
+++ b/docs/source/format/images/columnar-diagram_1.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
690.1456502907071 311.0564381927252" width="690.1456502907071"
height="311.0564381927252">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(99.43623868525026
102.80643819272518) rotate(0 65.5 22.5)"><path d="M11.25 0 C50.75 -2.78, 89.74
3.83, 119.75 0 C128.42 -2.37, 129.02 5.08, 131 11.25 C129.47 20.04, 133.13
26.89, 131 33.75 C128.51 40.02, 130.63 43.49, 119.75 45 C99.59 44.84, 72.22
42.43, 11.25 45 C6.52 45.47, -0.54 40.07, 0 33.75 C-3.31 26.87, -3.35 16.81, 0
11.25 C-1.38 1.24, 2.76 -1.14, 11.25 0" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M11.25 0 C42.6 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/columnar-diagram_2.svg
b/docs/source/format/images/columnar-diagram_2.svg
new file mode 100644
index 0000000000..5f7b72486c
--- /dev/null
+++ b/docs/source/format/images/columnar-diagram_2.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1001.5000000000001 44.332324548691304" width="1001.5000000000001"
height="44.332324548691304">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 11.657679799899825)
rotate(0 30.442437446757367 11.046034554472271)"><path d="M5.52 0 C21.88 -2.65,
40.67 2.49, 55.36 0 C58.46 -2.61, 62.26 1.35, 60.88 5.52 C62.04 10.72, 61.65
11.55, 60.88 16.57 C64.42 21.44, 58.7 21.6, 55.36 22.09 C34.67 20.35, 23.7
20.6, 5.52 22.09 C0.96 21.99, -1.58 17.34, 0 16.57 C0.98 12.47, 1.18 10.39, 0
5.52 C-3.14 3.82, 1.19 1.61, 5.52 0" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M5.52 0 C16 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/columnar-diagram_3.svg
b/docs/source/format/images/columnar-diagram_3.svg
new file mode 100644
index 0000000000..ce4838d301
--- /dev/null
+++ b/docs/source/format/images/columnar-diagram_3.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1011.5000000000007 43.63151057202731" width="1011.5000000000007"
height="43.63151057202731">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 10.769720731541383)
rotate(0 30.442437446757367 11.046034554472257)"><path d="M5.52 0 C19.38 2.82,
40.63 -3.61, 55.36 0 C58.48 3.36, 61.95 -1.15, 60.88 5.52 C60.52 6.95, 61.75
10.2, 60.88 16.57 C58.79 22.44, 60.89 22.91, 55.36 22.09 C46.75 21.39, 30.06
21.24, 5.52 22.09 C2.4 19.4, 2.33 22.84, 0 16.57 C-1 15.2, -1.47 11.68, 0 5.52
C-1.1 4.07, 5.12 3.14, 5.52 0" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M5.52 0 C23.52 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/dense-union-diagram.svg
b/docs/source/format/images/dense-union-diagram.svg
new file mode 100644
index 0000000000..a2fadf8bb7
--- /dev/null
+++ b/docs/source/format/images/dense-union-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1487.031130125909 1301.0285072437619" width="1487.031130125909"
height="1301.0285072437619">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <rect x="0" y="0" width="1487.031130125909" height="1301.0285072437619"
fill="#ffffff"></rect><g stroke-linecap="round" transform="translate(10 10)
rotate(0 733.5155650629545 29.721659908090714)"><path d="M-2.7 3.59 L1465.64
0.4 L1466.62 59.3 L3.31 56.68" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M-1.16 0.94 C558.06 -4.73, 1120.05 -3.71,
1465.85 0.18 M-0.36 -0.58 C521.98 -1.01, 1044.34 -0.57, 1467.03 -0.26 M1467.68
-0.32 C1470.41 13.58, 1469.49 31.85, 1468.45 55.86 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/dictionary-diagram.svg
b/docs/source/format/images/dictionary-diagram.svg
new file mode 100644
index 0000000000..be1facb3f4
--- /dev/null
+++ b/docs/source/format/images/dictionary-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1562.414263431941 532.5616745076773" width="1562.414263431941"
height="532.5616745076773">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 10) rotate(0
733.5155650629545 29.721659908090714)"><path d="M2.54 3.74 L1470.49 -2.71
L1467.93 62.65 L1.56 56.85" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M0.94 -0.79 C427.88 7.82, 857.07 7.69, 1467.55
-0.07 M0.03 -0.41 C361.87 -5.79, 723.06 -6.61, 1467.04 -0.4 M1463.97 -3.85
C1467.9 17.36, 1466.25 40.35, 1465.21 58.41 M1466.33 -1.02 C1465.32 16.12,
1467.54 32.58, 1468.93 57.61 M1466.67 59.1 C1083.84 60.71, 701.4 6 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/fixed-list-diagram.svg
b/docs/source/format/images/fixed-list-diagram.svg
new file mode 100644
index 0000000000..e0be404e54
--- /dev/null
+++ b/docs/source/format/images/fixed-list-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1171.3426535634085 500.629901274448" width="1171.3426535634085"
height="500.629901274448">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 10) rotate(0
575.6713267817042 27.293180622390082)"><path d="M2.22 0.07 L1152.11 2.79
L1153.28 56.88 L1.03 52.85" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M-0.1 -0.48 C287.86 -5.6, 578.23 -5.42, 1152.3
-0.17 M0.54 -0.47 C420.69 -3.37, 839.68 -3.07, 1150.73 -0.32 M1154.68 -2.59
C1149.62 21.94, 1152.35 42.01, 1154.3 56.71 M1149.81 -1.96 C1150.12 16.03,
1148.94 35.16, 1152.91 54.34 M1152.07 55.89 C697.42 47.72, 243.81 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/fixed-string-diagram.svg
b/docs/source/format/images/fixed-string-diagram.svg
new file mode 100644
index 0000000000..2f857ef149
--- /dev/null
+++ b/docs/source/format/images/fixed-string-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1110.702151340734 382.15429186395613" width="1110.702151340734"
height="382.15429186395613">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 10) rotate(0
545.351075670367 29.721659908090714)"><path d="M0.62 -3.77 L1089.62 2.92
L1090.79 57.88 L-3.84 56.59" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M0.91 1.18 C317.81 -5.78, 636.49 -4.35, 1090.78
0.15 M0.18 0.4 C406.08 0.46, 811.67 0.67, 1091.5 0.33 M1092.46 1.77 C1088.27
13.3, 1088.32 32.24, 1087.21 55.97 M1089.04 -1.82 C1091.44 19.53, 1091.53
36.75, 1088.73 57.52 M1091.64 58.21 C842.22 59.1, 593.08 57.69, [...]
\ No newline at end of file
diff --git a/docs/source/format/images/map-diagram.svg
b/docs/source/format/images/map-diagram.svg
new file mode 100644
index 0000000000..b50c2a9e2c
--- /dev/null
+++ b/docs/source/format/images/map-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
2745.1732053852093 1065.8656221982455" width="2745.1732053852093"
height="1065.8656221982455">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 141.48173764525836)
rotate(0 390.4353389905203 151.59257627027318)"><path d="M1.33 -1.19 L779.75
-0.52 L779.26 303.69 L0.45 304.39" stroke="none" stroke-width="0"
fill="#ffc9c9"></path><path d="M0 0 C188.12 -3.2, 377.82 -3.58, 780.87 0 M0 0
C289.4 -1.64, 578.94 -1.66, 780.87 0 M780.87 0 C782.45 79.5, 780.98 157.21,
780.87 303.19 M780.87 0 C781.67 119.15, 781.12 237.46, 780.87 303.19 M780.87
303.19 C521.07 304.62, 261.58 305.7, 0 303.19 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/primitive-diagram.svg
b/docs/source/format/images/primitive-diagram.svg
new file mode 100644
index 0000000000..c7158ef4d9
--- /dev/null
+++ b/docs/source/format/images/primitive-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1487.031130125909 503.1402763085464" width="1487.031130125909"
height="503.1402763085464">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 10) rotate(0
733.5155650629545 29.721659908090714)"><path d="M3.27 -3.31 L1465.95 -3.25
L1463.97 60.74 L-1.33 62.79" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M1.37 -1.55 C522.46 0.72, 1043.12 -1.32, 1465.99
-0.74 M-0.64 -0.54 C490.84 -2.79, 979.48 -2.87, 1467.69 -0.23 M1463.27 -0.24
C1464.77 25.86, 1466.81 47.1, 1464.18 56.8 M1467.96 -0.01 C1469.43 18.23,
1466.74 36.38, 1465.99 59.47 M1467.99 59.52 C917.67 54.05, 36 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/ree-diagram.svg
b/docs/source/format/images/ree-diagram.svg
new file mode 100644
index 0000000000..8913ef91e1
--- /dev/null
+++ b/docs/source/format/images/ree-diagram.svg
@@ -0,0 +1,13 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1544.3797629384087 652.8954222118855" width="1544.3797629384087"
height="652.8954222118855">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: Virgil;
+ src:
url(data:font/woff2;base64,d09GMgABAAAAABhIAAsAAAAAJqAAABf6AAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAABmAAgTQRCArDELFCC1YAATYCJAOBKAQgBYMcByAbuxwjA3Vq0NKS/eUBNzfQJzQAy7OMwR4SdwEJFmxZL779p4k5wZcJKP4prDQ8zem/u8slTkyBOGIJ4i0xQUIQLQRJEG+L1rCaUnOoODpRKqJj/4/OrDIqSs/Tjea/KsbVKJVcDu22AP77XyA84P/p1j/5v4M24Vn9Re30Jl0z085tLcKSlUkCQfYJrJhG/i+nlXStuv76JVtOZ8CSId2D6+ACd/PJHr95S3S5rq/pa5UsKYaAJFNgwNDEzgCXVAvmM/dWN/YT0uD5u2QEQvgZN1cyak1tAMhnzkfYWAe4/ywD5B7UyziycSLCsMKyykeVQH2lr1C6RleYLrWNnvaO7FGt/eNG
[...]
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 10) rotate(0
762.1898814692044 29.721659908090714)"><path d="M-3.83 -3.81 L1523.49 -0.28
L1525.98 56.58 L-1.3 57.69" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M0.44 -0.42 C588.59 -12.39, 1173.47 -12.16,
1524.48 0.83 M0.07 0.79 C311.97 0.9, 623.2 0.88, 1523.9 0.61 M1524.14 -1.5
C1528.53 21.4, 1527.83 47.89, 1527.63 62.28 M1525.11 -1.43 C1521.73 24.66,
1524.27 46.75, 1523.33 60.33 M1525.52 59.79 C1157.45 50.81, 790.94 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/sparse-union-diagram.svg
b/docs/source/format/images/sparse-union-diagram.svg
new file mode 100644
index 0000000000..cc150f0e70
--- /dev/null
+++ b/docs/source/format/images/sparse-union-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1543.8254002417002 1214.2513084672646" width="1543.8254002417002"
height="1214.2513084672646">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <rect x="0" y="0" width="1543.8254002417002" height="1214.2513084672646"
fill="#ffffff"></rect><g stroke-linecap="round" transform="translate(10 10)
rotate(0 761.9127001208501 29.721659908090714)"><path d="M3.91 1.78 L1524.84
-1.13 L1523.32 63.15 L-1.89 60.67" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M-0.16 0.21 C429.21 0.26, 857.98 -2.33, 1523.56
-0.12 M-0.76 0.32 C515.43 2.99, 1030.78 3.42, 1523.32 0.75 M1525.91 2.76
C1526.69 21.39, 1527.92 37.41, 1526.31 62.96 M [...]
\ No newline at end of file
diff --git a/docs/source/format/images/struct-diagram.svg
b/docs/source/format/images/struct-diagram.svg
new file mode 100644
index 0000000000..81ff29689a
--- /dev/null
+++ b/docs/source/format/images/struct-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1495.3089493858786 991.8194870076786" width="1495.3089493858786"
height="991.8194870076786">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(18.277819259969647 10)
rotate(0 733.5155650629545 29.721659908090714)"><path d="M-3.55 1.29 L1465.25
1.38 L1465.08 60.09 L1.55 57.91" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M0.61 -0.33 C513.17 -8.83, 1026.67 -8.07,
1467.57 0.93 M-0.53 -0.4 C530.65 0.22, 1061.37 -0.37, 1467.57 0.51 M1469.88
3.47 C1469.48 11.73, 1466.32 32.23, 1468.13 62.18 M1465.16 -0.72 C1466.59
13.58, 1466.37 26.07, 1466.84 61.15 M1467.08 60.93 C100 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/var-list-diagram.svg
b/docs/source/format/images/var-list-diagram.svg
new file mode 100644
index 0000000000..a3024371d3
--- /dev/null
+++ b/docs/source/format/images/var-list-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1266.933473875909 601.8372074955921" width="1266.933473875909"
height="601.8372074955921">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 10) rotate(0
623.4667369379545 29.721659908090714)"><path d="M-0.37 4 L1244.02 2.35 L1247.14
59.08 L-1.97 63.01" stroke="none" stroke-width="0" fill="#b2f2bb"></path><path
d="M0.27 0.85 C473.54 -1.79, 949.4 0.43, 1247.11 -0.44 M-0.45 0.18 C337.82
-7.85, 677.01 -8.67, 1246.74 0.69 M1250.75 -2.52 C1243.07 20.05, 1246.34 43.45,
1244.05 62.5 M1248.56 -0.21 C1247.57 15.21, 1246.44 28.89, 1246.17 61.43
M1246.45 59.98 C846.67 62.65, 450.47 61. [...]
\ No newline at end of file
diff --git a/docs/source/format/images/var-list-view-diagram.svg
b/docs/source/format/images/var-list-view-diagram.svg
new file mode 100644
index 0000000000..dab451556d
--- /dev/null
+++ b/docs/source/format/images/var-list-view-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1277.753786375909 704.8071429714123" width="1277.753786375909"
height="704.8071429714123">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10 10) rotate(0
628.8768931879545 29.721659908090714)"><path d="M-3.44 3.79 L1258.4 -1.46
L1254.08 60.23 L-0.69 60.47" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M-0.16 0.3 C486.53 -6.75, 975.42 -7.08, 1259.07
0.09 M0.04 -0.69 C395.9 -4.66, 792.42 -3.96, 1257.27 -0.78 M1261.35 2.09
C1257.78 16.25, 1253.57 28.5, 1260.97 57.03 M1257.7 1.74 C1259.59 18.49,
1257.54 35.1, 1256.92 60.59 M1257.48 60.76 C997.52 59.16, 733.76 59. [...]
\ No newline at end of file
diff --git a/docs/source/format/images/var-string-diagram.svg
b/docs/source/format/images/var-string-diagram.svg
new file mode 100644
index 0000000000..8cf17498ba
--- /dev/null
+++ b/docs/source/format/images/var-string-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
1487.031130125909 493.0622872014401" width="1487.031130125909"
height="493.0622872014401">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(1028.6245753901771
277.43154790352264) rotate(0 135.3646112146389 37.32296614216284)"><path
d="M-0.16 -0.13 L270.38 0.88 L269.2 74.76 L-1.1 76.09" stroke="none"
stroke-width="0" fill="#a5d8ff"></path><path d="M0 0 C99.08 -0.77, 197.98 0.42,
270.73 0 M0 0 C103.23 -0.36, 205.67 -0.22, 270.73 0 M270.73 0 C268.9 15.58,
268.92 36.77, 270.73 74.65 M270.73 0 C269.41 29.71, 270.65 56.86, 270.73 74.65
M270.73 74.65 C188.37 75.73, 105.98 73.88, 0 74 [...]
\ No newline at end of file
diff --git a/docs/source/format/images/var-string-view-diagram.svg
b/docs/source/format/images/var-string-view-diagram.svg
new file mode 100644
index 0000000000..c3aa87ac4a
--- /dev/null
+++ b/docs/source/format/images/var-string-view-diagram.svg
@@ -0,0 +1,21 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0
2152.333404158228 1361.4275658159222" width="2152.333404158228"
height="1361.4275658159222">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts">
+ @font-face {
+ font-family: "Virgil";
+ src: url("https://excalidraw.com/Virgil.woff2";);
+ }
+ @font-face {
+ font-family: "Cascadia";
+ src: url("https://excalidraw.com/Cascadia.woff2";);
+ }
+ @font-face {
+ font-family: "Assistant";
+ src: url("https://excalidraw.com/Assistant-Regular.woff2";);
+ }
+ </style>
+
+ </defs>
+ <g stroke-linecap="round" transform="translate(10.09719590731902 10)
rotate(0 1066.1181041254545 29.721659908090714)"><path d="M-0.87 -1.43 L2133.76
-0.72 L2133.99 59.39 L-0.83 58.72" stroke="none" stroke-width="0"
fill="#b2f2bb"></path><path d="M0 0 C797.85 5.71, 1596.48 5.14, 2132.24 0 M0 0
C493.86 4.05, 988.53 3.44, 2132.24 0 M2132.24 0 C2133.57 16.88, 2133.51 32.26,
2132.24 59.44 M2132.24 0 C2132.86 15.36, 2131.89 29.78, 2132.24 59.44 M2132.24
59.44 C1631.5 52.08, 1131.24 52.21, 0 [...]
\ No newline at end of file
diff --git a/docs/source/format/index.rst b/docs/source/format/index.rst
index 44ea3e8e7e..ce31a15a1f 100644
--- a/docs/source/format/index.rst
+++ b/docs/source/format/index.rst
@@ -23,8 +23,10 @@ Specifications
.. toctree::
:maxdepth: 2
- Versioning
+ Intro
Columnar
+ Versioning
+ Changing
CanonicalExtensions
Other
CDataInterface
@@ -34,6 +36,5 @@ Specifications
Flight
FlightSql
ADBC
- Changing
Integration
Glossary
