Copilot commented on code in PR #104: URL: https://github.com/apache/arrow-erlang/pull/104#discussion_r3610749971
########## src/arrow_variable_list_array.erl: ########## @@ -15,30 +15,29 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides support for Arrow's Fixed-Size List Layout. -%% -%% This module provides support for the Fixed-Size List Layout[1], which is an -%% layout that supports storing a list of lists of a specific length and -%% nesting. -%% -%% == Invalid Input == -%% -%% It is important that care is taken when passing input values to this module. -%% For performance reasons, the input is not validated. The function crashes on -%% nesting that is inconsitent: a. with the type, b. between elements. The -%% behaviour on invalid input <strong>CANNOT BE GUARANTEED</strong>. Therefore, -%% one must be careful to not to <strong>CRASH THE PROCESS</strong> or worse -%% still, <strong>PRODUCE INVALID OUTPUT</strong>. -%% -%% Any input must follow the following rules: -%% -%% <ol> -%% <li>The nesting of each element must be consistent with the type</li> -%% <li>The nesting of each element must be consistent with each other</li> -%% </ol> -%% -%% [1]: https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout -module(arrow_variable_list_array). +-moduledoc """ +Provides support for Arrow's Fixed-Size List Layout. + +This module provides support for the [Fixed-Size List +Layout](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout), +which is an layout that supports storing a list of lists of a specific length +and nesting. Review Comment: The moduledoc describes this as “Fixed-Size List Layout”, but the module implements `variable_list` (and links to the variable-size list layout). This makes the rendered docs misleading. ########## src/arrow_type.erl: ########## @@ -15,123 +15,94 @@ % specific language governing permissions and limitations % under the License. -%%%----------------------------------------------------------------------------- -%% @doc Provides representation for as well as functions for working with Arrow -%% specific datatypes. -%% -%% This module provides a longhand (a 2 tuple) as well as a shorthand (an atom) -%% represenations for representing a primitive types. Additionally it represents -%% Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types -%% that are supported by this package). -%% -%% Generally, when we say an array A has type X, we mean that each element in A -%% has type X. Thus, an array of type of `Int8' would look like `[1, 2, 3]', but -%% an array of type `List<Int8>' would look like `[[1, 2], [3]]'. -%% -%% == Primitive Types == -%% -%% A Primitive Type is any type that refers to a single value (as opposed to a -%% list of values). Currently it includes Booleans, Signed Integers, Unsigned -%% Integers, Floats and Binaries. Both a longhand and shorthand syntax had been -%% provided to initiate a type. -%% -%% The following is a comprehensive list of all supported primitive types: -%% -%% <ol> -%% <li>Booleans</li> -%% <li>Signed Integers: Includes `Int 8', `Int 16', `Int 32' and `Int 64'</li> -%% <li> -%% Unsigned Integers: Includes `UInt 8', `UInt 16', `UInt 32' and `UInt 64' -%% </li> -%% <li>Floats: Includes `Float 16', `Float 32' and `Float 64'</li> -%% <li>Binaries</li> -%% </ol> -%% -%% === Shorthand Syntax === -%% -%% In the shorthand syntax, you can just use a single atom to refer to a type, -%% such that the atom is `<name><size>'. The name is a single letter that refers -%% to the type, and the size is that type's size in bits. Take the example of -%% `Int 8'. Its name is `s', and size is `8'. Thus in shorthand, it is `s8'. In -%% the case of datatypes without variations, such as Booleans and Binaries, they -%% are represented by just their datatype. Do note that internally the longhand -%% syntax is used. -%% -%% Here you can find the name of each type: -%% -%% <table border="1"> -%% <tr> -%% <td>Type</td> -%% <td>Name</td> -%% <td>Example Atom</td> -%% </tr> -%% <tr> -%% <td>Booleans</td> -%% <td>N/A</td> -%% <td>`bool'</td> -%% </tr> -%% <tr> -%% <td>Signed Integer</td> -%% <td>`s'</td> -%% <td>`s64'</td> -%% </tr> -%% <tr> -%% <td>Unsigned Integer</td> -%% <td>`u'</td> -%% <td>`u16'</td> -%% </tr> -%% <tr> -%% <td>Float</td> -%% <td>`f'</td> -%% <td>`f32'</td> -%% </tr> -%% <tr> -%% <td>Binaries</td> -%% <td>N/A</td> -%% <td>`bin'</td> -%% </tr> -%% </table> -%% -%% Types that are shorthand are under the `t:arrow_shorthand_type()' and -%% `arrow_short_*()' types. -%% -%% === Longhand Syntax === -%% -%% In longhand syntax, you use a 2 tuple to to refer to the type, such that the -%% tuple is `{Name, Size}'. Thus, `Int 8' would be represented as `{s, 8}'. Do -%% note that in the case or Booleans and Binaries their `Size' is `undefined', as -%% the have variable or undefined size. -%% -%% Types that are longhand are under the `t:arrow_longhand_type()' and -%% `arrow_long_*()' types. -%% -%% == Nested Type == -%% -%% A Nested Type is any data structure that supports nesting. This can include -%% Lists, FixedLists, Maps, Structs and more, but is curently limited to -%% FixedLists. Nested Types are represented using a 3 tuple: `{NestedType, Type, -%% Length}'. `NestedType' refers to the data structure that is nesting your -%% primitive type. `Type' refers to the type that is being nested in the data -%% structure. Do note that `Type' can be a Primitive Type, or another Nested -%% Type. Both the shorthand and the longhand syntax can be used or Primitive -%% Types.`Length' refers to the length of each element, and will only be -%% required to represent FixedLists. -%% -%% It is important to remember while nesting that the type refers to the type of -%% each element in an array. A list of `Int8' is 2 dimensional, and a list of -%% `List<Int8>' is 3 dimensional. -%% -%% == Native Type == -%% -%% A Native Type is any Erlang Datatype that can be serialized by `arrow', -%% and is represented by the type `t:native_type()'. It currently only includes -%% values that are primitive in nature such as booleans, integers, floats, -%% binaries, and additionally the atoms `undefined' and `nil' to represent a -%% `NULL' value. -%% -%% @end -%%%----------------------------------------------------------------------------- -module(arrow_type). +-moduledoc """ +Provides representation for as well as functions for working with Arrow +specific datatypes. + +This module provides a longhand (a 2 tuple) as well as a shorthand (an atom) +represenations for representing a primitive types. Additionally it represents +Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types +that are supported by this package). + +Generally, when we say an array A has type X, we mean that each element in A +has type X. Thus, an array of type of `Int8` would look like `[1, 2, 3]`, but +an array of type `List<Int8>` would look like `[[1, 2], [3]]`. + +## Primitive Types + +A Primitive Type is any type that refers to a single value (as opposed to a +list of values). Currently it includes Booleans, Signed Integers, Unsigned +Integers, Floats and Binaries. Both a longhand and shorthand syntax had been +provided to initiate a type. + +The following is a comprehensive list of all supported primitive types: + +1. Booleans +2. Signed Integers: Includes `Int 8`, `Int 16`, `Int 32` and + `Int 64` +3. Unsigned Integers: Includes `UInt 8`, `UInt 16`, `UInt 32` and + `UInt 64` +4. Floats: Includes `Float 16`, `Float 32` and `Float 64` +5. Binaries + +### Shorthand Syntax + +In the shorthand syntax, you can just use a single atom to refer to a type, +such that the atom is `<name><size>`. The name is a single letter that refers +to the type, and the size is that type's size in bits. Take the example of +`Int 8`. Its name is `s`, and size is `8`. Thus in shorthand, it is `s8`. In +the case of datatypes without variations, such as Booleans and Binaries, they +are represented by just their datatype. Do note that internally the longhand +syntax is used. + +Here you can find the name of each table: + +| Type | Name | Example Atom | +|------------------|-------|--------------| +| Booleans | N/A | `bool` | +| Signed Integer | `s` | `s64` | +| Unsigned Integer | `u ` | `u16 ` | +| Float | `f ` | `f32 ` | +| Binaries | N/A | `bin ` | + +Types that are shorthand are under the `t:arrow_shorthand_type()` and +`arrow_short_*()` types. + +### Longhand Syntax + +In longhand syntax, you use a 2 tuple to to refer to the type, such that the +tuple is `{Name, Size}`. Thus, `Int 8` would be represented as `{s, 8}`. Do +note that in the case or Booleans and Binaries their `Size` is `undefined`, as +the have variable or undefined size. + +Types that are longhand are under the `t:arrow_longhand_type()` and +`arrow_long_*()` types. + +### Type Review Comment: The heading “### Type” introduces the Nested Type section; as-written it reads like a generic ‘type’ section and is inconsistent with the surrounding terminology (Primitive Types / Native Type). ########## src/arrow_type.erl: ########## @@ -15,123 +15,94 @@ % specific language governing permissions and limitations % under the License. -%%%----------------------------------------------------------------------------- -%% @doc Provides representation for as well as functions for working with Arrow -%% specific datatypes. -%% -%% This module provides a longhand (a 2 tuple) as well as a shorthand (an atom) -%% represenations for representing a primitive types. Additionally it represents -%% Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types -%% that are supported by this package). -%% -%% Generally, when we say an array A has type X, we mean that each element in A -%% has type X. Thus, an array of type of `Int8' would look like `[1, 2, 3]', but -%% an array of type `List<Int8>' would look like `[[1, 2], [3]]'. -%% -%% == Primitive Types == -%% -%% A Primitive Type is any type that refers to a single value (as opposed to a -%% list of values). Currently it includes Booleans, Signed Integers, Unsigned -%% Integers, Floats and Binaries. Both a longhand and shorthand syntax had been -%% provided to initiate a type. -%% -%% The following is a comprehensive list of all supported primitive types: -%% -%% <ol> -%% <li>Booleans</li> -%% <li>Signed Integers: Includes `Int 8', `Int 16', `Int 32' and `Int 64'</li> -%% <li> -%% Unsigned Integers: Includes `UInt 8', `UInt 16', `UInt 32' and `UInt 64' -%% </li> -%% <li>Floats: Includes `Float 16', `Float 32' and `Float 64'</li> -%% <li>Binaries</li> -%% </ol> -%% -%% === Shorthand Syntax === -%% -%% In the shorthand syntax, you can just use a single atom to refer to a type, -%% such that the atom is `<name><size>'. The name is a single letter that refers -%% to the type, and the size is that type's size in bits. Take the example of -%% `Int 8'. Its name is `s', and size is `8'. Thus in shorthand, it is `s8'. In -%% the case of datatypes without variations, such as Booleans and Binaries, they -%% are represented by just their datatype. Do note that internally the longhand -%% syntax is used. -%% -%% Here you can find the name of each type: -%% -%% <table border="1"> -%% <tr> -%% <td>Type</td> -%% <td>Name</td> -%% <td>Example Atom</td> -%% </tr> -%% <tr> -%% <td>Booleans</td> -%% <td>N/A</td> -%% <td>`bool'</td> -%% </tr> -%% <tr> -%% <td>Signed Integer</td> -%% <td>`s'</td> -%% <td>`s64'</td> -%% </tr> -%% <tr> -%% <td>Unsigned Integer</td> -%% <td>`u'</td> -%% <td>`u16'</td> -%% </tr> -%% <tr> -%% <td>Float</td> -%% <td>`f'</td> -%% <td>`f32'</td> -%% </tr> -%% <tr> -%% <td>Binaries</td> -%% <td>N/A</td> -%% <td>`bin'</td> -%% </tr> -%% </table> -%% -%% Types that are shorthand are under the `t:arrow_shorthand_type()' and -%% `arrow_short_*()' types. -%% -%% === Longhand Syntax === -%% -%% In longhand syntax, you use a 2 tuple to to refer to the type, such that the -%% tuple is `{Name, Size}'. Thus, `Int 8' would be represented as `{s, 8}'. Do -%% note that in the case or Booleans and Binaries their `Size' is `undefined', as -%% the have variable or undefined size. -%% -%% Types that are longhand are under the `t:arrow_longhand_type()' and -%% `arrow_long_*()' types. -%% -%% == Nested Type == -%% -%% A Nested Type is any data structure that supports nesting. This can include -%% Lists, FixedLists, Maps, Structs and more, but is curently limited to -%% FixedLists. Nested Types are represented using a 3 tuple: `{NestedType, Type, -%% Length}'. `NestedType' refers to the data structure that is nesting your -%% primitive type. `Type' refers to the type that is being nested in the data -%% structure. Do note that `Type' can be a Primitive Type, or another Nested -%% Type. Both the shorthand and the longhand syntax can be used or Primitive -%% Types.`Length' refers to the length of each element, and will only be -%% required to represent FixedLists. -%% -%% It is important to remember while nesting that the type refers to the type of -%% each element in an array. A list of `Int8' is 2 dimensional, and a list of -%% `List<Int8>' is 3 dimensional. -%% -%% == Native Type == -%% -%% A Native Type is any Erlang Datatype that can be serialized by `arrow', -%% and is represented by the type `t:native_type()'. It currently only includes -%% values that are primitive in nature such as booleans, integers, floats, -%% binaries, and additionally the atoms `undefined' and `nil' to represent a -%% `NULL' value. -%% -%% @end -%%%----------------------------------------------------------------------------- -module(arrow_type). +-moduledoc """ +Provides representation for as well as functions for working with Arrow +specific datatypes. + +This module provides a longhand (a 2 tuple) as well as a shorthand (an atom) +represenations for representing a primitive types. Additionally it represents +Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types +that are supported by this package). + +Generally, when we say an array A has type X, we mean that each element in A +has type X. Thus, an array of type of `Int8` would look like `[1, 2, 3]`, but +an array of type `List<Int8>` would look like `[[1, 2], [3]]`. + +## Primitive Types + +A Primitive Type is any type that refers to a single value (as opposed to a +list of values). Currently it includes Booleans, Signed Integers, Unsigned +Integers, Floats and Binaries. Both a longhand and shorthand syntax had been +provided to initiate a type. + +The following is a comprehensive list of all supported primitive types: + +1. Booleans +2. Signed Integers: Includes `Int 8`, `Int 16`, `Int 32` and + `Int 64` +3. Unsigned Integers: Includes `UInt 8`, `UInt 16`, `UInt 32` and + `UInt 64` +4. Floats: Includes `Float 16`, `Float 32` and `Float 64` +5. Binaries + +### Shorthand Syntax + +In the shorthand syntax, you can just use a single atom to refer to a type, +such that the atom is `<name><size>`. The name is a single letter that refers +to the type, and the size is that type's size in bits. Take the example of +`Int 8`. Its name is `s`, and size is `8`. Thus in shorthand, it is `s8`. In +the case of datatypes without variations, such as Booleans and Binaries, they +are represented by just their datatype. Do note that internally the longhand +syntax is used. + +Here you can find the name of each table: + +| Type | Name | Example Atom | +|------------------|-------|--------------| +| Booleans | N/A | `bool` | +| Signed Integer | `s` | `s64` | +| Unsigned Integer | `u ` | `u16 ` | +| Float | `f ` | `f32 ` | +| Binaries | N/A | `bin ` | Review Comment: The shorthand table includes trailing spaces inside backticks (e.g. `` `u ` ``, `` `u16 ` ``), which makes the documented atoms incorrect and will confuse users copying them. ########## src/arrow_ipc_file.erl: ########## @@ -15,27 +15,32 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides records and functions to deal with the IPC File. -%% -%% The IPC File[1] is an extension of the IPC Stream[2] that supports random -%% access with the help of a footer which contains the offsets of all the -%% messages. -%% -%% [1]: https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format -%% [2]: https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format -%% @end -module(arrow_ipc_file). +-moduledoc """ +Provides records and functions to deal with the IPC File. The [IPC +File](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format) is an +extension of the [IPC Stream]( +https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format) that +supports random access with the help of a footer which contains the offsets of +all the messages. Review Comment: The `IPC Stream` Markdown link is broken because the URL is split onto a new line inside the link parentheses, which will likely render as plain text instead of a hyperlink. ########## src/arrow_buffer.erl: ########## @@ -74,15 +71,16 @@ from_erlang(Data, Type) -> end, from_erlang(Data, Type, Len). -%% @doc Creates a new buffer from a list of Erlang values or binaries, given its -%% type and length -%% @end +-doc """ +Creates a new buffer from a list of Erlang values or binaries, given its +and length +""". Review Comment: The `from_erlang/3` docstring is missing the word “type” (“given its and length”), which reads like a formatting error in generated docs. ########## src/arrow_buffer.erl: ########## @@ -15,55 +15,52 @@ % specific language governing permissions and limitations % under the License. -%% @doc Buffer implementation for `arrow'. -%% This module adds suppport for buffers, or Contiguous Memory Regions. -%% -%% There are multiple things to know about buffers[1]: -%% -%% <ol> -%% <li> -%% Each value it stores is called an element or a slot[2]. -%% </li> -%% <li> -%% Each slot's length (in bytes) is a positive integer. As a result when we -%% say that a slot has a length of 1, we mean that each slot has a length of -%% 1 byte. -%% </li> -%% <li> -%% The buffer's length in the metadata refers to the unpadded binary's size in bytes. -%% </li> -%% <li> -%% All buffers have a size that is a multiple of 64. If their data's length -%% is not a multiple of 64, it must be padded (in this implementation, by -%% zeros). -%% </li> -%% <li> -%% Null values are represented in this implementation by zeros. -%% </li> -%% <li> -%% In this implementation buffers can be initialized from raw bytes as data -%% apart datatypes supported by Arrow. This is so that the Validity Bitmap -%% Buffer can be initialized. -%% </li> -%% </ol> -%% -%% [1]: [https://arrow.apache.org/docs/format/Glossary.html#term-buffer] -%% -%% [2]: [https://arrow.apache.org/docs/format/Glossary.html#term-slot] -%% @end -module(arrow_buffer). + +%% TODO Check formatting with erlfmt +%% erlfmt:ignore +-moduledoc """ +Buffer implementation for `arrow`. + +This module adds suppport for buffers, or Contiguous Memory Regions. There are Review Comment: Spelling: “suppport” should be “support” in the moduledoc. ########## src/arrow_offsets.erl: ########## @@ -15,68 +15,66 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides support for Apache Arrow's Offsets -%% -%% Arrow has a concept of offsets[1] in order to tell the length of a slot[2] or -%% a single element in an array of variable-size elements. This module provides -%% support for generating offsets. -%% -%% There are couple of things to remember about offsets: -%% -%% <ol> -%% <li> -%% Each element in the offsets coresponds to the distance in bytes of the -%% coresponding element in the values from the beginning of the buffer. -%% -%% I.E., distance of values[j] from beginning of the buffer = offsets[j]. -%% </li> -%% <li> -%% The very last element in the offsets is the length of the buffer, or -%% distance of the end of the last slot from the beginning of the buffer. -%% </li> -%% <li> -%% Thus, the offsets is one element longer than the values, or: -%% -%% length(offsets) == length(values) + 1 -%% </li> -%% <li> -%% Therefore, in order to find the length of a slot, we subtract the offset -%% the current element from the offset of the next element, or: -%% -%% slot[j] = offsets[j + 1] - offsets[j] -%% </li> -%% <li> -%% Null values have an offset of 0 as they take no memory in the buffer. -%% Thus, the previous offset and the current offset are equivalent if the -%% current element is a null. -%% </li> -%% </ol> -%% -%% [1]: [https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout] -%% -%% [2]: [https://arrow.apache.org/docs/format/Columnar.html#terminology] -%% @end -module(arrow_offsets). --export([new/2, new/3]). +-moduledoc """ +Provides support for Apache Arrow's Offsets + +Arrow has a concept of +[offsets](https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout) +in order to tell the length of a +[slot](https://arrow.apache.org/docs/format/Columnar.html#terminology) or a +single element in an array of variable-size elements. This module provides +support for generating offsets. + +There are couple of things to remember about offsets: + +Firstly, each element in the offsets coresponds to the distance in bytes of the +coresponding element in the values from the beginning of the buffer. Review Comment: Spelling in moduledoc: “coresponds/coresponding” should be “corresponds/corresponding”. ########## src/arrow_ipc_type.erl: ########## @@ -90,7 +87,9 @@ %% from_erlang/1 %% %%%%%%%%%%%%%%%%%%% -%% @doc Returns the IPC Type for an `#array{}'. +-doc """ +Returns the IPC Type for an `t:arrow_array:array()`. +""". Review Comment: The doc comment refers to `t:arrow_array:array()`, which isn’t a valid Erlang type expression. Using `arrow_array:array()` keeps the docs copy/pasteable. ########## src/arrow_ipc_schema.erl: ########## @@ -15,58 +15,48 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides a record and functions to deal with Schemas -%% -%% A Schema[1] represents a table, or a list of arrays of equal length. This -%% module provides a record and a function to manage all the metadata required -%% to represent a schema. Metadata such as: -%% -%% <ol> -%% <li> -%% `endianness': The Endianness of the table. One of `little' or `big'. -%% Defaults to `little'. -%% </li> -%% <li> -%% `fields': The list of fields[2] in a table. -%% </li> -%% <li> -%% `type': The Layout of the column -%% </li> -%% <li> -%% `custom_metadata': A list of custom metadata in key-value format -%% </li> -%% <li> -%% `features': Any features used by the table which may not be present in -%% other implementations of Arrow. -%% </li> -%% </ol> -%% -%% Currently, big endianness, custom metadata and features are not supported, -%% but they have been added for forwards comapatibility. -%% -%% You can find Schemas in the Arrow spec here[3]. -%% -%% [1]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530] -%% -%% [2]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492] -%% -%% [3]: [https://arrow.apache.org/docs/format/Columnar.html#schema-message] -%% @end -module(arrow_ipc_schema). +-moduledoc """ +Provides a record and functions to deal with Schemas A +[Schema](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530)[1] +represents a table, or a list of arrays of equal length. This module provides a +record and a function to manage all the metadata required to represent a schema. +Metadata such as: + +1. `endianness`: The Endianness of the table. One of `little` or + `big`. Defaults to `little`. +2. `fields`: The list of [fields](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492) in a table. +3. `type`: The Layout of the column +4. `custom_metadata`: A list of custom metadata in key-value format +5. `features`: Any features used by the table which may not be + present in other implementations of Arrow. + +Currently, big endianness, custom metadata and features are not supported, but +they have been added for forwards comapatibility. Review Comment: Spelling in moduledoc: “comapatibility” should be “compatibility”. ########## src/arrow_ipc_record_batch.erl: ########## @@ -15,57 +15,47 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides a record and functions to deal with RecordBatches -%% -%% A RecordBatch[1] represents a list of equal length arrays and their -%% coresponding buffers. This module provides a record and a function to manage -%% all the metadata required to represent a RecordBatch. Metadata such as: -%% -%% <ol> -%% <li> -%% `length': The number of rows or records. In other words, the length of -%% an array. -%% </li> -%% <li> -%% `nodes': A list of maps, where each map has the length and null count -%% of an array -%% </li> -%% <li> -%% `buffers': A list of maps, where each map has the length and the offset -%% (from the beginning of the message body) of a buffer of an array. -%% </li> -%% <li> -%% `compression': The compression applied on the body of the Record Batch. -%% Can either by `undefined' (i.e. no compression), `zstd' for Zstandard[2], -%% or `lz4_frame' for LZ4 Frame[3]. Defaults to `undefined'. -%% </li> -%% </ol> -%% -%% Currently, compression is not supported, but it has been added for forwards -%% comapatibility. -%% -%% You can find RecordBatches in the Arrow spec here[4]. -%% -%% [1]: [https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102] -%% -%% [2]: [https://facebook.github.io/zstd/] -%% -%% [3]: [https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md] -%% -%% [4]: [https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message] -%% @end -module(arrow_ipc_record_batch). +-moduledoc """ +Provides a record and functions to deal with RecordBatches +A [RecordBatch](https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102) represents a list of equal length arrays and their +coresponding buffers. This module provides a record and a function to manage +all the metadata required to represent a RecordBatch. Metadata such as: Review Comment: The moduledoc opening has a missing sentence break and a spelling error (“coresponding”), which reduces readability of generated docs. ########## src/arrow_ipc_record_batch.erl: ########## @@ -15,57 +15,47 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides a record and functions to deal with RecordBatches -%% -%% A RecordBatch[1] represents a list of equal length arrays and their -%% coresponding buffers. This module provides a record and a function to manage -%% all the metadata required to represent a RecordBatch. Metadata such as: -%% -%% <ol> -%% <li> -%% `length': The number of rows or records. In other words, the length of -%% an array. -%% </li> -%% <li> -%% `nodes': A list of maps, where each map has the length and null count -%% of an array -%% </li> -%% <li> -%% `buffers': A list of maps, where each map has the length and the offset -%% (from the beginning of the message body) of a buffer of an array. -%% </li> -%% <li> -%% `compression': The compression applied on the body of the Record Batch. -%% Can either by `undefined' (i.e. no compression), `zstd' for Zstandard[2], -%% or `lz4_frame' for LZ4 Frame[3]. Defaults to `undefined'. -%% </li> -%% </ol> -%% -%% Currently, compression is not supported, but it has been added for forwards -%% comapatibility. -%% -%% You can find RecordBatches in the Arrow spec here[4]. -%% -%% [1]: [https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102] -%% -%% [2]: [https://facebook.github.io/zstd/] -%% -%% [3]: [https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md] -%% -%% [4]: [https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message] -%% @end -module(arrow_ipc_record_batch). +-moduledoc """ +Provides a record and functions to deal with RecordBatches +A [RecordBatch](https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102) represents a list of equal length arrays and their +coresponding buffers. This module provides a record and a function to manage +all the metadata required to represent a RecordBatch. Metadata such as: + +1. `length`: The number of rows or records. In other words, the + length of an array. +2. `nodes`: A list of maps, where each map has the length and null + count of an array +3. `buffers`: A list of maps, where each map has the length and the + offset (from the beginning of the message body) of a buffer of an + array. + +4. `compression`: The compression applied on the body of the Record + Batch. Can either by `undefined` (i.e. no compression), `zstd` for + [Zstandard](https://facebook.github.io/zstd/), or `lz4_frame` for [LZ4 + Frame](https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md). + Defaults to `undefined`. + +Currently, compression is not supported, but it has been added for forwards +comapatibility. Review Comment: Spelling in moduledoc: “comapatibility” should be “compatibility”. ########## src/arrow_ipc_schema.erl: ########## @@ -15,58 +15,48 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides a record and functions to deal with Schemas -%% -%% A Schema[1] represents a table, or a list of arrays of equal length. This -%% module provides a record and a function to manage all the metadata required -%% to represent a schema. Metadata such as: -%% -%% <ol> -%% <li> -%% `endianness': The Endianness of the table. One of `little' or `big'. -%% Defaults to `little'. -%% </li> -%% <li> -%% `fields': The list of fields[2] in a table. -%% </li> -%% <li> -%% `type': The Layout of the column -%% </li> -%% <li> -%% `custom_metadata': A list of custom metadata in key-value format -%% </li> -%% <li> -%% `features': Any features used by the table which may not be present in -%% other implementations of Arrow. -%% </li> -%% </ol> -%% -%% Currently, big endianness, custom metadata and features are not supported, -%% but they have been added for forwards comapatibility. -%% -%% You can find Schemas in the Arrow spec here[3]. -%% -%% [1]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530] -%% -%% [2]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492] -%% -%% [3]: [https://arrow.apache.org/docs/format/Columnar.html#schema-message] -%% @end -module(arrow_ipc_schema). +-moduledoc """ +Provides a record and functions to deal with Schemas A +[Schema](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530)[1] +represents a table, or a list of arrays of equal length. This module provides a +record and a function to manage all the metadata required to represent a schema. Review Comment: The moduledoc has a leftover footnote marker (`[1]`) and a missing sentence break (“Schemas A ...”), which hurts readability in generated docs. ########## src/arrow_bitmap.erl: ########## @@ -15,57 +15,49 @@ % specific language governing permissions and limitations % under the License. -%% @doc Validity Bitmap implementation for `arrow'. -%% -%% Defines a function `validity_bitmap/1' to return the Validity Bitmap[1] along -%% with the Null Count[2], of an Array. -%% -%% An important thing to consider about our implementation of the Null Count is -%% that we need to support both `undefined' and `nil' as null values as they are -%% the conventions for null values in Erlang, and Elixir respectively. -%% -%% There are 5 important characteristics to remember about the validity bitmap: -%% -%% <ol> -%% <li> -%% A null value is represented by a 0 bit, and a non null value by a 1 bit. -%% </li> -%% <li> -%% Every 8 elements's validities are batched into a byte, which are then -%% reversed as Arrow uses least-significant bit (LSB) numbering (more in -%% attached reference). -%% </li> -%% <li> -%% If a "batch" consists of less than 8 elements, its validity needs to be -%% padded by 0 bits so that it can make a byte. -%% </li> -%% <li> -%% Each byte is stored in a slot of a Buffer (see docs for -%% `arrow_buffer'). This buffer with the validities of each batch of 8 -%% elements make up what is called the Validity Bitmap. -%% </li> -%% <li> -%% If the Null Count is 0, we can allocate the Validity Bitmap as a NULL -%% pointer (which in Erlang's case is `undefined'). -%% </li> -%% </ol> -%% [1]: [https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps] -%% -%% [2]: [https://arrow.apache.org/docs/format/Columnar.html#null-count] -%% @end -module(arrow_bitmap). --export([validity_bitmap/1]). +-moduledoc """ +Validity Bitmap implementation for `arrow`. + +Defines a function `validity_bitmap/1` to return the [Validity +Bitmap](https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps) +along with the [Null +Count](https://arrow.apache.org/docs/format/Columnar.html#null-count), of an +Array. + +An important thing to consider about our implementation of the Null Count is +that we need to support both `undefined` and `nil` as null values as they are +the conventions for null values in Erlang, and Elixir respectively. + +There are 5 important characteristics to remember about the validity bitmap: --include("arrow_buffer.hrl"). +1. A null value is represented by a 0 bit, and a non null value by a 1 bit. + +2. Every 8 elements's validities are batched into a byte, which are then + reversed as Arrow uses least-significant bit (LSB) numbering (more in + attached reference). + +3. If a "batch" consists of less than 8 elements, its validity needs to be + padded by 0 bits so that it can make a byte. + +4. Each byte is stored in a slot of a Buffer (see docs for `arrow_buffer`). This + buffer with the validities of each batch of 8 elements make up what is called + the Validity Bitmap. + +5. If the Null Count is 0, we can allocate the Validity Bitmap as a NULL pointer + (which in Erlang's case is `undefined`). +""". +-export([validity_bitmap/1]). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Validity Bitmap & Null Count %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%% @doc Returns the Validity Bitmap along with the Null Count, of -%% an Array. +-doc """ +Returns the Validity Bitmap along with the Null Count, of an Array. +""". -spec validity_bitmap(Value :: [arrow_type:native_type()] | list()) -> - {Bitmap :: #buffer{}, non_neg_integer()}. + {Bitmap :: arrow_buffer:buffer(), non_neg_integer()}. Review Comment: `validity_bitmap/1` returns `{undefined, 0}` when there are no nulls, but the typespec currently states the bitmap is always `arrow_buffer:buffer()`. This is a spec/implementation mismatch (and will cause Dialyzer warnings). ########## src/arrow_type.erl: ########## @@ -15,123 +15,94 @@ % specific language governing permissions and limitations % under the License. -%%%----------------------------------------------------------------------------- -%% @doc Provides representation for as well as functions for working with Arrow -%% specific datatypes. -%% -%% This module provides a longhand (a 2 tuple) as well as a shorthand (an atom) -%% represenations for representing a primitive types. Additionally it represents -%% Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types -%% that are supported by this package). -%% -%% Generally, when we say an array A has type X, we mean that each element in A -%% has type X. Thus, an array of type of `Int8' would look like `[1, 2, 3]', but -%% an array of type `List<Int8>' would look like `[[1, 2], [3]]'. -%% -%% == Primitive Types == -%% -%% A Primitive Type is any type that refers to a single value (as opposed to a -%% list of values). Currently it includes Booleans, Signed Integers, Unsigned -%% Integers, Floats and Binaries. Both a longhand and shorthand syntax had been -%% provided to initiate a type. -%% -%% The following is a comprehensive list of all supported primitive types: -%% -%% <ol> -%% <li>Booleans</li> -%% <li>Signed Integers: Includes `Int 8', `Int 16', `Int 32' and `Int 64'</li> -%% <li> -%% Unsigned Integers: Includes `UInt 8', `UInt 16', `UInt 32' and `UInt 64' -%% </li> -%% <li>Floats: Includes `Float 16', `Float 32' and `Float 64'</li> -%% <li>Binaries</li> -%% </ol> -%% -%% === Shorthand Syntax === -%% -%% In the shorthand syntax, you can just use a single atom to refer to a type, -%% such that the atom is `<name><size>'. The name is a single letter that refers -%% to the type, and the size is that type's size in bits. Take the example of -%% `Int 8'. Its name is `s', and size is `8'. Thus in shorthand, it is `s8'. In -%% the case of datatypes without variations, such as Booleans and Binaries, they -%% are represented by just their datatype. Do note that internally the longhand -%% syntax is used. -%% -%% Here you can find the name of each type: -%% -%% <table border="1"> -%% <tr> -%% <td>Type</td> -%% <td>Name</td> -%% <td>Example Atom</td> -%% </tr> -%% <tr> -%% <td>Booleans</td> -%% <td>N/A</td> -%% <td>`bool'</td> -%% </tr> -%% <tr> -%% <td>Signed Integer</td> -%% <td>`s'</td> -%% <td>`s64'</td> -%% </tr> -%% <tr> -%% <td>Unsigned Integer</td> -%% <td>`u'</td> -%% <td>`u16'</td> -%% </tr> -%% <tr> -%% <td>Float</td> -%% <td>`f'</td> -%% <td>`f32'</td> -%% </tr> -%% <tr> -%% <td>Binaries</td> -%% <td>N/A</td> -%% <td>`bin'</td> -%% </tr> -%% </table> -%% -%% Types that are shorthand are under the `t:arrow_shorthand_type()' and -%% `arrow_short_*()' types. -%% -%% === Longhand Syntax === -%% -%% In longhand syntax, you use a 2 tuple to to refer to the type, such that the -%% tuple is `{Name, Size}'. Thus, `Int 8' would be represented as `{s, 8}'. Do -%% note that in the case or Booleans and Binaries their `Size' is `undefined', as -%% the have variable or undefined size. -%% -%% Types that are longhand are under the `t:arrow_longhand_type()' and -%% `arrow_long_*()' types. -%% -%% == Nested Type == -%% -%% A Nested Type is any data structure that supports nesting. This can include -%% Lists, FixedLists, Maps, Structs and more, but is curently limited to -%% FixedLists. Nested Types are represented using a 3 tuple: `{NestedType, Type, -%% Length}'. `NestedType' refers to the data structure that is nesting your -%% primitive type. `Type' refers to the type that is being nested in the data -%% structure. Do note that `Type' can be a Primitive Type, or another Nested -%% Type. Both the shorthand and the longhand syntax can be used or Primitive -%% Types.`Length' refers to the length of each element, and will only be -%% required to represent FixedLists. -%% -%% It is important to remember while nesting that the type refers to the type of -%% each element in an array. A list of `Int8' is 2 dimensional, and a list of -%% `List<Int8>' is 3 dimensional. -%% -%% == Native Type == -%% -%% A Native Type is any Erlang Datatype that can be serialized by `arrow', -%% and is represented by the type `t:native_type()'. It currently only includes -%% values that are primitive in nature such as booleans, integers, floats, -%% binaries, and additionally the atoms `undefined' and `nil' to represent a -%% `NULL' value. -%% -%% @end -%%%----------------------------------------------------------------------------- -module(arrow_type). +-moduledoc """ +Provides representation for as well as functions for working with Arrow +specific datatypes. + +This module provides a longhand (a 2 tuple) as well as a shorthand (an atom) +represenations for representing a primitive types. Additionally it represents +Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types +that are supported by this package). Review Comment: There are a couple of spelling/grammar errors in the moduledoc (“represenations”, “a primitive types”) that will show up in generated docs. ########## src/arrow_variable_list_array.erl: ########## @@ -15,30 +15,29 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides support for Arrow's Fixed-Size List Layout. -%% -%% This module provides support for the Fixed-Size List Layout[1], which is an -%% layout that supports storing a list of lists of a specific length and -%% nesting. -%% -%% == Invalid Input == -%% -%% It is important that care is taken when passing input values to this module. -%% For performance reasons, the input is not validated. The function crashes on -%% nesting that is inconsitent: a. with the type, b. between elements. The -%% behaviour on invalid input <strong>CANNOT BE GUARANTEED</strong>. Therefore, -%% one must be careful to not to <strong>CRASH THE PROCESS</strong> or worse -%% still, <strong>PRODUCE INVALID OUTPUT</strong>. -%% -%% Any input must follow the following rules: -%% -%% <ol> -%% <li>The nesting of each element must be consistent with the type</li> -%% <li>The nesting of each element must be consistent with each other</li> -%% </ol> -%% -%% [1]: https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout -module(arrow_variable_list_array). +-moduledoc """ +Provides support for Arrow's Fixed-Size List Layout. + +This module provides support for the [Fixed-Size List +Layout](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout), +which is an layout that supports storing a list of lists of a specific length +and nesting. + +## Invalid Input + +It is important that care is taken when passing input values to this +module. For performance reasons, the input is not validated. The +function crashes on nesting that is inconsitent: a. with the type, b. Review Comment: Spelling in moduledoc: “inconsitent” should be “inconsistent”. ########## src/arrow_fixed_list_array.erl: ########## @@ -15,50 +15,46 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides support for Arrow's Fixed-Size List Layout. -%% -%% This module provides support for the Fixed-Size List Layout[1], which is an -%% layout that supports storing a list of lists of a specific length and -%% nesting. -%% -%% == Invalid Input == -%% -%% It is important that care is taken when passing input values to this module. -%% For performance reasons, the input is not validated. The function crashes on -%% nesting that is inconsitent: a. with the type, b. between elements. The lists -%% are checked to have the same length as each other at the top level, but not -%% for deeper levels. The behaviour of the module on invalid input -%% <strong>CANNOT BE GUARANTEED</strong>. Therefore, one must be careful to not -%% to <strong>CRASH THE PROCESS</strong> or worse still, <strong>PRODUCE INVALID -%% OUTPUT</strong>. -%% -%% Any input must follow the following rules: -%% -%% <ol> -%% <li>The length of each element must be consistent with the type</li> -%% <li>The nesting of each element must be consistent with the type</li> -%% <li>The length of each element must be consistent with each other</li> -%% <li>The nesting of each element must be consistent with each other</li> -%% <li> -%% The nested type is a `fixed_list' (as only then can fixed size be -%% guaranteed) -%% </li> -%% </ol> -%% -%% [1]: https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout -module(arrow_fixed_list_array). +-moduledoc """ +Provides support for Arrow's Fixed-Size List Layout. + +This module provides support for the [Fixed-Size List +Layout](https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout), +which is an layout that supports storing a list of lists of a specific length +and nesting. + +## Invalid Input + +It is important that care is taken when passing input values to this module. For +performance reasons, the input is not validated. The function crashes on nesting +that is inconsitent: a. with the type, b. between elements. The lists are Review Comment: Spelling in moduledoc: “inconsitent” should be “inconsistent”. ########## src/arrow_ipc_message.erl: ########## @@ -15,73 +15,79 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides a record and functions to deal with the Encapsulated Message -%% Format. -%% -%% A Message is a serialized form of a Schema[1] or a RecordBatch[2] (which may -%% be required to read a serialized array) along with some metadata. This module -%% provides a record and a function to manage all the metadata required to -%% represent a message. Metadata such as: -%% -%% <ol> -%% <li> -%% `version': The Apache Arrow Format Version. One of v1..v5. Defaults to v5. -%% </li> -%% <li> -%% `header': The metadata of the Schema or RecordBatch -%% </li> -%% <li> -%% `body_length': The length of the body in bytes -%% </li> -%% <li> -%% `custom_metadata': A list of custom metadata in key-value format -%% </li> -%% <li> -%% `body': The actual body. Can be undefined (in the case of Schema) or a -%% binary (in the case of Record Batch). -%% </li> -%% </ol> -%% -%% Currently, changing the version and custom metadata are not supported, but -%% they have been added for forwards comapatibility. -%% -%% This module also provides the `to_ipc/1' function which serializes the -%% message into the Encapsulated Message Format[3]. However, this function gives -%% <em>incomplete output</em> with invalid metadata, because of an unsatisfied -%% dependency on flatbuffers, which is required for serializing the metadata. -%% -%% [1]: [https://arrow.apache.org/docs/format/Columnar.html#schema-message] -%% -%% [2]: [https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message] -%% -%% [3]: [https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format] -%% @end -module(arrow_ipc_message). +-moduledoc """ +Provides a record and functions to deal with the Encapsulated Message +Format. + +A Message is a serialized form of a +[Schema](https://arrow.apache.org/docs/format/Columnar.html#schema-message) or a +[RecordBatch](https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message) +(which may be required to read a serialized array) along with some metadata. +This module provides a record and a function to manage all the metadata required +to represent a message. Metadata such as: + +1. `version`: The Apache Arrow Format Version. One of `v1..v5`. + Defaults to v5. +2. `header`: The metadata of the Schema or RecordBatch +3. `body_length`: The length of the body in bytes +4. `custom_metadata`: A list of custom metadata in key-value format +5. `body`: The actual body. Can be undefined (in the case of Schema) + or a binary (in the case of Record Batch). + +Currently, changing the version and custom metadata are not supported, but they +have been added for forwards comapatibility. Review Comment: Spelling in moduledoc: “comapatibility” should be “compatibility”. ########## src/arrow_ipc_field.erl: ########## @@ -15,64 +15,64 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides a record and functions to deal with individual fields in -%% a Schema -%% -%% A Field[1] represents a single column in a table (which is represented by a -%% Schema[2]). This module provides a record and a function to manage all the -%% metadata required to represent a column. Metadata such as: -%% -%% <ol> -%% <li> -%% `name': The Name of the column. This can either be a string or null. -%% </li> -%% <li> -%% `nullable': A boolean representing whether a column can have null values. -%% Generally true. -%% </li> -%% <li> -%% `type': The Type of the column, and is of type -%% `arrow_ipc_type:ipc_type()' -%% </li> -%% <li> -%% `dictionary': A boolean representing if the column is dictionary encoded -%% </li> -%% <li> -%% `children': The child fields of a nested datatype -%% </li> -%% <li> -%% `custom_metadata': A list of custom metadata in key-value format -%% </li> -%% </ol> -%% -%% Currently, dictionary encoding and custom metadata are not supported, but -%% they have been added for forwards comapatibility. -%% -%% [1]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492] -%% -%% [2]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530] -%% @end -module(arrow_ipc_field). +-moduledoc """ +Provides a record and functions to deal with individual fields in +a Schema + +A +[Field](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492) +represents a single column in a table (which is represented by a +[Schema](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530). +This module provides a record and a function to manage all the metadata required +to represent a column. Metadata such as: + +1. `name`: The Name of the column. This can either be a string or + null. +2. `nullable`: A boolean representing whether a column can have null + values. Generally true. +3. `type`: The Type of the column, and is of type + `t:arrow_ipc_type:ipc_type/0` +4. `dictionary`: A boolean representing if the column is dictionary + encoded +5. `children`: The child fields of a nested datatype +6. `custom_metadata`: A list of custom metadata in key-value format + +Currently, dictionary encoding and custom metadata are not supported, but +they have been added for forwards comapatibility. Review Comment: Spelling in moduledoc: “comapatibility” should be “compatibility”. ########## src/arrow_ipc_record_batch.erl: ########## @@ -15,57 +15,47 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides a record and functions to deal with RecordBatches -%% -%% A RecordBatch[1] represents a list of equal length arrays and their -%% coresponding buffers. This module provides a record and a function to manage -%% all the metadata required to represent a RecordBatch. Metadata such as: -%% -%% <ol> -%% <li> -%% `length': The number of rows or records. In other words, the length of -%% an array. -%% </li> -%% <li> -%% `nodes': A list of maps, where each map has the length and null count -%% of an array -%% </li> -%% <li> -%% `buffers': A list of maps, where each map has the length and the offset -%% (from the beginning of the message body) of a buffer of an array. -%% </li> -%% <li> -%% `compression': The compression applied on the body of the Record Batch. -%% Can either by `undefined' (i.e. no compression), `zstd' for Zstandard[2], -%% or `lz4_frame' for LZ4 Frame[3]. Defaults to `undefined'. -%% </li> -%% </ol> -%% -%% Currently, compression is not supported, but it has been added for forwards -%% comapatibility. -%% -%% You can find RecordBatches in the Arrow spec here[4]. -%% -%% [1]: [https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102] -%% -%% [2]: [https://facebook.github.io/zstd/] -%% -%% [3]: [https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md] -%% -%% [4]: [https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message] -%% @end -module(arrow_ipc_record_batch). +-moduledoc """ +Provides a record and functions to deal with RecordBatches +A [RecordBatch](https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102) represents a list of equal length arrays and their +coresponding buffers. This module provides a record and a function to manage +all the metadata required to represent a RecordBatch. Metadata such as: + +1. `length`: The number of rows or records. In other words, the + length of an array. +2. `nodes`: A list of maps, where each map has the length and null + count of an array +3. `buffers`: A list of maps, where each map has the length and the + offset (from the beginning of the message body) of a buffer of an + array. + +4. `compression`: The compression applied on the body of the Record + Batch. Can either by `undefined` (i.e. no compression), `zstd` for Review Comment: Grammar in moduledoc: “Can either by” should be “Can either be”. ########## src/arrow_fixed_list_array.erl: ########## @@ -15,50 +15,46 @@ % specific language governing permissions and limitations % under the License. -%% @doc Provides support for Arrow's Fixed-Size List Layout. -%% -%% This module provides support for the Fixed-Size List Layout[1], which is an -%% layout that supports storing a list of lists of a specific length and -%% nesting. -%% -%% == Invalid Input == -%% -%% It is important that care is taken when passing input values to this module. -%% For performance reasons, the input is not validated. The function crashes on -%% nesting that is inconsitent: a. with the type, b. between elements. The lists -%% are checked to have the same length as each other at the top level, but not -%% for deeper levels. The behaviour of the module on invalid input -%% <strong>CANNOT BE GUARANTEED</strong>. Therefore, one must be careful to not -%% to <strong>CRASH THE PROCESS</strong> or worse still, <strong>PRODUCE INVALID -%% OUTPUT</strong>. -%% -%% Any input must follow the following rules: -%% -%% <ol> -%% <li>The length of each element must be consistent with the type</li> -%% <li>The nesting of each element must be consistent with the type</li> -%% <li>The length of each element must be consistent with each other</li> -%% <li>The nesting of each element must be consistent with each other</li> -%% <li> -%% The nested type is a `fixed_list' (as only then can fixed size be -%% guaranteed) -%% </li> -%% </ol> -%% -%% [1]: https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout -module(arrow_fixed_list_array). +-moduledoc """ +Provides support for Arrow's Fixed-Size List Layout. + +This module provides support for the [Fixed-Size List +Layout](https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout), +which is an layout that supports storing a list of lists of a specific length Review Comment: Grammar in moduledoc: “an layout” should be “a layout”. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
