alamb commented on code in PR #1840: URL: https://github.com/apache/arrow-rs/pull/1840#discussion_r894740032
########## arrow/src/lib.rs: ########## @@ -15,114 +15,215 @@ // specific language governing permissions and limitations // under the License. -//! A native Rust implementation of [Apache Arrow](https://arrow.apache.org), a cross-language +//! A complete, safe, native Rust implementation of [Apache Arrow](https://arrow.apache.org), a cross-language //! development platform for in-memory data. //! -//! ### DataType +//! # Columnar Format //! -//! Every [`Array`](array::Array) in this crate has an associated [`DataType`](datatypes::DataType), -//! that specifies how its data is layed in memory and represented. -//! Thus, a central enum of this crate is [`DataType`](datatypes::DataType), that contains the set of valid -//! DataTypes in the specification. For example, [`DataType::Utf8`](datatypes::DataType::Utf8). +//! The [`array`] module provides statically typed implementations of all the array +//! types as defined by the [Arrow Columnar Format](https://arrow.apache.org/docs/format/Columnar.html). //! -//! ## Array -//! -//! The central trait of this package is the dynamically-typed [`Array`](array::Array) that -//! represents a fixed-sized, immutable, Send + Sync Array of nullable elements. An example of such an array is [`UInt32Array`](array::UInt32Array). -//! One way to think about an arrow [`Array`](array::Array) is a `Arc<[Option<T>; len]>` where T can be anything ranging from an integer to a string, or even -//! another [`Array`](array::Array). -//! -//! [`Arrays`](array::Array) have [`len()`](array::Array::len), [`data_type()`](array::Array::data_type), and the nullability of each of its elements, -//! can be obtained via [`is_null(index)`](array::Array::is_null). To downcast an [`Array`](array::Array) to a specific implementation, you can use +//! For example, an [`Int32Array`](array::Int32Array) represents a nullable array of `i32` //! //! ```rust -//! use arrow::array::{Array, UInt32Array}; -//! let array = UInt32Array::from(vec![Some(1), None, Some(3)]); +//! # use arrow::array::{Array, Int32Array}; +//! let array = Int32Array::from(vec![Some(1), None, Some(3)]); //! assert_eq!(array.len(), 3); //! assert_eq!(array.value(0), 1); //! assert_eq!(array.is_null(1), true); -//! ``` //! -//! To make the array dynamically typed, we wrap it in an [`Arc`](std::sync::Arc): -//! -//! ```rust -//! # use std::sync::Arc; -//! use arrow::datatypes::DataType; -//! use arrow::array::{UInt32Array, ArrayRef}; -//! # let array = UInt32Array::from(vec![Some(1), None, Some(3)]); -//! let array: ArrayRef = Arc::new(array); -//! assert_eq!(array.len(), 3); -//! // array.value() is not available in the dynamically-typed version -//! assert_eq!(array.is_null(1), true); -//! assert_eq!(array.data_type(), &DataType::UInt32); +//! let collected: Vec<_> = array.iter().collect(); +//! assert_eq!(collected, vec![Some(1), None, Some(3)]); +//! assert_eq!(array.values(), [1, 0, 3]) //! ``` //! -//! to downcast, use `as_any()`: +//! It is also possible to write generic code. For example, the following is generic over +//! all primitively typed arrays: //! //! ```rust -//! # use std::sync::Arc; -//! # use arrow::array::{UInt32Array, ArrayRef}; -//! # let array = UInt32Array::from(vec![Some(1), None, Some(3)]); -//! # let array: ArrayRef = Arc::new(array); -//! let array = array.as_any().downcast_ref::<UInt32Array>().unwrap(); -//! assert_eq!(array.value(0), 1); +//! # use std::iter::Sum; +//! # use arrow::array::{Float32Array, PrimitiveArray, TimestampNanosecondArray}; +//! # use arrow::datatypes::ArrowPrimitiveType; +//! # +//! fn sum<T: ArrowPrimitiveType>(array: &PrimitiveArray<T>) -> T::Native +//! where +//! T: ArrowPrimitiveType, +//! T::Native: Sum +//! { +//! array.iter().map(|v| v.unwrap_or_default()).sum() +//! } +//! +//! assert_eq!(sum(&Float32Array::from(vec![1.1, 2.9, 3.])), 7.); +//! assert_eq!(sum(&TimestampNanosecondArray::from(vec![1, 2, 3])), 6); //! ``` //! -//! ## Memory and Buffers +//! For more examples, consult the [`array`] docs. //! -//! Data in [`Array`](array::Array) is stored in [`ArrayData`](array::ArrayData), that in turn -//! is a collection of other [`ArrayData`](array::ArrayData) and [`Buffers`](buffer::Buffer). -//! [`Buffers`](buffer::Buffer) is the central struct that array implementations use to keep allocated memory and pointers. -//! The [`MutableBuffer`](buffer::MutableBuffer) is the mutable counter-part of [`Buffer`](buffer::Buffer). -//! These are the lowest abstractions of this crate, and are used throughout the crate to -//! efficiently allocate, write, read and deallocate memory. +//! # Type Erasure / Trait Objects //! -//! ## Field, Schema and RecordBatch +//! It is often the case that code wishes to handle any type of array, without necessarily knowing Review Comment: thank you for this -- it is a good contribution ########## arrow/src/array/mod.rs: ########## @@ -15,40 +15,66 @@ // specific language governing permissions and limitations // under the License. -//! The central type in Apache Arrow are arrays, represented -//! by the [`Array` trait](crate::array::Array). -//! An array represents a known-length sequence of values all -//! having the same type. +//! The central type in Apache Arrow are arrays, which are a known-length sequence of values +//! all having the same type. This module provides concrete implementations of each type, as +//! well as an [`Array`] trait that can be used for type-erasure. //! -//! Internally, those values are represented by one or several -//! [buffers](crate::buffer::Buffer), the number and meaning -//! of which depend on the array’s data type, as documented in -//! [the Arrow data layout specification](https://arrow.apache.org/docs/format/Columnar.html). -//! For example, the type `Int16Array` represents an Apache -//! Arrow array of 16-bit integers. +//! # Downcasting an Array //! -//! Those buffers consist of the value data itself and an -//! optional [bitmap buffer](crate::bitmap::Bitmap) that -//! indicates which array entries are null values. -//! The bitmap buffer can be entirely omitted if the array is -//! known to have zero null values. +//! Arrays are often passed around as a dynamically typed [`&dyn Array`] or [`ArrayRef`]. +//! For example, [`RecordBatch`](`crate::record_batch::RecordBatch`) stores columns as [`ArrayRef`]. //! -//! There are concrete implementations of this trait for each -//! data type, that help you access individual values of the -//! array. +//! Whilst these arrays can be passed directly to the [`compute`](crate::compute), +//! [`csv`](crate::csv), [`json`](crate::json), etc... APIs, it is often the case that you wish +//! to interact with the data directly. This requires downcasting to the concrete type of the array: +//! +//! ``` +//! # use arrow::array::{Array, Float32Array, Int32Array}; +//! # +//! fn sum_int32(array: &dyn Array) -> i32 { +//! let integers: &Int32Array = array.as_any().downcast_ref().unwrap(); +//! integers.iter().map(|val| val.unwrap_or_default()).sum() +//! } +//! +//! // Note: the values for positions corresponding to nulls will be arbitrary +//! fn as_f32_slice(array: &dyn Array) -> &[f32] { +//! array.as_any().downcast_ref::<Float32Array>().unwrap().values() +//! } +//! ``` //! //! # Building an Array //! -//! Arrow's `Arrays` are immutable, but there is the trait -//! [`ArrayBuilder`](crate::array::ArrayBuilder) -//! that helps you with constructing new `Arrays`. As with the -//! `Array` trait, there are builder implementations for all -//! concrete array types. +//! Most [`Array`] implementations can be constructed directly from iterators or [`Vec`] //! -//! # Example //! ``` -//! use arrow::array::Int16Array; +//! # use arrow::array::Int32Array; +//! # use arrow::array::StringArray; +//! # use arrow::array::ListArray; +//! # use arrow::datatypes::Int32Type; +//! # +//! Int32Array::from(vec![1, 2]); Review Comment: Agree -- this is very nice ########## arrow/src/lib.rs: ########## @@ -15,114 +15,215 @@ // specific language governing permissions and limitations // under the License. -//! A native Rust implementation of [Apache Arrow](https://arrow.apache.org), a cross-language +//! A complete, safe, native Rust implementation of [Apache Arrow](https://arrow.apache.org), a cross-language //! development platform for in-memory data. //! -//! ### DataType +//! # Columnar Format //! -//! Every [`Array`](array::Array) in this crate has an associated [`DataType`](datatypes::DataType), -//! that specifies how its data is layed in memory and represented. -//! Thus, a central enum of this crate is [`DataType`](datatypes::DataType), that contains the set of valid -//! DataTypes in the specification. For example, [`DataType::Utf8`](datatypes::DataType::Utf8). +//! The [`array`] module provides statically typed implementations of all the array +//! types as defined by the [Arrow Columnar Format](https://arrow.apache.org/docs/format/Columnar.html). //! -//! ## Array -//! -//! The central trait of this package is the dynamically-typed [`Array`](array::Array) that -//! represents a fixed-sized, immutable, Send + Sync Array of nullable elements. An example of such an array is [`UInt32Array`](array::UInt32Array). -//! One way to think about an arrow [`Array`](array::Array) is a `Arc<[Option<T>; len]>` where T can be anything ranging from an integer to a string, or even -//! another [`Array`](array::Array). -//! -//! [`Arrays`](array::Array) have [`len()`](array::Array::len), [`data_type()`](array::Array::data_type), and the nullability of each of its elements, -//! can be obtained via [`is_null(index)`](array::Array::is_null). To downcast an [`Array`](array::Array) to a specific implementation, you can use +//! For example, an [`Int32Array`](array::Int32Array) represents a nullable array of `i32` //! //! ```rust -//! use arrow::array::{Array, UInt32Array}; -//! let array = UInt32Array::from(vec![Some(1), None, Some(3)]); +//! # use arrow::array::{Array, Int32Array}; +//! let array = Int32Array::from(vec![Some(1), None, Some(3)]); //! assert_eq!(array.len(), 3); //! assert_eq!(array.value(0), 1); //! assert_eq!(array.is_null(1), true); -//! ``` //! -//! To make the array dynamically typed, we wrap it in an [`Arc`](std::sync::Arc): -//! -//! ```rust -//! # use std::sync::Arc; -//! use arrow::datatypes::DataType; -//! use arrow::array::{UInt32Array, ArrayRef}; -//! # let array = UInt32Array::from(vec![Some(1), None, Some(3)]); -//! let array: ArrayRef = Arc::new(array); -//! assert_eq!(array.len(), 3); -//! // array.value() is not available in the dynamically-typed version -//! assert_eq!(array.is_null(1), true); -//! assert_eq!(array.data_type(), &DataType::UInt32); +//! let collected: Vec<_> = array.iter().collect(); +//! assert_eq!(collected, vec![Some(1), None, Some(3)]); +//! assert_eq!(array.values(), [1, 0, 3]) //! ``` //! -//! to downcast, use `as_any()`: +//! It is also possible to write generic code. For example, the following is generic over +//! all primitively typed arrays: //! //! ```rust -//! # use std::sync::Arc; -//! # use arrow::array::{UInt32Array, ArrayRef}; -//! # let array = UInt32Array::from(vec![Some(1), None, Some(3)]); -//! # let array: ArrayRef = Arc::new(array); -//! let array = array.as_any().downcast_ref::<UInt32Array>().unwrap(); -//! assert_eq!(array.value(0), 1); +//! # use std::iter::Sum; +//! # use arrow::array::{Float32Array, PrimitiveArray, TimestampNanosecondArray}; +//! # use arrow::datatypes::ArrowPrimitiveType; +//! # +//! fn sum<T: ArrowPrimitiveType>(array: &PrimitiveArray<T>) -> T::Native +//! where +//! T: ArrowPrimitiveType, +//! T::Native: Sum +//! { +//! array.iter().map(|v| v.unwrap_or_default()).sum() +//! } +//! +//! assert_eq!(sum(&Float32Array::from(vec![1.1, 2.9, 3.])), 7.); +//! assert_eq!(sum(&TimestampNanosecondArray::from(vec![1, 2, 3])), 6); //! ``` //! -//! ## Memory and Buffers +//! For more examples, consult the [`array`] docs. //! -//! Data in [`Array`](array::Array) is stored in [`ArrayData`](array::ArrayData), that in turn -//! is a collection of other [`ArrayData`](array::ArrayData) and [`Buffers`](buffer::Buffer). -//! [`Buffers`](buffer::Buffer) is the central struct that array implementations use to keep allocated memory and pointers. -//! The [`MutableBuffer`](buffer::MutableBuffer) is the mutable counter-part of [`Buffer`](buffer::Buffer). -//! These are the lowest abstractions of this crate, and are used throughout the crate to -//! efficiently allocate, write, read and deallocate memory. +//! # Type Erasure / Trait Objects //! -//! ## Field, Schema and RecordBatch +//! It is often the case that code wishes to handle any type of array, without necessarily knowing +//! its concrete type. This use-case is catered for by a combination of [`Array`] +//! and [`DataType`](datatypes::DataType), with the former providing a type-erased container for +//! the array, and the latter identifying the concrete type of array. //! -//! [`Field`](datatypes::Field) is a struct that contains an array's metadata (datatype and whether its values -//! can be null), and a name. [`Schema`](datatypes::Schema) is a vector of fields with optional metadata. -//! Together, they form the basis of a schematic representation of a group of [`Arrays`](array::Array). +//! ```rust +//! # use arrow::array::{Array, Float32Array}; +//! # use arrow::array::StringArray; +//! # use arrow::datatypes::DataType; +//! # +//! fn impl_string(array: &StringArray) {} +//! fn impl_f32(array: &Float32Array) {} +//! +//! fn impl_dyn(array: &dyn Array) { +//! match array.data_type() { +//! DataType::Utf8 => impl_string(array.as_any().downcast_ref().unwrap()), +//! DataType::Float32 => impl_f32(array.as_any().downcast_ref().unwrap()), +//! _ => unimplemented!() +//! } +//! } +//! ``` //! -//! In fact, [`RecordBatch`](record_batch::RecordBatch) is a struct with a [`Schema`](datatypes::Schema) and a vector of -//! [`Array`](array::Array)s, all with the same `len`. A record batch is the highest order struct that this crate currently offers -//! and is broadly used to represent a table where each column in an `Array`. +//! It is also common to want to write a function that returns one of a number of possible +//! array implementations. [`ArrayRef`] is a type-alias for [`Arc<dyn Array>`](array::Array) +//! which is frequently used for this purpose //! -//! ## Compute +//! ```rust +//! # use std::str::FromStr; +//! # use std::sync::Arc; +//! # use arrow::array::{ArrayRef, Int32Array, PrimitiveArray}; +//! # use arrow::datatypes::{ArrowPrimitiveType, DataType, Int32Type, UInt32Type}; +//! # use arrow::compute::cast; +//! # +//! fn parse_to_primitive<'a, T, I>(iter: I) -> PrimitiveArray<T> +//! where +//! T: ArrowPrimitiveType, +//! I: IntoIterator<Item=&'a str>, +//! { +//! PrimitiveArray::from_iter(iter.into_iter().map(|val| T::Native::from_str(val).ok())) +//! } +//! +//! fn parse_strings<'a, I>(iter: I, to_data_type: DataType) -> ArrayRef +//! where +//! I: IntoIterator<Item=&'a str>, +//! { +//! match to_data_type { +//! DataType::Int32 => Arc::new(parse_to_primitive::<Int32Type, _>(iter)) as _, +//! DataType::UInt32 => Arc::new(parse_to_primitive::<UInt32Type, _>(iter)) as _, +//! _ => unimplemented!() +//! } +//! } +//! +//! let array = parse_strings(["1", "2", "3"], DataType::Int32); +//! let integers = array.as_any().downcast_ref::<Int32Array>().unwrap(); +//! assert_eq!(integers.values(), [1, 2, 3]) +//! ``` //! -//! This crate offers many operations (called kernels) to operate on [`Array`](array::Array)s, that you can find at [`Kernel`](compute::kernels). -//! It has both vertical and horizontal operations, and some of them have an SIMD implementation. +//! # Compute Kernels //! -//! ## Status +//! The [`compute`](compute) module provides optimised implementations of many common operations, +//! for example the `parse_strings` operation above could also be implemented as follows: //! -//! This crate has most of the implementation of the arrow specification. Specifically, it supports the following types: +//! ``` +//! # use std::sync::Arc; +//! # use arrow::error::Result; +//! # use arrow::array::{ArrayRef, StringArray, UInt32Array}; +//! # use arrow::datatypes::DataType; +//! # +//! fn parse_strings<'a, I>(iter: I, to_data_type: &DataType) -> Result<ArrayRef> +//! where +//! I: IntoIterator<Item=&'a str>, +//! { +//! let array = Arc::new(StringArray::from_iter(iter.into_iter().map(Some))) as _; +//! arrow::compute::cast(&array, to_data_type) +//! } +//! +//! let array = parse_strings(["1", "2", "3"], &DataType::UInt32).unwrap(); +//! let integers = array.as_any().downcast_ref::<UInt32Array>().unwrap(); +//! assert_eq!(integers.values(), [1, 2, 3]) +//! ``` //! -//! * All arrow primitive types, such as [`Int32Array`](array::UInt8Array), [`BooleanArray`](array::BooleanArray) and [`Float64Array`](array::Float64Array). -//! * All arrow variable length types, such as [`StringArray`](array::StringArray) and [`BinaryArray`](array::BinaryArray) -//! * All composite types such as [`StructArray`](array::StructArray) and [`ListArray`](array::ListArray) -//! * Dictionary types [`DictionaryArray`](array::DictionaryArray) - +//! This module also implements many common vertical operations: //! -//! This crate also implements many common vertical operations: -//! * all mathematical binary operators, such as [`subtract`](compute::kernels::arithmetic::subtract) -//! * all boolean binary operators such as [`equality`](compute::kernels::comparison::eq) +//! * All mathematical binary operators, such as [`subtract`](compute::kernels::arithmetic::subtract) +//! * All boolean binary operators such as [`equality`](compute::kernels::comparison::eq) //! * [`cast`](compute::kernels::cast::cast) //! * [`filter`](compute::kernels::filter::filter) //! * [`take`](compute::kernels::take::take) and [`limit`](compute::kernels::limit::limit) //! * [`sort`](compute::kernels::sort::sort) //! * some string operators such as [`substring`](compute::kernels::substring::substring) and [`length`](compute::kernels::length::length) //! -//! as well as some horizontal operations, such as +//! As well as some horizontal operations, such as: //! //! * [`min`](compute::kernels::aggregate::min) and [`max`](compute::kernels::aggregate::max) //! * [`sum`](compute::kernels::aggregate::sum) //! -//! Finally, this crate implements some readers and writers to different formats: +//! # Tabular Representation +//! +//! It is common to want to group one or more columns together into a tabular representation. This +//! is provided by [`RecordBatch`] which combines a [`Schema`](datatypes::Schema) +//! and a corresponding list of [`ArrayRef`]. //! -//! * JSON: [`Reader`](json::reader::Reader) +//! +//! ``` +//! # use std::sync::Arc; +//! # use arrow::array::{Float32Array, Int32Array}; +//! # use arrow::record_batch::RecordBatch; +//! # +//! let col_1 = Arc::new(Int32Array::from_iter([1, 2, 3])) as _; +//! let col_2 = Arc::new(Float32Array::from_iter([1., 6.3, 4.])) as _; +//! +//! let batch = RecordBatch::try_from_iter([("col1", col_1), ("col_2", col_2)]).unwrap(); +//! ``` +//! +//! # IO +//! +//! This crate provides readers and writers for various formats to/from [`RecordBatch`] +//! +//! * JSON: [`Reader`](json::reader::Reader) and [`Writer`](json::writer::Writer) //! * CSV: [`Reader`](csv::reader::Reader) and [`Writer`](csv::writer::Writer) //! * IPC: [`Reader`](ipc::reader::StreamReader) and [`Writer`](ipc::writer::FileWriter) //! -//! The parquet implementation is on a [separate crate](https://crates.io/crates/parquet) +//! Parquet is published as a [separate crate](https://crates.io/crates/parquet) +//! +//! # Memory and Buffers +//! +//! Advanced users may wish to interact with the underlying buffers of an [`Array`], for example, +//! for FFI or high-performance conversion from other formats. This interface is provided by +//! [`ArrayData`] which stores the [`Buffer`] comprising an [`Array`], and can be accessed +//! with [`Array::data`](array::Array::data) +//! +//! The APIs for constructing [`ArrayData`] come in safe, and unsafe variants, with the former +//! performing extensive, but potentially expensive validation to ensure the buffers are well-formed. +//! +//! An [`ArrayRef`] can be cheaply created from an [`ArrayData`] using [`make_array`], +//! or by using the appropriate [`From`] conversion on the concrete [`Array`] implementation. +//! +//! # Safety and Security +//! +//! Like many crates, this crate makes use of unsafe where prudent. However, it endeavours to be +//! sound. Specifically, **it should not be possible to trigger undefined behaviour using safe APIs.** Review Comment: 🎉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
