liurenjie1024 commented on code in PR #20: URL: https://github.com/apache/iceberg-rust/pull/20#discussion_r1282570157
########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), + /// Stored as 4-byte little-endian + Float(f32), + /// Stored as 8-byte little-endian + Double(f64), + /// Stores days from the 1970-01-01 in an 4-byte little-endian int + Date(i32), + /// Stores microseconds from midnight in an 8-byte little-endian long + Time(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + Timestamp(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + TimestampTZ(i64), Review Comment: For time values, how about using values from the `chrono` crate instead? Since these values are in memory representation, which should be easy to use? For example, implement display? You can refer to https://github.com/liurenjie1024/icelake/blob/aece12b6a9c9bd5c5d44ca70b24d43e4123f5d78/icelake/src/types/in_memory.rs#L163 https://github.com/icelake-io/icelake/blob/aece12b6a9c9bd5c5d44ca70b24d43e4123f5d78/icelake/src/types/in_memory.rs#L163 ########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), Review Comment: ```suggestion Long(i64), ``` ########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), + /// Stored as 4-byte little-endian + Float(f32), + /// Stored as 8-byte little-endian + Double(f64), + /// Stores days from the 1970-01-01 in an 4-byte little-endian int + Date(i32), + /// Stores microseconds from midnight in an 8-byte little-endian long + Time(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + Timestamp(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + TimestampTZ(i64), + /// UTF-8 bytes (without length) + String(String), + /// 16-byte big-endian value + UUID(i128), Review Comment: As above, use `Uuid` instead? ########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), + /// Stored as 4-byte little-endian + Float(f32), Review Comment: cc @ZENOTME If we are going to partition value specific ordering/hashing, do we still need `OrderedFloat`? ########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), + /// Stored as 4-byte little-endian + Float(f32), + /// Stored as 8-byte little-endian + Double(f64), + /// Stores days from the 1970-01-01 in an 4-byte little-endian int + Date(i32), + /// Stores microseconds from midnight in an 8-byte little-endian long + Time(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + Timestamp(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + TimestampTZ(i64), + /// UTF-8 bytes (without length) + String(String), + /// 16-byte big-endian value + UUID(i128), + /// Binary value + Fixed(usize, Vec<u8>), + /// Binary value (without length) + Binary(Vec<u8>), + /// Stores unscaled value as two’s-complement big-endian binary, + /// using the minimum number of bytes for the value + Decimal(Decimal), + /// A struct is a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. + /// Each field can be either optional or required, meaning that values can (or cannot) be null. Fields may be any type. + /// Fields may have an optional comment or doc string. Fields can have default values. + Struct(Struct), + /// A list is a collection of values with some element type. + /// The element field has an integer id that is unique in the table schema. + /// Elements can be either optional or required. Element types may be any type. + List(Vec<Option<Value>>), + /// A map is a collection of key-value pairs with a key type and a value type. + /// Both the key field and value field each have an integer id that is unique in the table schema. + /// Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types. + Map(HashMap<String, Option<Value>>), +} + +impl From<Value> for ByteBuf { + fn from(value: Value) -> Self { + match value { + Value::Boolean(val) => { + if val { + ByteBuf::from([0u8]) + } else { + ByteBuf::from([1u8]) + } + } + Value::Int(val) => ByteBuf::from(val.to_le_bytes()), + Value::LongInt(val) => ByteBuf::from(val.to_le_bytes()), + Value::Float(val) => ByteBuf::from(val.to_le_bytes()), + Value::Double(val) => ByteBuf::from(val.to_le_bytes()), + Value::Date(val) => ByteBuf::from(val.to_le_bytes()), + Value::Time(val) => ByteBuf::from(val.to_le_bytes()), + Value::Timestamp(val) => ByteBuf::from(val.to_le_bytes()), + Value::TimestampTZ(val) => ByteBuf::from(val.to_le_bytes()), + Value::String(val) => ByteBuf::from(val.as_bytes()), + Value::UUID(val) => ByteBuf::from(val.to_be_bytes()), + Value::Fixed(_, val) => ByteBuf::from(val), + Value::Binary(val) => ByteBuf::from(val), + _ => todo!(), + } + } +} + +/// The partition struct stores the tuple of partition values for each file. +/// Its type is derived from the partition fields of the partition spec used to write the manifest file. +/// In v2, the partition struct’s field ids must match the ids from the partition spec. +#[derive(Debug, Clone, PartialEq)] +pub struct Struct { + /// Vector to store the field values + fields: Vec<Option<Value>>, + /// A lookup that matches the field name to the entry in the vector + lookup: HashMap<String, usize>, +} + +impl Deref for Struct { + type Target = [Option<Value>]; + + fn deref(&self) -> &Self::Target { + &self.fields + } +} + +impl Struct { + /// Get reference to partition value + pub fn get(&self, name: &str) -> Option<&Value> { Review Comment: I'm wondering if we really need a `get` by name method. Java version's `StructLike` only allows to access by position: https://github.com/apache/iceberg/blob/1bb853191fd378fb1dfda5a5cb297475b7fc204b/api/src/main/java/org/apache/iceberg/StructLike.java#L26 I think maybe wrapping `StructType` in another data structure may avoid keeping the lookup table? It's quite memory inefficient to keep a lookup table for every struct. ########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), + /// Stored as 4-byte little-endian + Float(f32), + /// Stored as 8-byte little-endian + Double(f64), + /// Stores days from the 1970-01-01 in an 4-byte little-endian int + Date(i32), + /// Stores microseconds from midnight in an 8-byte little-endian long + Time(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + Timestamp(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + TimestampTZ(i64), + /// UTF-8 bytes (without length) + String(String), + /// 16-byte big-endian value + UUID(i128), + /// Binary value + Fixed(usize, Vec<u8>), Review Comment: Should we really keep `usize`? 🤔 ########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), + /// Stored as 4-byte little-endian + Float(f32), + /// Stored as 8-byte little-endian + Double(f64), + /// Stores days from the 1970-01-01 in an 4-byte little-endian int + Date(i32), + /// Stores microseconds from midnight in an 8-byte little-endian long + Time(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + Timestamp(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + TimestampTZ(i64), + /// UTF-8 bytes (without length) + String(String), + /// 16-byte big-endian value + UUID(i128), + /// Binary value + Fixed(usize, Vec<u8>), + /// Binary value (without length) + Binary(Vec<u8>), + /// Stores unscaled value as two’s-complement big-endian binary, + /// using the minimum number of bytes for the value + Decimal(Decimal), + /// A struct is a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. + /// Each field can be either optional or required, meaning that values can (or cannot) be null. Fields may be any type. + /// Fields may have an optional comment or doc string. Fields can have default values. + Struct(Struct), + /// A list is a collection of values with some element type. + /// The element field has an integer id that is unique in the table schema. + /// Elements can be either optional or required. Element types may be any type. + List(Vec<Option<Value>>), + /// A map is a collection of key-value pairs with a key type and a value type. + /// Both the key field and value field each have an integer id that is unique in the table schema. + /// Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types. + Map(HashMap<String, Option<Value>>), +} + +impl From<Value> for ByteBuf { + fn from(value: Value) -> Self { + match value { + Value::Boolean(val) => { + if val { + ByteBuf::from([0u8]) + } else { + ByteBuf::from([1u8]) + } + } + Value::Int(val) => ByteBuf::from(val.to_le_bytes()), + Value::LongInt(val) => ByteBuf::from(val.to_le_bytes()), + Value::Float(val) => ByteBuf::from(val.to_le_bytes()), + Value::Double(val) => ByteBuf::from(val.to_le_bytes()), + Value::Date(val) => ByteBuf::from(val.to_le_bytes()), + Value::Time(val) => ByteBuf::from(val.to_le_bytes()), + Value::Timestamp(val) => ByteBuf::from(val.to_le_bytes()), + Value::TimestampTZ(val) => ByteBuf::from(val.to_le_bytes()), + Value::String(val) => ByteBuf::from(val.as_bytes()), + Value::UUID(val) => ByteBuf::from(val.to_be_bytes()), + Value::Fixed(_, val) => ByteBuf::from(val), + Value::Binary(val) => ByteBuf::from(val), + _ => todo!(), + } + } +} + +/// The partition struct stores the tuple of partition values for each file. +/// Its type is derived from the partition fields of the partition spec used to write the manifest file. +/// In v2, the partition struct’s field ids must match the ids from the partition spec. +#[derive(Debug, Clone, PartialEq)] +pub struct Struct { + /// Vector to store the field values + fields: Vec<Option<Value>>, + /// A lookup that matches the field name to the entry in the vector + lookup: HashMap<String, usize>, Review Comment: This may introduce much memory overhead. ########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), + /// Stored as 4-byte little-endian + Float(f32), + /// Stored as 8-byte little-endian + Double(f64), + /// Stores days from the 1970-01-01 in an 4-byte little-endian int + Date(i32), + /// Stores microseconds from midnight in an 8-byte little-endian long + Time(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + Timestamp(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + TimestampTZ(i64), + /// UTF-8 bytes (without length) + String(String), + /// 16-byte big-endian value + UUID(i128), + /// Binary value + Fixed(usize, Vec<u8>), + /// Binary value (without length) + Binary(Vec<u8>), + /// Stores unscaled value as two’s-complement big-endian binary, + /// using the minimum number of bytes for the value + Decimal(Decimal), + /// A struct is a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. + /// Each field can be either optional or required, meaning that values can (or cannot) be null. Fields may be any type. + /// Fields may have an optional comment or doc string. Fields can have default values. + Struct(Struct), + /// A list is a collection of values with some element type. + /// The element field has an integer id that is unique in the table schema. + /// Elements can be either optional or required. Element types may be any type. + List(Vec<Option<Value>>), + /// A map is a collection of key-value pairs with a key type and a value type. + /// Both the key field and value field each have an integer id that is unique in the table schema. + /// Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types. + Map(HashMap<String, Option<Value>>), +} + +impl From<Value> for ByteBuf { + fn from(value: Value) -> Self { + match value { + Value::Boolean(val) => { + if val { + ByteBuf::from([0u8]) + } else { + ByteBuf::from([1u8]) + } + } + Value::Int(val) => ByteBuf::from(val.to_le_bytes()), + Value::LongInt(val) => ByteBuf::from(val.to_le_bytes()), + Value::Float(val) => ByteBuf::from(val.to_le_bytes()), + Value::Double(val) => ByteBuf::from(val.to_le_bytes()), + Value::Date(val) => ByteBuf::from(val.to_le_bytes()), + Value::Time(val) => ByteBuf::from(val.to_le_bytes()), + Value::Timestamp(val) => ByteBuf::from(val.to_le_bytes()), + Value::TimestampTZ(val) => ByteBuf::from(val.to_le_bytes()), + Value::String(val) => ByteBuf::from(val.as_bytes()), + Value::UUID(val) => ByteBuf::from(val.to_be_bytes()), + Value::Fixed(_, val) => ByteBuf::from(val), + Value::Binary(val) => ByteBuf::from(val), + _ => todo!(), Review Comment: How about create a type `PrimitiveValue` so that we can have type safety? ########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), + /// Stored as 4-byte little-endian + Float(f32), + /// Stored as 8-byte little-endian + Double(f64), + /// Stores days from the 1970-01-01 in an 4-byte little-endian int + Date(i32), + /// Stores microseconds from midnight in an 8-byte little-endian long + Time(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + Timestamp(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + TimestampTZ(i64), + /// UTF-8 bytes (without length) + String(String), + /// 16-byte big-endian value + UUID(i128), + /// Binary value + Fixed(usize, Vec<u8>), Review Comment: Should we really keep `usize`? 🤔 ########## crates/iceberg/src/spec/values.rs: ########## @@ -0,0 +1,417 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +/*! + * Value in iceberg + */ + +use std::{any::Any, collections::HashMap, fmt, ops::Deref}; + +use rust_decimal::Decimal; +use serde::{ + de::{MapAccess, Visitor}, + ser::SerializeStruct, + Deserialize, Deserializer, Serialize, +}; +use serde_bytes::ByteBuf; + +use crate::Error; + +use super::datatypes::{PrimitiveType, Type}; + +/// Values present in iceberg type +#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)] +#[serde(untagged)] +pub enum Value { + /// 0x00 for false, non-zero byte for true + Boolean(bool), + /// Stored as 4-byte little-endian + Int(i32), + /// Stored as 8-byte little-endian + LongInt(i64), + /// Stored as 4-byte little-endian + Float(f32), + /// Stored as 8-byte little-endian + Double(f64), + /// Stores days from the 1970-01-01 in an 4-byte little-endian int + Date(i32), + /// Stores microseconds from midnight in an 8-byte little-endian long + Time(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + Timestamp(i64), + /// Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long + TimestampTZ(i64), + /// UTF-8 bytes (without length) + String(String), + /// 16-byte big-endian value + UUID(i128), + /// Binary value + Fixed(usize, Vec<u8>), + /// Binary value (without length) + Binary(Vec<u8>), + /// Stores unscaled value as two’s-complement big-endian binary, + /// using the minimum number of bytes for the value + Decimal(Decimal), + /// A struct is a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. + /// Each field can be either optional or required, meaning that values can (or cannot) be null. Fields may be any type. + /// Fields may have an optional comment or doc string. Fields can have default values. + Struct(Struct), + /// A list is a collection of values with some element type. + /// The element field has an integer id that is unique in the table schema. + /// Elements can be either optional or required. Element types may be any type. + List(Vec<Option<Value>>), + /// A map is a collection of key-value pairs with a key type and a value type. + /// Both the key field and value field each have an integer id that is unique in the table schema. + /// Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types. + Map(HashMap<String, Option<Value>>), Review Comment: I think we should have `Value` as key here following iceberg's spec. Only allows `String` as key is the restriction of avro, not iceberg. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
