PDGGK commented on code in PR #1358:
URL: https://github.com/apache/paimon-rust/pull/1358#discussion_r4521273264


##########
crates/integrations/datafusion/src/sql_context.rs:
##########
@@ -3765,17 +3770,45 @@ fn parse_static_partitions(
         let field = field_map.get(col_name.as_str()).ok_or_else(|| {
             DataFusionError::Plan(format!("Column '{col_name}' not found in 
table schema"))
         })?;
-        let datum = sql_expr_to_datum(val_expr, field.data_type())?;
+        let datum = sql_expr_to_datum(val_expr, field.data_type(), 
legacy_partition_name)?;
         result.insert(col_name, Some(datum));
     }
 
     Ok(result)
 }
 
 /// Convert a SQL literal expression to a Paimon Datum.
-fn sql_expr_to_datum(expr: &SqlExpr, data_type: &PaimonDataType) -> 
DFResult<Datum> {
+fn sql_expr_to_datum(
+    expr: &SqlExpr,
+    data_type: &PaimonDataType,
+    legacy_partition_name: bool,
+) -> DFResult<Datum> {
+    let parse_text = |text: &str| {
+        crate::procedures::parse_partition_value(text, data_type, 
legacy_partition_name).ok_or_else(
+            || {
+                DataFusionError::Plan(format!(
+                    "Invalid partition value '{text}' for type {data_type:?}"
+                ))
+            },
+        )
+    };
     let (value, negate) = match expr {
         SqlExpr::Value(v) => (&v.value, false),
+        SqlExpr::TypedString(typed)
+            if matches!(
+                (&typed.data_type, data_type),
+                (SqlDataType::Date, PaimonDataType::Date(_))
+                    | (
+                        SqlDataType::Timestamp(..),
+                        PaimonDataType::Timestamp(_) | 
PaimonDataType::LocalZonedTimestamp(_)
+                    )
+            ) =>
+        {
+            let text = typed.value.value.clone().into_string().ok_or_else(|| {
+                DataFusionError::Plan(format!("Unsupported partition value 
expression: {expr}"))
+            })?;
+            return parse_text(&text);

Review Comment:
   Reproduced it: on the previous head, with the session at `+08:00`, a static 
overwrite of `k = TIMESTAMP WITH TIME ZONE '2024-03-01 10:00:00'` replaced the 
18:00 row.
   
   Fixed in fe71531 by no longer parsing the literal text. For DECIMAL, 
TIMESTAMP and TIMESTAMP WITH LOCAL TIME ZONE keys, and for `DATE '...'`, the 
partition value is now planned with 
`SessionState::create_logical_expr_from_sql_expr`, cast to the column's Arrow 
type and evaluated, then turned into a `Datum` with `scalar_to_datum`. That is 
how INSERT gets the value it writes, so the session time zone applies. 
`test_partition_specs_evaluate_values_like_insert` sets `+08:00` and uses your 
two instants for INSERT OVERWRITE, TRUNCATE and DROP PARTITION.



##########
crates/integrations/datafusion/src/sql_context.rs:
##########
@@ -3942,14 +3985,20 @@ fn datum_to_constant_array(
             Datum::Double(v) => Ok(Arc::new(Float64Array::from(vec![*v; 
num_rows]))),
             Datum::String(v) => Ok(Arc::new(StringArray::from(vec![v.as_str(); 
num_rows]))),
             Datum::Date(v) => Ok(Arc::new(Date32Array::from(vec![*v; 
num_rows]))),
-            Datum::Time(_)
-            | Datum::Timestamp { .. }
-            | Datum::LocalZonedTimestamp { .. }
-            | Datum::Decimal { .. }
-            | Datum::Bytes(_)
-            | Datum::Variant { .. } => Err(DataFusionError::Plan(format!(
-                "Unsupported datum type for partition column: {d}"
-            ))),
+            Datum::Timestamp { .. } | Datum::LocalZonedTimestamp { .. } | 
Datum::Decimal { .. } => {
+                let arrow_type =
+                    
paimon::arrow::paimon_type_to_arrow(data_type).map_err(to_datafusion_error)?;
+                crate::physical_plan::scan::datum_to_scalar(d.clone(), 
&arrow_type)

Review Comment:
   Reproduced with the same `does not belong to this partition` error. With the 
change above, the spec value is cast to the column's `TIMESTAMP(3)` type before 
it becomes a `Datum`, so `.123456` becomes `.123` in both the written column 
and the commit filter, as it does in INSERT. The test has a `.123856` value 
next to a `.124` row (the extra digits are dropped, not rounded, which is what 
INSERT stores) and the exact `.123` case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to