PDGGK commented on code in PR #1358:
URL: https://github.com/apache/paimon-rust/pull/1358#discussion_r4521273264
##########
crates/integrations/datafusion/src/sql_context.rs:
##########
@@ -3765,17 +3770,45 @@ fn parse_static_partitions(
let field = field_map.get(col_name.as_str()).ok_or_else(|| {
DataFusionError::Plan(format!("Column '{col_name}' not found in
table schema"))
})?;
- let datum = sql_expr_to_datum(val_expr, field.data_type())?;
+ let datum = sql_expr_to_datum(val_expr, field.data_type(),
legacy_partition_name)?;
result.insert(col_name, Some(datum));
}
Ok(result)
}
/// Convert a SQL literal expression to a Paimon Datum.
-fn sql_expr_to_datum(expr: &SqlExpr, data_type: &PaimonDataType) ->
DFResult<Datum> {
+fn sql_expr_to_datum(
+ expr: &SqlExpr,
+ data_type: &PaimonDataType,
+ legacy_partition_name: bool,
+) -> DFResult<Datum> {
+ let parse_text = |text: &str| {
+ crate::procedures::parse_partition_value(text, data_type,
legacy_partition_name).ok_or_else(
+ || {
+ DataFusionError::Plan(format!(
+ "Invalid partition value '{text}' for type {data_type:?}"
+ ))
+ },
+ )
+ };
let (value, negate) = match expr {
SqlExpr::Value(v) => (&v.value, false),
+ SqlExpr::TypedString(typed)
+ if matches!(
+ (&typed.data_type, data_type),
+ (SqlDataType::Date, PaimonDataType::Date(_))
+ | (
+ SqlDataType::Timestamp(..),
+ PaimonDataType::Timestamp(_) |
PaimonDataType::LocalZonedTimestamp(_)
+ )
+ ) =>
+ {
+ let text = typed.value.value.clone().into_string().ok_or_else(|| {
+ DataFusionError::Plan(format!("Unsupported partition value
expression: {expr}"))
+ })?;
+ return parse_text(&text);
Review Comment:
Reproduced it: on the previous head, with the session at `+08:00`, a static
overwrite of `k = TIMESTAMP WITH TIME ZONE '2024-03-01 10:00:00'` replaced the
18:00 row.
Fixed in fe71531 by no longer parsing the literal text. For DECIMAL,
TIMESTAMP and TIMESTAMP WITH LOCAL TIME ZONE keys, and for `DATE '...'`, the
partition value is now planned with
`SessionState::create_logical_expr_from_sql_expr`, cast to the column's Arrow
type and evaluated, then turned into a `Datum` with `scalar_to_datum`. That is
how INSERT gets the value it writes, so the session time zone applies.
`test_partition_specs_evaluate_values_like_insert` sets `+08:00` and uses your
two instants for INSERT OVERWRITE, TRUNCATE and DROP PARTITION.
##########
crates/integrations/datafusion/src/sql_context.rs:
##########
@@ -3942,14 +3985,20 @@ fn datum_to_constant_array(
Datum::Double(v) => Ok(Arc::new(Float64Array::from(vec![*v;
num_rows]))),
Datum::String(v) => Ok(Arc::new(StringArray::from(vec![v.as_str();
num_rows]))),
Datum::Date(v) => Ok(Arc::new(Date32Array::from(vec![*v;
num_rows]))),
- Datum::Time(_)
- | Datum::Timestamp { .. }
- | Datum::LocalZonedTimestamp { .. }
- | Datum::Decimal { .. }
- | Datum::Bytes(_)
- | Datum::Variant { .. } => Err(DataFusionError::Plan(format!(
- "Unsupported datum type for partition column: {d}"
- ))),
+ Datum::Timestamp { .. } | Datum::LocalZonedTimestamp { .. } |
Datum::Decimal { .. } => {
+ let arrow_type =
+
paimon::arrow::paimon_type_to_arrow(data_type).map_err(to_datafusion_error)?;
+ crate::physical_plan::scan::datum_to_scalar(d.clone(),
&arrow_type)
Review Comment:
Reproduced with the same `does not belong to this partition` error. With the
change above, the spec value is cast to the column's `TIMESTAMP(3)` type before
it becomes a `Datum`, so `.123456` becomes `.123` in both the written column
and the commit filter, as it does in INSERT. The test has a `.123856` value
next to a `.124` row (the extra digits are dropped, not rounded, which is what
INSERT stores) and the exact `.123` case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]