github-actions[bot] commented on code in PR #68161:
URL: https://github.com/apache/doris/pull/68161#discussion_r4044588204
##########
be/src/format_v2/expr/cast.cpp:
##########
@@ -116,6 +148,13 @@ Status Cast::_do_execute(VExprContext* context, const
Block* block, const Select
ColumnPtr tmp_arg_column;
RETURN_IF_ERROR(_children[0]->execute_column(context, block, selector,
count, tmp_arg_column));
auto arg_type = _children[0]->execute_type(block);
+ if (_truncate_datetimev2_precision &&
can_truncate_datetimev2_precision(arg_type, _data_type)) {
+ auto result = _data_type->create_column();
+ result->insert_range_from(*tmp_arg_column, 0, count);
Review Comment:
[P1] Handle source/target nullability before copying here. Paimon
intentionally exposes every top-level Doris column as nullable, while a
historical `NOT NULL TIMESTAMP(6)` file leaf is non-nullable. When that field
is narrowed to p0, `_data_type->create_column()` returns `ColumnNullable`, but
`tmp_arg_column` is `ColumnDateTimeV2`; `ColumnNullable::insert_range_from`
assert-casts the source to `ColumnNullable` and the scan fails with FatalError.
The runtime-nullability fallback does not catch required source -> nullable
target. Please make this copy wrapper-aware and add a NOT NULL
precision-evolution case.
##########
be/src/format_v2/column_mapper.cpp:
##########
@@ -2898,6 +2901,12 @@ Status TableColumnMapper::_create_direct_mapping(const
ColumnDefinition& table_c
mapping->projected_file_children = file_field.children;
mapping->timestamp_is_adjusted_to_utc =
file_field.timestamp_is_adjusted_to_utc;
mapping->file_type = file_field.type;
+ const auto file_type = remove_nullable(mapping->file_type);
+ const auto table_type = remove_nullable(mapping->table_type);
+ mapping->truncate_datetimev2_precision =
_options.truncate_datetimev2_precision_for_paimon &&
+ file_type->get_primitive_type()
== TYPE_DATETIMEV2 &&
Review Comment:
[P1] Apply the Paimon rule to TIMESTAMPTZ narrowing too. With
`enable.mapping.timestamp_tz=true`, FE maps `TIMESTAMP_LTZ` to
TIMESTAMPTZ(scale), and the native Paimon schema keeps that primitive. This
DATETIMEV2-only gate stays false; V2 then treats same-primitive TIMESTAMPTZ
scale mismatches as trivial pass-through, and V1 selects `ConsistentConverter`.
An old p6 LTZ value ending in `.600000` therefore remains fractional under a p0
schema on both native routes while JNI truncates it. Please carry the
Paimon-specific rule through TIMESTAMPTZ without changing ordinary SQL CAST
semantics, and test LTZ evolution with the option enabled.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonColumnValue.java:
##########
@@ -167,7 +168,8 @@ public LocalDate getDate() {
@Override
public LocalDateTime getDateTime() {
- Timestamp ts = record.getTimestamp(idx, dorisType.getPrecision());
+ Timestamp ts = DateTimeUtils.truncate(
Review Comment:
[P1] Avoid Paimon's truncation helper for evolved JNI timestamps. The pinned
1.3.1 implementation checks the decimal length of an unpadded nanosecond value
and truncates epoch milliseconds with Java division toward zero. Consequently a
p6 value ending in `.000001` remains `.000001` when narrowed to p4/p5, and
`1969-12-31 23:59:59.600` narrowed to p0 becomes the epoch instead of
`23:59:59`. Both changed getters take this path (including nested/LTZ uses),
while the added tests only cover a positive `.600` -> p0 case. Please use
fixed-width fractional truncation with floor-safe negative handling and cover
these two boundaries.
##########
be/src/format/column_type_convert.cpp:
##########
@@ -373,6 +373,11 @@ std::unique_ptr<ColumnTypeConverter>
ColumnTypeConverter::get_converter(const Da
return _decimal_converter(src_type, dst_type);
}
+ if (src_primitive_type == TYPE_DATETIMEV2 && dst_primitive_type ==
TYPE_DATETIMEV2 &&
+ src_type->get_scale() > dst_type->get_scale()) {
+ return
std::make_unique<DateTimeV2PrecisionConverter>(dst_type->get_scale());
Review Comment:
[P1] Disable V1 predicate pushdown when this converter changes timestamp
values. Conversion happens after file pruning, while Parquet `_type_matches()`
only compares primitive types and ORC still builds a TIMESTAMP search argument.
For an old p6 row `...01.600000` evolved to p0, `WHERE ts = ...01` should match
after truncation, but raw Parquet min/max or an ORC SARG can discard its
group/stripe first. V2 marks this mapping `FINALIZE_ONLY`; V1 needs equivalent
raw metadata/page pushdown gating plus filtered Parquet/ORC regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
