OIiveirra commented on code in PR #68161:
URL: https://github.com/apache/doris/pull/68161#discussion_r4271828715
##########
be/src/format_v2/column_mapper.cpp:
##########
@@ -60,6 +60,16 @@ namespace doris::format {
namespace {
+bool requires_paimon_datetime_precision_conversion(const DataTypePtr&
source_type,
+ const DataTypePtr&
target_type) {
+ const auto source = remove_nullable(source_type);
+ const auto target = remove_nullable(target_type);
+ const auto primitive = source->get_primitive_type();
+ return primitive == target->get_primitive_type() &&
+ (primitive == TYPE_DATETIMEV2 || primitive == TYPE_TIMESTAMPTZ) &&
Review Comment:
Fixed in ff1efbafc860a2fd9fa22b42fd29eee23875d0d7.
Affected cross-family precision-narrowing raw files now route to JNI rather
than the rounding native cast, preserving the SDK's zone conversion. Added
automatic (force_jni_scanner=false) coverage for both directions,
UTC/Asia-Shanghai and negative-epoch values; projections and equality queries
pass.
Verification: 135 focused FE tests and 59 JNI tests passed; full ASAN FE/BE
build and targeted paimon_timestamp_types External regression passed (1 suite,
0 failed/fatal/skipped). Scope remains V2/JNI; V1 is unchanged.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -381,21 +421,37 @@ private Split getSplit() {
}
private void resetDatetimeV2Precision() {
+ boolean repairHistoricalPrecision = Boolean.parseBoolean(
+ params.getOrDefault(PAIMON_OPTION_PREFIX +
DORIS_TIMESTAMP_PRECISION_REPAIR, "true"));
Review Comment:
Fixed in ff1efbafc860a2fd9fa22b42fd29eee23875d0d7.
For producers omitting the repair option, timestamp-dependent predicate
subtrees are removed before widened reads. Independent top-level AND siblings
remain pushed; mixed OR remains residual. Added a real historical Parquet
reader test with the option omitted, positive/negative timestamps and a
retained integer filter.
Verification: 135 focused FE tests and 59 JNI tests passed; full ASAN FE/BE
build and targeted paimon_timestamp_types External regression passed (1 suite,
0 failed/fatal/skipped). Scope remains V2/JNI; V1 is unchanged.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -224,11 +227,11 @@ private void initReader() throws IOException {
: null;
variantProjections.add(variantProjection);
if (variantProjection == null) {
- // The engine may have pruned this complex column down to the
sub-fields the query touches.
- // Mirroring that shape with paimon's own types lets
withReadType push the same projection
- // through the ROW/ARRAY/MAP readers instead of reading the
whole column and discarding it.
DataType projectedType =
PaimonReadTypeProjection.project(tableField.type(),
types[outputIndex]);
+ if (requiresDatetimeV2PrecisionRepair) {
Review Comment:
Fixed in ff1efbafc860a2fd9fa22b42fd29eee23875d0d7.
Policy-required fields are added before SDK reader creation while retaining
widened timestamp types, so SDK schema augmentation cannot restore the lossy
current precision. Authorization still delegates to the original SDK result;
the original output projection is restored afterwards. Historical
QueryAuthSplit tests cover hidden integer fields, negative epoch and nested
unprojected policy siblings.
Verification: 135 focused FE tests and 59 JNI tests passed; full ASAN FE/BE
build and targeted paimon_timestamp_types External regression passed (1 suite,
0 failed/fatal/skipped). Scope remains V2/JNI; V1 is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
