Gabriel39 commented on code in PR #69163:
URL: https://github.com/apache/doris/pull/69163#discussion_r4296158231
##########
fe/be-java-extensions/paimon-connector/src/main/java/org/apache/doris/paimon/PaimonArrowBatchAdapter.java:
##########
@@ -163,9 +164,12 @@ public Timestamp getTimestamp(int index, int precision) {
case MICROSECOND:
return Timestamp.fromMicros(value);
case NANOSECOND:
- return Timestamp.fromEpochMillis(
- Math.floorDiv(value, 1_000_000L),
- (int) Math.floorMod(value,
1_000_000L));
+ // Paimon serializers trust target precision.
Truncate the positive
+ // fractional remainder so negative epochs and
leading zero digits
+ // obey the same narrowing rule, including
nested values and keys.
+ int nanos = (int) Math.floorMod(value,
1_000_000L);
+ return
Timestamp.fromEpochMillis(Math.floorDiv(value, 1_000_000L),
+ nanos / nanosPerUnit * nanosPerUnit);
Review Comment:
Fixed in 7f5dc6148cd. The Paimon Arrow adapter now checks the
signed-nanosecond range after flooring to precision 7 or 8, before handing the
value to the writer. Scalar and nested tests reject Long.MIN_VALUE when
narrowing would move it below the representable range; the existing precision
and round-trip tests also pass.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergNereidsUtils.java:
##########
@@ -524,8 +524,13 @@ static Object extractNereidsLiteralValue(
}
// accept string value for date
return literal.getStringValue();
- case TIMESTAMP:
- case TIMESTAMP_NANO: {
+ case TIMESTAMP_NANO:
+ if (((org.apache.iceberg.types.Types.TimestampNanoType)
icebergType).shouldAdjustToUTC()) {
+ return null;
+ }
+ return Expressions.nanos(IcebergTimestampUtils.parseNanos(
+ literal.getStringValue(),
java.time.ZoneOffset.UTC));
Review Comment:
Fixed in 7f5dc6148cd. Raw rewrite WHERE strings now use the session
destination timezone before their civil fields are encoded as nanos. The unit
test covers an explicit +08:00 literal in Asia/Shanghai. The locally executed
Parquet and ORC regression creates separate midnight and previous-day 16:00
files, verifies that only the requested midnight files are rewritten, and
checks that both rows remain.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##########
@@ -2407,11 +2409,12 @@ protected List<FileScanTask>
loadFileScanTasksWithManifestCache(
}
ExternalTable targetExternalTable = (ExternalTable)
source.getTargetTable();
Schema scanSchema = scan.schema();
+ Set<Integer> unsafeNanosFields =
IcebergTimestampUtils.unsafeNanosFields(icebergTable, scanSchema);
Review Comment:
Fixed in 7f5dc6148cd. When an equality-delete key has unsafe retained-DATE
metrics, manifest-cache planning builds and probes the delete index without
column statistics. Partition/spec and sequence-number applicability are
retained, and the scan task keeps the original data file. A test writes real
historical DATE and later nanos equality-delete manifests and verifies that the
delete remains attached to the data task.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1866,24 +2001,104 @@ Status
OrcReader::_select_stripe_ranges_by_statistics() {
return Status::OK();
}
-void OrcReader::_apply_current_stripe_range() {
- if (!_state->stripe_pruning_applied ||
_state->selected_stripe_ranges.empty()) {
- return;
+Status OrcReader::_prepare_timestamp_stripe_ranges() {
+ if (!_state->timestamp_plan.requires_stripe_reader()) {
+ return Status::OK();
+ }
+ std::vector<uint64_t> stripes;
+ std::set<uint64_t> unsafe_date_stripes;
+ if (_state->stripe_ranges_applied) {
+ for (const auto& range : _state->selected_stripe_ranges) {
+ for (auto stripe = range.first_stripe; stripe < range.last_stripe;
++stripe) {
+ stripes.push_back(stripe);
+ if (range.disable_sarg) {
+ unsafe_date_stripes.insert(stripe);
+ }
+ }
+ }
+ } else {
+ RETURN_IF_ERROR(_collect_split_stripes(&stripes));
+ }
+ _state->selected_stripe_ranges.clear();
+ _state->stripe_ranges_applied = true;
+ _reader_statistics.read_row_groups = cast_set<int32_t>(stripes.size());
+ if (stripes.empty()) {
+ return Status::OK();
+ }
+ try {
+ uint64_t first_row = 0;
+ size_t selected = 0;
+ std::string previous_timezone;
+ for (uint64_t index = 0; index <= stripes.back(); ++index) {
+ const auto stripe = _state->reader->getStripe(index);
+ const auto end_row = first_row + stripe->getNumberOfRows();
+ if (index == stripes[selected]) {
+ const auto& name = stripe->getWriterTimezone();
Review Comment:
Fixed in 7f5dc6148cd. Timestamp stripe ranges now retain cheap file-tail
descriptors and read/cache writer timezones lazily. The first scan window
contains one stripe; subsequent compatible windows grow to amortize SDK
RowReader creation. Row-ID scans skip unrequested stripes. The ASAN test
verifies one footer read at open, one footer for a late-stripe row-ID lookup,
and at most seven RowReaders for 64 same-zone stripes.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1152,7 +1255,12 @@ DataTypePtr OrcReader::_convert_to_doris_type(const
::orc::Type& type) const {
data_type = std::make_shared<DataTypeDateV2>();
break;
case ::orc::TypeKind::TIMESTAMP:
- data_type = std::make_shared<DataTypeDateTimeV2>(6);
+ if (type.hasAttributeKey("iceberg.timestamp-unit") &&
+ type.getAttributeValue("iceberg.timestamp-unit") == "NANOS") {
+ data_type = std::make_shared<DataTypeTimeStampNs>();
Review Comment:
Fixed in 7f5dc6148cd. Metadata COUNT(column) declines pushdown when the
argument contains an annotated NANOS timestamp, so normal materialization
performs signed-nanosecond validation. COUNT(*) remains metadata-only. ASAN
tests cover scalar, array, map and struct arguments with out-of-range, valid
and null values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
