This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-4.2 in repository https://gitbox.apache.org/repos/asf/doris.git
commit 20a412ebe479b4e3e60558c9a279f8ed54dac193 Author: Gabriel <[email protected]> AuthorDate: Tue Sep 29 21:57:29 2026 +0800 [fix](arrow) Reject out-of-range Flight timestamps (#68604) ### What problem does this PR solve? Flight SQL can publish out-of-range timestamps from legacy INT96 date materialization. For example, a non-null zero DATETIME becomes `timestamp[us]` with value `-62169984000000000`: Arrow validation succeeds, but PyArrow scalar conversion raises `OverflowError`. Validate timestamps in `ArrowFlightArrowBlockConvertor` before publishing each batch. Check the actual timestamp unit and both UTC/local calendar bounds for zoned values. Recursively check arrays, map keys/values, and structs while skipping values masked by a NULL parent. Reject unsupported timestamps with an error identifying the field path and block row. Other Arrow consumers and Parquet scan compatibility remain unchanged. Related PR: #68596 adds separate UTF-8 validation. This PR is independent of that change; both checks must be retained when integrating the shared Flight conversion entry point. ### Testing - Added seven BE tests covering zero dates, years 0/10000, seconds/milliseconds/microseconds, valid calendar boundaries, pre-epoch fractions, timezone offsets, slices/subsequent batches, nested timestamps, NULL parents, and other consumers. - Before the fix: four new test groups failed because malformed timestamps were published; three valid/NULL groups passed. - After the fix: all 36 tests in `ArrowFlightTimestampTest.*:DataTypeSerDeArrowTest.*` passed under ASAN. Recompiled the changed converter and new tests against existing ASAN BE libraries, with C++ access control enabled. Test dates are constructed through the public unchecked setter so invalid calendar values reach the validator. - Added a Flight JDBC regression suite and a separate HDFS INT96 suite: four existing group4 fixtures must be read successfully with microsecond truncation at the upper calendar boundary; the zero-year fixture must return a range error. Both suites preserve the configured Flight credentials and avoid the TLS-dependent `connect` shortcut. The HDFS suite discovers the live FE Flight endpoint through MySQL because external clusters can override the configured port. The HDFS suite requires `enableHiveTest` and the external test environment. - Fixed the nested NULL assertion that caused the P0/Cloud P0 Groovy `getAt()` failure by consuming the JDBC array directly. - Clang-format 16 passed for all affected C++ files. Groovy 4.0.19 executed both scripts against mock JDBC connections in 12 checks covering TLS on/off, successful assertions, missing expected errors, and unrelated errors. These are script-level checks; the live external validation below additionally exercises real SQL and HDFS data. ### Live external regression validation - Used the CI-built FE/BE binaries for `8d1158d3f5` in an isolated local cluster and rebuilt the regression framework from this branch. This follow-up changes only test scripts and expected results. - Before the update: all four affected suites failed locally, reproducing the stale Flight port and zero-year timestamp failures. - After the update: all four suites passed, with zero failures and zero skips: `test_flight_int96_timestamp_range`, `test_remote_doris_all_types_select`, `test_remote_doris_statistics`, and `test_remote_doris_table_stats`. - The configured Flight port intentionally remained incorrect. The INT96 suite discovered the running FE endpoint, consumed all four valid HDFS fixtures, and asserted the expected error for the invalid fixture. - Remote Doris success fixtures now use year 0001 as the minimum timestamp, with matching golden results. A separate assertion preserves coverage of zero-year rejection through a Remote Doris catalog. ### Release note Flight SQL now returns a server-side error for timestamps outside the supported 0001–9999 calendar range instead of publishing values that fail client conversion. Valid timestamps and NULLs retain their existing values. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - Behavior changed: - [x] Yes: reject out-of-range Flight timestamps before publishing a batch. - Does this need documentation? - [x] No. --- be/src/format/arrow/arrow_block_convertor.cpp | 137 +++++++++++++ .../format/arrow/arrow_flight_timestamp_test.cpp | 220 +++++++++++++++++++++ .../test_remote_doris_all_types_select.out | 4 +- .../remote_doris/test_remote_doris_statistics.out | 2 +- .../test_flight_int96_timestamp_range.groovy | 83 ++++++++ .../test_flight_timestamp_range.groovy | 81 ++++++++ .../test_remote_doris_all_types_select.groovy | 15 +- .../test_remote_doris_statistics.groovy | 3 +- .../test_remote_doris_table_stats.groovy | 3 +- 9 files changed, 541 insertions(+), 7 deletions(-) diff --git a/be/src/format/arrow/arrow_block_convertor.cpp b/be/src/format/arrow/arrow_block_convertor.cpp index b62ca98cde9..6b2ed5a71e8 100644 --- a/be/src/format/arrow/arrow_block_convertor.cpp +++ b/be/src/format/arrow/arrow_block_convertor.cpp @@ -17,6 +17,8 @@ #include "format/arrow/arrow_block_convertor.h" +#include <arrow/array/array_nested.h> +#include <arrow/array/array_primitive.h> #include <arrow/array/builder_base.h> #include <arrow/array/builder_binary.h> #include <arrow/array/builder_decimal.h> @@ -34,6 +36,7 @@ #include <cctz/time_zone.h> #include <glog/logging.h> +#include <algorithm> #include <array> #include <cstring> #include <ctime> @@ -63,6 +66,134 @@ namespace doris { namespace { +class FlightTimestampValidator { +public: + Status init(std::shared_ptr<arrow::Array> array, std::string path, + const cctz::time_zone& timezone) { + _array = std::move(array); + _path = std::move(path); + switch (_array->type_id()) { + case arrow::Type::TIMESTAMP: { + const auto& type = static_cast<const arrow::TimestampType&>(*_array->type()); + int64_t units_per_second = 1; + switch (type.unit()) { + case arrow::TimeUnit::SECOND: + units_per_second = 1; + break; + case arrow::TimeUnit::MILLI: + units_per_second = 1000; + break; + case arrow::TimeUnit::MICRO: + units_per_second = 1000000; + break; + case arrow::TimeUnit::NANO: + units_per_second = 1000000000; + break; + } + int64_t min_seconds = -62135596800LL; + int64_t end_seconds = 253402300800LL; + if (!type.timezone().empty()) { + // The ordinary Arrow writer has already checked the schema/timezone binding. + const auto local_start = cctz::convert(cctz::civil_second(1, 1, 1), timezone); + const auto local_end = cctz::convert(cctz::civil_second(10000, 1, 1), timezone); + // Python first constructs UTC and then applies the Arrow timezone. Both calendar + // representations must fit, including offsets at the first and last supported day. + min_seconds = + std::max<int64_t>(min_seconds, local_start.time_since_epoch().count()); + end_seconds = std::min<int64_t>(end_seconds, local_end.time_since_epoch().count()); + } + // Nanosecond bounds do not fit int64_t, even though every encoded value does. + _min = static_cast<__int128>(min_seconds) * units_per_second; + _end = static_cast<__int128>(end_seconds) * units_per_second; + break; + } + case arrow::Type::LIST: { + const auto& list = static_cast<const arrow::ListArray&>(*_array); + RETURN_IF_ERROR(add_child(list.values(), _path + "[]", timezone)); + break; + } + case arrow::Type::MAP: { + const auto& map = static_cast<const arrow::MapArray&>(*_array); + RETURN_IF_ERROR(add_child(map.keys(), _path + ".key", timezone)); + RETURN_IF_ERROR(add_child(map.items(), _path + ".value", timezone)); + break; + } + case arrow::Type::STRUCT: { + const auto& structure = static_cast<const arrow::StructArray&>(*_array); + for (int i = 0; i < structure.num_fields(); ++i) { + RETURN_IF_ERROR(add_child(structure.field(i), + _path + "." + _array->type()->field(i)->name(), + timezone)); + } + break; + } + default: + break; + } + return Status::OK(); + } + + Status validate(int64_t start, int64_t end, size_t block_start, int64_t parent_row = -1) const { + if (!has_timestamps()) { + return Status::OK(); + } + for (int64_t row = start; row < end; ++row) { + // Children of a NULL struct/list/map are not observable, even if their physical + // buffers contain default zero dates. Never validate those masked values. + if (_array->IsNull(row)) { + continue; + } + const int64_t output_row = parent_row < 0 ? row : parent_row; + if (_array->type_id() == arrow::Type::TIMESTAMP) { + const int64_t value = static_cast<const arrow::TimestampArray&>(*_array).Value(row); + if (value < _min || value >= _end) { + return Status::InvalidArgument( + "Arrow Flight timestamp in column '{}' at row {} is outside the " + "supported 0001-9999 range (type: {})", + _path, block_start + output_row + 1, _array->type()->ToString()); + } + continue; + } + int64_t child_start = row; + int64_t child_end = row + 1; + if (_array->type_id() == arrow::Type::LIST) { + const auto& list = static_cast<const arrow::ListArray&>(*_array); + child_start = list.value_offset(row); + child_end = child_start + list.value_length(row); + } else if (_array->type_id() == arrow::Type::MAP) { + const auto& map = static_cast<const arrow::MapArray&>(*_array); + child_start = map.value_offset(row); + child_end = child_start + map.value_length(row); + } + for (const auto& child : _children) { + RETURN_IF_ERROR(child.validate(child_start, child_end, block_start, output_row)); + } + } + return Status::OK(); + } + +private: + bool has_timestamps() const { + return _array->type_id() == arrow::Type::TIMESTAMP || !_children.empty(); + } + + Status add_child(std::shared_ptr<arrow::Array> array, std::string path, + const cctz::time_zone& timezone) { + FlightTimestampValidator child; + RETURN_IF_ERROR(child.init(std::move(array), std::move(path), timezone)); + if (child.has_timestamps()) { + _children.push_back(std::move(child)); + } + return Status::OK(); + } + + std::shared_ptr<arrow::Array> _array; + std::string _path; + std::vector<FlightTimestampValidator> _children; + __int128 _min = 0; + __int128 _end = 0; +}; + int hex_value(char c) { if (c >= '0' && c <= '9') { return c - '0'; @@ -355,6 +486,12 @@ Status ArrowFlightArrowBlockConvertor::convert_to_arrow(const Block& block, arro i + 1, batch->schema()->field(i)->name(), status.ToString()); } + // Arrow accepts the full int64 timestamp domain; ValidateFull cannot enforce the calendar + // range required by Flight clients. Check the encoded values before publishing the batch. + FlightTimestampValidator validator; + RETURN_IF_ERROR( + validator.init(batch->column(i), batch->schema()->field(i)->name(), _timezone)); + RETURN_IF_ERROR(validator.validate(0, batch->num_rows(), start_row)); } *result = std::move(batch); return Status::OK(); diff --git a/be/test/format/arrow/arrow_flight_timestamp_test.cpp b/be/test/format/arrow/arrow_flight_timestamp_test.cpp new file mode 100644 index 00000000000..ebcafc04a91 --- /dev/null +++ b/be/test/format/arrow/arrow_flight_timestamp_test.cpp @@ -0,0 +1,220 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include <arrow/api.h> +#include <gtest/gtest.h> + +#include "core/column/column_nullable.h" +#include "core/column/column_vector.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_date_or_datetime_v2.h" +#include "core/data_type/data_type_map.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" +#include "format/arrow/arrow_block_convertor.h" +#include "util/timezone_utils.h" + +namespace doris { +namespace { +using DateTime = DateV2Value<DateTimeV2ValueType>; + +DateTime make_datetime(uint16_t year, uint8_t month, uint8_t day, uint8_t hour, uint8_t minute, + uint16_t second, uint32_t microsecond) { + DateTime value; + // Invalid calendar values must reach the converter to exercise its range validation. + value.unchecked_set_time(year, month, day, hour, minute, second, microsecond); + return value; +} + +Block timestamp_block(int scale, const std::vector<DateTime>& values) { + auto column = ColumnDateTimeV2::create(); + for (const auto& value : values) { + column->insert_value(value); + } + return Block {{std::move(column), std::make_shared<DataTypeDateTimeV2>(scale), "event_time"}}; +} + +class ArrowFlightTimestampTest : public testing::Test { +protected: + static void SetUpTestSuite() { TimezoneUtils::load_timezones_to_cache(); } +}; + +TEST_F(ArrowFlightTimestampTest, RejectsOutOfRangeInEveryUnitWithoutPublishingBatch) { + for (int scale : {0, 3, 6}) { + for (const auto& value : {DateTime {}, make_datetime(0, 12, 31, 0, 0, 0, 0), + make_datetime(10000, 1, 1, 0, 0, 0, 0)}) { + SCOPED_TRACE(scale); + auto block = timestamp_block(scale, {value}); + ArrowFlightArrowBlockConvertor flight(block, "UTC", cctz::utc_time_zone(), true); + ASSERT_TRUE(flight.init().ok()); + const ArrowBlockConvertor& converter = flight; + std::shared_ptr<arrow::RecordBatch> batch; + const auto status = + converter.convert_to_arrow(block, arrow::default_memory_pool(), &batch); + EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code()) << status; + EXPECT_NE(std::string::npos, status.to_string().find("event_time")); + EXPECT_NE(std::string::npos, status.to_string().find("row 1")); + EXPECT_NE(std::string::npos, status.to_string().find("0001-9999")); + EXPECT_EQ(nullptr, batch); + + // Other Arrow consumers retain their existing date semantics. + DorisArrowBlockConvertor ordinary(block, "UTC", cctz::utc_time_zone(), true); + ASSERT_TRUE(ordinary.init().ok()); + ASSERT_TRUE( + ordinary.convert_to_arrow(block, arrow::default_memory_pool(), &batch).ok()); + EXPECT_TRUE(batch->ValidateFull().ok()); + } + } +} + +TEST_F(ArrowFlightTimestampTest, PreservesCalendarBoundariesAndPreEpochFractions) { + for (int scale : {0, 3, 6}) { + const int64_t factor = scale == 0 ? 1 : scale == 3 ? 1000 : 1000000; + const uint32_t fraction = scale == 0 ? 0 : scale == 3 ? 999000 : 999999; + auto block = timestamp_block(scale, {make_datetime(1, 1, 1, 0, 0, 0, 0), + make_datetime(9999, 12, 31, 23, 59, 59, fraction), + make_datetime(1969, 12, 31, 23, 59, 59, fraction)}); + ArrowFlightArrowBlockConvertor converter(block, "UTC", cctz::utc_time_zone(), true); + ASSERT_TRUE(converter.init().ok()); + std::shared_ptr<arrow::RecordBatch> batch; + ASSERT_TRUE(converter.convert_to_arrow(block, arrow::default_memory_pool(), &batch).ok()); + const auto& values = static_cast<const arrow::TimestampArray&>(*batch->column(0)); + EXPECT_EQ(-62135596800LL * factor, values.Value(0)); + EXPECT_EQ(253402300800LL * factor - 1, values.Value(1)); + EXPECT_EQ(-1, values.Value(2)); + } +} + +TEST_F(ArrowFlightTimestampTest, ChecksSlicesAndSubsequentBatches) { + auto block = timestamp_block(6, {make_datetime(2024, 1, 1, 0, 0, 0, 0), DateTime {}}); + ArrowFlightArrowBlockConvertor converter(block, "UTC", cctz::utc_time_zone(), true); + ASSERT_TRUE(converter.init().ok()); + std::shared_ptr<arrow::RecordBatch> batch; + ASSERT_TRUE(converter.convert_to_arrow(block, arrow::default_memory_pool(), &batch, 0, 1).ok()); + auto previous = batch; + const auto status = + converter.convert_to_arrow(block, arrow::default_memory_pool(), &batch, 1, 2); + EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code()); + EXPECT_NE(std::string::npos, status.to_string().find("row 2")); + EXPECT_EQ(previous, batch); +} + +TEST_F(ArrowFlightTimestampTest, RejectsNestedTimestampValuesAndMapKeys) { + auto datetime = make_nullable(std::make_shared<DataTypeDateTimeV2>(6)); + const auto invalid = Field::create_field<TYPE_DATETIMEV2>(DateTime {}); + const auto valid = Field::create_field<TYPE_DATETIMEV2>(make_datetime(2024, 1, 1, 0, 0, 0, 0)); + DataTypes types {std::make_shared<DataTypeArray>(datetime), + std::make_shared<DataTypeStruct>(DataTypes {datetime}, Strings {"child"}), + std::make_shared<DataTypeMap>(datetime, datetime), + std::make_shared<DataTypeMap>(datetime, datetime), + std::make_shared<DataTypeArray>(std::make_shared<DataTypeStruct>( + DataTypes {datetime}, Strings {"child"}))}; + FieldVector fields { + Field::create_field<TYPE_ARRAY>(Array {valid, invalid}), + Field::create_field<TYPE_STRUCT>(Struct {invalid}), + Field::create_field<TYPE_MAP>(Map {Field::create_field<TYPE_ARRAY>(Array {valid}), + Field::create_field<TYPE_ARRAY>(Array {invalid})}), + Field::create_field<TYPE_MAP>(Map {Field::create_field<TYPE_ARRAY>(Array {invalid}), + Field::create_field<TYPE_ARRAY>(Array {valid})}), + Field::create_field<TYPE_ARRAY>( + Array {Field::create_field<TYPE_STRUCT>(Struct {invalid})})}; + for (size_t i = 0; i < types.size(); ++i) { + SCOPED_TRACE(types[i]->get_name()); + auto column = types[i]->create_column(); + column->insert_default(); + column->insert(fields[i]); + Block block {{std::move(column), types[i], "nested"}}; + ArrowFlightArrowBlockConvertor converter(block, "UTC", cctz::utc_time_zone(), true); + ASSERT_TRUE(converter.init().ok()); + std::shared_ptr<arrow::RecordBatch> batch; + const auto status = + converter.convert_to_arrow(block, arrow::default_memory_pool(), &batch, 1, 2); + EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code()) << status; + EXPECT_NE(std::string::npos, status.to_string().find("nested")); + EXPECT_NE(std::string::npos, status.to_string().find("row 2")); + EXPECT_EQ(nullptr, batch); + } +} + +TEST_F(ArrowFlightTimestampTest, ChecksBothUtcAndZonedCalendarBounds) { + const std::vector<std::pair<std::string, DateTime>> cases { + {"+08:00", make_datetime(1, 1, 1, 0, 0, 0, 0)}, + {"-08:00", make_datetime(9999, 12, 31, 23, 0, 0, 0)}, + {"-08:00", make_datetime(0, 12, 31, 23, 0, 0, 0)}, + {"+08:00", make_datetime(10000, 1, 1, 1, 0, 0, 0)}}; + for (const auto& [zone, value] : cases) { + SCOPED_TRACE(zone); + cctz::time_zone timezone; + ASSERT_TRUE(TimezoneUtils::find_cctz_time_zone(zone, timezone)); + auto block = timestamp_block(6, {value}); + ArrowFlightArrowBlockConvertor converter(block, zone, timezone); + ASSERT_TRUE(converter.init().ok()); + std::shared_ptr<arrow::RecordBatch> batch; + const auto status = converter.convert_to_arrow(block, arrow::default_memory_pool(), &batch); + EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code()) << status; + EXPECT_EQ(nullptr, batch); + + auto valid = timestamp_block( + 6, {make_datetime(1, 1, 2, 0, 0, 0, 0), make_datetime(9999, 12, 30, 23, 0, 0, 0)}); + ASSERT_TRUE(converter.convert_to_arrow(valid, arrow::default_memory_pool(), &batch).ok()); + } +} + +TEST_F(ArrowFlightTimestampTest, NaiveBoundsDoNotDependOnSessionTimezone) { + cctz::time_zone timezone; + ASSERT_TRUE(TimezoneUtils::find_cctz_time_zone("+08:00", timezone)); + auto block = timestamp_block(6, {make_datetime(1, 1, 1, 0, 0, 0, 0), + make_datetime(9999, 12, 31, 23, 59, 59, 999999)}); + ArrowFlightArrowBlockConvertor converter(block, "+08:00", timezone, true); + ASSERT_TRUE(converter.init().ok()); + std::shared_ptr<arrow::RecordBatch> batch; + ASSERT_TRUE(converter.convert_to_arrow(block, arrow::default_memory_pool(), &batch).ok()); + const auto& values = static_cast<const arrow::TimestampArray&>(*batch->column(0)); + EXPECT_EQ(-62135596800000000LL, values.Value(0)); + EXPECT_EQ(253402300799999999LL, values.Value(1)); +} + +TEST_F(ArrowFlightTimestampTest, IgnoresTimestampsMaskedByNullParents) { + auto datetime = std::make_shared<DataTypeDateTimeV2>(6); + const auto invalid = Field::create_field<TYPE_DATETIMEV2>(DateTime {}); + DataTypes types { + datetime, std::make_shared<DataTypeStruct>(DataTypes {datetime}, Strings {"child"}), + std::make_shared<DataTypeArray>(datetime), + std::make_shared<DataTypeMap>(make_nullable(datetime), make_nullable(datetime))}; + FieldVector fields { + invalid, Field::create_field<TYPE_STRUCT>(Struct {invalid}), + Field::create_field<TYPE_ARRAY>(Array {invalid}), + Field::create_field<TYPE_MAP>(Map {Field::create_field<TYPE_ARRAY>(Array {invalid}), + Field::create_field<TYPE_ARRAY>(Array {invalid})})}; + for (size_t i = 0; i < types.size(); ++i) { + auto data = types[i]->create_column(); + data->insert(fields[i]); + auto nulls = ColumnUInt8::create(); + nulls->insert_value(1); + Block block {{ColumnNullable::create(std::move(data), std::move(nulls)), + make_nullable(types[i]), "masked"}}; + ArrowFlightArrowBlockConvertor converter(block, "UTC", cctz::utc_time_zone(), true); + ASSERT_TRUE(converter.init().ok()); + std::shared_ptr<arrow::RecordBatch> batch; + const auto status = converter.convert_to_arrow(block, arrow::default_memory_pool(), &batch); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(batch->column(0)->IsNull(0)); + } +} + +} // namespace +} // namespace doris diff --git a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_all_types_select.out b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_all_types_select.out index a8823bb027e..70fb18a6da0 100644 --- a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_all_types_select.out +++ b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_all_types_select.out @@ -1,12 +1,12 @@ -- This file is automatically generated. You should know what you did if you want to edit this -- !sql -- -2025-05-18T01:00 true -128 -32768 -2147483648 -9223372036854775808 -1234567890123456790 -123.456 -123456.789 -123457 -123456789012346 -1234567890123456789012345678 1970-01-01 0000-01-01T00:00 A Hello Hello, Doris! ["apple", "banana", "orange"] {"Emily":101, "age":25} {"f1":11, "f2":3.14, "f3":"Emily"} {"k1":"v31","k2":300,"k3":[123,456],"k4":[],"k5":{"i1":"iv1"}} +2025-05-18T01:00 true -128 -32768 -2147483648 -9223372036854775808 -1234567890123456790 -123.456 -123456.789 -123457 -123456789012346 -1234567890123456789012345678 1970-01-01 0001-01-01T00:00 A Hello Hello, Doris! ["apple", "banana", "orange"] {"Emily":101, "age":25} {"f1":11, "f2":3.14, "f3":"Emily"} {"k1":"v31","k2":300,"k3":[123,456],"k4":[],"k5":{"i1":"iv1"}} 2025-05-18T02:00 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N 2025-05-18T03:00 false 127 32767 2147483647 9223372036854775807 1234567890123456789 123.456 123456.789 123457 123456789012346 1234567890123456789012345678 9999-12-31 9999-12-31T23:59:59 [] {} {"f1":11, "f2":3.14, "f3":"Emily"} {} 2025-05-18T04:00 true 0 0 0 0 0 0.0 0 0 0 0 2023-10-01 2023-10-01T12:34:56 A Hello Hello, Doris! ["apple", "banana", "orange"] {"Emily":101, "age":25} {"f1":11, "f2":3.14, "f3":"Emily"} [] -- !sql -- -2025-05-18T01:00 [1] [-128] [-32768] [-2147483648] [-9223372036854775808] [-1234567890123456790] [-123.456] [-123456.789] [-123457] [-123456789012346] [-1234567890123456789012345678] ["0000-01-01"] ["0000-01-01 00:00:00"] ["A"] ["Hello"] ["Hello, Doris!"] +2025-05-18T01:00 [1] [-128] [-32768] [-2147483648] [-9223372036854775808] [-1234567890123456790] [-123.456] [-123456.789] [-123457] [-123456789012346] [-1234567890123456789012345678] ["0000-01-01"] ["0001-01-01 00:00:00"] ["A"] ["Hello"] ["Hello, Doris!"] 2025-05-18T02:00 [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] [null] 2025-05-18T03:00 [0] [127] [32767] [2147483647] [9223372036854775807] [1234567890123456789] [123.456] [123456.789] [123457] [123456789012346] [1234567890123456789012345678] ["9999-12-31"] ["9999-12-31 23:59:59"] [""] [""] [""] 2025-05-18T04:00 [1] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] ["2023-10-01"] ["2023-10-01 12:34:56"] ["A"] ["Hello"] ["Hello, Doris!"] diff --git a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_statistics.out b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_statistics.out index eecba5cbcd6..aebd716491e 100644 --- a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_statistics.out +++ b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_statistics.out @@ -4,7 +4,7 @@ c_bigint 4 3 1 -9223372036854775808 9223372036854775807 32 c_boolean 4 2 1 0 1 4 c_char 4 2 1 A 2 c_date 4 3 1 1970-01-01 9999-12-31 16 -c_datetime 4 3 1 0000-01-01 00:00:00 9999-12-31 23:59:59 32 +c_datetime 4 3 1 0001-01-01 00:00:00 9999-12-31 23:59:59 32 c_decimal18 4 3 1 -123456789012346 123456789012346 32 c_decimal32 4 3 1 -1234567890123456789012345678 1234567890123456789012345678 64 c_decimal9 4 3 1 -123457 123457 16 diff --git a/regression-test/suites/arrow_flight_sql_p0/test_flight_int96_timestamp_range.groovy b/regression-test/suites/arrow_flight_sql_p0/test_flight_int96_timestamp_range.groovy new file mode 100644 index 00000000000..7cdc5f1bbfe --- /dev/null +++ b/regression-test/suites/arrow_flight_sql_p0/test_flight_int96_timestamp_range.groovy @@ -0,0 +1,83 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import java.sql.DriverManager +import java.sql.Types +import java.time.LocalDateTime + +suite("test_flight_int96_timestamp_range", "arrow_flight_sql,external,hive,tvf,external_docker") { + if (!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableHiveTest"))) { + return + } + def host = context.config.otherConfigs.get("externalEnvIp") + def port = context.config.otherConfigs.get("hive2HdfsPort") + // External clusters can override the configured Flight port; use the running FE's endpoint. + def frontend = jdbc_sql_return_maparray("SHOW FRONTENDS").find { + it.IsMaster.toString().toBoolean() && it.Alive.toString().toBoolean() + } + assertNotNull(frontend) + assertTrue(frontend.ArrowFlightSqlPort.toString().toInteger() > 0) + def flightUrl = "jdbc:arrow-flight-sql://${frontend.Host}:${frontend.ArrowFlightSqlPort}" + + "/catalog=${context.dbName}?useServerPrepStmts=false&useSSL=false&useEncryption=false" + Class.forName("org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver") + def query = { String file -> + """SELECT * FROM HDFS( + "uri" = "hdfs://${host}:${port}/user/doris/tvf_data/test_hdfs_parquet/group4/${file}", + "hadoop.username" = "doris", "format" = "parquet") LIMIT 10""" + } + // Keep the configured Flight identity without the TLS-dependent connect() wrapper. + DriverManager.getConnection(flightUrl, context.config.otherConfigs.get("extArrowFlightSqlUser"), + context.config.otherConfigs.get("extArrowFlightSqlPassword")).withCloseable { flight -> + flight.createStatement().withCloseable { statement -> + // Nanosecond fractions in these fixtures truncate to valid DATETIME(6) values. + for (def file : ["part-00000-570d8e52-652d-4892-8bdc-7fa5466ffa69.c000.snappy.parquet", + "part-00000-b945dfb5-9982-4f86-b903-dabef99caba1.c000.snappy.parquet", + "part-00000-721700d2-26d7-42a3-a8f9-b6601628ccd4.c000.snappy.parquet", + "part-00000-afeef968-a917-4d51-a652-e5a4214df453.c000.snappy.parquet"]) { + logger.info("Read INT96 Flight fixture: ${file}") + statement.executeQuery(query(file)).withCloseable { rows -> + assertTrue(rows.next()) + def metadata = rows.getMetaData() + for (int column = 1; column <= metadata.getColumnCount(); ++column) { + if (metadata.getColumnType(column) == Types.TIMESTAMP) { + assertNotNull(rows.getObject(column, LocalDateTime.class)) + } + } + assertEquals(LocalDateTime.parse("9999-12-31T23:59:59.999999"), + rows.getObject(metadata.getColumnCount(), LocalDateTime.class)) + assertFalse(rows.next()) + } + } + } + // Consume the result on the same discovered endpoint so zero-year values must fail over Flight. + try { + flight.createStatement().withCloseable { statement -> + statement.executeQuery(query("int96_timestamps_nanos_outside_day_range.parquet")) + .withCloseable { rows -> + while (rows.next()) { + for (int column = 1; column <= rows.getMetaData().getColumnCount(); ++column) { + rows.getObject(column) + } + } + } + } + assertTrue(false, "Expected an out-of-range Flight timestamp error") + } catch (Exception error) { + assertTrue(error.toString().contains("outside the supported 0001-9999 range"), error.toString()) + } + } +} diff --git a/regression-test/suites/arrow_flight_sql_p0/test_flight_timestamp_range.groovy b/regression-test/suites/arrow_flight_sql_p0/test_flight_timestamp_range.groovy new file mode 100644 index 00000000000..c494bd84684 --- /dev/null +++ b/regression-test/suites/arrow_flight_sql_p0/test_flight_timestamp_range.groovy @@ -0,0 +1,81 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import java.time.LocalDateTime + +suite("test_flight_timestamp_range", "arrow_flight_sql") { + // Reuse the configured Flight connection so TLS settings cannot skip the assertions or change credentials. + def flight = context.getArrowFlightSqlConnection() + def input = "${context.dbName}.flight_timestamp_input" + def expectRangeError = { String query -> + try { + arrow_flight_sql(query) + assertTrue(false, "Expected an out-of-range Flight timestamp error") + } catch (Exception error) { + assertTrue(error.toString().contains("outside the supported 0001-9999 range"), error.toString()) + } + } + arrow_flight_sql "DROP TABLE IF EXISTS ${input}" + arrow_flight_sql """CREATE TABLE ${input} (id INT, value STRING) + DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 + PROPERTIES ("replication_num" = "1")""" + try { + arrow_flight_sql """INSERT INTO ${input} VALUES + (1, '0001-01-01 00:00:00.000000'), + (2, '9999-12-31 23:59:59.999999'), + (3, '1969-12-31 23:59:59.999999'), + (4, NULL), (5, '0000-12-31 00:00:00.000000')""" + // Check that the source value reaches the date conversion instead of becoming SQL NULL. + assertEquals([['0000-12-31 00:00:00.000000']], + arrow_flight_sql("SELECT CAST(CAST(value AS DATETIME(6)) AS STRING) FROM ${input} WHERE id = 5")) + for (def scale : [0, 3, 6]) { + expectRangeError("SELECT CAST(value AS DATETIME(${scale})) AS event_time FROM ${input} WHERE id = 5") + } + expectRangeError("SELECT CAST('0000-12-31 00:00:00' AS DATETIME(6)) AS event_time") + for (def expression : ["array(CAST(value AS DATETIME(6)))", + "named_struct('child', CAST(value AS DATETIME(6)))", + "map('key', CAST(value AS DATETIME(6)))", + "map(CAST(value AS DATETIME(6)), 'value')"]) { + expectRangeError("SELECT ${expression} AS nested FROM ${input} WHERE id = 5") + } + expectRangeError("SELECT CAST(value AS DATETIME(6)) AS event_time FROM ${input} ORDER BY id") + flight.createStatement().withCloseable { statement -> + statement.executeQuery("SELECT CAST(value AS DATETIME(6)) FROM ${input} WHERE id <= 4 ORDER BY id") + .withCloseable { rows -> + // Typed access preserves proleptic calendar dates and microseconds. + for (def expected : ["0001-01-01T00:00:00", "9999-12-31T23:59:59.999999", + "1969-12-31T23:59:59.999999"]) { + assertTrue(rows.next()) + assertEquals(LocalDateTime.parse(expected), rows.getObject(1, LocalDateTime.class)) + } + assertTrue(rows.next()) + assertNull(rows.getObject(1, LocalDateTime.class)) + assertFalse(rows.next()) + } + statement.executeQuery("SELECT array(CAST(value AS DATETIME(6))) FROM ${input} WHERE id = 4") + .withCloseable { rows -> + assertTrue(rows.next()) + def values = rows.getArray(1).getArray() + assertEquals(1, values.length) + assertNull(values[0]) + assertFalse(rows.next()) + } + } + } finally { + arrow_flight_sql "DROP TABLE IF EXISTS ${input}" + } +} diff --git a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_all_types_select.groovy b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_all_types_select.groovy index 31530d7bfc1..c6919d62b57 100644 --- a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_all_types_select.groovy +++ b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_all_types_select.groovy @@ -67,8 +67,9 @@ suite("test_remote_doris_all_types_select", "p0,external,doris,external_docker,e ); """ + // Successful Flight reads require timestamps within the supported 0001-9999 range. sql """ - INSERT INTO `test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"}, '{"k1":"v31", "k2": 300, "k3": [123, 456], "k4": [], " [...] + INSERT INTO `test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0001-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"}, '{"k1":"v31", "k2": 300, "k3": [123, 456], "k4": [], " [...] """ sql """ INSERT INTO `test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t` values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL) @@ -108,7 +109,7 @@ suite("test_remote_doris_all_types_select", "p0,external,doris,external_docker,e """ sql """ - INSERT INTO `test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t2` values('2025-05-18 01:00:00.000', [true], [-128], [-32768], [-2147483648], [-9223372036854775808], [-1234567890123456790], [-123.456], [-123456.789], [-123457], [-123456789012346], [-1234567890123456789012345678], ['0000-01-01'], ['0000-01-01 00:00:00'], ['A'], ['Hello'], ['Hello, Doris!']) + INSERT INTO `test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t2` values('2025-05-18 01:00:00.000', [true], [-128], [-32768], [-2147483648], [-9223372036854775808], [-1234567890123456790], [-123.456], [-123456.789], [-123457], [-123456789012346], [-1234567890123456789012345678], ['0000-01-01'], ['0001-01-01 00:00:00'], ['A'], ['Hello'], ['Hello, Doris!']) """ sql """ INSERT INTO `test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t2` values('2025-05-18 02:00:00.000', [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL]) @@ -172,6 +173,16 @@ suite("test_remote_doris_all_types_select", "p0,external,doris,external_docker,e select * from `test_remote_doris_all_types_select_catalog`.`test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t3` order by id """ + // Keep zero-year rejection covered separately from successful all-type round trips. + sql """INSERT INTO test_remote_doris_all_types_select_db.test_remote_doris_all_types_select_t3 + (id, datetime_0) VALUES ('2025-05-18 02:00:00', '0000-01-01 00:00:00')""" + test { + sql """SELECT datetime_0 FROM test_remote_doris_all_types_select_catalog + .test_remote_doris_all_types_select_db.test_remote_doris_all_types_select_t3 + WHERE id = '2025-05-18 02:00:00'""" + exception "outside the supported 0001-9999 range" + } + sql """ DROP DATABASE IF EXISTS test_remote_doris_all_types_select_db """ sql """ DROP CATALOG IF EXISTS `test_remote_doris_all_types_select_catalog` """ } diff --git a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_statistics.groovy b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_statistics.groovy index 223c294d811..210946f397d 100644 --- a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_statistics.groovy +++ b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_statistics.groovy @@ -82,8 +82,9 @@ suite("test_remote_doris_statistics", "p0,external,doris,external_docker,externa ); """ + // Successful Flight reads require timestamps within the supported 0001-9999 range. sql """ - INSERT INTO `test_remote_doris_statistics_db`.`test_remote_doris_statistics_t1` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"}) + INSERT INTO `test_remote_doris_statistics_db`.`test_remote_doris_statistics_t1` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0001-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"}) """ sql """ INSERT INTO `test_remote_doris_statistics_db`.`test_remote_doris_statistics_t1` values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL) diff --git a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_table_stats.groovy b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_table_stats.groovy index e6de44e005a..9486e888a6d 100644 --- a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_table_stats.groovy +++ b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_table_stats.groovy @@ -82,8 +82,9 @@ suite("test_remote_doris_table_stats", "p0,external,doris,external_docker,extern ); """ + // Successful Flight reads require timestamps within the supported 0001-9999 range. sql """ - INSERT INTO `test_remote_doris_table_stats_db`.`test_remote_doris_table_stats_t1` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"}) + INSERT INTO `test_remote_doris_table_stats_db`.`test_remote_doris_table_stats_t1` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0001-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"}) """ sql """ INSERT INTO `test_remote_doris_table_stats_db`.`test_remote_doris_table_stats_t1` values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
