This is an automated email from the ASF dual-hosted git repository.
mrhhsg pushed a change to branch adaptive_batch_size
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 520fbd968f7 [feature](be) Implement adaptive batch size with EWMA
prediction for byte-budget control
add 157a35e2580 [opt](explode) Optimize explode_outer and posexplode
(#62069)
add 0a3bea51271 [refactor](fe) Decouple SlotRef dependencies:
GlobalVariable, Column serialization, Expr visitor (#61918)
add 405541b2de5 [fix](tvf) fix wrong backend assignment for insert into
local tvf (#62143)
add a770f16d9f3 [Feature](compaction) add CompactionTaskTracker with
system table and HTTP API (#61696)
add 4ad3989c712 [fix](fe) Fix null pointer exception in sessionVariables
after upgrade (#61959)
add 16682030e14 [bugfix] fix show partitions command for external catalog
validations incompatible with partitions tvf (#62134)
add d224c08fbb6 [opt](memory) Remove unused fields from CloudReplica
(#62125)
add f754d728b99 [refactor](build) Extract third-party filesystem JAR
packaging into post-build.sh (#62144)
add 03290d2399a [feature](tso) Add global monotonically increasing
Timestamp Oracle(TSO) (#61199)
add a0b9e3784f9 [Bug](function) fix wrong result of window funnel v2 +
deduplication/fixed mode (#62043)
add 9c7ec9e132c [fix](tpcds) Change sr_ticket_number and ss_ticket_number
to bigint in SF10000 schema (#62158)
add 2cf8a386e28 [fix](lineage) Propagate connectAttributes through FE
forward path for scheduleInfo (#62082)
add 958731935a3 [chore](beut) using uuid as dir name to make the test more
stable (#62127)
add 32d40262c3f [fix](metrics) Fix prepared statement QPS metrics not
counted when audit log disabled (#61621)
add 0e4ffd0b40a [refine](be) remove -Wno-deprecated-declarations and fix
all deprecated usages (#62098)
add 63654958c67 [fix](audit) Fix UnboundAlias digest when name comes from
child (#62160)
add 361bbed335a [fix](profile) populate inverted index metrics for
SEARCH() query path (#62121)
add 094438f551c [refactor](fe) Introduce pluggable filesystem SPI to unify
cloud/storage and fe-core/fs (#62023)
add a2bc37f021e [fix](auth) Fix HTTP API authentication framework for
admin operations (#60761)
add 0ba5b455250 [refactor](fe) Decouple DecimalLiteral from
SessionVariable (#62166)
add b31412ae784 [chore](cloud) Change range iterator check pattern from
do...while() to while() to get better robustness (#59019)
add 7bd2ddae9fc [bugfix] fix doris build failure because of no cuda
support scene, and add FAISS_ENABLE_GPU switch to enable it or not by user,
default value as OFF (#62133)
add c39633520a0 [fix](fe) Fix NoSuchElementException when count with
MATCH_ALL expression (#62172)
add b95bf645c4b [refactor](fe) Decouple analysis.DateLiteral from
nereids.util.DateUtils (#62154)
add 7ad64890477 [fix](memtable) using memtable memory instead of load
memory to calculate adaptive write buffer size (#62104)
add 9c226f5bf0f [fix](regression) isolate ann_index_p0 table names (#62178)
add ebf08f296db [fix](doris catalog) FragmentMgr should not cancel virtual
doris cluster query (#62135)
add e41a7fbe85e [chore](fe) clean some code (#62171)
add 1cf74d131ef [fix](paimon) add jindo oss support and token propagation
(#62106)
add 6ea8fa5d44e [fix](pipeline) Prevent wake_up() from submitting
finalized task to scheduler (#62155)
add 629ab2fcae5 [improvement](be) Optimize array distance functions to
avoid material… (#61969)
add 2f172a943dd [refine](code) Add comments and benchmarks for ColumnView.
(#61678)
add c9c2a5c1353 [feature](RoutineLoad) Support RoutineLoad IAM auth
(#61324)
add 06d114b4ce1 [Fix](pyudf) Fix import conflicts for modules with the
same top-level name (#62167)
add ff36db5b6fb [fix](memory) track IO layer read buffers via
MemTrackerLimiter using PODArray (#62032)
add 78da3c9d4e2 [fix](delete) avoid quorum stalls on delete push failures
(#61647)
add bb217873838 [improve](job) Support internal pass-through properties
with "__" prefix in streaming job (#62193)
add 0ccaaf423da [fix](test) Make S3FileWriter boundary test multipart mock
thread-safe (#62149)
add 2ea624eb2e3 [chore](fe-dep)Update aircompressor version to 2.0.3
(#62239)
add d2a9be1cb0d [fix](test) fix unstable test agg_strategy_variable
(#62169)
add 91d2bc2566c [opt](memory) limit max block bytes per batch in the write
path (#61821)
add dc9b25f1ca2 [chore](physcial plan) remove useless call of SlotDesc's
parent (#62233)
new 32f5950571d [feature](be) Implement adaptive batch size with EWMA
prediction for byte-budget control
new e88ad3a5cd5 [fix](be) Fix adaptive batch size review issues
new ad43a025dbf [fix](be) Address remaining review comments for adaptive
batch size
new c64212666cc [fix](be) Enforce byte budget in distinct streaming agg
and topN path
new 741616fb778 [refactor](be) Extract BlockBudget utility for block
row/byte budget control
new b4f1d52451c [test](be) Add byte-budget unit tests for 6 operators
new 90823c02911 [fix](be) Fix ASAN heap-buffer-overflow in segment
iterator with adaptive batch size
new 1c426b7f3d8 fix regression test cases
new 7d63a11f9e4 remove config::load_reader_max_block_bytes
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (520fbd968f7)
\
N -- N -- N refs/heads/adaptive_batch_size (7d63a11f9e4)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.gitleaks.toml | 14 +
be/CMakeLists.txt | 1 -
be/benchmark/benchmark_column_view.hpp | 484 +++++++++++
be/benchmark/benchmark_fastunion.hpp | 2 +-
be/benchmark/benchmark_hll_merge.hpp | 2 +-
be/benchmark/benchmark_main.cpp | 7 +-
be/benchmark/benchmark_string.hpp | 4 +-
be/benchmark/binary_cast_benchmark.hpp | 2 +-
be/src/agent/task_worker_pool.cpp | 8 +-
be/src/cloud/cloud_base_compaction.h | 7 +
be/src/cloud/cloud_compaction_action.cpp | 12 +-
be/src/cloud/cloud_cumulative_compaction.h | 7 +
be/src/cloud/cloud_full_compaction.h | 7 +
be/src/cloud/cloud_storage_engine.cpp | 129 ++-
be/src/cloud/cloud_storage_engine.h | 10 +-
be/src/cloud/pb_convert.cpp | 12 +
be/src/common/block_budget.h | 93 ++
be/src/common/config.cpp | 3 +
be/src/common/config.h | 3 +
be/src/common/multi_version.h | 9 +-
be/src/core/column/column_array.cpp | 4 -
be/src/core/column/column_execute_util.h | 24 +
be/src/core/column/column_vector.cpp | 1 -
be/src/core/column/column_vector.h | 7 +-
be/src/core/data_type/data_type.cpp | 2 -
be/src/core/data_type/data_type_number_base.cpp | 3 +-
be/src/core/data_type/data_type_number_base.h | 2 +-
be/src/core/data_type/primitive_type.h | 10 +-
.../data_type_serde/data_type_number_serde.cpp | 13 +-
.../core/data_type_serde/data_type_number_serde.h | 6 +-
be/src/core/field.cpp | 6 -
be/src/exec/common/hash_table/join_hash_table.h | 65 +-
.../exec/operator/aggregation_source_operator.cpp | 17 +-
.../distinct_streaming_aggregation_operator.cpp | 29 +-
.../distinct_streaming_aggregation_operator.h | 1 +
.../operator/join/process_hash_table_probe_impl.h | 55 +-
.../operator/nested_loop_join_probe_operator.cpp | 31 +-
be/src/exec/operator/olap_scan_operator.cpp | 40 +-
be/src/exec/operator/operator.h | 1 -
be/src/exec/operator/schema_scan_operator.cpp | 6 +-
be/src/exec/operator/set_source_operator.cpp | 9 +-
.../operator/streaming_aggregation_operator.cpp | 9 +-
be/src/exec/operator/table_function_operator.cpp | 230 +++--
be/src/exec/operator/union_source_operator.cpp | 6 +-
be/src/exec/pipeline/pipeline_task.cpp | 23 +-
be/src/exec/sort/vsorted_run_merger.cpp | 5 +-
.../aggregate_function_approx_count_distinct.h | 2 +-
.../aggregate/aggregate_function_min_max_impl.h | 1 -
.../aggregate_function_window_funnel_v2.h | 266 +++---
.../exprs/function/array/function_array_distance.h | 173 ++--
be/src/exprs/function/function_search.cpp | 113 ++-
be/src/exprs/function/functions_comparison.h | 1 -
be/src/exprs/function/round.h | 8 +-
.../exprs/lambda_function/varray_sort_function.cpp | 3 +-
be/src/exprs/table_function/table_function.h | 1 +
be/src/exprs/table_function/vexplode.cpp | 2 +-
be/src/exprs/table_function/vexplode_v2.cpp | 3 +-
be/src/format/csv/csv_reader.cpp | 42 +-
be/src/format/json/new_json_reader.cpp | 27 +-
be/src/format/orc/vorc_reader.cpp | 3 +
be/src/format/orc/vorc_reader.h | 1 +
be/src/format/parquet/vparquet_reader.cpp | 1 +
be/src/format/parquet/vparquet_reader.h | 1 +
.../schema_compaction_tasks_scanner.cpp | 504 +++++++++++
.../schema_compaction_tasks_scanner.h} | 32 +-
.../information_schema/schema_rowsets_scanner.cpp | 11 +
be/src/information_schema/schema_scanner.cpp | 3 +
be/src/io/fs/buffered_reader.cpp | 22 +-
be/src/io/fs/buffered_reader.h | 4 +-
be/src/io/fs/hdfs_file_system.cpp | 8 +-
be/src/io/fs/http_file_reader.cpp | 14 +-
be/src/io/fs/http_file_reader.h | 3 +-
be/src/load/memtable/memtable.cpp | 10 +-
be/src/load/routine_load/data_consumer.cpp | 43 +-
be/src/load/routine_load/data_consumer.h | 4 +
be/src/runtime/aws_msk_iam_auth.cpp | 517 +++++++++++
be/src/runtime/aws_msk_iam_auth.h | 169 ++++
be/src/runtime/fragment_mgr.cpp | 18 +-
be/src/runtime/query_context.cpp | 2 +
be/src/runtime/query_context.h | 3 +-
be/src/runtime/runtime_state.h | 8 +-
be/src/service/http/action/compaction_action.cpp | 51 +-
.../http/action/compaction_profile_action.cpp | 270 ++++++
...health_action.h => compaction_profile_action.h} | 15 +-
be/src/service/http/action/health_action.h | 4 +-
be/src/service/http/action/http_stream.cpp | 11 +-
be/src/service/http/action/http_stream.h | 5 +-
be/src/service/http/action/pad_rowset_action.cpp | 2 +-
be/src/service/http/action/stream_load.cpp | 11 +-
be/src/service/http/action/stream_load.h | 6 +-
be/src/service/http/action/stream_load_2pc.cpp | 6 +-
be/src/service/http/action/stream_load_2pc.h | 5 +-
be/src/service/http/http_handler_with_auth.cpp | 4 +
be/src/service/http_service.cpp | 12 +
be/src/storage/compaction/base_compaction.h | 6 +
be/src/storage/compaction/compaction.cpp | 90 +-
be/src/storage/compaction/compaction.h | 22 +
be/src/storage/compaction/cumulative_compaction.h | 6 +
be/src/storage/compaction/full_compaction.h | 6 +
be/src/storage/compaction_task_tracker.cpp | 271 ++++++
be/src/storage/compaction_task_tracker.h | 203 +++++
be/src/storage/data_dir.cpp | 29 +-
.../storage/index/ann/cmake-protect/CMakeLists.txt | 6 +-
.../inverted/inverted_index_compound_reader.cpp | 2 +-
be/src/storage/iterator/vcollect_iterator.cpp | 51 ++
be/src/storage/iterator/vcollect_iterator.h | 5 +
be/src/storage/merger.cpp | 10 +-
be/src/storage/merger.h | 6 +-
be/src/storage/olap_server.cpp | 71 +-
be/src/storage/rowset/beta_rowset_reader.cpp | 9 +-
be/src/storage/rowset/rowset.cpp | 3 +-
be/src/storage/rowset/rowset.h | 5 +-
be/src/storage/rowset/rowset_meta.h | 4 +
be/src/storage/rowset_builder.cpp | 4 +-
.../segment/adaptive_block_size_predictor.cpp | 4 +-
be/src/storage/segment/segment_iterator.cpp | 10 +-
be/src/storage/storage_engine.h | 14 +-
be/src/storage/tablet/tablet_reader.h | 7 +-
.../storage/task/engine_publish_version_task.cpp | 65 +-
be/src/storage/task/engine_publish_version_task.h | 37 +-
be/src/storage/txn/txn_manager.cpp | 14 +-
be/src/storage/txn/txn_manager.h | 6 +-
be/src/udf/python/python_server.py | 8 +-
be/test/common/block_budget_test.cpp | 132 +++
be/test/exec/operator/agg_operator_test.cpp | 53 ++
.../exec/operator/analytic_sink_operator_test.cpp | 2 +-
...istinct_streaming_aggregation_operator_test.cpp | 37 +-
.../operator/exchange_source_operator_test.cpp | 2 +-
.../exec/operator/hashjoin_probe_operator_test.cpp | 5 +-
.../operator/partition_sort_sink_operator_test.cpp | 2 +-
.../exec/operator/query_cache_operator_test.cpp | 2 +-
be/test/exec/operator/repeat_operator_test.cpp | 2 +-
be/test/exec/operator/set_operator_test.cpp | 45 +-
be/test/exec/operator/sort_operator_test.cpp | 5 +-
.../exec/operator/streaming_agg_operator_test.cpp | 56 ++
.../exec/operator/table_function_operator_test.cpp | 683 ++++++++++++++-
be/test/exec/operator/union_operator_test.cpp | 60 +-
be/test/exec/pipeline/dummy_task_queue.h | 9 +-
be/test/exec/pipeline/pipeline_task_test.cpp | 19 +-
.../exec/sink/arrow_result_block_buffer_test.cpp | 8 +-
be/test/exec/sink/result_block_buffer_test.cpp | 6 +-
.../exprs/aggregate/vec_window_funnel_v2_test.cpp | 743 ++++++++++++++++
be/test/exprs/function/cast/cast_test.h | 7 +-
be/test/format/parquet/parquet_expr_test.cpp | 2 +-
be/test/io/fs/s3_file_writer_test.cpp | 30 +-
be/test/runtime/aws_msk_iam_auth_test.cpp | 244 ++++++
.../fragment_mgr_cross_cluster_cancel_test.cpp | 156 ++++
.../workload_group/workload_group_manager_test.cpp | 26 +-
be/test/service/http/http_auth_test.cpp | 99 ++-
be/test/storage/compaction_task_tracker_test.cpp | 769 +++++++++++++++++
.../iterator/block_reader_batch_max_rows_test.cpp | 16 +
.../vcollect_iterator_collected_enough_test.cpp | 5 +-
.../segment/adaptive_block_size_predictor_test.cpp | 3 +-
.../inverted_index_compound_reader_test.cpp | 57 ++
be/test/storage/storage_engine_test.cpp | 28 +-
be/test/storage/txn/txn_manager_test.cpp | 28 +-
be/test/testutil/mock/mock_runtime_state.h | 6 +-
build.sh | 117 +--
cloud/src/meta-service/http_encode_key.cpp | 5 +-
cloud/src/meta-service/meta_service.cpp | 17 +-
cloud/src/meta-service/meta_service_job.cpp | 8 +-
cloud/src/meta-service/meta_service_resource.cpp | 8 +-
.../src/meta-service/meta_service_tablet_stats.cpp | 4 +-
cloud/src/meta-service/meta_service_txn.cpp | 16 +-
cloud/src/meta-store/blob_message.cpp | 9 +-
cloud/src/meta-store/txn_kv.cpp | 2 +-
cloud/src/recycler/checker.cpp | 84 +-
cloud/src/recycler/meta_checker.cpp | 4 +-
cloud/src/recycler/recycler.cpp | 14 +-
cloud/src/recycler/recycler.h | 11 +-
cloud/src/recycler/util.cpp | 4 +-
cloud/src/resource-manager/resource_manager.cpp | 4 +-
cloud/test/bvars_test.cpp | 4 +-
cloud/test/mem_txn_kv_test.cpp | 4 +-
cloud/test/meta_service_test.cpp | 14 +-
cloud/test/recycler_test.cpp | 28 +-
cloud/test/txn_kv_test.cpp | 12 +-
conf/fe.conf | 5 +
docker/thirdparties/jindofs-helpers.sh | 116 ++-
fe/.mvn/maven-build-cache-config.xml | 16 +
fe/be-java-extensions/paimon-scanner/pom.xml | 8 +-
.../paimon-scanner/src/main/resources/package.xml | 6 +
fe/be-java-extensions/preload-extensions/pom.xml | 1 -
.../apache/doris/catalog/NameSpaceContext.java} | 36 +-
fe/fe-common/pom.xml | 48 ++
.../main/java/org/apache/doris/common/Config.java | 53 ++
.../java/org/apache/doris/qe/GlobalVariable.java | 107 ++-
.../main/java/org/apache/doris/qe/VarAttrDef.java | 87 ++
fe/fe-core/pom.xml | 132 +--
.../main/java/org/apache/doris/alter/Alter.java | 5 +-
.../apache/doris/alter/SchemaChangeHandler.java | 17 +-
.../java/org/apache/doris/analysis/ColumnDef.java | 8 +-
.../org/apache/doris/analysis/ColumnRefExpr.java | 12 -
.../org/apache/doris/analysis/DateLiteral.java | 218 +----
.../apache/doris/analysis/DateLiteralUtils.java | 282 ++++++
.../org/apache/doris/analysis/DecimalLiteral.java | 53 +-
.../apache/doris/analysis/DecimalLiteralUtils.java | 84 ++
.../main/java/org/apache/doris/analysis/Expr.java | 14 -
.../doris/analysis/ExprToColumnLabelVisitor.java | 51 --
.../doris/analysis/ExprToExprNameVisitor.java | 65 ++
.../doris/analysis/ExprToStringValueVisitor.java | 2 +-
.../apache/doris/analysis/FunctionCallExpr.java | 11 -
.../org/apache/doris/analysis/LiteralExpr.java | 12 +-
.../java/org/apache/doris/analysis/ParseNode.java | 13 -
.../apache/doris/analysis/PartitionExprUtil.java | 2 +-
.../org/apache/doris/analysis/SchemaTableType.java | 4 +-
.../java/org/apache/doris/analysis/Separator.java | 51 +-
.../java/org/apache/doris/analysis/SlotRef.java | 9 -
.../org/apache/doris/analysis/StatementBase.java | 25 +-
.../org/apache/doris/analysis/StorageBackend.java | 42 +-
.../org/apache/doris/analysis/TableSample.java | 16 +-
.../org/apache/doris/analysis/VariableExpr.java | 5 -
.../org/apache/doris/analysis/VirtualSlotRef.java | 4 -
.../org/apache/doris/backup/BackupHandler.java | 14 +-
.../java/org/apache/doris/backup/BackupJob.java | 6 +-
.../java/org/apache/doris/backup/Repository.java | 639 +++++++++-----
.../org/apache/doris/backup/RepositoryMgr.java | 8 +-
.../java/org/apache/doris/backup/RestoreJob.java | 13 +-
.../org/apache/doris/catalog/AliasFunction.java | 12 +-
.../org/apache/doris/catalog/AzureResource.java | 127 ++-
.../apache/doris/catalog/CatalogRecycleBin.java | 3 +-
.../apache/doris/catalog/CloudTabletStatMgr.java | 20 +-
.../main/java/org/apache/doris/catalog/Column.java | 314 +------
.../org/apache/doris/catalog/ColumnToProtobuf.java | 201 +++++
.../org/apache/doris/catalog/ColumnToThrift.java | 173 ++++
.../java/org/apache/doris/catalog/Database.java | 60 +-
.../java/org/apache/doris/catalog/DatabaseIf.java | 27 -
.../main/java/org/apache/doris/catalog/Env.java | 76 +-
.../org/apache/doris/catalog/HdfsStorageVault.java | 35 +-
.../doris/catalog/LocalTabletInvertedIndex.java | 6 +-
.../main/java/org/apache/doris/catalog/MTMV.java | 3 +
.../doris/catalog/MaterializedIndexMeta.java | 3 +
.../java/org/apache/doris/catalog/OlapTable.java | 29 +-
.../java/org/apache/doris/catalog/Replica.java | 19 -
.../java/org/apache/doris/catalog/S3Resource.java | 130 ++-
.../java/org/apache/doris/catalog/SchemaTable.java | 42 +
.../org/apache/doris/catalog/TableIndexes.java | 9 +-
.../org/apache/doris/catalog/TableProperty.java | 13 +
.../main/java/org/apache/doris/catalog/View.java | 4 +
.../catalog/constraint/ForeignKeyConstraint.java | 5 +-
.../cloud/alter/CloudSchemaChangeHandler.java | 3 +
.../apache/doris/cloud/backup/CloudRestoreJob.java | 4 +-
.../apache/doris/cloud/catalog/CloudReplica.java | 65 --
.../cloud/datasource/CloudInternalCatalog.java | 9 +-
.../apache/doris/cloud/load/CleanCopyJobTask.java | 27 +-
.../java/org/apache/doris/cloud/load/CopyJob.java | 2 +-
.../doris/cloud/load/CopyLoadPendingTask.java | 68 +-
.../org/apache/doris/cloud/stage/StageUtil.java | 74 +-
.../apache/doris/cloud/storage/AzureRemote.java | 336 --------
.../org/apache/doris/cloud/storage/BosRemote.java | 60 --
.../org/apache/doris/cloud/storage/CosRemote.java | 179 ----
.../apache/doris/cloud/storage/DefaultRemote.java | 364 --------
.../doris/cloud/storage/ListObjectsResult.java | 45 -
.../org/apache/doris/cloud/storage/MockRemote.java | 110 ---
.../org/apache/doris/cloud/storage/ObjectFile.java | 44 -
.../org/apache/doris/cloud/storage/ObjectInfo.java | 144 ++++
.../doris/cloud/storage/ObjectInfoAdapter.java | 201 +++++
.../org/apache/doris/cloud/storage/ObsRemote.java | 104 ---
.../org/apache/doris/cloud/storage/OssRemote.java | 191 -----
.../org/apache/doris/cloud/storage/RemoteBase.java | 272 ------
.../org/apache/doris/cloud/storage/S3Remote.java | 105 ---
.../doris/common/plugin/CloudPluginDownloader.java | 44 +-
.../doris/common/proc/BuildIndexProcDir.java | 3 +-
.../doris/common/proc/PartitionsProcDir.java | 3 +-
.../apache/doris/common/proc/RollupProcDir.java | 3 +-
.../doris/common/proc/SchemaChangeProcDir.java | 3 +-
.../org/apache/doris/common/util/BrokerReader.java | 19 +-
.../org/apache/doris/common/util/BrokerUtil.java | 274 ------
.../org/apache/doris/common/util/LocationPath.java | 2 +-
.../apache/doris/common/util/PropertyAnalyzer.java | 27 +-
.../org/apache/doris/datasource/CatalogIf.java | 3 -
.../apache/doris/datasource/ExternalCatalog.java | 10 +-
.../apache/doris/datasource/ExternalDatabase.java | 5 -
.../org/apache/doris/datasource/FileGroupInfo.java | 5 +-
.../doris/datasource/FilePartitionUtils.java | 126 +++
.../apache/doris/datasource/FileQueryScanNode.java | 3 +-
.../apache/doris/datasource/InternalCatalog.java | 21 +-
.../org/apache/doris/datasource/hive/AcidUtil.java | 98 +--
.../doris/datasource/hive/HMSExternalCatalog.java | 22 +-
.../doris/datasource/hive/HMSTransaction.java | 277 +++---
.../datasource/hive/HiveExternalMetaCache.java | 62 +-
.../org/apache/doris/datasource/hive/HiveUtil.java | 9 +-
.../doris/datasource/iceberg/IcebergUtils.java | 19 +-
.../datasource/iceberg/fileio/DelegateFileIO.java | 62 +-
.../iceberg/fileio/DelegateInputFile.java | 4 +-
.../iceberg/fileio/DelegateOutputFile.java | 32 +-
.../fileio/DelegateSeekableInputStream.java | 4 +-
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 7 +-
.../maxcompute/source/MaxComputeScanNode.java | 10 +-
.../paimon/source/PaimonPredicateConverter.java | 3 +-
.../storage/AbstractS3CompatibleProperties.java | 9 +
.../property/storage/AzureProperties.java | 8 +
.../datasource/property/storage/COSProperties.java | 9 +
.../datasource/property/storage/OBSProperties.java | 9 +
.../datasource/property/storage/OSSProperties.java | 3 +
.../source/TrinoConnectorPredicateConverter.java | 2 +-
.../java/org/apache/doris/fs/DirectoryLister.java | 7 +-
.../main/java/org/apache/doris/fs/FileSystem.java | 147 ----
.../java/org/apache/doris/fs/FileSystemCache.java | 33 +-
.../org/apache/doris/fs/FileSystemDescriptor.java | 105 +++
.../apache/doris/fs/FileSystemDirectoryLister.java | 23 +-
.../org/apache/doris/fs/FileSystemFactory.java | 193 ++++-
.../org/apache/doris/fs/FileSystemIOException.java | 65 --
.../apache/doris/fs/FileSystemPluginManager.java | 146 ++++
.../apache/doris/fs/FileSystemProviderImpl.java | 42 -
.../org/apache/doris/fs/FsStorageTypeAdapter.java | 59 ++
.../org/apache/doris/fs/LocalDfsFileSystem.java | 199 -----
.../org/apache/doris/fs/PersistentFileSystem.java | 57 --
.../apache/doris/fs/RemoteFileRemoteIterator.java | 47 -
.../java/org/apache/doris/fs/SchemaTypeMapper.java | 6 +
.../apache/doris/fs/SpiSwitchingFileSystem.java | 206 +++++
.../doris/fs/StoragePropertiesConverter.java | 92 ++
.../org/apache/doris/fs/StorageTypeMapper.java | 68 --
.../fs/TransactionScopeCachingDirectoryLister.java | 35 +-
.../java/org/apache/doris/fs/io/DorisInput.java | 54 --
.../org/apache/doris/fs/io/DorisInputFile.java | 75 --
.../org/apache/doris/fs/io/DorisInputStream.java | 51 --
.../org/apache/doris/fs/io/DorisOutputFile.java | 59 --
.../org/apache/doris/fs/io/hdfs/HdfsInput.java | 107 ---
.../org/apache/doris/fs/io/hdfs/HdfsInputFile.java | 155 ----
.../apache/doris/fs/io/hdfs/HdfsInputStream.java | 180 ----
.../apache/doris/fs/io/hdfs/HdfsOutputFile.java | 95 ---
.../apache/doris/fs/io/hdfs/HdfsOutputStream.java | 135 ---
.../org/apache/doris/fs/obj/AzureObjStorage.java | 626 --------------
.../java/org/apache/doris/fs/obj/ObjStorage.java | 70 --
.../java/org/apache/doris/fs/obj/S3ObjStorage.java | 862 -------------------
.../doris/fs/operations/BrokerFileOperations.java | 163 ----
.../apache/doris/fs/operations/BrokerOpParams.java | 57 --
.../doris/fs/operations/HDFSFileOperations.java | 135 ---
.../apache/doris/fs/operations/HDFSOpParams.java | 72 --
.../org/apache/doris/fs/operations/OpParams.java | 54 --
.../apache/doris/fs/remote/AzureFileSystem.java | 89 --
.../apache/doris/fs/remote/BrokerFileSystem.java | 712 ----------------
.../org/apache/doris/fs/remote/ObjFileSystem.java | 187 ----
.../org/apache/doris/fs/remote/RemoteFile.java | 117 ---
.../apache/doris/fs/remote/RemoteFileSystem.java | 72 --
.../org/apache/doris/fs/remote/S3FileSystem.java | 143 ----
.../doris/fs/remote/SwitchingFileSystem.java | 131 ---
.../apache/doris/fs/remote/dfs/DFSFileSystem.java | 597 -------------
.../remote/dfs/DFSFileSystemPhantomReference.java | 44 -
.../apache/doris/fs/remote/dfs/OFSFileSystem.java | 27 -
.../fs/remote/dfs/RemoteFSPhantomManager.java | 127 ---
.../doris/httpv2/controller/BaseController.java | 1 +
.../apache/doris/httpv2/rest/BackendsAction.java | 4 -
.../doris/httpv2/rest/BootstrapFinishAction.java | 4 +-
.../doris/httpv2/rest/CheckDecommissionAction.java | 6 +-
.../doris/httpv2/rest/CheckWalSizeAction.java | 6 +-
.../apache/doris/httpv2/rest/ConnectionAction.java | 6 +-
.../apache/doris/httpv2/rest/CopyIntoAction.java | 15 +-
.../apache/doris/httpv2/rest/DebugPointAction.java | 15 +-
.../doris/httpv2/rest/ExtraBasepathAction.java | 4 +-
.../doris/httpv2/rest/FeVersionInfoAction.java | 7 +-
.../apache/doris/httpv2/rest/GetLogFileAction.java | 7 +-
.../doris/httpv2/rest/GetSmallFileAction.java | 7 +-
.../org/apache/doris/httpv2/rest/HealthAction.java | 6 +-
.../apache/doris/httpv2/rest/MetaInfoAction.java | 16 +-
.../doris/httpv2/rest/MetaReplayerCheckAction.java | 7 +-
.../apache/doris/httpv2/rest/MetricsAction.java | 5 +-
.../apache/doris/httpv2/rest/MinidumpAction.java | 7 +-
.../apache/doris/httpv2/rest/ProfileAction.java | 11 +-
.../apache/doris/httpv2/rest/QueryStatsAction.java | 26 +-
.../doris/httpv2/rest/RestBaseController.java | 18 +
.../apache/doris/httpv2/rest/RowCountAction.java | 7 +-
.../apache/doris/httpv2/rest/SetConfigAction.java | 6 +-
.../doris/httpv2/rest/StorageTypeCheckAction.java | 7 +-
.../org/apache/doris/httpv2/rest/TSOAction.java | 97 +++
.../doris/httpv2/rest/manager/ClusterAction.java | 11 +-
.../doris/httpv2/rest/manager/NodeAction.java | 13 +-
.../httpv2/rest/manager/QueryProfileAction.java | 9 +-
.../apache/doris/httpv2/restv2/ImportAction.java | 30 +-
.../java/org/apache/doris/info/TableNameInfo.java | 75 +-
.../org/apache/doris/info/TableNameInfoUtils.java | 104 +++
.../java/org/apache/doris/info/TableRefInfo.java | 4 +-
.../insert/streaming/StreamingJobProperties.java | 7 +
.../job/offset/s3/S3SourceOffsetProvider.java | 51 +-
.../org/apache/doris/journal/JournalEntity.java | 6 +
.../apache/doris/journal/local/LocalJournal.java | 8 +
.../main/java/org/apache/doris/load/DeleteJob.java | 21 +-
.../main/java/org/apache/doris/load/ExportJob.java | 3 +-
.../main/java/org/apache/doris/load/ExportMgr.java | 12 +-
.../doris/load/loadv2/BrokerLoadPendingTask.java | 28 +-
.../load/routineload/kafka/KafkaConfiguration.java | 15 +
.../kafka/KafkaDataSourceProperties.java | 80 +-
.../java/org/apache/doris/master/Checkpoint.java | 2 -
.../java/org/apache/doris/metric/MetricRepo.java | 28 +
.../org/apache/doris/mtmv/MTMVPartitionUtil.java | 9 +-
.../java/org/apache/doris/mtmv/MTMVService.java | 12 +-
.../org/apache/doris/mysql/MysqlAuthPacket.java | 2 +-
.../apache/doris/mysql/privilege/RoleManager.java | 2 +-
.../doris/mysql/privilege/TablePrivEntry.java | 4 +-
.../doris/nereids/analyzer/UnboundAlias.java | 2 +-
.../glue/translator/PhysicalPlanTranslator.java | 113 +--
.../glue/translator/PlanTranslatorContext.java | 15 +-
.../doris/nereids/load/NereidsFileGroupInfo.java | 6 +-
.../nereids/load/NereidsLoadPlanInfoCollector.java | 4 +-
.../doris/nereids/minidump/MinidumpUtils.java | 4 +-
.../rules/implementation/AggregateStrategies.java | 14 +-
.../nereids/rules/rewrite/ForeignKeyContext.java | 5 +-
.../doris/nereids/stats/FilterEstimation.java | 3 +-
.../trees/plans/commands/AddConstraintCommand.java | 9 +-
.../commands/AdminCancelRepairTableCommand.java | 2 +-
.../plans/commands/AdminCompactTableCommand.java | 2 +-
.../plans/commands/AdminRepairTableCommand.java | 2 +-
.../commands/AdminSetPartitionVersionCommand.java | 2 +-
.../plans/commands/AdminSetTableStatusCommand.java | 2 +-
.../plans/commands/AlterColumnStatsCommand.java | 2 +-
.../trees/plans/commands/AlterTableCommand.java | 2 +-
.../plans/commands/AlterTableStatsCommand.java | 2 +-
.../trees/plans/commands/AnalyzeTableCommand.java | 2 +-
.../plans/commands/CancelAlterTableCommand.java | 2 +-
.../plans/commands/CancelBuildIndexCommand.java | 2 +-
.../plans/commands/CleanQueryStatsCommand.java | 2 +-
.../commands/CreateMaterializedViewCommand.java | 2 +-
.../trees/plans/commands/CreatePolicyCommand.java | 2 +-
.../trees/plans/commands/CreateStageCommand.java | 23 +-
.../trees/plans/commands/DescribeCommand.java | 2 +-
.../plans/commands/DropCachedStatsCommand.java | 2 +-
.../plans/commands/DropConstraintCommand.java | 4 +-
.../commands/DropMaterializedViewCommand.java | 2 +-
.../trees/plans/commands/DropRowPolicyCommand.java | 2 +-
.../trees/plans/commands/DropStatsCommand.java | 2 +-
.../trees/plans/commands/DropStreamCommand.java | 2 +-
.../trees/plans/commands/DropTableCommand.java | 2 +-
.../trees/plans/commands/DropViewCommand.java | 2 +-
.../trees/plans/commands/ExecuteActionCommand.java | 2 +-
.../trees/plans/commands/LockTablesCommand.java | 2 +-
.../plans/commands/RecoverPartitionCommand.java | 2 +-
.../trees/plans/commands/RecoverTableCommand.java | 2 +-
.../trees/plans/commands/ShowAnalyzeCommand.java | 2 +-
.../commands/ShowColumnHistogramStatsCommand.java | 2 +-
.../plans/commands/ShowColumnStatsCommand.java | 2 +-
.../trees/plans/commands/ShowColumnsCommand.java | 2 +-
.../plans/commands/ShowConstraintsCommand.java | 4 +-
.../ShowCreateMaterializedViewCommand.java | 2 +-
.../plans/commands/ShowCreateStreamCommand.java | 2 +-
.../plans/commands/ShowCreateTableCommand.java | 2 +-
.../plans/commands/ShowCreateViewCommand.java | 2 +-
.../trees/plans/commands/ShowDataCommand.java | 2 +-
.../trees/plans/commands/ShowDataSkewCommand.java | 4 +-
.../plans/commands/ShowIndexStatsCommand.java | 2 +-
.../plans/commands/ShowPartitionsCommand.java | 4 +-
.../plans/commands/ShowQueryStatsCommand.java | 2 +-
.../commands/ShowQueuedAnalyzeJobsCommand.java | 2 +-
.../commands/ShowReplicaDistributionCommand.java | 2 +-
.../plans/commands/ShowReplicaStatusCommand.java | 2 +-
.../plans/commands/ShowTableStatsCommand.java | 2 +-
.../commands/ShowTabletsFromTableCommand.java | 2 +-
.../trees/plans/commands/ShowViewCommand.java | 4 +-
.../trees/plans/commands/TruncateTableCommand.java | 2 +-
.../trees/plans/commands/UnsetVariableCommand.java | 3 +-
.../trees/plans/commands/WarmUpClusterCommand.java | 2 +-
.../trees/plans/commands/info/AlterMTMVInfo.java | 2 +-
.../trees/plans/commands/info/AlterViewInfo.java | 2 +-
.../trees/plans/commands/info/BuildIndexOp.java | 2 +-
.../plans/commands/info/CancelMTMVTaskInfo.java | 2 +-
.../trees/plans/commands/info/CopyIntoInfo.java | 17 +-
.../trees/plans/commands/info/CreateIndexOp.java | 2 +-
.../trees/plans/commands/info/CreateMTMVInfo.java | 2 +-
.../plans/commands/info/CreateStreamInfo.java | 4 +-
.../plans/commands/info/CreateTableLikeInfo.java | 4 +-
.../trees/plans/commands/info/CreateViewInfo.java | 2 +-
.../trees/plans/commands/info/DropIndexOp.java | 2 +-
.../trees/plans/commands/info/DropMTMVInfo.java | 2 +-
.../trees/plans/commands/info/LabelNameInfo.java | 4 +-
.../commands/info/ModifyTablePropertiesOp.java | 16 +
.../trees/plans/commands/info/PauseMTMVInfo.java | 2 +-
.../trees/plans/commands/info/RefreshMTMVInfo.java | 2 +-
.../trees/plans/commands/info/ResumeMTMVInfo.java | 2 +-
.../plans/commands/info/ShowCreateMTMVInfo.java | 2 +-
.../commands/insert/InsertIntoTVFCommand.java | 15 +-
.../commands/refresh/RefreshTableCommand.java | 2 +-
.../worker/job/UnassignedJobBuilder.java | 16 +-
.../worker/job/UnassignedLocalTVFSinkJob.java | 68 ++
.../plans/logical/LogicalCatalogRelation.java | 3 +-
.../plans/physical/PhysicalCatalogRelation.java | 3 +-
.../apache/doris/nereids/types/VariantField.java | 20 +-
.../java/org/apache/doris/persist/EditLog.java | 9 +
.../apache/doris/persist/GlobalVarPersistInfo.java | 8 +-
.../org/apache/doris/persist/OperationType.java | 2 +
.../org/apache/doris/persist/gson/GsonUtils.java | 23 +-
.../doris/persist/meta/MetaPersistMethod.java | 6 +
.../doris/persist/meta/PersistMetaModules.java | 2 +-
.../planner/BackendPartitionedSchemaScanNode.java | 1 +
.../apache/doris/planner/MaterializationNode.java | 3 +-
.../org/apache/doris/planner/OlapScanNode.java | 5 +-
.../org/apache/doris/planner/OlapTableSink.java | 5 +-
.../org/apache/doris/planner/TVFTableSink.java | 3 +-
.../java/org/apache/doris/qe/AuditLogHelper.java | 152 ++--
.../java/org/apache/doris/qe/ConnectContext.java | 37 +-
.../java/org/apache/doris/qe/ConnectProcessor.java | 12 +-
.../java/org/apache/doris/qe/FEOpExecutor.java | 5 +
.../org/apache/doris/qe/MysqlConnectProcessor.java | 3 +
.../java/org/apache/doris/qe/SessionVariable.java | 942 ++++++++++-----------
.../java/org/apache/doris/qe/StmtExecutor.java | 11 +-
.../main/java/org/apache/doris/qe/VariableMgr.java | 96 +--
.../apache/doris/statistics/AnalysisManager.java | 5 +-
.../doris/statistics/StatisticsJobAppender.java | 9 +-
.../doris/statistics/util/StatisticsUtil.java | 185 +---
.../ExternalFileTableValuedFunction.java | 33 +-
.../ParquetMetadataTableValuedFunction.java | 14 +-
.../apache/doris/task/AlterInvertedIndexTask.java | 3 +-
.../org/apache/doris/task/AlterReplicaTask.java | 3 +-
.../org/apache/doris/task/CreateReplicaTask.java | 3 +-
.../main/java/org/apache/doris/task/PushTask.java | 6 +
.../doris/transaction/DatabaseTransactionMgr.java | 70 +-
.../doris/transaction/HiveTransactionManager.java | 10 +-
.../apache/doris/transaction/TableCommitInfo.java | 17 +-
.../transaction/TransactionManagerFactory.java | 6 +-
.../apache/doris/transaction/TransactionState.java | 11 +
.../main/java/org/apache/doris/tso/TSOService.java | 526 ++++++++++++
.../java/org/apache/doris/tso/TSOTimestamp.java | 247 ++++++
.../org/apache/doris/analysis/DateLiteralTest.java | 99 ---
.../doris/analysis/DateLiteralUtilsTest.java | 427 ++++++++++
.../doris/analysis/DecimalLiteralUtilsTest.java | 277 ++++++
.../org/apache/doris/analysis/ExprToSqlTest.java | 37 +-
.../analysis/ExprToStringValueVisitorTest.java | 10 +-
.../org/apache/doris/backup/BackupJobTest.java | 11 +-
.../org/apache/doris/backup/BrokerStorageTest.java | 192 -----
.../org/apache/doris/backup/RepositoryTest.java | 498 +++++++++--
.../org/apache/doris/backup/RestoreJobTest.java | 3 +-
.../doris/catalog/SessionVariablesNullFixTest.java | 127 +++
.../doris/cloud/backup/CloudRestoreJobTest.java | 3 +-
.../org/apache/doris/cloud/load/CopyJobTest.java | 2 +-
.../doris/cloud/load/CopyLoadPendingTaskTest.java | 226 +++--
.../apache/doris/cloud/stage/StageUtilTest.java | 100 ++-
.../apache/doris/common/PropertyAnalyzerTest.java | 3 +-
.../apache/doris/common/util/BrokerUtilTest.java | 116 +--
.../apache/doris/common/util/LocationPathTest.java | 2 +-
.../apache/doris/common/util/TimeUtilsTest.java | 4 +-
.../apache/doris/datasource/PathVisibleTest.java | 33 +-
.../datasource/hive/HMSTransactionPathTest.java | 208 +++--
.../apache/doris/datasource/hive/HiveAcidTest.java | 255 +++---
.../doris/datasource/hive/HmsCommitTest.java | 110 +--
.../datasource/iceberg/IcebergPredicateTest.java | 7 +-
.../datasource/jdbc/source/JdbcScanNodeTest.java | 24 +-
.../datasource/lakesoul/LakeSoulPredicateTest.java | 7 +-
.../metastore/PaimonDlfRestCatalogTest.java | 42 -
.../property/storage/OSSPropertiesTest.java | 2 +
.../TrinoConnectorPredicateTest.java | 15 +-
.../external/iceberg/IcebergHadoopCatalogTest.java | 90 --
.../doris/fs/FileSystemTransferUtilTest.java | 218 +++++
.../java/org/apache/doris/fs/MemoryFileSystem.java | 355 ++++++++
.../org/apache/doris/fs/MemoryFileSystemTest.java | 373 ++++++++
.../org/apache/doris/fs/SchemaTypeMapperTest.java | 1 +
.../doris/fs/SpiSwitchingFileSystemTest.java | 285 +++++++
...TransactionScopeCachingDirectoryListerTest.java | 52 +-
.../apache/doris/fs/obj/AzureObjStorageTest.java | 272 ------
.../org/apache/doris/fs/obj/MockedS3Client.java | 131 ---
.../org/apache/doris/fs/obj/S3FileSystemTest.java | 246 ------
.../doris/fs/obj/S3ObjStorageGlobListTest.java | 187 ----
.../org/apache/doris/fs/obj/S3ObjStorageTest.java | 557 ------------
.../doris/fs/remote/RemoteFileSystemTest.java | 195 -----
.../org/apache/doris/http/ForwardToMasterTest.java | 30 +-
.../apache/doris/httpv2/rest/HttpApiAuthTest.java | 158 ++++
.../apache/doris/info/TableNameInfoUtilsTest.java | 181 ++++
.../java/org/apache/doris/load/DeleteJobTest.java | 157 ++++
.../load/loadv2/BrokerLoadPendingTaskTest.java | 84 +-
.../load/routineload/KafkaAwsMskIamAuthTest.java | 491 +++++++++++
.../load/routineload/RoutineLoadManagerTest.java | 6 -
.../doris/master/MasterImplDeleteTaskTest.java | 152 ++++
.../java/org/apache/doris/mtmv/MTMVUtilTest.java | 2 +-
.../nereids/parser/NereidsParserDigestTest.java | 8 +-
.../doris/nereids/trees/plans/ConstraintTest.java | 37 +-
.../trees/plans/CreateTableCommandTest.java | 21 +-
.../commands/CreateRepositoryCommandTest.java | 6 +-
.../org/apache/doris/qe/AuditLogHelperTest.java | 104 +++
.../doris/qe/ConnectAttributesForwardTest.java | 128 +++
.../org/apache/doris/qe/ConnectContextTest.java | 43 +
.../org/apache/doris/qe/SessionVariablesTest.java | 2 +-
.../doris/statistics/util/StatisticsUtilTest.java | 30 -
.../transaction/DatabaseTransactionMgrTest.java | 120 +++
.../java/org/apache/doris/tso/TSOServiceTest.java | 563 ++++++++++++
.../org/apache/doris/tso/TSOTimestampTest.java | 154 ++++
.../extension/loader/ChildFirstClassLoader.java | 28 +
.../loader/DirectoryPluginRuntimeManager.java | 104 ++-
fe/fe-filesystem/ENV_TEST_README.md | 189 +++++
fe/fe-filesystem/fe-filesystem-api/pom.xml | 73 ++
.../org/apache/doris/filesystem/BlockInfo.java} | 34 +-
.../apache/doris/filesystem/DorisInputFile.java | 75 ++
.../apache/doris/filesystem/DorisInputStream.java | 46 +
.../apache/doris/filesystem/DorisOutputFile.java} | 20 +-
.../org/apache/doris/filesystem/FileEntry.java | 103 +++
.../org/apache/doris/filesystem/FileIterator.java} | 19 +-
.../org/apache/doris/filesystem/FileSystem.java | 180 ++++
.../doris/filesystem/FileSystemIOException.java} | 18 +-
.../doris/filesystem/FileSystemTransferUtil.java | 176 ++++
.../apache/doris/filesystem}/FileSystemType.java | 2 +-
.../apache/doris/filesystem}/FileSystemUtil.java | 47 +-
.../org/apache/doris/filesystem/GlobListing.java} | 51 +-
.../java/org/apache/doris/filesystem/Location.java | 140 +++
.../apache/doris/filesystem}/RemoteIterator.java | 2 +-
.../doris/filesystem}/SimpleRemoteIterator.java | 19 +-
.../org/apache/doris/filesystem/BlockInfoTest.java | 63 ++
.../org/apache/doris/filesystem/EnvTestBase.java | 100 +++
.../org/apache/doris/filesystem/FileEntryTest.java | 101 +++
.../filesystem/FileSystemDefaultMethodsTest.java | 223 +++++
.../filesystem/FileSystemTransferUtilTest.java | 85 ++
.../doris/filesystem/FileSystemUtilTest.java | 133 +++
.../apache/doris/filesystem/GlobListingTest.java | 65 ++
.../org/apache/doris/filesystem/LocationTest.java | 143 ++++
.../doris/filesystem/SimpleRemoteIteratorTest.java | 60 ++
.../fe-filesystem-azure}/pom.xml | 72 +-
.../src/main/assembly/plugin-zip.xml | 65 ++
.../doris/filesystem/azure/AzureFileSystem.java | 396 +++++++++
.../filesystem/azure/AzureFileSystemProvider.java | 60 ++
.../doris/filesystem/azure/AzureObjStorage.java | 531 ++++++++++++
.../apache/doris/filesystem/azure/AzureUri.java | 146 ++++
....apache.doris.filesystem.spi.FileSystemProvider | 18 +
.../filesystem/azure/AzureFileSystemEnvTest.java | 211 +++++
.../azure/AzureObjStorageExtensionTest.java | 361 ++++++++
.../doris/filesystem/azure/AzureUriTest.java | 120 +++
fe/fe-filesystem/fe-filesystem-broker/pom.xml | 110 +++
.../src/main/assembly/plugin-zip.xml | 65 ++
.../filesystem/broker/BrokerClientFactory.java | 76 ++
.../doris/filesystem/broker/BrokerClientPool.java | 84 ++
.../broker/BrokerFileSystemProvider.java | 85 ++
.../doris/filesystem/broker/BrokerInputFile.java | 104 +++
.../doris/filesystem/broker/BrokerInputStream.java | 170 ++++
.../doris/filesystem/broker/BrokerOutputFile.java | 95 +++
.../filesystem/broker/BrokerOutputStream.java | 113 +++
.../filesystem/broker/BrokerSpiFileSystem.java | 258 ++++++
....apache.doris.filesystem.spi.FileSystemProvider | 18 +
.../broker/BrokerFileSystemProviderTest.java | 149 ++++
.../broker/BrokerSpiFileSystemEnvTest.java | 176 ++++
.../filesystem/broker/BrokerSpiFileSystemTest.java | 268 ++++++
.../fe-filesystem-cos}/pom.xml | 67 +-
.../src/main/assembly/plugin-zip.xml | 65 ++
.../filesystem/cos/CosFileSystemProvider.java | 72 ++
.../apache/doris/filesystem/cos/CosObjStorage.java | 222 +++++
....apache.doris.filesystem.spi.FileSystemProvider | 18 +
.../doris/filesystem/cos/CosFileSystemEnvTest.java | 207 +++++
.../doris/filesystem/cos/CosObjStorageTest.java | 207 +++++
.../fe-filesystem-hdfs}/pom.xml | 75 +-
.../src/main/assembly/plugin-zip.xml | 65 ++
.../doris/filesystem/hdfs/DFSFileSystem.java | 276 ++++++
.../doris/filesystem/hdfs/HdfsConfigBuilder.java | 58 ++
.../doris/filesystem/hdfs/HdfsFileIterator.java | 62 ++
.../filesystem/hdfs/HdfsFileSystemProvider.java | 66 ++
.../doris/filesystem/hdfs/HdfsInputFile.java | 73 ++
.../doris/filesystem/hdfs/HdfsOutputFile.java | 60 ++
.../filesystem/hdfs/HdfsSeekableInputStream.java | 94 ++
.../hdfs/KerberosHadoopAuthenticator.java | 65 ++
.../filesystem/hdfs/SimpleHadoopAuthenticator.java | 63 ++
....apache.doris.filesystem.spi.FileSystemProvider | 18 +
.../filesystem/hdfs/DFSFileSystemEnvTest.java | 215 +++++
.../doris/filesystem/hdfs/DFSFileSystemTest.java | 125 +++
.../filesystem/hdfs/HdfsConfigBuilderTest.java | 85 ++
.../hdfs/HdfsSeekableInputStreamTest.java | 191 +++++
.../hdfs/KerberosHadoopAuthenticatorEnvTest.java | 115 +++
.../hdfs/SimpleHadoopAuthenticatorTest.java | 70 ++
.../fe-filesystem-local}/pom.xml | 62 +-
.../src/main/assembly/plugin-zip.xml | 65 ++
.../doris/filesystem/local/LocalFileSystem.java | 248 ++++++
.../filesystem/local/LocalFileSystemProvider.java} | 29 +-
....apache.doris.filesystem.spi.FileSystemProvider | 18 +
.../filesystem/local/LocalFileSystemTest.java | 221 +++++
.../fe-filesystem-obs}/pom.xml | 67 +-
.../src/main/assembly/plugin-zip.xml | 65 ++
.../filesystem/obs/ObsFileSystemProvider.java | 72 ++
.../apache/doris/filesystem/obs/ObsObjStorage.java | 220 +++++
....apache.doris.filesystem.spi.FileSystemProvider | 18 +
.../doris/filesystem/obs/ObsFileSystemEnvTest.java | 202 +++++
.../doris/filesystem/obs/ObsObjStorageTest.java | 197 +++++
.../fe-filesystem-oss}/pom.xml | 65 +-
.../src/main/assembly/plugin-zip.xml | 65 ++
.../filesystem/oss/OssFileSystemProvider.java | 72 ++
.../apache/doris/filesystem/oss/OssObjStorage.java | 227 +++++
....apache.doris.filesystem.spi.FileSystemProvider | 18 +
.../doris/filesystem/oss/OssFileSystemEnvTest.java | 202 +++++
.../doris/filesystem/oss/OssObjStorageTest.java | 195 +++++
fe/fe-filesystem/fe-filesystem-s3/pom.xml | 126 +++
.../src/main/assembly/plugin-zip.xml | 65 ++
.../apache/doris/filesystem/s3/S3FileSystem.java | 538 ++++++++++++
.../doris/filesystem/s3/S3FileSystemProvider.java | 59 ++
.../apache/doris/filesystem/s3/S3ObjStorage.java | 644 ++++++++++++++
.../apache/doris/filesystem/s3/S3OutputStream.java | 93 ++
.../java/org/apache/doris/filesystem/s3/S3Uri.java | 67 ++
....apache.doris.filesystem.spi.FileSystemProvider | 18 +
.../doris/filesystem/s3/S3FileSystemEnvTest.java | 238 ++++++
.../doris/filesystem/s3/S3FileSystemTest.java | 207 +++++
.../doris/filesystem/s3/S3ObjStorageEnvTest.java | 225 +++++
.../doris/filesystem/s3/S3ObjStorageMockTest.java | 377 +++++++++
.../doris/filesystem/s3/S3ObjStorageTest.java | 214 +++++
.../doris/filesystem/s3/S3OutputStreamTest.java | 228 +++++
.../org/apache/doris/filesystem/s3/S3UriTest.java | 76 ++
fe/fe-filesystem/fe-filesystem-spi/pom.xml | 82 ++
.../doris/filesystem/spi/FileSystemProvider.java | 78 ++
.../doris/filesystem/spi/HadoopAuthenticator.java | 45 +
.../apache/doris/filesystem/spi/IOCallable.java} | 22 +-
.../apache/doris/filesystem/spi/ObjFileSystem.java | 124 +++
.../apache/doris/filesystem/spi/ObjStorage.java | 135 +++
.../apache/doris/filesystem/spi}/RemoteObject.java | 19 +-
.../doris/filesystem/spi}/RemoteObjects.java | 26 +-
.../apache/doris/filesystem/spi/RequestBody.java} | 33 +-
.../doris/filesystem/spi/StsCredentials.java} | 35 +-
.../doris/filesystem/spi/UploadPartResult.java} | 27 +-
.../doris/filesystem/spi/ObjFileSystemTest.java | 354 ++++++++
fe/fe-filesystem/pom.xml | 74 ++
.../apache/doris/foundation/fs/FsStorageType.java} | 23 +-
.../org/apache/doris/catalog/PatternType.java} | 13 +-
.../org/apache/doris/catalog/VariantField.java | 11 +-
fe/pom.xml | 33 +-
gensrc/proto/olap_file.proto | 5 +
gensrc/script/Makefile | 2 +-
gensrc/script/gen_build_version.sh | 4 +-
gensrc/thrift/AgentService.thrift | 1 +
gensrc/thrift/Descriptors.thrift | 1 +
gensrc/thrift/FrontendService.thrift | 1 +
post-build.sh | 232 +++++
regression-test/conf/regression-conf.groovy | 1 +
.../data/ann_index_p0/ivf_index_test.out | 12 +
.../data/ann_index_p0/ivf_on_disk_index_test.out | 30 +
.../tvf/test_hdfs_tvf_error_uri.out | 5 -
.../test_count_match_all_pushdown.out | 16 +
.../agg_strategy/test_variables.out | 24 -
.../adaptive_batch_size/adaptive_batch_size.out | 18 +-
.../data/query_p0/aggregate/window_funnel_v2.out | 56 ++
.../query_p0/system/test_query_sys_rowsets.out | 1 +
.../system/test_query_sys_scan_rowsets.out | 1 +
.../data/query_p0/system/test_table_properties.out | 8 +-
.../search/test_search_inverted_index_profile.out | 22 +
.../suites/ann_index_p0/ann_index_basic.groovy | 22 +-
.../suites/ann_index_p0/ivf_index_test.groovy | 31 +-
.../ann_index_p0/ivf_on_disk_index_test.groovy | 18 +-
.../suites/auth/test_http_api_auth.groovy | 216 +++++
.../compaction/test_be_compaction_tasks.groovy | 167 ++++
.../test_compaction_profile_action.groovy | 193 +++++
.../test_inlineview_with_window_function.groovy | 5 +-
.../backup_restore_object_storage.groovy | 4 +-
.../tvf/test_hdfs_tvf_error_uri.groovy | 14 +-
.../test_skip_calc_between_segments.groovy | 3 +-
.../test_count_match_all_pushdown.groovy | 83 ++
.../test_streaming_mysql_job_create_alter.groovy | 14 +-
.../test_insert_random_distribution_table.groovy | 1 +
.../load_p0/msk_load/test_routine_load_msk.groovy | 177 ++++
.../test_routine_load_msk_public_access.groovy | 105 +++
.../agg_strategy/test_variables.groovy | 7 +-
.../adaptive_batch_size/adaptive_batch_size.groovy | 24 +-
.../query_p0/aggregate/window_funnel_v2.groovy | 412 +++++++++
.../test_search_inverted_index_profile.groovy | 302 +++++++
regression-test/suites/tso_p0/test_tso_api.groovy | 117 +++
.../tso_p0/test_tso_rowset_commit_tso.groovy | 78 ++
.../test_compact_multi_segments.groovy | 3 +
.../test_schema_change_add_key_column.groovy | 2 +
run-fe-ut.sh | 4 +-
run-fs-env-test.sh | 105 +++
.../ddl/create-tpcds-tables-sf10000.sql | 4 +-
747 files changed, 36851 insertions(+), 15591 deletions(-)
create mode 100644 be/benchmark/benchmark_column_view.hpp
create mode 100644 be/src/common/block_budget.h
create mode 100644
be/src/information_schema/schema_compaction_tasks_scanner.cpp
copy be/src/{service/http/action/stream_load_2pc.h =>
information_schema/schema_compaction_tasks_scanner.h} (54%)
create mode 100644 be/src/runtime/aws_msk_iam_auth.cpp
create mode 100644 be/src/runtime/aws_msk_iam_auth.h
create mode 100644 be/src/service/http/action/compaction_profile_action.cpp
copy be/src/service/http/action/{health_action.h =>
compaction_profile_action.h} (78%)
create mode 100644 be/src/storage/compaction_task_tracker.cpp
create mode 100644 be/src/storage/compaction_task_tracker.h
create mode 100644 be/test/common/block_budget_test.cpp
create mode 100644 be/test/runtime/aws_msk_iam_auth_test.cpp
create mode 100644 be/test/runtime/fragment_mgr_cross_cluster_cancel_test.cpp
create mode 100644 be/test/storage/compaction_task_tracker_test.cpp
copy fe/{fe-core/src/main/java/org/apache/doris/analysis/VirtualSlotRef.java
=> fe-catalog/src/main/java/org/apache/doris/catalog/NameSpaceContext.java}
(51%)
rename fe/{fe-core =>
fe-common}/src/main/java/org/apache/doris/qe/GlobalVariable.java (77%)
create mode 100644
fe/fe-common/src/main/java/org/apache/doris/qe/VarAttrDef.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteralUtils.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/DecimalLiteralUtils.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/ExprToColumnLabelVisitor.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/ExprToExprNameVisitor.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnToProtobuf.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnToThrift.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/AzureRemote.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/BosRemote.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/CosRemote.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/DefaultRemote.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ListObjectsResult.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/MockRemote.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectFile.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfo.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObsRemote.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/OssRemote.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/RemoteBase.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/storage/S3Remote.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/FilePartitionUtils.java
delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/fs/FileSystem.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemDescriptor.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemIOException.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemProviderImpl.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/FsStorageTypeAdapter.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/LocalDfsFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/PersistentFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/RemoteFileRemoteIterator.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/SpiSwitchingFileSystem.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/StoragePropertiesConverter.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/StorageTypeMapper.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/DorisInput.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/DorisInputFile.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/DorisInputStream.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/DorisOutputFile.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsInput.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsInputFile.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsInputStream.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsOutputFile.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsOutputStream.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/obj/AzureObjStorage.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/obj/ObjStorage.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/obj/S3ObjStorage.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/operations/BrokerFileOperations.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/operations/BrokerOpParams.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/operations/HDFSFileOperations.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/operations/HDFSOpParams.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/operations/OpParams.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/AzureFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/BrokerFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/ObjFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/RemoteFile.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/RemoteFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/S3FileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/SwitchingFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/DFSFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/DFSFileSystemPhantomReference.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/OFSFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/RemoteFSPhantomManager.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/TSOAction.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/info/TableNameInfoUtils.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/distribute/worker/job/UnassignedLocalTVFSinkJob.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/tso/TSOService.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/tso/TSOTimestamp.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/analysis/DateLiteralTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/analysis/DateLiteralUtilsTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/analysis/DecimalLiteralUtilsTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/backup/BrokerStorageTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/catalog/SessionVariablesNullFixTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/external/iceberg/IcebergHadoopCatalogTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemTransferUtilTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/MemoryFileSystem.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/MemoryFileSystemTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/SpiSwitchingFileSystemTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/obj/AzureObjStorageTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/obj/MockedS3Client.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/obj/S3FileSystemTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/obj/S3ObjStorageGlobListTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/obj/S3ObjStorageTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/remote/RemoteFileSystemTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/httpv2/rest/HttpApiAuthTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/info/TableNameInfoUtilsTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/load/DeleteJobTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/load/routineload/KafkaAwsMskIamAuthTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/master/MasterImplDeleteTaskTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/qe/AuditLogHelperTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/qe/ConnectAttributesForwardTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/tso/TSOServiceTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/tso/TSOTimestampTest.java
create mode 100644 fe/fe-filesystem/ENV_TEST_README.md
create mode 100644 fe/fe-filesystem/fe-filesystem-api/pom.xml
copy fe/{fe-core/src/main/java/org/apache/doris/analysis/VirtualSlotRef.java
=>
fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/BlockInfo.java}
(59%)
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/DorisInputFile.java
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/DorisInputStream.java
rename
fe/{fe-core/src/main/java/org/apache/doris/fs/remote/dfs/OSSHdfsFileSystem.java
=>
fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/DorisOutputFile.java}
(70%)
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileEntry.java
copy
fe/{fe-core/src/main/java/org/apache/doris/fs/operations/FileOperations.java =>
fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileIterator.java}
(69%)
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileSystem.java
rename fe/{fe-core/src/main/java/org/apache/doris/fs/RemoteFiles.java =>
fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileSystemIOException.java}
(72%)
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileSystemTransferUtil.java
rename fe/{fe-core/src/main/java/org/apache/doris/fs =>
fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem}/FileSystemType.java
(98%)
rename fe/{fe-core/src/main/java/org/apache/doris/fs =>
fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem}/FileSystemUtil.java
(62%)
rename fe/{fe-core/src/main/java/org/apache/doris/fs/GlobListResult.java =>
fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/GlobListing.java}
(50%)
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/Location.java
rename fe/{fe-core/src/main/java/org/apache/doris/fs =>
fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem}/RemoteIterator.java
(96%)
rename fe/{fe-core/src/main/java/org/apache/doris/fs =>
fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem}/SimpleRemoteIterator.java
(80%)
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/BlockInfoTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/EnvTestBase.java
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/FileEntryTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/FileSystemDefaultMethodsTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/FileSystemTransferUtilTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/FileSystemUtilTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/GlobListingTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/LocationTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/SimpleRemoteIteratorTest.java
copy fe/{be-java-extensions/paimon-scanner =>
fe-filesystem/fe-filesystem-azure}/pom.xml (54%)
create mode 100644
fe/fe-filesystem/fe-filesystem-azure/src/main/assembly/plugin-zip.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystem.java
create mode 100644
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProvider.java
create mode 100644
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureObjStorage.java
create mode 100644
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureUri.java
create mode 100644
fe/fe-filesystem/fe-filesystem-azure/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
create mode 100644
fe/fe-filesystem/fe-filesystem-azure/src/test/java/org/apache/doris/filesystem/azure/AzureFileSystemEnvTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-azure/src/test/java/org/apache/doris/filesystem/azure/AzureObjStorageExtensionTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-azure/src/test/java/org/apache/doris/filesystem/azure/AzureUriTest.java
create mode 100644 fe/fe-filesystem/fe-filesystem-broker/pom.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/assembly/plugin-zip.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/java/org/apache/doris/filesystem/broker/BrokerClientFactory.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/java/org/apache/doris/filesystem/broker/BrokerClientPool.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/java/org/apache/doris/filesystem/broker/BrokerFileSystemProvider.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/java/org/apache/doris/filesystem/broker/BrokerInputFile.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/java/org/apache/doris/filesystem/broker/BrokerInputStream.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/java/org/apache/doris/filesystem/broker/BrokerOutputFile.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/java/org/apache/doris/filesystem/broker/BrokerOutputStream.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/java/org/apache/doris/filesystem/broker/BrokerSpiFileSystem.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/test/java/org/apache/doris/filesystem/broker/BrokerFileSystemProviderTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/test/java/org/apache/doris/filesystem/broker/BrokerSpiFileSystemEnvTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-broker/src/test/java/org/apache/doris/filesystem/broker/BrokerSpiFileSystemTest.java
copy fe/{be-java-extensions/paimon-scanner =>
fe-filesystem/fe-filesystem-cos}/pom.xml (55%)
create mode 100644
fe/fe-filesystem/fe-filesystem-cos/src/main/assembly/plugin-zip.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-cos/src/main/java/org/apache/doris/filesystem/cos/CosFileSystemProvider.java
create mode 100644
fe/fe-filesystem/fe-filesystem-cos/src/main/java/org/apache/doris/filesystem/cos/CosObjStorage.java
create mode 100644
fe/fe-filesystem/fe-filesystem-cos/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
create mode 100644
fe/fe-filesystem/fe-filesystem-cos/src/test/java/org/apache/doris/filesystem/cos/CosFileSystemEnvTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-cos/src/test/java/org/apache/doris/filesystem/cos/CosObjStorageTest.java
copy fe/{be-java-extensions/paimon-scanner =>
fe-filesystem/fe-filesystem-hdfs}/pom.xml (52%)
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/assembly/plugin-zip.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/DFSFileSystem.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilder.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsFileIterator.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsFileSystemProvider.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsInputFile.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsOutputFile.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsSeekableInputStream.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticator.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/SimpleHadoopAuthenticator.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/DFSFileSystemEnvTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/DFSFileSystemTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilderTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/HdfsSeekableInputStreamTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticatorEnvTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/SimpleHadoopAuthenticatorTest.java
copy fe/{be-java-extensions/paimon-scanner =>
fe-filesystem/fe-filesystem-local}/pom.xml (51%)
create mode 100644
fe/fe-filesystem/fe-filesystem-local/src/main/assembly/plugin-zip.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-local/src/main/java/org/apache/doris/filesystem/local/LocalFileSystem.java
copy fe/{fe-core/src/main/java/org/apache/doris/analysis/VirtualSlotRef.java
=>
fe-filesystem/fe-filesystem-local/src/main/java/org/apache/doris/filesystem/local/LocalFileSystemProvider.java}
(54%)
create mode 100644
fe/fe-filesystem/fe-filesystem-local/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
create mode 100644
fe/fe-filesystem/fe-filesystem-local/src/test/java/org/apache/doris/filesystem/local/LocalFileSystemTest.java
copy fe/{be-java-extensions/paimon-scanner =>
fe-filesystem/fe-filesystem-obs}/pom.xml (55%)
create mode 100644
fe/fe-filesystem/fe-filesystem-obs/src/main/assembly/plugin-zip.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-obs/src/main/java/org/apache/doris/filesystem/obs/ObsFileSystemProvider.java
create mode 100644
fe/fe-filesystem/fe-filesystem-obs/src/main/java/org/apache/doris/filesystem/obs/ObsObjStorage.java
create mode 100644
fe/fe-filesystem/fe-filesystem-obs/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
create mode 100644
fe/fe-filesystem/fe-filesystem-obs/src/test/java/org/apache/doris/filesystem/obs/ObsFileSystemEnvTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-obs/src/test/java/org/apache/doris/filesystem/obs/ObsObjStorageTest.java
copy fe/{be-java-extensions/paimon-scanner =>
fe-filesystem/fe-filesystem-oss}/pom.xml (54%)
create mode 100644
fe/fe-filesystem/fe-filesystem-oss/src/main/assembly/plugin-zip.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssFileSystemProvider.java
create mode 100644
fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssObjStorage.java
create mode 100644
fe/fe-filesystem/fe-filesystem-oss/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
create mode 100644
fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssFileSystemEnvTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssObjStorageTest.java
create mode 100644 fe/fe-filesystem/fe-filesystem-s3/pom.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/main/assembly/plugin-zip.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystem.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3ObjStorage.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3OutputStream.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3Uri.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemEnvTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3ObjStorageEnvTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3ObjStorageMockTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3ObjStorageTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3OutputStreamTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3UriTest.java
create mode 100644 fe/fe-filesystem/fe-filesystem-spi/pom.xml
create mode 100644
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java
create mode 100644
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/HadoopAuthenticator.java
rename
fe/{fe-core/src/main/java/org/apache/doris/fs/operations/FileOperations.java =>
fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/IOCallable.java}
(67%)
create mode 100644
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjFileSystem.java
create mode 100644
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjStorage.java
rename fe/{fe-core/src/main/java/org/apache/doris/fs/obj =>
fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi}/RemoteObject.java
(76%)
rename fe/{fe-core/src/main/java/org/apache/doris/fs/obj =>
fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi}/RemoteObjects.java
(72%)
rename fe/{fe-core/src/main/java/org/apache/doris/cloud/storage/TosRemote.java
=>
fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/RequestBody.java}
(53%)
copy fe/{fe-core/src/main/java/org/apache/doris/analysis/VirtualSlotRef.java
=>
fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/StsCredentials.java}
(53%)
rename fe/{fe-core/src/main/java/org/apache/doris/fs/io/ParsedPath.java =>
fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/UploadPartResult.java}
(65%)
create mode 100644
fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/ObjFileSystemTest.java
create mode 100644 fe/fe-filesystem/pom.xml
rename
fe/{fe-core/src/main/java/org/apache/doris/fs/remote/dfs/JFSFileSystem.java =>
fe-foundation/src/main/java/org/apache/doris/foundation/fs/FsStorageType.java}
(66%)
rename fe/{fe-core/src/main/java/org/apache/doris/fs/FileSystemProvider.java
=> fe-type/src/main/java/org/apache/doris/catalog/PatternType.java} (78%)
create mode 100755 post-build.sh
create mode 100644
regression-test/data/inverted_index_p0/test_count_match_all_pushdown.out
create mode 100644
regression-test/data/search/test_search_inverted_index_profile.out
create mode 100644 regression-test/suites/auth/test_http_api_auth.groovy
create mode 100644
regression-test/suites/compaction/test_be_compaction_tasks.groovy
create mode 100644
regression-test/suites/compaction/test_compaction_profile_action.groovy
create mode 100644
regression-test/suites/inverted_index_p0/test_count_match_all_pushdown.groovy
create mode 100644
regression-test/suites/load_p0/msk_load/test_routine_load_msk.groovy
create mode 100644
regression-test/suites/load_p0/msk_load/test_routine_load_msk_public_access.groovy
create mode 100644
regression-test/suites/search/test_search_inverted_index_profile.groovy
create mode 100644 regression-test/suites/tso_p0/test_tso_api.groovy
create mode 100644
regression-test/suites/tso_p0/test_tso_rowset_commit_tso.groovy
create mode 100755 run-fs-env-test.sh
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
