This is an automated email from the ASF dual-hosted git repository.
jiangtian pushed a change to branch use_pam_for_insert_tablet
in repository https://gitbox.apache.org/repos/asf/iotdb.git
from 3641826e1e0 multiple fixes
add 0f0aaf410da Pipe: Fixed the authority for table model insertion
(#15186)
add 77b7541eb7e Fix incorrect os dirs creation (#15196)
add 91fb32a5c3a Pipe: Added default logic for username / password in tree
sql-dialect pipes & Fixed the NPE for configNode table schema snapshot parsing
need (#15200)
add e86ec17bc95 Pipe: Add metrics for tsfile to tablets invocation call
count and time (#15202)
add c986324b0e6 Pipe: Enable showing WRITE_BACK_SINK (#15207)
add 3805b47e58b remove export tsfile script (#15206)
add e62bbf52833 Pipe: Enable showing WRITE_BACK_SINK (Fix IoTDBDatabaseIT)
(#15209)
add a81c5c6ac70 Load: Support multi-disks awareness for IoTV2 and Pipe
(#15181)
add 7fe614c7cc2 Pipe: Added close for tree session in WriteBackConnector
(#15211)
add 7bfcb4965c1 fix: clone & delete issues (#15153)
add 4678776f5c1 Pipe Plugin: Fix ClassLoader and Visibility not
initialized from CN snapshot after reboot (#15212)
add f66fcd91642 Pipe: Fixed the NPE of OPC UA Sink in table model when
there are null values & Deleted the extra folder of opc ua sink in table model
(#15215)
add ebb1bcf8910 Load: Add default value for multi disks awareness (#15216)
add 3d1cd0ea7f6 Pipe: Fix the problem of missing points in TsFile of
parsing table model (#15217)
add 0ee458e991f Added retry for insertion when fetch table schema failed
due to connection error & Improved the retry mechanism of client rpc
add a6088a28a07 [RTO/RPO] Chore: Fix remaining issues in PR 15014 (#15171)
add c5d0c397d98 Pipe: Added userName / password logic for alter pipe
replace statement (#15219)
add 630a3d255d9 Set safeDeletedSearchIndex for iotv2 to accelerate wal
deletion #15221
add 680026eefea Pipe: Prevent Duplicate Time Usage Reporting on Close in
PipeTsFileToTabletsMetrics (#15220)
add a8c80ecb140 Load: Batched tablet insertion during conversion (#15125)
add 08c29635d38 Pipe Plugin: Only Allow Create Model Compatible Plugins
(#15227)
add c746da2faec Deleted the useless read lock in getDeviceAttribute
add 8889255c0c4 Load: Implement region operations cache for load × region
migration detection (#15210)
add 7e4e01cc632 Load & Pipe: Support Active Load Table Model TsFiles &
Support Async Load in SQL & Support Async Load Strategy in Pipe (#15208)
add bf2eee9ad1e Fix: Null value settings not applying correctly in tablet.
(#15228)
add c9ec0fda2d1 Subscription: distinguish between reference count of ack
and clean in tsfile batch to avoid cleaning before ack (#15229)
add 4256ea48ad1 Fix BigArray NPE of some aggregation functions (first,
last, sum, extreme) when groups are more than 1024 in aggregation query
add 9b406e15353 update description when file_type is csv, sql_dialect is
table and table is not required (#15232)
add 1697f545307 Fixed the NullPointerException caused by non-current table
procedures #15238
add b6fea634d61 Add some supplementary logs during partition allocation
(#15234)
add 4350a56c237 Description add the file name when importing file
exception (#15237)
add 46fc1e0a6fa Pipe: Fixed the tsFile parsing & write-back-sink auto
create db bug (#15240)
add e237f723974 [Python-Client] Use executeV2 interface to query
add 304fbab00bd Unifying Exception Messages for Aggregation Functions
(#15231)
add b1d4fedb33c Pipe IT: Fixed the issue that the Pipe IT test error log
could not be printed (#15242)
add 5f28e331d29 Pipe: Optimized the error log when source/sink's username
is without password in alter (#15241)
add c0a642892f7 Pipe: Added memory control for aligned chunk reader in
TsFileInsertionEventScanParser & TsFileInsertionEventTableParserTabletIterator
(#15222)
add 7c301137057 Add GroupNode and parallelize TableFunctionProcessorNode
add 91d067b161e Pipe: Fix stuck caused by async connector client not
returned after transferring tsfiles & Fix validateTsFile and
shouldMarkAsPipeRequest may not be effective (#15245) (#15249)
add 5ddd4234666 using correct encoding in read chunk compaction (#15250)
add b3e77b36b32 Merge branch 'master' into use_pam_for_insert_tablet
No new revisions were added by this update.
Summary of changes:
.github/workflows/multi-language-client.yml | 10 +-
.github/workflows/pipe-it.yml | 54 ++-
.../org/apache/iotdb/db/it/IoTDBRestartIT.java | 3 +-
.../it/schema/IoTDBDeleteAlignedTimeseriesIT.java | 42 ++
.../pipe/it/dual/tablemodel/TableModelUtils.java | 8 +-
.../manual/basic/IoTDBPipePermissionIT.java | 90 +++-
.../pipe/it/single/IoTDBPipePermissionIT.java | 43 ++
.../it/db/it/IoTDBAuthenticationTableIT.java | 4 +-
.../relational/it/db/it/IoTDBWindowTVFIT.java | 229 ++++++++++
.../udf/IoTDBUserDefinedAggregateFunctionIT.java | 1 -
.../relational/it/schema/IoTDBDatabaseIT.java | 10 +-
.../apache/iotdb/tools/it/ExportTsFileTestIT.java | 4 +-
.../relational/table/TableFunctionAnalysis.java | 28 +-
.../src/assembly/resources/tools/export-tsfile.bat | 62 ---
.../src/assembly/resources/tools/export-tsfile.sh | 59 ---
.../apache/iotdb/tool/data/AbstractImportData.java | 10 +-
.../org/apache/iotdb/tool/data/ImportData.java | 6 +
.../apache/iotdb/tool/data/ImportDataTable.java | 7 +-
iotdb-client/client-cpp/src/main/Session.cpp | 12 +-
iotdb-client/client-py/iotdb/Session.py | 324 +++++++-------
.../client-py/iotdb/template/MeasurementNode.py | 2 +-
iotdb-client/client-py/iotdb/template/Template.py | 6 +-
.../{TsFileConstant.py => tsfile_constant.py} | 0
.../tsfile/utils/{DateUtils.py => date_utils.py} | 0
.../iotdb/tsfile/utils/{Pair.py => pair.py} | 0
...{ReadWriteIOUtils.py => read_write_io_utils.py} | 0
.../client-py/iotdb/tsfile/utils/tsblock_serde.py | 266 ++++++++++++
iotdb-client/client-py/iotdb/utils/Field.py | 6 +-
.../client-py/iotdb/utils/IoTDBRpcDataSet.py | 463 ---------------------
iotdb-client/client-py/iotdb/utils/NumpyTablet.py | 2 +-
.../client-py/iotdb/utils/SessionDataSet.py | 48 ++-
iotdb-client/client-py/iotdb/utils/Tablet.py | 2 +-
.../{IoTDBConnectionException.py => exception.py} | 20 +
.../client-py/iotdb/utils/iotdb_rpc_dataset.py | 396 ++++++++++++++++++
iotdb-client/client-py/iotdb/utils/rpc_utils.py | 69 +++
iotdb-client/client-py/resources/pyproject.toml | 3 +-
.../tests/integration/test_tablemodel_query.py | 456 ++++++++++++++++++++
.../iotdb/confignode/manager/ProcedureManager.java | 54 ++-
.../manager/load/service/HeartbeatService.java | 12 +
.../manager/load/service/TopologyService.java | 60 ++-
.../manager/partition/PartitionManager.java | 67 ++-
.../pipe/extractor/IoTDBConfigRegionExtractor.java | 11 +-
.../persistence/pipe/PipePluginInfo.java | 51 ++-
.../procedure/env/ConfigNodeProcedureEnv.java | 11 +-
.../impl/region/RegionMigrateProcedure.java | 8 +-
.../impl/schema/DeleteDatabaseProcedure.java | 4 +-
.../java/org/apache/iotdb/db/conf/IoTDBConfig.java | 35 ++
.../org/apache/iotdb/db/conf/IoTDBDescriptor.java | 22 +-
.../db/pipe/agent/task/PipeDataNodeTaskAgent.java | 11 +-
.../agent/task/connection/PipeEventCollector.java | 4 +-
.../subtask/processor/PipeProcessorSubtask.java | 15 +-
.../client/IoTDBDataNodeAsyncClientManager.java | 6 +-
.../connector/protocol/opcua/OpcUaNameSpace.java | 11 +-
.../async/IoTDBDataRegionAsyncConnector.java | 12 +-
.../async/handler/PipeTransferTsFileHandler.java | 56 ++-
.../protocol/writeback/WriteBackConnector.java | 13 +
.../common/tsfile/PipeTsFileInsertionEvent.java | 6 +-
.../tsfile/parser/TsFileInsertionEventParser.java | 23 +
.../parser/TsFileInsertionEventParserProvider.java | 61 ++-
.../query/TsFileInsertionEventQueryParser.java | 19 +-
.../scan/TsFileInsertionEventScanParser.java | 25 +-
.../table/TsFileInsertionEventTableParser.java | 16 +-
...ileInsertionEventTableParserTabletIterator.java | 164 ++++----
.../dataregion/IoTDBDataRegionExtractor.java | 2 +
.../iotdb/db/pipe/metric/PipeDataNodeMetrics.java | 3 +
.../overview/PipeTsFileToTabletsMetrics.java | 170 ++++++++
.../protocol/thrift/IoTDBDataNodeReceiver.java | 75 ++--
.../visitor/PipeStatementExceptionVisitor.java | 5 +
.../protocol/thrift/impl/ClientRPCServiceImpl.java | 111 ++---
.../impl/DataNodeInternalRPCServiceImpl.java | 9 +
.../process/function/PartitionRecognizer.java | 66 ++-
.../process/function/TableFunctionOperator.java | 94 +++--
.../{SliceCache.java => PartitionCache.java} | 14 +-
.../operator/process/function/partition/Slice.java | 61 ++-
.../aggregation/CountAllAccumulator.java | 2 +-
.../relational/aggregation/FirstByAccumulator.java | 11 +-
.../relational/aggregation/LastByAccumulator.java | 6 +-
.../relational/aggregation/MaxAccumulator.java | 6 +-
.../relational/aggregation/MinAccumulator.java | 4 +-
.../relational/aggregation/SumAccumulator.java | 4 +-
.../aggregation/TableVarianceAccumulator.java | 4 +-
.../aggregation/grouped/GroupedAvgAccumulator.java | 2 +-
.../grouped/GroupedExtremeAccumulator.java | 17 +-
.../grouped/GroupedFirstAccumulator.java | 17 +-
.../grouped/GroupedFirstByAccumulator.java | 18 +-
.../grouped/GroupedLastAccumulator.java | 37 +-
.../grouped/GroupedLastByAccumulator.java | 18 +-
.../aggregation/grouped/GroupedMaxAccumulator.java | 16 +-
.../grouped/GroupedMaxMinByBaseAccumulator.java | 30 +-
.../aggregation/grouped/GroupedMinAccumulator.java | 16 +-
.../aggregation/grouped/GroupedSumAccumulator.java | 3 +-
.../grouped/GroupedVarianceAccumulator.java | 2 +-
.../grouped/StreamingAggregationOperator.java | 32 +-
.../grouped/StreamingHashAggregationOperator.java | 32 +-
.../plan/analyze/load/LoadTsFileAnalyzer.java | 76 ++++
.../analyze/load/LoadTsFileTableSchemaCache.java | 6 +-
.../analyze/load/LoadTsFileTreeSchemaCache.java | 6 +-
.../execution/config/TableConfigTaskVisitor.java | 55 ++-
.../execution/config/TreeConfigTaskVisitor.java | 43 +-
.../config/executor/ClusterConfigTaskExecutor.java | 90 +++-
.../config/metadata/CreatePipePluginTask.java | 2 +
.../execution/config/sys/pipe/AlterPipeTask.java | 3 +-
.../db/queryengine/plan/parser/ASTVisitor.java | 19 +-
.../plan/planner/TableOperatorGenerator.java | 17 +-
.../SimpleFragmentParallelPlanner.java | 126 +-----
.../exceptions/RootFIPlacementException.java | 2 +-
.../plan/AbstractFragmentParallelPlanner.java | 168 ++++++++
.../plan/planner/plan/node/PlanGraphPrinter.java | 20 +
.../plan/planner/plan/node/PlanNodeType.java | 4 +
.../plan/planner/plan/node/PlanVisitor.java | 5 +
.../relational/analyzer/StatementAnalyzer.java | 13 +-
.../TableFunctionInvocationAnalysis.java | 9 +-
.../metadata/fetcher/TableDeviceSchemaFetcher.java | 7 +-
.../fetcher/cache/TableDeviceSchemaCache.java | 11 +-
.../plan/relational/planner/RelationPlanner.java | 3 +-
.../distribute/TableDistributedPlanGenerator.java | 161 ++++++-
.../distribute/TableDistributedPlanner.java | 2 +-
.../distribute/TableModelQueryFragmentPlanner.java | 167 ++------
.../TableModelTypeProviderExtractor.java | 7 +
.../rule/ImplementTableFunctionSource.java | 78 ++--
.../rule/PruneTableFunctionProcessorColumns.java | 4 +-
.../PruneTableFunctionProcessorSourceColumns.java | 4 +-
.../node/{StreamSortNode.java => GroupNode.java} | 86 ++--
.../relational/planner/node/TableFunctionNode.java | 20 +-
.../planner/node/TableFunctionProcessorNode.java | 42 +-
.../optimizations/LogicalOptimizeFactory.java | 5 +-
.../planner/optimizations/ParallelizeGrouping.java | 266 ++++++++++++
.../PushLimitOffsetIntoTableScan.java | 6 +
.../TransformAggregationToStreamable.java | 22 +
.../optimizations/TransformSortToStreamSort.java | 6 +
.../optimizations/UnaliasSymbolReferences.java | 29 +-
.../plan/relational/sql/ast/LoadTsFile.java | 6 +
.../plan/scheduler/AsyncSendPlanNodeHandler.java | 2 +-
.../plan/scheduler/load/LoadTsFileScheduler.java | 5 +-
.../plan/statement/crud/LoadTsFileStatement.java | 10 +
.../metadata/pipe/AlterPipeStatement.java | 9 +
.../metadata/pipe/CreatePipePluginStatement.java | 10 +
.../iotdb/db/service/RegionMigrateService.java | 69 ++-
.../db/storageengine/dataregion/DataRegion.java | 73 +++-
.../ReadChunkAlignedSeriesCompactionExecutor.java | 5 +-
.../memtable/AlignedWritableMemChunk.java | 169 +++++---
.../load/active/ActiveLoadTsFileLoader.java | 102 +++--
.../load/config/LoadTsFileConfigurator.java | 20 +
...ertedInsertTabletStatementExceptionVisitor.java | 5 +
...vertedInsertTabletStatementTSStatusVisitor.java | 7 +
...leStatementDataTypeConvertExecutionVisitor.java | 1 +
...eeStatementDataTypeConvertExecutionVisitor.java | 186 ++++++---
.../storageengine/load/disk/ILoadDiskSelector.java | 66 +++
.../InheritSystemMultiDisksStrategySelector.java | 58 +++
.../db/storageengine/load/disk/MinIOSelector.java | 112 +++++
...MemoryBlock.java => LoadTsFileMemoryBlock.java} | 13 +-
.../load/memory/LoadTsFileMemoryManager.java | 19 +-
.../load/metrics/LoadTsFileMemMetricSet.java | 6 +-
.../db/storageengine/rescon/disk/TierManager.java | 2 +-
.../batch/SubscriptionPipeTsFileEventBatch.java | 5 +-
.../pipe/SubscriptionPipeTsFileBatchEvents.java | 16 +-
.../apache/iotdb/db/utils/ErrorHandlingUtils.java | 8 +-
.../db/utils/datastructure/AlignedTVList.java | 24 +-
.../iotdb/db/utils/datastructure/TVList.java | 5 +-
.../process/tvf/TableFunctionOperatorTest.java | 6 +-
.../analyzer/AggregationCornerCaseTest.java | 179 ++++++++
.../relational/analyzer/TableFunctionTest.java | 108 +++--
.../planner/assertions/PlanMatchPattern.java | 5 +
...atchedAlignedSeriesReadChunkCompactionTest.java | 2 +-
.../apache/iotdb/metrics/utils/FileStoreUtils.java | 2 +-
.../conf/iotdb-system.properties.template | 21 +
.../apache/iotdb/commons/conf/CommonConfig.java | 9 +
.../iotdb/commons/conf/CommonDescriptor.java | 5 +
.../agent/plugin/builtin/BuiltinPipePlugin.java | 1 -
.../iotdb/commons/pipe/config/PipeConfig.java | 5 +
.../iotdb/commons/service/metric/enums/Metric.java | 2 +
.../relational/TableBuiltinTableFunction.java | 17 +-
.../relational/tvf/CapacityTableFunction.java | 89 ++--
.../builtin/relational/tvf/HOPTableFunction.java | 29 +-
...ableFunction.java => SessionTableFunction.java} | 88 ++--
...leFunction.java => VariationTableFunction.java} | 111 +++--
.../udf/builtin/relational/tvf/WindowTVFUtils.java | 55 +++
.../commons/udf/service/UDFManagementService.java | 3 +
.../org/apache/iotdb/commons/utils/FileUtils.java | 49 +++
.../apache/iotdb/commons/utils/JVMCommonUtils.java | 2 +-
.../apache/iotdb/commons/utils/StatusUtils.java | 5 +
.../db/relational/grammar/sql/RelationalSql.g4 | 4 -
.../src/main/thrift/datanode.thrift | 9 +-
183 files changed, 5610 insertions(+), 2208 deletions(-)
create mode 100644
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBWindowTVFIT.java
delete mode 100644
iotdb-client/cli/src/assembly/resources/tools/export-tsfile.bat
delete mode 100644
iotdb-client/cli/src/assembly/resources/tools/export-tsfile.sh
rename iotdb-client/client-py/iotdb/tsfile/common/constant/{TsFileConstant.py
=> tsfile_constant.py} (100%)
rename iotdb-client/client-py/iotdb/tsfile/utils/{DateUtils.py =>
date_utils.py} (100%)
rename iotdb-client/client-py/iotdb/tsfile/utils/{Pair.py => pair.py} (100%)
rename iotdb-client/client-py/iotdb/tsfile/utils/{ReadWriteIOUtils.py =>
read_write_io_utils.py} (100%)
create mode 100644 iotdb-client/client-py/iotdb/tsfile/utils/tsblock_serde.py
delete mode 100644 iotdb-client/client-py/iotdb/utils/IoTDBRpcDataSet.py
rename iotdb-client/client-py/iotdb/utils/{IoTDBConnectionException.py =>
exception.py} (64%)
create mode 100644 iotdb-client/client-py/iotdb/utils/iotdb_rpc_dataset.py
create mode 100644 iotdb-client/client-py/iotdb/utils/rpc_utils.py
create mode 100644
iotdb-client/client-py/tests/integration/test_tablemodel_query.py
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/overview/PipeTsFileToTabletsMetrics.java
rename
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/function/partition/{SliceCache.java
=> PartitionCache.java} (86%)
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/AbstractFragmentParallelPlanner.java
copy
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/node/{StreamSortNode.java
=> GroupNode.java} (51%)
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/optimizations/ParallelizeGrouping.java
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/ILoadDiskSelector.java
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/InheritSystemMultiDisksStrategySelector.java
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/MinIOSelector.java
rename
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/memory/{LoadTsFileAnalyzeSchemaMemoryBlock.java
=> LoadTsFileMemoryBlock.java} (88%)
copy
integration-test/src/main/java/org/apache/iotdb/db/query/udf/example/relational/MyRepeatWithIndex.java
=>
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/relational/tvf/CapacityTableFunction.java
(55%)
copy
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/relational/tvf/{HOPTableFunction.java
=> SessionTableFunction.java} (61%)
copy
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/relational/tvf/{HOPTableFunction.java
=> VariationTableFunction.java} (54%)
create mode 100644
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/relational/tvf/WindowTVFUtils.java
