This is an automated email from the ASF dual-hosted git repository.
marklau99 pushed a change to branch wal-compress-formal-branch
in repository https://gitbox.apache.org/repos/asf/iotdb.git
omit 5f3c1e3d453 fix bug
omit 0c22b438180 temp for debug
omit a65ecb683de add log for WAL size
omit 70d84004231 edit configuration item
omit de9e810029b support multi version WAL
omit 3d59603d767 support compression type in WAL Compress Header
omit 648c18ee43e recover some code
omit 175a5f78491 remove useless code
omit dcb13a5b893 fix compilation problem
omit b8057212491 fix bug
omit 2c59762ad5c enable wal compression
add 214d8d2d77d Pipe: fix pipe plugin instance not closed issue after
failed initialization (#12509)
add 5e6737108dd Bump org.springframework:spring-web from 5.3.33 to 5.3.34
(#12506)
add 0df3700054b add allow compaction check when inner space compaction
task after start (#12494)
add b80aa0ae708 Degrade configNode leader redirection log level and avoid
meaningless log print when ConfigNodeClient is first constructed. (#12510)
add be6e889312a [IOTDB-6325] Support RegionScan for active metadata
queries [FE Part]
add 234dab783ee Fix return value of fetch schmea with incorrect template
info
add e2bd04d87c4 Pipe: Fix iterator.hasNext() == true after removal from
empty ConcurrentIterableLinkedQueue (#12514)
add feb61e8d83a Pipe: Fix InsertRowsNode is not supported in batch mode
(#12517)
add 6d445f92a1e Pipe: Degraded the lock in PipeEventCollector to avoid
waitForTsFileClose() blocking pipe drop (#12518)
add c893101de1b Pipe: cache leader support batch mode (#12486)
add 741a218fb75 fix filter push down is not valid in template align by
device situation
add c12c84101c8 Fixed invalid links in En version & CH version for README
(#12512)
add 98d185d5694 fixed default parameters, output ulimit as empty, remove
excess logs from the bat script (#12522)
add 3e57c3fc78c Adjust the hard connection error log information to debug
level (#12523)
add f16df817848 Add new optimization rule: remove `Project` under
`RawDataAggregation`
add 2cb13ed1613 Pipe: timely restart pipes according to the deletion of
original TsFiles (#12406)
add 48d87bc4fd8 Pipe: Fixed the bug that events that do not trigger
collection in processorSubtask may not be reported & the pipes that extract
deletion in dataRegion extractor may be auto-restarted (#12497)
add d506eb78cb4 Load: Update TsFile Resource when
enable_auto_create_schema and verify schema are FALSE in Analysis Stage (#12525)
add 8f23949384d Change TsFile version
add 796dd5a5399 add transformStatus method in TsFileResource (#12526)
add 504810cd406 Pipe: Fixed the bug that schema region and config region
cannot report progress (#12528)
add 257d9aabac0 [IOTDB-6327] Random choosing available nodes to send sql
requests
add cfe19176edb Fix compaction write rate can not hot load
add e3aa4d8954c Pipe: RPC compression for data sync connectors (SNAPPY,
GZIP, LZ4, ZSTD and LZMA2) (#12519)
add 44ed96111ab Add device number metric
add 58331a6c73d bump ratis to 3.1.0-snapshot (#12533)
add b50ec7d64a4 Revert "bump ratis to 3.1.0-snapshot (#12533)"
add e0eea1d14bf JDBC: Fix maxRows cannot be set to 0 (#12540)
add 2f1ca39f391 Fix non-root user query data returns unexpected path
add 29fab685489 Pipe IT: Added flush on
IoTDBPipeExtractorIT#testExtractorPatternMatch() to avoid pipe on previous
follower getting unclosed historical tsfile from the old leader (#12534)
add 8562eead2bc add remote storage block for tiered storage (#12535)
add 95708409a01 [RatisConsensus] Use empty map for create peer (#12463)
add 9f92b1ae829 bump ratis 3.1.0-d09cbac-SNAPSHOT (#12547)
add 81382d71c05 Fix client cpp IT warning (#12548)
add 867f78373f9 enable wal compression
add 99f7f741047 fix bug
add 35bdf6f28d2 fix compilation problem
add e84647486d1 remove useless code
add 326e2c7ed92 recover some code
add 911d3dc4ada support compression type in WAL Compress Header
add 19b5cda1a35 support multi version WAL
add 23382f32313 edit configuration item
add 9f70798aa6a add log for WAL size
add c7065c4306e temp for debug
add 42d3c8b1fc7 fix bug
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (5f3c1e3d453)
\
N -- N -- N refs/heads/wal-compress-formal-branch (42d3c8b1fc7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/client-cpp.yml | 7 +-
README.md | 2 +-
README_ZH.md | 4 +-
.../org/apache/iotdb/db/it/auth/IoTDBAuthIT.java | 29 ++
...T.java => IoTDBPipeConnectorCompressionIT.java} | 111 ++++---
.../pipe/it/autocreate/IoTDBPipeExtractorIT.java | 36 ++-
.../parameter/PipeParameterValidator.java | 4 +-
.../org/apache/iotdb/tool/IoTDBDataBackTool.java | 2 +-
.../org/apache/iotdb/isession/INodeSupplier.java | 3 +
.../java/org/apache/iotdb/jdbc/IoTDBStatement.java | 4 +-
.../apache/iotdb/session/DummyNodesSupplier.java | 12 +
.../org/apache/iotdb/session/NodesSupplier.java | 26 +-
.../apache/iotdb/session/QueryEndPointPolicy.java} | 7 +-
.../apache/iotdb/session/RoundRobinPolicy.java} | 17 +-
.../java/org/apache/iotdb/session/Session.java | 22 +-
.../org/apache/iotdb/db/qp/sql/IoTDBSqlParser.g4 | 12 +-
.../protocol/IoTDBConfigRegionAirGapConnector.java | 27 +-
.../protocol/IoTDBConfigRegionConnector.java | 20 +-
.../pipe/execution/PipeConfigNodeSubtask.java | 73 +++--
.../pipe/extractor/IoTDBConfigRegionExtractor.java | 23 ++
.../receiver/protocol/IoTDBConfigNodeReceiver.java | 3 +
.../org/apache/iotdb/db/conf/IoTDBDescriptor.java | 2 +-
.../db/pipe/agent/task/PipeDataNodeTaskAgent.java | 57 +++-
.../client/IoTDBDataNodeAsyncClientManager.java | 10 +-
.../IoTDBDataNodeCacheLeaderClientManager.java | 2 +-
.../client/IoTDBDataNodeSyncClientManager.java | 9 +
...oTDBThriftAsyncPipeTransferBatchReqBuilder.java | 36 ---
...IoTDBThriftSyncPipeTransferBatchReqBuilder.java | 29 --
...ferBatchReqBuilder.java => PipeEventBatch.java} | 61 ++--
.../builder/PipeTransferBatchReqBuilder.java | 219 +++++---------
.../evolvable/request/PipeTransferPlanNodeReq.java | 1 +
.../request/PipeTransferTabletBatchReq.java | 23 +-
.../request/PipeTransferTabletBinaryReq.java | 57 ++--
.../request/PipeTransferTabletInsertNodeReq.java | 13 +-
.../airgap/IoTDBDataNodeAirGapConnector.java | 12 +-
.../airgap/IoTDBDataRegionAirGapConnector.java | 24 +-
.../airgap/IoTDBSchemaRegionAirGapConnector.java | 15 +-
.../async/IoTDBDataRegionAsyncConnector.java | 67 ++--
.../PipeTransferTabletBatchEventHandler.java | 24 +-
.../PipeTransferTsFileInsertionEventHandler.java | 19 +-
.../thrift/sync/IoTDBDataNodeSyncConnector.java | 5 +-
.../thrift/sync/IoTDBDataRegionSyncConnector.java | 72 +++--
.../thrift/sync/IoTDBSchemaRegionConnector.java | 15 +-
.../db/pipe/connector/util/LeaderCacheUtils.java | 69 +++++
.../common/tsfile/PipeTsFileInsertionEvent.java | 4 +-
.../dataregion/IoTDBDataRegionExtractor.java | 6 +
.../iotdb/db/pipe/metric/PipeProcessorMetrics.java | 24 +-
.../protocol/thrift/IoTDBDataNodeReceiver.java | 54 +++-
.../pipe/resource/tsfile/PipeTsFileResource.java | 33 +-
.../resource/tsfile/PipeTsFileResourceManager.java | 35 ++-
.../pipe/task/connection/PipeEventCollector.java | 21 +-
.../db/pipe/task/stage/PipeTaskExtractorStage.java | 13 +
.../connector/PipeConnectorSubtaskManager.java | 13 +
.../subtask/processor/PipeProcessorSubtask.java | 20 +-
.../iotdb/db/protocol/client/ConfigNodeClient.java | 35 ++-
.../impl/DataNodeInternalRPCServiceImpl.java | 19 +-
.../queryengine/common/TimeseriesSchemaInfo.java | 123 ++++++++
.../common/schematree/ClusterSchemaTree.java | 31 +-
.../common/schematree/DeviceSchemaInfo.java | 4 +
.../queryengine/common/schematree/ISchemaTree.java | 2 +
.../common/schematree/node/SchemaInternalNode.java | 5 +
.../schematree/node/SchemaMeasurementNode.java | 6 +
.../common/schematree/node/SchemaNode.java | 2 +
.../schema/source/TimeSeriesSchemaSource.java | 2 +-
.../db/queryengine/plan/analyze/Analysis.java | 29 ++
.../queryengine/plan/analyze/AnalyzeVisitor.java | 194 +++++++++++-
.../plan/analyze/LoadTsfileAnalyzer.java | 39 +--
.../plan/analyze/schema/ClusterSchemaFetcher.java | 7 +-
.../plan/analyze/schema/ISchemaFetcher.java | 16 +
.../plan/optimization/AggregationPushDown.java | 18 +-
.../db/queryengine/plan/parser/ASTVisitor.java | 41 ++-
.../plan/planner/LogicalPlanBuilder.java | 21 ++
.../plan/planner/LogicalPlanVisitor.java | 37 ++-
.../plan/planner/TemplatedLogicalPlanBuilder.java | 5 +-
.../distribution/DistributionPlanContext.java | 4 +
.../planner/distribution/ExchangeNodeAdder.java | 12 +
.../plan/planner/distribution/SourceRewriter.java | 65 ++++
.../plan/planner/plan/node/PlanNodeType.java | 14 +-
.../plan/planner/plan/node/PlanVisitor.java | 20 ++
.../{IntoNode.java => RegionMergeNode.java} | 85 +++---
.../source/AlignedSeriesAggregationScanNode.java | 6 +-
.../plan/node/source/DeviceRegionScanNode.java | 175 +++++++++++
.../planner/plan/node/source/RegionScanNode.java | 94 ++++++
.../plan/node/source/TimeseriesRegionScanNode.java | 336 +++++++++++++++++++++
.../crud/InsertMultiTabletsStatement.java | 8 +
.../statement/metadata/CountDevicesStatement.java | 16 +
.../metadata/CountTimeSeriesStatement.java | 15 +
.../statement/metadata/ShowDevicesStatement.java | 14 +
.../metadata/ShowTimeSeriesStatement.java | 16 +-
.../schemaengine/metric/SchemaEngineMemMetric.java | 15 +-
.../schemaengine/metric/SchemaRegionMemMetric.java | 21 ++
.../rescon/ISchemaEngineStatistics.java | 2 +
.../rescon/MemSchemaEngineStatistics.java | 14 +
.../rescon/MemSchemaRegionStatistics.java | 3 +
.../db/storageengine/dataregion/DataRegion.java | 2 +-
.../execute/task/AbstractCompactionTask.java | 12 +-
.../execute/task/InnerSpaceCompactionTask.java | 3 +
.../dataregion/tsfile/TsFileResource.java | 5 +
.../dataregion/tsfile/TsFileResourceBlockType.java | 1 +
.../SubscriptionConnectorSubtaskManager.java | 14 +
.../resource/PipeTsFileResourceManagerTest.java | 16 +-
.../plan/optimization/AggregationPushDownTest.java | 69 +++++
.../plan/optimization/TestPlanBuilder.java | 20 ++
.../distribution/AggregationAlignByDeviceTest.java | 44 ---
.../planner/distribution/AlignedByDeviceTest.java | 328 ++------------------
.../distribution/RegionScanPlanningTest.java | 149 +++++++++
.../logical/RegionScanLogicalPlannerTest.java | 206 +++++++++++++
.../src/assembly/resources/sbin/health_check.bat | 2 -
.../src/assembly/resources/sbin/health_check.sh | 1 +
.../apache/iotdb/commons/conf/CommonConfig.java | 11 +
.../iotdb/commons/conf/CommonDescriptor.java | 5 +
.../org/apache/iotdb/commons/path/AlignedPath.java | 6 +
.../apache/iotdb/commons/path/MeasurementPath.java | 4 +
.../commons/pipe/agent/plugin/PipePluginAgent.java | 10 +-
.../iotdb/commons/pipe/config/PipeConfig.java | 7 +
.../config/constant/PipeConnectorConstant.java | 18 ++
.../pipe/connector/compressor/PipeCompressor.java} | 51 ++--
.../compressor/PipeCompressorFactory.java | 80 +++++
.../connector/compressor/PipeGZIPCompressor.java} | 34 ++-
.../connector/compressor/PipeLZ4Compressor.java} | 34 ++-
.../connector/compressor/PipeLZMA2Compressor.java} | 34 ++-
.../compressor/PipeSnappyCompressor.java} | 34 ++-
.../connector/compressor/PipeZSTDCompressor.java} | 34 ++-
.../payload/thrift/request/PipeRequestType.java | 3 +
.../thrift/request/PipeTransferCompressedReq.java | 146 +++++++++
.../connector/protocol/IoTDBAirGapConnector.java | 7 +-
.../pipe/connector/protocol/IoTDBConnector.java | 62 ++++
.../connector/protocol/IoTDBSslSyncConnector.java | 8 +-
.../queue/ConcurrentIterableLinkedQueue.java | 20 +-
.../commons/pipe/event/PipeWritePlanEvent.java | 15 +-
.../ConcurrentIterableLinkedQueueTest.java | 84 +++---
pom.xml | 6 +-
132 files changed, 3454 insertions(+), 1210 deletions(-)
copy
integration-test/src/test/java/org/apache/iotdb/pipe/it/autocreate/{IoTDBPipeProcessorIT.java
=> IoTDBPipeConnectorCompressionIT.java} (56%)
copy
iotdb-client/{isession/src/main/java/org/apache/iotdb/isession/INodeSupplier.java
=> session/src/main/java/org/apache/iotdb/session/QueryEndPointPolicy.java}
(85%)
copy
iotdb-client/{isession/src/main/java/org/apache/iotdb/isession/INodeSupplier.java
=> session/src/main/java/org/apache/iotdb/session/RoundRobinPolicy.java} (72%)
delete mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/payload/evolvable/builder/IoTDBThriftAsyncPipeTransferBatchReqBuilder.java
delete mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/payload/evolvable/builder/IoTDBThriftSyncPipeTransferBatchReqBuilder.java
copy
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/payload/evolvable/builder/{PipeTransferBatchReqBuilder.java
=> PipeEventBatch.java} (76%)
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/util/LeaderCacheUtils.java
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/TimeseriesSchemaInfo.java
copy
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/process/{IntoNode.java
=> RegionMergeNode.java} (55%)
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/source/DeviceRegionScanNode.java
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/source/RegionScanNode.java
create mode 100644
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/source/TimeseriesRegionScanNode.java
create mode 100644
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/planner/distribution/RegionScanPlanningTest.java
create mode 100644
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/planner/logical/RegionScanLogicalPlannerTest.java
copy
iotdb-core/{consensus/src/main/java/org/apache/iotdb/consensus/ratis/metrics/CounterProxy.java
=>
node-commons/src/main/java/org/apache/iotdb/commons/pipe/connector/compressor/PipeCompressor.java}
(51%)
create mode 100644
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/connector/compressor/PipeCompressorFactory.java
copy
iotdb-core/{datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MergeSortKey.java
=>
node-commons/src/main/java/org/apache/iotdb/commons/pipe/connector/compressor/PipeGZIPCompressor.java}
(52%)
copy
iotdb-core/{datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MergeSortKey.java
=>
node-commons/src/main/java/org/apache/iotdb/commons/pipe/connector/compressor/PipeLZ4Compressor.java}
(52%)
copy
iotdb-core/{datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MergeSortKey.java
=>
node-commons/src/main/java/org/apache/iotdb/commons/pipe/connector/compressor/PipeLZMA2Compressor.java}
(52%)
copy
iotdb-core/{datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MergeSortKey.java
=>
node-commons/src/main/java/org/apache/iotdb/commons/pipe/connector/compressor/PipeSnappyCompressor.java}
(52%)
copy
iotdb-core/{datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/MergeSortKey.java
=>
node-commons/src/main/java/org/apache/iotdb/commons/pipe/connector/compressor/PipeZSTDCompressor.java}
(52%)
create mode 100644
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/connector/payload/thrift/request/PipeTransferCompressedReq.java
