Caideyipi opened a new pull request, #18031:
URL: https://github.com/apache/iotdb/pull/18031
## Motivation
When a receiver disables `enable_auto_create_schema`, a full pipe can still
need to load historical tree TsFiles before the corresponding schema
snapshot/realtime schema events have been applied on the receiver. In the
reported case, the history pipe is auto-split to transfer data only, and
`LoadTsFile` runs with schema verification enabled. The receiver then fails to
load the TsFile because a device such as `root.sg.device_nonAligned` does not
yet exist and the global auto-create-schema switch blocks schema creation.
This is not simply caused by delete operations removing data on the
receiver. The direct failure is that pipe-triggered `LoadTsFile` cannot create
or verify schema while loading the historical TsFile, so the data in that file
is never loaded.
## Changes
This PR adds a pipe-specific schema auto-create path for tree `LoadTsFile`
analysis:
- Exposes the existing `LoadTsFileAnalyzer` pipe-generated marker to the
tree schema verifier.
- Allows pipe-generated tree `LoadTsFile` to auto-create required databases
during analysis, even if the receiver disables global auto-create schema.
- Adds a forced auto-create flag to the schema fetch/validation path used by
tree `LoadTsFile`.
- Keeps existing behavior for non-pipe load and normal insert validation:
they still honor `enable_auto_create_schema=false`.
- Adds an integration regression test for auto-split full pipe history sync
with deletions when the receiver disables auto-create schema.
## Behavior after this change
For a pipe-generated tree TsFile load:
- The receiver may create the database and missing timeseries schema
required by the TsFile during schema verification.
- Schema verification still runs and can still reject real mismatches, such
as alignment or type mismatches.
- Historical TsFile data can be loaded even when schema transfer is handled
separately by the realtime/schema side of an auto-split pipe.
For user-issued `LOAD TSFILE`, normal inserts, and other non-pipe paths:
- `enable_auto_create_schema=false` continues to prevent automatic schema
creation.
## Benefits for Pipe
This makes full/history pipe sync more robust when the receiver is
configured conservatively with auto schema creation disabled:
- Avoids data loss caused by history TsFile load failure when schema events
arrive later or are transferred through another auto-split pipe.
- Preserves strict receiver behavior for ordinary client writes while
allowing pipe replay to reconstruct the sender-side schema needed for
historical data files.
- Keeps verification enabled, so Pipe does not silently load incompatible
data just to avoid the schema creation failure.
- Improves auto-split full sync behavior for workloads that include
deletes/mods and data-only history transfer.
## Tests
- `mvn spotless:apply -pl iotdb-core/datanode`
- `mvn spotless:apply -pl integration-test -P with-integration-tests`
- `git diff --check`
- `mvn -Ddevelocity.off=true verify -DskipUTs
-Dit.test=IoTDBPipeReceiverAutoCreateDisabledIT#testAutoSplitHistoryTsFileWithDeletionWhenReceiverAutoCreateSchemaDisabled
-DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false -pl
integration-test -P with-integration-tests`
- test sources compiled successfully
- build stopped before executing IT at `cluster-test-assembly-share`
because the local workspace lacks
`external-service-impl/mqtt/target/mqtt-2.0.7-SNAPSHOT-jar-with-dependencies.jar`
- `mvn -Ddevelocity.off=true compile -pl iotdb-core/datanode`
- reached javac
- failed on existing unrelated compile errors outside the touched files,
including generated aggregation/fill classes and missing constants in
`PipeSourceConstant`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
