Caideyipi opened a new pull request, #18314:
URL: https://github.com/apache/iotdb/pull/18314
## Description
### Problem and observed behavior
Historical Pipe events from ConfigRegion, SchemaRegion, and DataRegion are
transferred independently. A historical TsFile can therefore reach the receiver
before the schema and template events that describe it, making the result
dependent on event arrival order.
The reported screenshots show this timing-dependent failure:
- The replicated data and all three expected devices are already visible on
the receiver.
- `SHOW PATHS SET DEVICE TEMPLATE aligned_template` does not return the
expected `root.sg_aligned.device_aligned`.
- The receiver log reports that setting `aligned_template` on
`root.sg_aligned.device_aligned` failed because ordinary timeseries had already
been created under that path.
The early TsFile load auto-creates ordinary timeseries. When the historical
template-set event arrives later, it conflicts with those timeseries and the
receiver cannot restore the original template binding.
### Fix
- Mark a TsFile transfer as requiring schema readiness only when all of the
following are true:
- the source is the built-in IoTDB source;
- the sink sends a Pipe request rather than a general write request;
- the inclusion options contain all schema events required before loading
data: ordinary timeseries creation and template create/alter/set/activate.
- Carry this marker in V2 TsFile seal parameters through synchronous Thrift,
asynchronous Thrift, and AirGap sinks.
- For a marked transfer, force schema verification and disable schema
auto-creation for that individual load.
- Treat missing schema as temporary unavailability so Pipe retries the
TsFile after the corresponding schema/template events arrive.
- Preserve the per-load option through the Active Load path used by
asynchronous loading.
- Verify existing schema under the schema-deletion read lock to avoid racing
with deletion while auto-creation is disabled.
This avoids imposing cross-region transfer ordering: data loading simply
waits and retries until the schema history has been applied.
### Compatibility
The new behavior is opt-in through an internal system parameter and defaults
to disabled on the wire. Existing behavior is preserved for external sources,
incomplete schema inclusion, general-write requests,
`mark-as-pipe-request=false`, legacy requests, and ordinary `LOAD TSFILE`
operations. The existing default that permits per-load schema auto-creation
also remains unchanged unless the receiver gets the new marker.
### Tests
- Added focused unit coverage for parameter gating, V2 request
serialization, synchronous/asynchronous receiver configuration, retry
classification, and Active Load attribute persistence.
- Extended `IoTDBPipeAutoConflictIT` to assert that
`root.sg_aligned.device_aligned` is bound to `aligned_template` on the receiver.
- Focused test run: `OK (76 tests)` across 6 test classes.
- Spotless, whitespace checks, and isolated compilation of the affected
sources passed.
- A full local Maven/IT run could not reach the target IT because this
checkout has unrelated stale Freemarker-generated sources and an incompatible
`template-node` artifact.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent write
- [x] added comments explaining non-obvious intent.
- [x] added or updated unit tests for the new code paths.
- [x] updated integration-test coverage.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `PipeDataNodeTaskBuilder`: determines whether a Pipe has complete schema
history and injects the internal wait marker.
- `PipeTransferTsFileSealWithModReq` and DataRegion sinks: propagate the
marker across all V2 transfer paths.
- `IoTDBDataNodeReceiver`: configures marked sync and async loads to verify
without auto-creating schema.
- `LoadTsFileAnalyzer`, `TreeSchemaAutoCreatorAndVerifier`, and
`SchemaValidator`: support per-load schema auto-creation control and retryable
missing-schema handling.
- Active Load helpers/configuration: persist the per-load option across
asynchronous handoff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
