kkzi opened a new issue, #885:
URL: https://github.com/apache/tsfile/issues/885
## Problem
`TsFileWriter::do_check_schema` shows noticeable CPU overhead when writing
repeated tablets with the same device and schema.
In our workload, each writer repeatedly writes tablets for a fixed device
with a fixed set of measurements. However,
`write_tablet()` calls `do_check_schema()` for every tablet write. This
appears to repeatedly perform schema lookup and
measurement-name matching even though the schema has already been
registered and does not change during the file lifecycle.
## Evidence
Using Windows Performance Analyzer on a 60-second TsFile archive workload,
the active processing window showed:
-
`TsFileArchive.dll!storage::TsFileWriter::do_check_schema<MeasurementNamesFromTablet>`
as the top TsFileArchive function
hotspot
- `std::string::compare` also appeared as a related hotspot
- The workload writes tablets with the same device and the same schema
repeatedly
- TsFile writing took longer than the input data duration
Example hotspot:
```text
TsFileArchive.dll!storage::TsFileWriter::do_check_schema<storage::MeasurementNamesFromTablet>
```
## Suspected Cause
write_tablet() creates or looks up schema information on every call:
- creates a StringArrayDeviceID from tablet.insert_target_name_
- looks up the device in schemas_
- iterates all measurement names
- looks up each measurement in measurement_schema_map_
- fills chunk_writers and data_types
For wide tablets and high-frequency writes, this repeated per-tablet
schema validation becomes expensive.
## Suggested Optimization
Add a cached or prepared schema path for repeated tablet writes.
Possible approaches:
1. Cache the resolved schema for the last used device/tablet schema inside
TsFileWriter.
2. Add an explicit prepared API, for example:
PreparedTabletSchema prepare_tablet_schema(device_id, schema_vec);
int write_tablet_prepared(const Tablet& tablet, const
PreparedTabletSchema& prepared);
The cached/prepared data could include:
- MeasurementSchemaGroup*
- resolved ChunkWriter* list
- resolved TSDataType list
This would allow repeated writes with the same device and schema to skip
per-column name lookup.
## Expected Benefit
Reduce CPU overhead in high-throughput TsFile writing workloads,
especially for wide schemas where the same tablet schema is
written repeatedly.
## Notes
This should preserve the existing validation behavior for dynamic schemas
or changing devices. The optimization can be limited
to cache hits where both device and schema identity match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
