Caideyipi commented on code in PR #18816:
URL: https://github.com/apache/iotdb/pull/18816#discussion_r4244734328
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedWritableMemChunk.java:
##########
@@ -262,35 +262,33 @@ public void writeAlignedTablet(
}
/**
- * Check metadata of columns and return array that mapping existed metadata
to index of data
- * column.
+ * Register missing measurements in insertion order, then reorder values and
bitmaps into the
+ * memchunk's column order.
*
* @param schemaListInInsertPlan Contains all existed schema in InsertPlan.
If some timeseries
* have been deleted, there will be null in its slot.
- * @return columnIndexArray: schemaList[i] is schema of
columns[columnIndexArray[i]]
+ * @return values and bitmaps aligned with this memchunk's schema list
*/
private Pair<Object[], BitMap[]> checkAndReorderColumnValuesInInsertPlan(
List<IMeasurementSchema> schemaListInInsertPlan, Object[] columnValues,
BitMap[] bitMaps) {
+ List<TSDataType> newDataTypes = new ArrayList<>();
+ for (IMeasurementSchema measurementSchema : schemaListInInsertPlan) {
+ if (measurementSchema != null
+ &&
!measurementIndexMap.containsKey(measurementSchema.getMeasurementName())) {
+ measurementIndexMap.put(measurementSchema.getMeasurementName(),
schemaList.size());
+ schemaList.add(measurementSchema);
+ newDataTypes.add(measurementSchema.getType());
+ }
+ }
Review Comment:
Applied in `5f27eb1`. The new-type list is created only after the first
missing measurement. Existing-schema writes now use one lookup/reorder pass;
the first missing measurement triggers a batch scan of the remaining schemas,
one TVList extension, and one resize of the reordered arrays. I also added
coverage for no-extension reordering and for retaining values/bitmaps already
processed before the first missing measurement.
I measured the additional iteration against the base implementation and the
initial PR, using the actual classes:
| No-extension operation | Columns | Rows | Base (µs/op) | Initial PR
(µs/op) | Revised (µs/op) |
| --- | ---: | ---: | ---: | ---: | ---: |
| Schema reorder | 10 | - | 0.105 ± 0.018 | 0.176 ± 0.029 | 0.117 ± 0.016 |
| Schema reorder | 100 | - | 1.144 ± 0.186 | 2.033 ± 0.465 | 1.139 ± 0.142 |
| Schema reorder | 1000 | - | 13.353 ± 1.557 | 22.187 ± 2.471 | 13.476 ±
2.471 |
| writeAlignedTablet | 10 | 1 | 0.523 ± 0.051 | 0.572 ± 0.083 | 0.453 ±
0.108 |
| writeAlignedTablet | 10 | 32 | 0.564 ± 0.144 | 0.556 ± 0.091 | 0.443 ±
0.068 |
| writeAlignedTablet | 100 | 1 | 3.247 ± 0.296 | 5.604 ± 1.486 | 3.262 ±
0.386 |
| writeAlignedTablet | 100 | 32 | 4.218 ± 0.858 | 5.069 ± 0.940 | 3.947 ±
0.619 |
| writeAlignedTablet | 1000 | 1 | 31.928 ± 2.716 | 42.896 ± 4.491 | 31.957 ±
2.164 |
| writeAlignedTablet | 1000 | 32 | 37.621 ± 3.040 | 51.825 ± 7.318 | 40.975
± 5.591 |
| Columns | Base reorder allocation (B/op) | Initial PR (B/op) | Revised
(B/op) |
| ---: | ---: | ---: | ---: |
| 10 | 136 | 136 | 136 |
| 100 | 856 | 856 | 856 |
| 1000 | 8056 | 8080 | 8056 |
The initial PR's additional schema scan increased isolated reorder time by
approximately 66%–78% in these runs. The revised path removes that scan.
Revised/base latency confidence intervals overlap for every tested scenario,
and revised reorder allocation matches the base at each width. For the
1,000-column/32-row tablet case, the revised point estimate is 8.9% above the
base (40.975 ± 5.591 vs 37.621 ± 3.040 µs); these measurements do not establish
a statistically distinguishable regression, and they do not prove zero
overhead. At 1,000 columns the initial PR also allocated about 24 extra bytes
per isolated reorder; lazy initialization removes that allocation.
Local microbenchmarks used Oracle JDK 17.0.15, Windows 11, an Intel Core
i9-13900H, G1 and a fixed 512 MiB heap. Base is `21d53dd`, initial PR is
`1f05d5d`, and revised is `5f27eb1`. Each variant compiles the actual
`AlignedWritableMemChunk` and `AlignedTVList` sources against the same freshly
built reactor dependencies.
No-extension measurements use JMH 1.37: one thread, three independent JVM
forks, three 300 ms warmup iterations and five 300 ms measurement iterations
per fork. Every measurement already exists, incoming schema objects are
distinct and reverse column order, values are dense INT64, and a bitmap array
is present with unmarked columns. The schema-reorder benchmark invokes the
private method through the same cached MethodHandle and consumes its result.
The tablet benchmark times the public `writeAlignedTablet` call; clearing and
prepopulating one row are outside the timed call, so the measured write uses
existing value blocks. Values below are means with JMH's 99.9% confidence
interval. Allocation is reported for the isolated reorder only; the tablet
profiler also counts invocation setup.
I also rechecked the 10,000-column expansion path:
| New INT64 columns | Values | Base (ms) | Initial PR (ms) | Revised (ms) |
Base allocated (MiB) | Revised allocated (MiB) |
| ---: | --- | ---: | ---: | ---: | ---: | ---: |
| 10,000 | Sparse | 298.12 | 7.00 | 6.13 | 956.22 | 1.95 |
| 10,000 | Dense | 354.01 | 18.55 | 17.19 | 962.26 | 7.98 |
Expansion measurements write one tablet row after one existing populated
INT64 column. Setup and release are excluded. Sparse values supply only the
first and last new columns; dense values supply every column. Each JVM warms up
with 30 expansions of 256 columns; the reported values are medians of three
independent JVMs. Current-thread allocation during `writeAlignedTablet` is
measured with `ThreadMXBean`, rather than retained heap size. These are local
microbenchmarks and do not measure distributed throughput or region-lock wait
time.
All 66 targeted tests passed. The PR description now includes the
measurements and their scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]