tarun11Mavani commented on code in PR #19040:
URL: https://github.com/apache/pinot/pull/19040#discussion_r3682625364
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/MutableOpenStructDataSource.java:
##########
@@ -62,13 +69,19 @@ public ComplexFieldSpec getFieldSpec() {
@Override
@Nullable
public DataSource getDataSource(String key) {
- Map<IndexType, IndexReader> indexes = _index.getIndexes(key);
- if (indexes == null || indexes.isEmpty()) {
+ // Live lookup, outside the memo: a key not yet observed may still be
created by the ingestion thread.
+ MutableKeyColumn col = _index.getKeyColumn(key);
+ if (col == null) {
return null;
}
- ColumnMetadata metadata = _index.getColumnMetadata(key);
- return new ImmutableDataSource(metadata,
- new ColumnIndexContainer.FromMap.Builder().withAll(indexes).build());
+ return _perKeyDataSourceCache.computeIfAbsent(key, k -> {
+ Map<IndexType, IndexReader> indexes = new
HashMap<>(_index.getIndexes(k));
Review Comment:
Confirmed both failure modes: docIds past the last written chunk hit
`_readers.get(bufferId)` OOB, and in-range holes deserialized as dictId 0
(first inserted value). Fixed by making dictId 0 *be* the answer: the
dictionary now reserves it for the key's default null value at construction, so
zero-initialized chunks read absent docs as the default (same value the sealed
build folds in), and a last-written-docId watermark guards reads past the
allocated tail.
Added `OpenStructConsumingSealedParityTest` covering a key absent from
middle + tail rows, consuming vs sealed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
