raghavyadav01 commented on code in PR #18637:
URL: https://github.com/apache/pinot/pull/18637#discussion_r3336469194
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReader.java:
##########
@@ -92,7 +96,34 @@ public Dictionary getDictionary() {
}
public int getDictId(int docId) {
- return _forwardIndexReader.getDictId(docId, _forwardIndexReaderContext);
+ if (_forwardIndexReader.isDictionaryEncoded()) {
+ return _forwardIndexReader.getDictId(docId, _forwardIndexReaderContext);
+ }
+ if (_dictionary == null) {
+ throw new UnsupportedOperationException(
+ "Cannot resolve dictId: forward index is raw and no dictionary is
materialized for column: " + (
+ _columnName != null ? _columnName : "<unknown>"));
+ }
+ // If we have separate dictionary on a RAW forward index column, use that
dictionary.
+ switch (_valueType.getStoredType()) {
+ case INT:
+ return _dictionary.indexOf(_forwardIndexReader.getInt(docId,
_forwardIndexReaderContext));
+ case LONG:
+ return _dictionary.indexOf(_forwardIndexReader.getLong(docId,
_forwardIndexReaderContext));
+ case FLOAT:
+ return _dictionary.indexOf(_forwardIndexReader.getFloat(docId,
_forwardIndexReaderContext));
+ case DOUBLE:
+ return _dictionary.indexOf(_forwardIndexReader.getDouble(docId,
_forwardIndexReaderContext));
+ case BIG_DECIMAL:
+ return _dictionary.indexOf(_forwardIndexReader.getBigDecimal(docId,
_forwardIndexReaderContext));
+ case STRING:
+ return _dictionary.indexOf(_forwardIndexReader.getString(docId,
_forwardIndexReaderContext));
+ case BYTES:
+ return _dictionary.indexOf(new
ByteArray(_forwardIndexReader.getBytes(docId, _forwardIndexReaderContext)));
Review Comment:
this allocates a fresh `byte[]` + `ByteArray` wrapper per row during the
star-tree build (`numDocs × numBytesDims` allocations total). Have you measured
the build-time cost of this path on a representative segment? Can we optimize?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
