siddharthteotia commented on a change in pull request #4791: Support STRING and
BYTES for no dictionary columns in realtime consuming segments
URL: https://github.com/apache/incubator-pinot/pull/4791#discussion_r365487755
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
##########
@@ -183,15 +187,22 @@ public long getLatestIngestionTimestamp() {
_maxNumValuesMap.put(column, 0);
// Check whether to generate raw index for the column while consuming
- // Only support generating raw index on single-value non-string columns
that do not have inverted index while
+ // Only support generating raw index on single-value columns that do not
have inverted index while
// consuming. After consumption completes and the segment is built, all
single-value columns can have raw index
FieldSpec.DataType dataType = fieldSpec.getDataType();
- int indexColumnSize = FieldSpec.DataType.INT.size();
- if (noDictionaryColumns.contains(column) &&
fieldSpec.isSingleValueField()
- && dataType != FieldSpec.DataType.STRING &&
!invertedIndexColumns.contains(column)) {
- // No dictionary
- indexColumnSize = dataType.size();
+ boolean isFixedWidthColumn = dataType.isFixedWidth();
+ int forwardIndexColumnSize = -1;
+ if (isNoDictionarySupportedForColumn(noDictionaryColumns,
invertedIndexColumns, fieldSpec, column)) {
+ // no dictionary
+ // each forward index entry will be equal to size of data
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
