mcvsubbu commented on a change in pull request #5470:
URL: https://github.com/apache/incubator-pinot/pull/5470#discussion_r432905837
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
##########
@@ -213,6 +215,14 @@ public void init(SegmentGeneratorConfig
segmentCreationSpec, SegmentIndexCreatio
}
}
+ public static boolean shouldDeriveNumDocsPerChunk(String columnName,
Map<String, Map<String, String>> columnProperties) {
+ if (columnProperties != null) {
+ Map<String, String> properties = columnProperties.get(columnName);
+ return properties != null &&
Boolean.parseBoolean(properties.get(FieldConfig.DERIVE_NUM_DOCS_PER_CHUNK_RAW_INDEX_KEY));
Review comment:
The reason I ask is that if we introduce a config it is hard to
remove/deprecate, etc. if we make it a default for text column, we can always
introduce a config later to adjust. In both offline and realtime cases, we know
the average column size (or, can compute easily) at the segment generation
time, so it seems to me that this can be done automatically without introducing
a configuration. I would propose to NOT introduce a config at this time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
