siddharthteotia commented on a change in pull request #6876:
URL: https://github.com/apache/incubator-pinot/pull/6876#discussion_r626272227
##########
File path:
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java
##########
@@ -262,6 +266,21 @@ private void
extractH3IndexConfigsFromTableConfig(TableConfig tableConfig) {
}
}
+ private void
extractNoDictionaryColumnCompressionCodecConfigsFromTableConfig(TableConfig
tableConfig) {
+ Map<String, String> rawIndexColumnsToCompressionTypeMap = new HashMap<>();
+ List<String> rawIndexColumns = new ArrayList<>();
+ List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+ if (fieldConfigList != null) {
+ for (FieldConfig fieldConfig : fieldConfigList) {
+ if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW &&
fieldConfig.getNoDictionaryColumnCompressionCodec() != null) {
+ rawIndexColumns.add(fieldConfig.getName());
+ rawIndexColumnsToCompressionTypeMap.put(fieldConfig.getName(),
fieldConfig.getNoDictionaryColumnCompressionCodec().name());
+ }
+ }
+ }
+ setRawIndexColumnCompressionType(rawIndexColumns,
rawIndexColumnsToCompressionTypeMap);
Review comment:
The current code won't be able to handle the compression config set
through old way
Lets say column1 and column2 in an existing table T.
For column1, someone specified compression config through old way
(noDictionaryConfig map)
For column2, someone specified compression config (SNAPPY, ZSTD) through new
way (FieldConfig)
The current code won't be able to preserve the config of column1 which we
need to handle until everything is migrated from existing way to FieldConfig
I haven't seen the old way of config being used so far at Li in production.
But we don't know if someone in open source is using it or not. If they are, it
will break things for them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
