Jackie-Jiang commented on a change in pull request #5147: Support default
star-tree
URL: https://github.com/apache/incubator-pinot/pull/5147#discussion_r399876497
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/startree/v2/builder/StarTreeV2BuilderConfig.java
##########
@@ -58,6 +69,73 @@ public static StarTreeV2BuilderConfig
fromIndexConfig(StarTreeIndexConfig indexC
return builder.build();
}
+ /**
+ * Generates default config based on the segment metadata.
+ * <ul>
+ * <li>
+ * All dictionary-encoded single-value dimensions (including date-time
columns) with cardinality smaller or equal
+ * to the threshold will be included in the split order, sorted by their
cardinality in descending order
+ * </li>
+ * <li>Time column (if exists and dictionary-encoded) will be appended to
the split order as the last element</li>
+ * <li>Use COUNT(*) and SUM for all numeric metrics as function column
pairs</li>
+ * <li>Use default value for max leaf records</li>
+ * </ul>
+ */
+ public static StarTreeV2BuilderConfig
generateDefaultConfig(SegmentMetadataImpl segmentMetadata) {
+ Schema schema = segmentMetadata.getSchema();
+ List<ColumnMetadata> dimensionColumnMetadataList = new ArrayList<>();
+ String timeColumn = null;
+ List<String> numericMetrics = new ArrayList<>();
+
+ for (FieldSpec fieldSpec : schema.getAllFieldSpecs()) {
+ if (!fieldSpec.isSingleValueField() || fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+ String column = fieldSpec.getName();
+ switch (fieldSpec.getFieldType()) {
+ case DIMENSION:
+ case DATE_TIME:
+ ColumnMetadata columnMetadata =
segmentMetadata.getColumnMetadataFor(column);
+ if (columnMetadata.hasDictionary()
+ && columnMetadata.getCardinality() <=
DIMENSION_CARDINALITY_THRESHOLD_FOR_DEFAULT_CONFIG) {
+ dimensionColumnMetadataList.add(columnMetadata);
+ }
+ break;
+ case TIME:
+ columnMetadata = segmentMetadata.getColumnMetadataFor(column);
+ if (columnMetadata.hasDictionary()) {
+ timeColumn = column;
+ }
+ break;
+ case METRIC:
+ if (fieldSpec.getDataType().isNumeric()) {
Review comment:
For `BYTES` column, there is no way to tell what type of data it represents
from the metadata. Similar to all other aggregation types (such as `MIN`,
`MAX`, etc.), `DICTINCTCOUNTHLL` needs to be configured through the customized
star-tree config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
