Jackie-Jiang commented on code in PR #18368:
URL: https://github.com/apache/pinot/pull/18368#discussion_r3329396125
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -76,6 +76,8 @@ public class ColumnMetadataImpl implements ColumnMetadata {
private final PartitionFunction _partitionFunction;
private final Set<Integer> _partitions;
private final boolean _autoGenerated;
+ @Nullable
+ private final String _parentOpenStructColumn;
Review Comment:
Let's call it `parentColumn` to be more general. We don't need it to tie it
with a specific data type
##########
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/OpenStructIndexConfig.java:
##########
@@ -0,0 +1,180 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.config.table;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.utils.JsonUtils;
+
+
+/// Configuration for the OPEN_STRUCT index on an OPEN_STRUCT column.
+///
+/// **Dense vs sparse:** a key is materialized as its own column if (a) it
appears in the explicit
+/// `denseKeys` set, or (b) its fill rate (fraction of documents containing
the key) is ≥
+/// `denseKeyMinFillRate`. Keys not satisfying either criterion go into a
sparse OPEN_STRUCT column.
+///
+/// **maxDenseKeys cutoff:** when more keys qualify as dense than
`maxDenseKeys` allows, the top
+/// `maxDenseKeys` keys ranked by fill rate are materialized; the rest fall
back to the sparse
+/// column. A non-positive value (default `0`) means unlimited — every
qualifying key is
+/// materialized. Use `denseKeys` to pin specific keys regardless of fill rate
ranking.
+///
+/// **Per-key index settings** are specified via `valueFieldConfigs` — each
entry is a standard
+/// [FieldConfig] (modern `indexes` format) for one materialized OPEN_STRUCT
key. Keys without an
+/// entry fall back to `defaultValueFieldConfig`. When neither is set, the
built-in default is
+/// DICTIONARY encoding with an inverted index.
+public class OpenStructIndexConfig extends IndexConfig {
+ public static final OpenStructIndexConfig DISABLED = new
OpenStructIndexConfig(false);
+ public static final OpenStructIndexConfig DEFAULT = new
OpenStructIndexConfig(true);
+
+ public static final double DEFAULT_DENSE_KEY_MIN_FILL_RATE = 0.5;
+ /// Default `maxDenseKeys`. `0` means unlimited.
+ public static final int DEFAULT_MAX_DENSE_KEYS = 0;
Review Comment:
Use `-1` to represent unlimited to avoid confusion. We can use `0` to
disable dense keys
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/V1Constants.java:
##########
@@ -173,6 +173,11 @@ public static class Column {
// Optional, default false
public static final String IS_AUTO_GENERATED = "isAutoGenerated";
+ // Optional. The parent OPEN_STRUCT column name for columns materialized
from an OPEN_STRUCT index.
+ // E.g. for column metrics$cpu with parent metrics:
+ // parentOpenStructColumn = metrics
+ public static final String PARENT_OPEN_STRUCT_COLUMN =
"parentOpenStructColumn";
Review Comment:
Simplify it to `parentColumn`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
