tarun11Mavani commented on code in PR #18643:
URL: https://github.com/apache/pinot/pull/18643#discussion_r3503466344
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java:
##########
@@ -552,6 +553,31 @@ protected void writeMetadata()
hasDictionary, dictionaryElementSize, fwdConfig.getEncodingType(),
false);
}
+ // OPEN_STRUCT splitters produce per-key materialized child columns
(col$key, col$__sparse__) that
+ // are not in _columnStatisticsMap. Merge their pre-built metadata into
the segment properties so
+ // each child appears as its own column at load time, and register them as
dimensions so the V3
+ // converter and SegmentMetadataImpl discover their index files.
+ List<String> openStructChildColumns = new ArrayList<>();
+ for (ColumnIndexCreators columnIndexCreators : _colIndexes.values()) {
+ for (IndexCreator indexCreator : columnIndexCreators.getIndexCreators())
{
+ if (indexCreator instanceof ColumnarOpenStructIndexCreator) {
+ ColumnarOpenStructIndexCreator splitter =
(ColumnarOpenStructIndexCreator) indexCreator;
+ for (Map.Entry<String, PropertiesConfiguration> childEntry
+ : splitter.getMaterializedColumnMetadata().entrySet()) {
+ String childCol = childEntry.getKey();
+ PropertiesConfiguration childProps = childEntry.getValue();
+ childProps.getKeys().forEachRemaining(key ->
properties.setProperty(key, childProps.getProperty(key)));
+ openStructChildColumns.add(childCol);
+ }
+ }
+ }
+ }
+ if (!openStructChildColumns.isEmpty()) {
+ List<String> dimensions = new ArrayList<>(_config.getDimensions());
+ dimensions.addAll(openStructChildColumns);
+ properties.setProperty(DIMENSIONS, dimensions);
Review Comment:
The materialized children are appended to DIMENSIONS because that's how the
segment loader discovers columns to load — it iterates the dimensions list. If
they're absent, the loader skips them and the segment loads without its
OPEN_STRUCT child columns.
Consumers that reconcile DIMENSIONS against the schema will see unknown
columns, but this is already handled: `ImmutableSegmentLoader` has a 9-line
exemption that skips columns whose `parentColumn` metadata points to a valid
OPEN_STRUCT schema field. `ColumnMetadataImpl.isMaterializedChild()` is the
signal that a column is synthetic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
