jihoonson commented on a change in pull request #12279:
URL: https://github.com/apache/druid/pull/12279#discussion_r815254439
##########
File path: processing/src/main/java/org/apache/druid/segment/IndexMergerV9.java
##########
@@ -809,6 +933,63 @@ private void mergeCapabilities(
}
}
+ /**
+ * Creates a merged columnCapabilities to merge two queryableIndexes.
+ * This method first snapshots a pair of capabilities and then merges them.
+ */
+ @Nullable
+ private static ColumnCapabilitiesImpl mergeCapabilities(
+ @Nullable final ColumnCapabilities capabilities,
+ @Nullable final ColumnCapabilities other,
+ CoercionLogic coercionLogic
+ )
+ {
+ ColumnCapabilitiesImpl merged =
ColumnCapabilitiesImpl.snapshot(capabilities, coercionLogic);
+ ColumnCapabilitiesImpl otherSnapshot =
ColumnCapabilitiesImpl.snapshot(other, coercionLogic);
+ if (merged == null) {
+ return otherSnapshot;
+ } else if (otherSnapshot == null) {
+ return merged;
+ }
+
+ if (!Objects.equals(merged.getType(), otherSnapshot.getType())
+ || !Objects.equals(merged.getComplexTypeName(),
otherSnapshot.getComplexTypeName())
+ || !Objects.equals(merged.getElementType(),
otherSnapshot.getElementType())) {
+ throw new ISE(
+ "Cannot merge columns of type[%s] and [%s]",
+ merged.getType(),
+ otherSnapshot.getType()
+ );
+ }
+
+
merged.setDictionaryEncoded(merged.isDictionaryEncoded().or(otherSnapshot.isDictionaryEncoded()).isTrue());
+
merged.setHasMultipleValues(merged.hasMultipleValues().or(otherSnapshot.hasMultipleValues()).isTrue());
+ merged.setDictionaryValuesSorted(
+
merged.areDictionaryValuesSorted().and(otherSnapshot.areDictionaryValuesSorted()).isTrue()
+ );
+ merged.setDictionaryValuesUnique(
+
merged.areDictionaryValuesUnique().and(otherSnapshot.areDictionaryValuesUnique()).isTrue()
+ );
+ merged.setHasNulls(merged.hasNulls().or(other.hasNulls()).isTrue());
+ // When merging persisted queryableIndexes in the same ingestion job,
+ // all queryableIndexes should have the exact same hasBitmapIndexes flag
set which is set in the ingestionSpec.
+ // One exception is null-only columns as they always do NOT have bitmap
indexes no matter whether the flag is set
+ // in the ingestionSpec. As a result, the mismatch checked in the if
clause below can happen
+ // when one of the columnCapability is from a real column and another is
from a null-only column.
+ // See NullColumnPartSerde for how columnCapability is created for
null-only columns.
+ // When the mismatch is found, we prefer the flag set in the ingestionSpec
over
+ // the columnCapability of null-only columns.
+ if (merged.hasBitmapIndexes() != otherSnapshot.hasBitmapIndexes()) {
+ merged.setHasBitmapIndexes(false);
+ }
Review comment:
This is actually the method that I migrated from
`ColumnCapabilities.merge()` and modified this line.
`ColumnCapabilities.merge()` is being used only for merging segments in
`IndexMergerV9` today, and I don't want other people to mistakenly use it after
I modify this line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
