clintropolis commented on code in PR #13803:
URL: https://github.com/apache/druid/pull/13803#discussion_r1136481488
##########
processing/src/main/java/org/apache/druid/segment/NestedDataColumnIndexer.java:
##########
@@ -59,16 +62,43 @@ public class NestedDataColumnIndexer implements
DimensionIndexer<StructuredData,
protected final StructuredDataProcessor indexerProcessor = new
StructuredDataProcessor()
{
@Override
- public ProcessedLiteral<?> processLiteralField(ArrayList<NestedPathPart>
fieldPath, Object fieldValue)
+ public ProcessedLiteral<?> processLiteralField(ArrayList<NestedPathPart>
fieldPath, @Nullable Object fieldValue)
{
- final String fieldName =
NestedPathFinder.toNormalizedJsonPath(fieldPath);
- LiteralFieldIndexer fieldIndexer = fieldIndexers.get(fieldName);
- if (fieldIndexer == null) {
- estimatedFieldKeySize +=
StructuredDataProcessor.estimateStringSize(fieldName);
- fieldIndexer = new LiteralFieldIndexer(globalDictionary);
- fieldIndexers.put(fieldName, fieldIndexer);
+ // null value is always added to the global dictionary as id 0, so we
can ignore them here
+ if (fieldValue != null) {
+ // why not
+ final String fieldName =
NestedPathFinder.toNormalizedJsonPath(fieldPath);
+ ExprEval<?> eval = ExprEval.bestEffortOf(fieldValue);
+ LiteralFieldIndexer fieldIndexer = fieldIndexers.get(fieldName);
+ if (fieldIndexer == null) {
+ estimatedFieldKeySize +=
StructuredDataProcessor.estimateStringSize(fieldName);
+ fieldIndexer = new LiteralFieldIndexer(globalDictionary);
+ fieldIndexers.put(fieldName, fieldIndexer);
+ }
+ return fieldIndexer.processValue(eval);
}
- return fieldIndexer.processValue(fieldValue);
+ return StructuredDataProcessor.ProcessedLiteral.NULL_LITERAL;
+ }
+
+ @Nullable
+ @Override
+ public ProcessedLiteral<?> processArrayOfLiteralsField(
+ ArrayList<NestedPathPart> fieldPath,
+ Object maybeArrayOfLiterals
+ )
+ {
+ final ExprEval<?> maybeLiteralArray =
ExprEval.bestEffortOf(maybeArrayOfLiterals);
+ if (maybeLiteralArray.type().isArray() &&
maybeLiteralArray.type().getElementType().isPrimitive()) {
+ final String fieldName =
NestedPathFinder.toNormalizedJsonPath(fieldPath);
+ LiteralFieldIndexer fieldIndexer = fieldIndexers.get(fieldName);
+ if (fieldIndexer == null) {
+ estimatedFieldKeySize +=
StructuredDataProcessor.estimateStringSize(fieldName);
+ fieldIndexer = new LiteralFieldIndexer(globalDictionary);
+ fieldIndexers.put(fieldName, fieldIndexer);
+ }
+ return fieldIndexer.processValue(maybeLiteralArray);
+ }
+ return null;
Review Comment:
the contract of `processArrayOfLiteralsField` is supposed to return a
`ProcessedLiteral` if and only if the value was an array of literals (it is
marked `@Nullable`). `processLiteralField` is not nullable, and must always
return a `ProcessedLiteral`.
The `StructuredDataProcessor` code when processing some input and it
encounters arrays will first attempt to `processArrayOfLiteralField`, if it
returns something, it was an array, else it must instead process the array
elements recursively. `processLiteralField` is called on everything that isn't
a map or array.
I'll see if i can clarify it better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
