J-HowHuang commented on code in PR #16624:
URL: https://github.com/apache/pinot/pull/16624#discussion_r2283702867
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/ComplexTypeTransformer.java:
##########
@@ -233,6 +241,14 @@ public List<GenericRow> transform(List<GenericRow>
records) {
}
}
}
+ if (_fieldsNeededForDownstreamTransformers != null) {
Review Comment:
An unnecessary column could be added even it's not the unnest column. For
example:
```
"complexTypeConfig": {
"fieldsToUnnest": [
"payload.commits"
]
}
```
With input
```
{
"payload": {
"col_unnecessary": {
"colA": "123"
},
"commits": {
"colA": "123",
"colB": "456"
}
}
}
```
We'll have `payload.col_unnecessary` in the transformed `GenericRow`. This
is because during the flattening, it's using BFS and adding one level at a
time--`payload.col_unnecessary` and `payload.commits` were added to
`GenericRow`, then `flattenMap` recursively called on these two column, then
`payload.col_unnecessary.colA`, `payload.commits.colA`, `payload.commits.colB`
were added.
We can't avoid adding `payload.col_unnecessary` into `GenericRow` unless we
re-write the flattening here with DFS approach
Plus we'll have to rename the prefix first to know if that field is
unnecessary or not since the columns passed in with
`withInputColumnsOfDownStreamTransformers` is in the context of the names after
prefix-renaming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
