[
https://issues.apache.org/jira/browse/KAFKA-12305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281258#comment-17281258
]
Chris Egerton commented on KAFKA-12305:
---------------------------------------
Thanks [~rhauch]. Do you have thoughts on whether elements inside arrays should
themselves be flattened? For example, with this input:
{code:java}
{
"f1": [
{
"f2": {
"f3": "v1"
}
}
]
}
{code}
should the output be this (option A):
{code:java}
{
"f1": [
{"f2.f3": "v1"}
]
}{code}
or this (option B):
{code:java}
{
"f1": [
{
"f2": {
"f3": "v1"
}
}
]
}
{code}
I'm leaning toward option A since, as you noted, the SMT definition deals
explicitly with structs and maps, and makes no distinction about whether
they're in arrays or top-level.
> Flatten SMT fails on arrays
> ---------------------------
>
> Key: KAFKA-12305
> URL: https://issues.apache.org/jira/browse/KAFKA-12305
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.0.1, 2.1.1, 2.2.2, 2.3.1, 2.4.1, 2.5.1, 2.7.0, 2.6.1,
> 2.8.0
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Major
>
> The {{Flatten}} SMT fails for array types. A sophisticated approach that
> tries to flatten arrays might be desirable in some cases, and may have been
> punted during the early design phase of the transform, but in the interim,
> it's probably not worth it to make array data and the SMT mutually exclusive.
> A naive approach that preserves arrays as-are and doesn't attempt to flatten
> them seems fair for now, but one alternative could be to traverse array
> elements and, if any are maps or structs, flatten those as well.
> Adding behavior to fully flatten arrays by essentially transforming them into
> maps whose elements are the elements of the array and whose keys are the
> indices of each element is likely out of scope for a bug fix and, although
> useful, might have to wait for a KIP.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
