[
https://issues.apache.org/jira/browse/KAFKA-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850866#comment-17850866
]
Adam Strickland commented on KAFKA-16858:
-----------------------------------------
*
{quote}Are you able to provide an anonymized form of your schema directly,
rather than just a high-level "Array of Structs"? I'm wondering if your schema
is capable of triggering the use of the mutable SchemaWrapper
[https://github.com/confluentinc/schema-registry/blob/7b886f309c83041d4f2a5b41b5910f3b8002413a/protobuf-converter/src/main/java/io/confluent/connect/protobuf/ProtobufData.java#L1779]
inside the ProtobufConverter.{quote}
Yes; see [^proto.proto]
*
{quote}w.r.t. the variable validateValue depth: Are you saying that in _error
cases_ the recursion depth is unpredictable, or in general? The validateValue
should be called at every or almost every location in the tree of values, so I
would expect to see lots of different recursion depths. Maybe you can share
some more stacktraces as examples.{quote}
We were only looking at error cases. I specifically recall seeing the same
message break on the 3rd recursion and on the 7th; the array for the message in
question contained 2 Structs.
*
{quote}So far in this investigation, I'm trying to find the source of the null
in hopes that we can prevent it, and get well-formed data to the Flatten SMT.
Regardless of the result of that investigation, I think we can consider this
input malformed, and throw an intentional DataException instead of
NullPointerException. Would that be an acceptable solution for you, or does
this data need to make it all the way through the pipeline?{quote}
Throwing a DataException would be preferable to the NPE. I'm not sure I
understand what you mean by "need to make it all the way through the
pipeline"... As it stands the particular problematic attribute is not something
we care about right now, which is why we can simply squash the Exception.
> Flatten SMT throws NPE
> ----------------------
>
> Key: KAFKA-16858
> URL: https://issues.apache.org/jira/browse/KAFKA-16858
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Affects Versions: 3.6.0
> Environment: Kafka 3.6 by way of CP 7.6.0
> Reporter: Adam Strickland
> Priority: Major
> Attachments: FlattenTest.java, proto.proto
>
>
> {{ConnectSchema.expectedClassesFor}} sometimes will throw an NPE as part of a
> call to an SMT chain. Stack trace snippet:
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.apply(MomentFlatten.java:84)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.applyWithSchema(MomentFlatten.java:173)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:274)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:203)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:216)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:255)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:213)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224)}}
> {{at
> org.apache.kafka.connect.data.ConnectSchema.expectedClassesFor(ConnectSchema.java:268)}}
> (the above transform is a sub-class of
> {{{}o.a.k.connect.transforms.Flatten{}}}; have confirmed that the error
> occurs regardless).
> The field being transformed is an array of structs. If the call to
> {{Schema#valueSchema()}} (o.a.k.connect.data.ConnectSchema.java:255) returns
> {{{}null{}}}, the subsequent call to {{Schema#name()}} at
> o.a.k.connect.data.ConnectSchema:268 throws an NPE.
> The strange thing that we have observed is that this doesn't always happen;
> *sometimes* the struct's schema is found and sometimes it is not. We have
> been unable to determine the root cause, but have constructed a test that
> replicates the problem as observed (see attachment).
> In our case we have worked around the issue with the aforementioned sub-class
> of {{{}Flatten{}}}, catching and logging the NPE on that specific use-case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
