rangareddy commented on issue #17171: URL: https://github.com/apache/hudi/issues/17171#issuecomment-5366104863
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9743). **Findings: needs a reproduction against a current build before it can be dispositioned.** The report is concrete: Hudi Streamer jobs on Dataproc fail with java.lang.IllegalArgumentException: Invalid lambda deserialization when ingesting Protobuf records from Confluent Cloud via ProtoKafkaSource. Reproducing it looks tractable - the report includes the full stack trace through InvocationTargetException; reproducing needs a Protobuf Kafka source, but 'Invalid lambda deserialization' is a classic serialized-lambda/classloader mismatch and is often reproducible locally with a ProtoKafkaSource unit test. I have not called this from the source. The surrounding code has been rewritten substantially since this was filed - table versions 8 and 9, the `BufferedRecordMerger` family replacing payload-based merging, the file-group-reader migration, the indexer abstraction in #18348 / #18372, and the `hudi-common` package moves in #19193 / #19195. Inferring either "still broken" or "silently fixed" through that much churn would be a guess. There is also no commit anywhere in the tree referencing HUDI-9743, so if it was fixed it was incidental rather than deliberate - another reason to verify rather than assume. 'Invalid lambda deserialization' almost always means a lambda was serialized by one class version and deserialized by another - so the first thing to check is whether a bundled jar and a cluster-provided jar both supply the offending class. That makes it worth reading alongside #17177 (HUDI-9758), which is about letting reflection utils use the thread context classloader; the two may share a root cause. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
