tkaymak opened a new issue, #41466: URL: https://github.com/apache/beam/issues/41466
### What happened? On Spark 4 the Structured Streaming runner fails in batch mode with `INTERNAL_ERROR Couldn't find method fromByteArray` when `Combine.perKey` groups by a key whose coder has no native Spark encoder, for example a `KV`, a `Void` or a POJO key. `CombineTest$BasicTests.testHotKeyCombining` is excluded from the Spark 4 batch ValidatesRunner suite for this. The Spark 4 `EncoderFactory` marks every Beam encoder as `AgnosticEncoders.StructEncoder`, also the ones that serialize to a single `BinaryType` column. Spark 3 decided this from the serializer, `TypedAggUtils.aggKeyColumn` checked `isSerializedAsStructForTopLevel`. Spark 4 checks `agnosticEncoder.isStruct`, wraps the binary grouping key in `CreateStruct`, and the key deserializer then calls `CoderHelpers.fromByteArray` with an `InternalRow`. Spark 3 is not affected. Part of #36841. ### Issue Priority Priority: 2 (default / most bugs should be filed as P2) ### Issue Components - [x] Component: Spark Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
