kennknowles commented on code in PR #40481:
URL: https://github.com/apache/beam/pull/40481#discussion_r4271773435
##########
sdks/java/io/splunk/src/main/java/org/apache/beam/sdk/io/splunk/SplunkIO.java:
##########
@@ -166,7 +166,8 @@ public PCollection<SplunkWriteError>
expand(PCollection<SplunkEvent> input) {
// Return a PCollection<SplunkWriteError>
return input
.apply("Create KV pairs", CreateKeys.of(parallelism()))
- .apply("Write Splunk events", ParDo.of(writer));
+ .apply("Write Splunk events", ParDo.of(writer))
+ .setCoder(SplunkWriteErrorCoder.of());
Review Comment:
As you've probably noticed, the trouble with this is that it requires a
special wrapper to be encoded as a row to be used by other languages. This is
not such a terrible thing, but it would be better to do a hand-written schema
coder so that `SplunkWriteError` is still encoded as a row, it just cannot use
the `@AutoValueSchema` annotation to do it (though fixing the `get`-prefix
limitation might be a one-shot prompt...)
This would look like making a new `SchemaCoder(toFunction, fromFunction)`
where the to/from are just the same logic that you have in the SchemaTransform
wrapper.
Does this work? Am I overlooking at issue?
---
Aside: if we were to switch to the new coder since the `@AutoValueSchema`
annotation is not working as intended, we should replace it in the other file
like so:
```
@DefaultCoder(SplunkWriteErrorCoder.class)
public class SplunkWriteError {
...
}
```
Then the coder will be automatic, here and everywhere.
But again, I think we can keep it encoded as a row so we don't have the
breaking change and it works smoothly with the schema transform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
