johannaojeling commented on issue #28981: URL: https://github.com/apache/beam/issues/28981#issuecomment-1783727490
I don't get any errors when using anonymous functions with Prism 🤔 Other example: if modifying the `textio` unit test `TestReadWithFilename` to the below, the test still passes: https://github.com/apache/beam/blob/ab2ab2ff72a4802812bb8820a1b94e23ebbe2f9f/sdks/go/pkg/beam/io/textio/textio_test.go#L92 ```diff - got := beam.ParDo(s, toKV, lines) + got := beam.ParDo(s, func(k, v string) kv { + return kv{k, v} + }, lines) ``` I saw your note in the PR @lostluck about anonymous functions not working on any portable runner. I remember I wrote a [doc example](https://github.com/apache/beam/blob/ab2ab2ff72a4802812bb8820a1b94e23ebbe2f9f/sdks/go/pkg/beam/io/fileio/example_test.go#L70-L77) for `fileio.ReadMatches` that uses one. That example runs on Prism and Dataflow, but I should probably change the code if we do not want to encourage users to use anonymous functions? Interesting - but seemingly unrelated to anonymous functions(?) - if I run a pipeline with `textio.Read` from a remote location such as GCS, sometimes it completes successfully immediately and sometimes it shows unexpected behavior when reading, which comes across as the pipeline being stuck. It makes a single initial split of the restriction but then does a lot of splitting while processing. When reading from a local file with lower latency this doesn't seem to happen. Hm.. I've encountered some strange splitting behavior when working on the [natsio.Read](https://github.com/apache/beam/issues/29000) unbounded source transform but thought it was some deficiency in my SDF code and haven't had the chance to debug it yet. I wonder if it may be related. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
