ahmedabu98 commented on issue #32781: URL: https://github.com/apache/beam/issues/32781#issuecomment-2422482604
> I try to stay with the pure Python code as much as possible We're ideally aiming for a multi-language experience where the user is not even aware that they are using a cross-language transform. They might still need a JVM or docker, but ideally they'd just worry about writing pure Python code like you say > I did try to use the StorageWrite API about 1 year ago, and I was having trouble getting it to work Might be worth a shot trying it again -- there was quite a bit of work done around 6 months ago to fix multiple issues with Storage Write API > why is there a strong push for cross-language Yep it's a lot easier to maintain, and significantly lowers the threshold for new SDKs to become usable. Also new features added to the main transform can almost instantly become available to foreign SDKs. With that said, there's nothing stopping SDKs from still developing and building native transforms. > I assume there a loss in efficiency or bandwidth by going cross language There's some -- cross-language introduces a fusion break between SDKs, and elements need to get serialized to pass through. > were considering moving our Dataflow work to Go once Go is more fully supported @lostluck can speak more on Go's roadmap -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
