[
https://issues.apache.org/jira/browse/BEAM-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637189#comment-16637189
]
Robert Burke commented on BEAM-3612:
------------------------------------
I've got a prototype tool I've been working on to simplify this.
It should be able to replace the necessity of the optimized package. There are
two avenues I'm exploring.
First: A vetting/verifying runner that can detect if a pipeline is "optimized"
via avoiding the default reflective paths. This will largely check if all the
various types and functions are registered with beam, as well as each having
the appropriate type assertion shims registered as well. Optionally it can also
generate a file to be included in the pipeline's package to optimize any types
and functions missing that are using exported types and functions. In
principle, this could be combined with the "compile for the execution
architecture" approach the universal runner takes
Second: A static analysis based approach for library authors, that could be
used with go generate. Having a package centric approach has the benefit
allowing library authors to produce performant transforms for pipeline authors
to use. It also avoids redundant type and function registrations. The main
redundancy is that the reflectx.Funcs might be regenerated and registered if
many packages provide the same signatures. In practice this is not likely a
huge concern, except for binary bloat. On the whole though, I suspect not
having all the speculative registrations of the optimized package will offset
this.
Other than submitting the tools I've got once they're ready, the main change to
the beam APIs is providing a way for the vetting/verifying runner to see if
Iterators and Emitters have type specializations registered, as the exec
package provides no way to examine the registry's contents at this time either
by invoking the maker/wrapper functions or by an explicit IsRegistered function
for each. These will need to be added.
> Make it easy to generate type-specialized Go SDK reflectx.Funcs
> ---------------------------------------------------------------
>
> Key: BEAM-3612
> URL: https://issues.apache.org/jira/browse/BEAM-3612
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Henning Rohde
> Assignee: Robert Burke
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
