[
https://issues.apache.org/jira/browse/BEAM-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik updated BEAM-3305:
----------------------------
Description:
We can add it as an optional field to beam.SideInput (see: pkg/beam/option.go).
The execution side needs to support it as well. However, with the various side
input forms, it's not clear how valuable such a feature would be.
It is strongly recommended to add Map Side Inputs
https://issues.apache.org/jira/browse/BEAM-3293 before implementing this
suggestion, and required to have caching implemented
https://issues.apache.org/jira/browse/BEAM-11097. Otherwise very little benefit
is acheived.
See https://issues.apache.org/jira/browse/BEAM-3293 for where code might need
to be changed.
Make providing a ViewFn part of the beam.SideInput struct, either as a method
to validate & update unexported fields, or adding an optional field.
The ViewFn would be a function that takes in the iterator/multimap version of
the Side Input and returns whatever the pre-processed type is. That value would
be what is cached to reduce work per common window.
The DoFn should then be able to use that type in the position of the
ProcessElement method for that side input. This avoids issues that only work in
of the Global Window like trying to process the side input in StartBundle.
Something would need to be changed about handling the DoFn method signature
analysis to support additional alternative side input types.
was:
We can add it as an optional field to beam.SideInput (see: pkg/beam/option.go).
The execution side needs to support it as well. However, with the various side
input forms, it's not clear how valuable such a feature would be.
It is strongly recommended to add Map Side Inputs
https://issues.apache.org/jira/browse/BEAM-3293 before implementing this
suggestion, and required to have caching implemented
https://issues.apache.org/jira/browse/BEAM-11097. Otherwise very little benefit
is acheived.
See https://issues.apache.org/jira/browse/BEAM-3293 for where code might need
to be changed.
Make providing a ViewFn part of the beam.SideInput struct, either as a method
to validate & update unexported fields, or adding an optional field.
The ViewFn would be a function that takes in the iterator version of the Side
Input and returns whatever the pre-processed type is. That value would be what
is cached to reduce work per common window.
The DoFn should then be able to use that type in the position of the
ProcessElement method for that side input. This avoids issues that only work in
of the Global Window like trying to process the side input in StartBundle.
Something would need to be changed about handling the DoFn method signature
analysis to support additional alternative side input types.
> Consider: Go ViewFn support
> ---------------------------
>
> Key: BEAM-3305
> URL: https://issues.apache.org/jira/browse/BEAM-3305
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Henning Rohde
> Priority: P3
>
> We can add it as an optional field to beam.SideInput (see:
> pkg/beam/option.go). The execution side needs to support it as well. However,
> with the various side input forms, it's not clear how valuable such a feature
> would be.
> It is strongly recommended to add Map Side Inputs
> https://issues.apache.org/jira/browse/BEAM-3293 before implementing this
> suggestion, and required to have caching implemented
> https://issues.apache.org/jira/browse/BEAM-11097. Otherwise very little
> benefit is acheived.
> See https://issues.apache.org/jira/browse/BEAM-3293 for where code might need
> to be changed.
> Make providing a ViewFn part of the beam.SideInput struct, either as a method
> to validate & update unexported fields, or adding an optional field.
> The ViewFn would be a function that takes in the iterator/multimap version of
> the Side Input and returns whatever the pre-processed type is. That value
> would be what is cached to reduce work per common window.
> The DoFn should then be able to use that type in the position of the
> ProcessElement method for that side input. This avoids issues that only work
> in of the Global Window like trying to process the side input in StartBundle.
> Something would need to be changed about handling the DoFn method signature
> analysis to support additional alternative side input types.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
