[
https://issues.apache.org/jira/browse/BEAM-3293?focusedWorklogId=680976&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-680976
]
ASF GitHub Bot logged work on BEAM-3293:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Nov/21 20:29
Start Date: 12/Nov/21 20:29
Worklog Time Spent: 10m
Work Description: jrmccluskey commented on a change in pull request
#15966:
URL: https://github.com/apache/beam/pull/15966#discussion_r748567812
##########
File path: sdks/go/pkg/beam/core/runtime/exec/sideinput.go
##########
@@ -58,7 +59,16 @@ func NewSideInputAdapter(sid StreamID, sideInputID string, c
*coder.Coder, wm Wi
}
func (s *sideInputAdapter) NewIterable(ctx context.Context, reader
StateReader, w typex.Window) (ReStream, error) {
- key, err := EncodeElement(s.kc, []byte(iterableSideInputKey))
+ return s.NewKeyedIterable(ctx, reader, w, iterableSideInputKey)
+}
+
+func (s *sideInputAdapter) NewKeyedIterable(ctx context.Context, reader
StateReader, w typex.Window, iterKey interface{}) (ReStream, error) {
+ // TODO(jrmccluskey): re-write to ensure support for non-string keys
Review comment:
I don't love this little hack, it will need a more substantial re-write
to support non-string keys. For now this establishes the function signature to
be correct; however, it will have to get tweaked once I get the end-to-end code
change completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 680976)
Time Spent: 2h (was: 1h 50m)
> Add lazy map side input form
> ----------------------------
>
> Key: BEAM-3293
> URL: https://issues.apache.org/jira/browse/BEAM-3293
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Henning Rohde
> Assignee: Jack McCluskey
> Priority: P3
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Add InputKinds LazyMap and LazyMultiMap that allow map lookup without reading
> everything to memory. They will be accessed through functions such as:
> func(K) func(*V) bool (a keyed function that returns an iterator)
> func(K) []V (a keyed function that returns a slice
> of values)
> On the execution layer, the new forms would need to be added to
> exec/sideinput.go
>
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sideinput.go]
>
> The inputs layer, for the actual abstraction using reflection:
>
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/input.go]
> The funcx package would need to be updated to detect the new parameter forms
>
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/funcx/fn.go]
>
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/funcx/sideinput.go]
> as well has the DoFn graph validation code
>
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/graph/fn.go#L566]
> They would need to be correctly translated into the pipeline protos:
>
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/graphx/translate.go#L315]
> and finally back to the newly created handlers in the exec package.
>
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/translate.go#L402]
> If implemented pre-generics, the code generator frontend, and backend would
> need to be updated to detect and generate code for efficient no-reflection
> overhead map access functions.
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/shimx/generate.go]
>
>
> [https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/starcgenx/starcgenx.go]
> Unit must be added throughout and Integration tests should be added to verify
> the functionality against portable beam runners.
>
> [https://github.com/apache/beam/tree/master/sdks/go/test/integration/primitives]
> And of course, the user GoDoc should be updated for the support.
> See this lengthy email response for a more indepth guide to how Side Inputs
> operate.
> [https://lists.apache.org/thread.html/ra42dc7ee30842f11740eff33f0afcd63702695878e427127e1268381%40%3Cdev.beam.apache.org%3E]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
