[
https://issues.apache.org/jira/browse/BEAM-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669181#comment-16669181
]
Robert Burke commented on BEAM-4124:
------------------------------------
This needs to be better documented somewhere, but for now this Jira is the
appropriate way to put the handling for it.
The default cap comes from
[https://github.com/grpc/grpc-go/blob/master/clientconn.go#L96,] which is the 4
MB previously stated.
The Beam Go SDK has a a GRPCx hook package to customize the interprocess
dialer, which can use any arbitrary GRPC configuration (assuming the runner can
handle things.)
[https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/grpcx/hook.go#L45]
{{In some package init somewhere:}}
{{[
|https://www.google.com/url?sa=D&q=http://godoc/pkg/google3/third_party/golang/apache_beam/pkg/beam/util/grpcx/grpcx&usg=AFQjCNFeFGAX19YyGRSEFcMHI_Ge_ABTjA]grpcx.RegisterHook("myCustomGRPCDialer",
func([]string) grpcx.Hook { return grpcx.Hook\{Dialer: DialContext} })}}
{{// DialContext is an override for grpc.Dial that increases received message
size.}}
{{func DialContext(ctx context.Context, endpoint string, timeout time.Duration)
(*grpc.ClientConn, error) {}}
{{ ctx, cancel := context.WithTimeout(ctx, timeout)}}
{{ defer cancel()}}{{ config := grpc.DefaultDialConfig()}}
{{ cc, err := grpc.DialContext(ctx, endpoint,}}
{{ grpc.WithDefaultCallOptions(}}
{{ grpc.MaxCallRecvMsgSize(1000<<20))) // Cap received messages to 1 GB
(1000 * 2^20)}}
{{ if err != nil {}}
{{ return nil, fmt.Errorf("failed to dial server at %v: %v", endpoint, err)}}
{{ }}}
{{ return cc, nil}}
{{}}}
{{Somewhere after beam.Init:}}
{{grpcx.EnableHook("myCustomGRPCDialer")}}
Given there are performance implications around this, we're likely to simply
keep the default GRPC connections generally, and require the runners to handle
these choices as appropriate for the runner. This does mean that at some point
the Dataflow runner might override the dialer context, but in that instance, it
would probably make it a bit more configurable. Similar for the in progress
flink runner, and others. We'll know more once Portability has settled.
> Support elements larger than 4 MB
> ---------------------------------
>
> Key: BEAM-4124
> URL: https://issues.apache.org/jira/browse/BEAM-4124
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Reporter: Cody Schroeder
> Priority: Major
>
> The Go SDK harness is limited by a gRPC message size limit of 4 MB.
> https://github.com/apache/beam/blob/4a32353/sdks/go/pkg/beam/core/runtime/harness/datamgr.go#L31
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
