[
https://issues.apache.org/jira/browse/BEAM-13939?focusedWorklogId=751743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-751743
]
ASF GitHub Bot logged work on BEAM-13939:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Apr/22 18:35
Start Date: 01/Apr/22 18:35
Worklog Time Spent: 10m
Work Description: thempatel commented on pull request #16961:
URL: https://github.com/apache/beam/pull/16961#issuecomment-1086214648
> Just to confirm, imports of the form `from
apache_beam.portability.api.beam_runner_api_pb2 import TestStreamPayload` are
no longer supported, and that's why the diff is so huge?
correct, this PR changes the the proto structure from a flat one to a
hierarchical one, like so:
```
model/
beam_fn_api.proto
beam_provision_api.proto
beam_interactive_api.proto
beam_artifact_api.proto
...
apache_beam/portability/api/
beam_fn_api_pb2.py
beam_provision_api_pb2.py
beam_interactive_api_pb2.py
beam_artifact_api_pb2.py
...
```
to
```
model/org/apache/beam/model/
beam_fn_api.proto
beam_provision_api.proto
beam_interactive_api.proto
beam_artifact_api.proto
...
apache_beam/portability/api/org/apache/beam/model/
beam_fn_api_pb2.py
beam_provision_api_pb2.py
beam_interactive_api_pb2.py
beam_artifact_api_pb2.py
...
```
so the reason you can no longer do `from
apache_beam.portability.api.beam_runner_api_pb2 import TestStreamPayload` is
because there is no module there! The fully qualified import is now
`apache_beam.portability.api.org.apache.beam.model.beam_runner_api_pb2 import
TestStreamPayload`.
In order to make this easier to work with, I updated the proto generator to
also generate module bindings in the `__init__` file of
`apache_beam.portability.api`, like so
```py
from .org.apache.beam.model import pipeline
# ...
external_transforms_pb2 = pipeline.external_transforms_pb2
# ...
```
so that you don't need to provide the fully qualified path. If we didn't do
this, this PR would be even more huge, since we'd have to update every import
of the generated bindings in the SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 751743)
Time Spent: 25h 40m (was: 25.5h)
> Go SDK: Protobuf namespace conflict
> -----------------------------------
>
> Key: BEAM-13939
> URL: https://issues.apache.org/jira/browse/BEAM-13939
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, sdk-go
> Affects Versions: 2.36.0
> Reporter: Milan Patel
> Priority: P2
> Attachments: demobug.zip
>
> Time Spent: 25h 40m
> Remaining Estimate: 0h
>
> The Go SDK generated grpc protobufs are not namespaced with enough
> granularity. If a user has another external dependency with the same protobuf
> file registered with the proto runtime, their compiled binary will panic at
> runtime pointing the user to this [doc
> page|https://developers.google.com/protocol-buffers/docs/reference/go/faq#fix-namespace-conflict].
>
> In the interim, following the instructions to add either ldflags to the
> compiler or an environment var to the binary works, but this is an unideal
> solution since only one of the duplicate proto specifications will be
> accessible from a [global
> registry|https://pkg.go.dev/google.golang.org/[email protected]/reflect/protoregistry].
>
> Ask: Regenerate the go protos such that descriptors like
> [these|https://github.com/apache/beam/blob/84353a7c973d3acaaa56d81c265dce7193a56be5/sdks/go/pkg/beam/model/pipeline_v1/metrics.pb.go#L797-L811]
> are outputted with filenames that are more granular, such as a filename that
> includes the directory structure of the repository.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
