Abacn commented on issue #33859:
URL: https://github.com/apache/beam/issues/33859#issuecomment-4462409236
> Does the ULR special case "javasdk" coders and now that it is a different
URN that is unknown to the ULR it is presuming it is a known coder? That's a
ULR bug. It should be resilient to unknown coders that have URNs.
The code path is generic, happens when translating a coder from proto
```
File
"/runner/_work/beam/beam/sdks/python/apache_beam/runners/portability/fn_api_runner/execution.py",
line 783, in _enqueue_stage_initial_inputs
coder = self.pipeline_context.coders[self.safe_coders.get(
File
"/runner/_work/beam/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 155, in __getitem__
return self.get_by_id(id)
File
"/runner/_work/beam/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 103, in get_by_id
self._id_to_obj[id] = self._obj_type.from_runner_api(
File "/runner/_work/beam/beam/sdks/python/apache_beam/coders/coders.py",
line 364, in from_runner_api
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File "/runner/_work/beam/beam/sdks/python/apache_beam/coders/coders.py",
line 364, in <listcomp>
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File
"/runner/_work/beam/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 103, in get_by_id
self._id_to_obj[id] = self._obj_type.from_runner_api(
File "/runner/_work/beam/beam/sdks/python/apache_beam/coders/coders.py",
line 364, in from_runner_api
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File "/runner/_work/beam/beam/sdks/python/apache_beam/coders/coders.py",
line 364, in <listcomp>
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File
"/runner/_work/beam/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 103, in get_by_id
self._id_to_obj[id] = self._obj_type.from_runner_api(
File "/runner/_work/beam/beam/sdks/python/apache_beam/coders/coders.py",
line 364, in from_runner_api
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File "/runner/_work/beam/beam/sdks/python/apache_beam/coders/coders.py",
line 364, in <listcomp>
[context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
File
"/runner/_work/beam/beam/sdks/python/apache_beam/runners/pipeline_context.py",
line 103, in get_by_id
self._id_to_obj[id] = self._obj_type.from_runner_api(
File "/runner/_work/beam/beam/sdks/python/apache_beam/coders/coders.py",
line 361, in from_runner_api
parameter_type, constructor = cls._known_urns[coder_proto.spec.urn]
KeyError: 'beam:coder:schema:v1'
```
Technically any runners that have Java side ended up with a schema coder
then get consumed in python side could trigger the same issue?
These tests happen to expose this issue. The same test was explicitly
excluded from other runners (Flink/Spark/twister2 has all excluded
"UsesPerKeyOrderedDelivery"), while Python validate runner test do not have
corresponding tests, persumably could also break if it does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
