David Young created SPARK-55062:
-----------------------------------
Summary: from_protobuf and to_protobuf do not support proto2
extensions
Key: SPARK-55062
URL: https://issues.apache.org/jira/browse/SPARK-55062
Project: Spark
Issue Type: Improvement
Components: Protobuf
Affects Versions: 4.1.1
Reporter: David Young
Proto2 extensions are currently silently dropped by from_protobuf and
to_protobuf.
This can be reproduced with any toy proto file containing an extension. For
example, given a message `Foo` with regular field `a` and extension field `b`,
from_protobuf will read only `a`.
This is because the dynamic message used for serde is created without an
extension registry. The fix will entail iterating through all file descriptors
in the descriptor set and finding all extensions applicable to the target
message. We cannot support extensions when we are not provided a file
descriptor set, as the Java class for the message will only have the descriptor
for the file it is defined in.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
