[
https://issues.apache.org/jira/browse/YARN-10508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
junwen yang updated YARN-10508:
-------------------------------
Description:
Regarding the issue YARN-5632 caused by the incompatibility of protobuf
message, we have created a static checker which keeps track of the proto file
change, and detect potential incompatibility:
# Add/delete required field, which is the case reported in HBASE-25238.
# The tag number of a field has been changed, as described in HDFS-9788. Also,
the [protobuf guidelines
|https://developers.google.com/protocol-buffers/docs/proto]suggests _each field
in the message definition has a *unique number*. These numbers are used to
identify your fields in the [message binary
format|https://developers.google.com/protocol-buffers/docs/encoding], and
should not be changed once your message type is in use_.
# A required field has been changed to optional, or an optional field has
been changed to required. According to the guidelines in [protobuf official
website|https://developers.google.com/protocol-buffers/docs/proto], _*Required
Is Forever* You should be very careful about marking fields as {{required}}. If
at some point you wish to stop writing or sending a required field, it will be
problematic to change the field to an optional field - old readers will
consider messages without this field to be incomplete and may reject or drop
them unintentionally. You should consider writing application-specific custom
validation routines for your buffers instead._
We have applied our checker on the frequently maintained YARN versions:
rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0,
rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0,
rel/release-3.3.0, and we found 28 potential problems as attached.
The results reported by our checker got confirmed by developers of HBASE and
IMPALA, and our checker is requested by HBASE, which can be found at
HBASE-25340.
was:
Regarding the issue YARN-5632 caused by the incompatibility of protobuf
message, we have created a static checker which keeps track of the proto file
change, and detect potential incompatibility:
# Add/delete required field, which is the case reported in HBASE-25238.
# The tag number of a field has been changed, as described in HDFS-9788. Also,
the [protobuf guidelines
|https://developers.google.com/protocol-buffers/docs/proto]suggests _each field
in the message definition has a *unique number*. These numbers are used to
identify your fields in the [message binary
format|https://developers.google.com/protocol-buffers/docs/encoding], and
should not be changed once your message type is in use_.
# A required field has been changed to optional, or an optional field has
been changed to required. According to the guidelines in [protobuf official
website|https://developers.google.com/protocol-buffers/docs/proto], _*Required
Is Forever* You should be very careful about marking fields as {{required}}. If
at some point you wish to stop writing or sending a required field, it will be
problematic to change the field to an optional field - old readers will
consider messages without this field to be incomplete and may reject or drop
them unintentionally. You should consider writing application-specific custom
validation routines for your buffers instead._
We have applied our checker on the frequently maintained YARN versions:
rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0,
rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0,
rel/release-3.3.0, and we found 28 potential problems as attached.
The results reported by our checker got confirmed by developers of HBASE and
our checker is requested by them, which can be found at HBASE-25340.
> Protobuf Mesage Incompatibility Detector
> ----------------------------------------
>
> Key: YARN-10508
> URL: https://issues.apache.org/jira/browse/YARN-10508
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: junwen yang
> Priority: Major
> Attachments: yarn_proto_incompatibility.txt
>
>
> Regarding the issue YARN-5632 caused by the incompatibility of protobuf
> message, we have created a static checker which keeps track of the proto file
> change, and detect potential incompatibility:
> # Add/delete required field, which is the case reported in HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788.
> Also, the [protobuf guidelines
> |https://developers.google.com/protocol-buffers/docs/proto]suggests _each
> field in the message definition has a *unique number*. These numbers are used
> to identify your fields in the [message binary
> format|https://developers.google.com/protocol-buffers/docs/encoding], and
> should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has
> been changed to required. According to the guidelines in [protobuf official
> website|https://developers.google.com/protocol-buffers/docs/proto],
> _*Required Is Forever* You should be very careful about marking fields as
> {{required}}. If at some point you wish to stop writing or sending a required
> field, it will be problematic to change the field to an optional field - old
> readers will consider messages without this field to be incomplete and may
> reject or drop them unintentionally. You should consider writing
> application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained YARN versions:
> rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0,
> rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0,
> rel/release-3.3.0, and we found 28 potential problems as attached.
> The results reported by our checker got confirmed by developers of HBASE and
> IMPALA, and our checker is requested by HBASE, which can be found at
> HBASE-25340.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
